Jev vs Luna: Is a 1-Point Win Worth Swapping Your Reviewer?

TypeSafe AI's Jev scored 67.8% to GPT-5.6 Luna's 66.8% on the vendor's own evals. GPT-6 Astra and Claude Fable 5.1 wrote the answer key.

That one-point margin is now cited as a reason to swap the model behind a production safety reviewer. Before you do, read what the score measures.

What Vercel said

Guillermo Rauch, Vercel's CEO, posted on X: "Default mode in fx is auto, with a safety reviewer analyzing every command. That reviewer runs on GPT Luna today. Jev is up to 18x faster (p95) and more accurate. It's coming to @vercel AI Gateway and likely new default."

Pranit from Vercel posted the result behind it: "We benchmarked fx auto mode (safety) classifier with @typesafeai's Jev. tl;dr: ~5-18x faster and more accurate than gpt-5.6-luna, our current top choice." Vercel's changelog said on September 16 that Jev "is now available on AI Gateway."

As of September 17, Vercel had not published the benchmark behind the claim. Beyond the speed ranges in those two posts, the result carries no dataset and no case count in public. That is normal for a post on X. It also leaves TypeSafe's own numbers as the only detailed ones on the table.

Whose labels

TypeSafe's evals page states that the reference labels were "generated via an average of the responses of GPT-6 Astra and Claude Fable 5.1, both at high thinking." The scored models ran at their providers' default reasoning settings. The page describes no human labeling.

So Jev's 67.8% measures how closely it agrees with two frontier models. Where both get a case wrong, a model that gets it right is marked down. Where they share a blind spot, a model with the same blind spot is rewarded. Labeling with strong models builds a large eval cheaply; it is not the same as measuring correctness.

For a command reviewer, the label that counts is the one an engineer on your team would put on rm -rf ./build running inside a CI job. No published benchmark has that label for your commands.

The numbers, as the vendor reports them

Aggregate results across four workflows, vendor-reported (the page scores nine models):

Model	Accuracy	Cost / case	Latency
Jev	67.8%	$0.0004	0.4s
GPT-5.6 Luna	66.8%	$0.0033	12.9s
GPT-5.6 Terra	67.9%	$0.0304	10.1s
Claude Sonnet 5	67.8%	$0.1174	78.1s
GPT-5.6 Sol	74.1%	$0.0836	23.3s
Claude Opus 5	73.1%	$0.1761	37.8s

Three models land within a tenth of a point of each other: Jev at 67.8%, Terra at 67.9%, Sonnet 5 at 67.8%. On these figures Terra costs roughly 76 times as much per case and Sonnet 5 roughly 294 times, with the costs rounded to four decimal places. The two models that clear Jev by a real margin are Sol, 6.3 points higher, and Opus 5, 5.3 points higher — at roughly 209x and 440x the cost per case.

TypeSafe's homepage banner reads "193.6x Faster, 444.6x Cheaper", footnoted as based on workflows for System One tasks. The launch post says it expects that figure to sit at "the higher end of real world gains," calls its side-by-side demo query "highly simplified," concedes the shorter input "paints our model in an advantageous light," and warns the workflow evals may carry "some bias." The vendor is telling you to read the banner as a best case.

The listed pricing shows where a gap that size can come from: input at $0.042 per million tokens, output listed as "FREE (too cheap to meter)." The Register gives GPT-5.6 Terra at $2.00 input and $12 output per million — about 48x on input, with no ratio on the output side because one side is zero.

The finding that missed the headline

The same eval page tested every model two ways: one large prompt, versus a workflow that decomposes the task into Choice, Score and Noul (yes/no) questions. Every LLM scored higher in workflow mode. Luna went from 51.9% to 66.8%.

Put that next to the switch. Decomposition bought Luna 14.9 points. Moving the decomposed workflow from Luna to Jev bought one more point, plus a roughly 8x lower cost per case. The ordering matters for anyone planning a migration: you can decompose today, on the model you already run, and measure what that alone is worth before changing vendors.

The aggregate hides the spread

Jev against the most accurate model on each workflow, as TypeSafe reports it:

Workflow	Jev	Best model	Gap
Customer service	76.0%	GPT-5.6 Sol, 78.3%	2.3
Agent trace observability	71.6%	GPT-5.6 Sol, 76.6%	5.0
Security incidents	61.7%	Claude Opus 5, 66.2%	4.5
Invoice processing	61.8%	GPT-5.6 Sol, 79.1%	17.3

With model, harness and labeling method held fixed, the gap runs from 2.3 points to 17.3 depending only on the task. The workflow closest in shape to a command reviewer — security incidents — is Jev's lowest raw score at 61.7%. None of these four is your reviewer, so none of these gaps predicts where yours lands.

"Zero hallucinations," read closely

TypeSafe's homepage claims "Zero Hallucinations." Its launch post explains the claim in its own words: "Our number is not empirical. Schema matching is guaranteed." Jev answers only inside the schema you declare — a yes/no question returns a probability, never a paragraph or a malformed object. It can still return the wrong option.

For a guardrail, that removes your parsing bugs and nothing else. A schema does not stop a confident 0.03 on a command that wipes a database volume.

Calibration is the load-bearing claim, and it is the one with the least public detail. TypeSafe says it trains Jev with RLCD, "Reinforcement Learning for Calibrated Decisions," for "epistemically honest probabilities." Anthony Maio, in a skeptical write-up dated September 15, counters that "the reward function, architecture, training procedure, and calibration methodology are all undisclosed," and that "individually calibrated judgments do not automatically compose into a calibrated workflow once you run them through thresholds, weights, and branches."

That sentence describes a safety reviewer exactly: a probability feeds a threshold, and the threshold picks allow or block. A well-calibrated P(unsafe) still does not tell you whether 0.5 is the right cutoff for your false-alarm budget. Maio's verdict is conditional — pending independent testing, "the strongest case for Jev is architectural rather than algorithmic."

Nathan Flurry of Rivet put the architectural case in one line on X: "jev is just a really smart switch statement." He also wrote that "jev does not replace gpt / claude."

Run the bake-off on your own labels

Pull commands from your agent's logs: the ones your current reviewer blocked, the ones a human overrode, and the dull majority that sailed through. Have a person mark each one safe or unsafe. Do not let either model label them, or you rebuild the vendor eval's problem inside your own repo.

One handling note before you start: logged commands can carry API tokens and customer data. Strip secrets and personal data from the sample before any of it reaches a third-party API, and check that your data-processing terms cover those calls.

Then run decomposition on your current model first. That is the change with a published 14.9-point result behind it, it costs you no vendor migration, and it sets the baseline that any switch has to beat.