Is the Cheaper LLM Actually Cheaper? Five Checks With Code

A cheaper model that looks fine on ten examples can cost more per finished task than the one you already run. The gap hides in retries, and your dashboard will not show it.

A public benchmark run published this week makes the arithmetic concrete: 150 banking questions, 77 possible labels, ten models, a 70% pass bar. Five checks, each a few lines of code.

1. Divide total spend by successes, not by calls

Your bill includes the retries, so the price you compare should too. Add up every attempt's cost and divide by the number of successes.

const costPerSuccess = attempts.reduce((sum, a) => sum + a.cost, 0) / successes;

On that run, DeepSeek V4.1 Flash made 32 retries and spent 31.3% of its money on them — and 11 of its final answers were still unreadable. Mistral Large 4 made 106 retries, 44.6% of its spend, and landed at $0.002274 per success. The baseline it was supposed to undercut, GPT-6.1 Sol, cost $0.001272. The cheap model was nearly twice the price.

2. Judge by the lower edge of the interval

On 150 rows, 77.3% is not the same as "clears 70%". Compute a Wilson lower bound and compare that against your bar.

Qwen 3.8 Flash passed 116 of 150 (77.3%), and its interval's lower edge lands at exactly 70.0% — borderline, not a pass. Gemini 3.5 Flash Lite (113 of 150) and Llama 4 Maverick (107 of 150) read the same way: above the bar as a point estimate, not clearly above it as an interval. GPT-6 Luna, at 123 of 150, has a lower edge of 75.1% and genuinely clears.

3. Count unreadable replies separately from wrong ones

These are different failures with different bills. A reply you cannot parse gets retried and billed again; a reply that parses to the wrong label is final and silently wrong. Keep two counters.

In that run the split was lopsided: Mistral Large 4 produced 45 unreadable finals, DeepSeek V4.1 Flash 11, Qwen 3.8 Flash 8 — and the other seven models had none at all.

4. Fix the bar before you look at results

Write the pass rate and the margin you need into a file, commit it, then run the evaluation. Pick the bar after seeing the table and it will land just under whichever model you were already hoping to ship.

This is the cheapest check on the list and the one teams skip most often.

5. Price the cheapest passing model against production, on the same rows

Not against its list price — against what you run today, on identical inputs.

On that run GPT-6 Luna was the cheapest model clearing the bar, at $0.00009 per success against $0.001272 for Sol, roughly 14 times less. But it passed 82.0% against Sol's 86.0%, and on the same rows it failed 8 that Sol passed while passing 2 that Sol failed.

That is the real decision: a 14x cost cut for eight specific regressions. Whether that trade is worth it depends on what those eight rows do in your product, and only your own examples can answer that.

The limit of this run

These numbers cover one case — single-label classification, where a row either matches its label or it does not. If your task is summarisation or open-ended generation, "pass" becomes a judge's call, and the interval has to widen to absorb the judge's own disagreement with your labels. The author raises that as an open question rather than answering it, and the honest reading is that nobody has a clean number for it yet.

The method still transfers. Count retries in the price, judge on the interval's lower edge, separate unparseable from wrong, commit the bar first, and compare on the same rows. Run those five on your own traffic before the next invoice tempts you into a swap.