Is a $0.05 model worth it? 92% vs 95% on one real task

A senior engineer running code security scans swapped a frontier model for a budget one and watched accuracy fall from 95% to 70% overnight. Three rounds of context engineering later, he reports pulling it back to 92% — at a twentieth of the cost.

The write-up is one practitioner's benchmark on one task, not an independent evaluation. Treat the numbers as his measurements. The shape of the tradeoff is still the useful part.

The numbers he reports

The workload was security scanning across hundreds of files. Running it on Claude 3 Opus out of the box cost about $1.00 per file at 95% accuracy. Swapping in Nemotron dropped the API cost to five cents a file immediately — and the raw accuracy to 70%, which he calls disastrous. Adding strict guardrails, targeted tools, and explicit examples of what a finding looks like brought it to 92%.

Configuration	Cost per file	Accuracy	Overhead
Claude 3 Opus, out of the box	$1.00	95%	None
Nemotron, raw prompt	$0.05	70%	None
Nemotron, context and tools optimized	$0.05	92%	Prompt design, validation schemas

His framing of the result: a 95% cost reduction for three points of accuracy made it an obvious production win. Whether three points is affordable is entirely a property of your task, and it is the only question that matters here. On a scan whose findings get human review, three points is noise. On an unattended gate, it is the whole decision.

What actually closed the gap

Nothing in his fix involved fine-tuning. He describes three changes, each aimed at removing reasoning load rather than adding capability:

• Explicit schemas. Force structured JSON output instead of free-form text, so the model cannot wander off-task.
• Targeted tools. Supply helper functions that parse code blocks before the model analyzes them, rather than letting it guess whether a pattern is valid.
• Few-shot patterns. Inject clear true-positive and false-positive examples into the system prompt so the model has a baseline for what counts as a real finding.

The underlying claim is that smaller models fail when asked to do too much reasoning in a single pass, and that the fix is to treat one like a junior engineer with a specific runbook rather than an oracle. His stated reason for preferring this over fine-tuning is agility, not accuracy: context engineering needs no training runs and no GPU time, so an iteration takes minutes.

When to stay on the frontier model

He names three cases where the swap is not worth attempting: work that needs creative synthesis, multi-step logical planning across unstructured domains, or request volumes low enough that your engineering setup time outweighs the API savings. That last one is the trap. The $0.05 figure is the API line only; the 92% configuration carries prompt design and validation schemas as ongoing overhead, and that cost does not show up per file.

One more constraint he flags: budget models often have smaller context windows or degrade on material buried mid-prompt, so the approach depends on chunking data and injecting only relevant reference material instead of dumping a codebase into the prompt.

How to test it on your own workload

The method transfers even if the models do not. Take a task you already run on a frontier model, build a labeled set from runs you have manually verified, then measure the cheap model raw before touching the prompt — the 70% number is the one that tells you how far you have to travel. Add schema constraints first, tools second, examples third, remeasuring at each step so you learn which lever moved the result. Then price the whole thing, including the hours spent building the harness, against your actual monthly volume rather than per file.

If the gap does not close within a few iterations, that is a finding too: it means the task needs the capability, and the cheapest configuration is the one you already have.