Strict Tool Contracts: 18 Wrong Answers per 1,000 vs 240

Two agent configurations, identical model and prompt and near-identical bills, shipped 18 versus 240 confidently wrong answers per thousand tasks. The variable was the tool contract.

The numbers come from a dependency-free JavaScript simulation published on Dev.to — "five worlds of 1,000 tasks," not measurements of a production agent. Worth reading anyway, because the two knobs it isolates are ones most teams have never set.

The two fates of a wrong tool call

The model starts from a distinction that is easy to state and easy to ignore. A wrong tool call "either errors - 404, schema violation, permission denied - in which case the agent sees it and retries with that tool excluded, or it returns something plausible - an empty list, a default object, the right shape with the wrong contents - in which case the agent proceeds on corrupted state."

The simulation names that second share silent, and names the chance a later call catches it trip — because any call that validates its inputs is an accidental verifier for everything upstream.

Where the 13.6x comes from

Baseline: at 96.5% success per step over twelve steps, compounding leaves 65.2%. Then the tool layer splits it:

Config	Success	Confidently wrong / 1,000	Bill
Strict (silent 0.20, trip 0.35)	91.5%	18	12.18
Permissive (silent 0.85, trip 0.05)	68.6%	240	11.92

Same model, same prompt, same retry policy. The author's summary: "a 13.6x gap in the thing that reaches a customer, bought entirely by whether a wrong call errors and whether the next call validates its input. Neither appears in any agent config."

Verifier placement beats verifier count

The more counterintuitive result concerns where checks go. At an identical check count, placement moved the outcome more than the budget did.

Verification	Cost	Confidently wrong	Success
None	11.86	7.7%	77.5%
3 checks at the front	13.29	7.6%	81.0%
3 checks sampled across	12.94	2.6%	81.2%
1 check on the last step	12.32	1.7%	78.8%
2 checks at the tail	12.62	0.7%	80.0%
All 12 steps	17.05	0.7%	93.9%

One check at the end beats three sampled across on both axes. Three at the front "cost 1.43 per task to move 7.7% to 7.6% - indistinguishable from having bought nothing," because over twelve steps a check at step 1 is reportedly worth 15.4x less than one at step 12 — everything downstream already checks it for free.

The distinction the author flags as the one that got the page rewritten: two tail checks and all twelve both reach 0.7% wrong, but success is 80.0% against 93.9%. A verifier does two separate jobs — it stops a wrong answer shipping, which only matters near the end, and it repairs the step, which matters everywhere. Only repair wants a uniform budget.

The upgrade that didn't pay

Two results argue against reaching for a bigger model first. Escalating retries to a 4x-priced model bought 0.3 points for 1.37 units per task, "because what makes a retry work is in the error message." And the 4x model on its own — 47.86 per task, 93.0% success, 2.7% wrong — was beaten on all three axes by cheap-plus-verify-all at 36% of the price.

What to check in your own stack

The figures are simulated, so treat them as a hypothesis about your agent, not a measurement of it. The testable version: log what your tools return on bad input. Every tool that answers a malformed call with an empty list or a default object instead of an error is buying you silent corruption at no discount. Fix those first, put your verifier at the end of the chain rather than the start, and only then argue about the model.