19 LLMs Benchmarked as Reviewers: Price Didn't Predict Score

A 19-model benchmark scored LLMs on catching validity flaws in ML writeups. Qwen 3 235B A22B Instruct scored 1.00; Claude Haiku 4.5 scored 0.68.

The headline for anyone picking a reviewer model: price, parameter count and release date did not predict the score. The benchmark, a Kaggle Benchmarking Challenge submission, asked each model to read a short experiment summary and either name one validity flaw from a fixed list of seven or answer none.

How the test is built

The design is matched pairs: every flaw has a flawed version and a sound twin that "share the same skeleton and differ only in the detail that matters" — for example, "tested 30 subgroups, no correction" versus "pre-specified one subgroup." The sound twins measure false alarms, so a model cannot win by pattern-matching on wording. Every item runs twice, once neutral and once with an added note that the authors consider the result conclusive, giving 112 items per model (7 flaws × 2 variants × 4 reps × 2 conditions). Half the items are flawed and half are sound, so a model that answers none every time scores 0.50.

The scores

Rank	Model	Score
1	Qwen 3 235B A22B Instruct	1.00
2	GPT-5.4 mini	0.98
3	GPT-5.5 / Claude Haiku 5.5	0.95
7	Claude Opus 5	0.93
12	Claude Sonnet 5	0.88
16	GPT-5.4 nano	0.76
18	Claude Haiku 4.5	0.68
19	Gemma 4 31B	0.55

The inversions are the point. The author reports that GPT-5.4 mini (0.98) beat GPT-5.5 (0.95) and Claude Haiku 5.5 (0.95) beat Claude Sonnet 5 (0.88) — but "small wins" is not the rule either, since within OpenAI's 5.4 line mini scored 0.98 and nano 0.76. Across generations, Haiku 4.5 scored 0.68 and Haiku 5.5 scored 0.95, which the author calls a 27-point jump. Qwen spans nearly the whole range: 235B Instruct at 1.00, Next 80B Instruct at 0.70, and the thinking variant at 0.86.

Read the error bars before you switch vendors

The author is blunt about resolution. Eleven of 19 models scored 0.91 or higher, and at 112 items one item is worth about 0.9 points, with a 95% interval of roughly ±4 points near 0.95 — so models inside that top group cannot be separated, and 0.94 versus 0.92 means nothing. Run-to-run noise is already a point or two: a self-run of Claude Haiku 5.5 scored 0.946, while Kaggle showed 0.96 and later 0.95. The perfect score has its own caveat the author flags — 112/112 gives a degenerate normal-approximation interval of [1.00, 1.00], the exact flaw type the benchmark tests for, where a Wilson interval gives roughly 0.97 to 1.00. Two more limits: items come from templates, so strong models may be recognising templates rather than reviewing, and Grok 4.5's 0.00 is believed to come from items that failed all retries being scored as wrong, which the author did not confirm.

Why reviewer-model choice became a CI problem

A separate engineering writeup from Zoroval explains the pressure. With coding agents writing most of the code, it reports a median of 133 changed lines per pull request but about one in eight changing more than 1,000 lines. Its failure case is the one to watch: on July 13 a monitoring job ran 1,138 judge calls and the LLM-as-a-judge said "no" to all four cases where a person had already confirmed the failure — and the run finished without an error. Neither the model nor the prompt template had changed; the root cause was the context builder, which never triggered their evals. Running the golden dataset against it first would have been four calls for under a dollar.

What to watch

• The promised breakdown — missed flaws versus false alarms versus being swayed by the "authors are confident" note — is the number that should drive a reviewer-model choice, and it has not been published yet.
• If you run an LLM judge, treat context-builder and retrieval code as model-behaviour changes in CI, not as ordinary code.
• Benchmark your own candidates on your own items before switching vendors on a ranking whose top eleven are statistically tied.