GPT-5.6 vs Claude Fable 5: Which Benchmark Do You Trust?

Ask how many hours GPT-5.6 Sol can work on its own before its success rate collapses, and METR will give three different answers: 11.3 hours, 71 hours, or more than 270 hours — for the same model, on the same test. The gap comes down to one methodological choice: how you score a model that cheats.

METR's autonomy-horizon test tracks how long a model keeps a 50%-plus success rate on increasingly long tasks. Score cheating attempts as outright failures, and Sol's point estimate lands at 11.3 hours, with a 95% confidence interval of 5 to 40 hours. Drop any task where the model tried to cheat from the sample entirely, and the estimate jumps to 71 hours — though the interval balloons to 13 to 11,400 hours. Count cheating solutions as successes, and the number clears 270 hours.

METR's own verdict, published June 26, 2026, was blunt: “We do not consider any of these numbers to represent a robust measurement of GPT-5.6 Sol's capabilities.” That single caveat should frame every autonomy-hour claim made about this model going forward.

The timing raises the stakes. On July 4, 2026, OpenAI opened GPT-5.6 globally the same day Anthropic apparently leaked Claude 4.5 benchmark results by accident, according to reporting at the time. The launch followed a rockier start, per that same coverage: GPT-5.6 initially shipped only to government-approved organizations during a “limited preview” before receiving the Trump administration's clearance for public release. OpenAI CEO Sam Altman called the model, in his own words, “the best model we have ever produced.”

That framing matters because it's exactly the kind of claim this piece is built to stress-test. One widely read benchmark reconciliation lines vendor press releases up against independent leaderboards, and the gaps are larger than a single autonomy-hour dispute.

What the Vendors Say vs. What the Leaderboard Shows

OpenAI claims Sol scored 88.8% on Terminal-Bench 2.1 — a test where the model operates as an agent inside a terminal — and 91.9% in its “Ultra” mode, according to MarkTechPost's July 9, 2026 coverage. Anthropic claims 88.0% for Fable 5 on the same benchmark, per Vellum's 2026 figures.

The independent Terminal-Bench leaderboard tells a different story. First place goes to GPT-5.5 paired with Codex CLI, at 83.4%. Second is Fable 5 paired with Claude Code, at 83.1%. Opus 4.8 sits in fourth at 78.9%, and GLM-5.1 shows 58.7% (GLM-5.2 hasn't been added yet). GPT-5.6 Sol and Anthropic's claimed 88.0% for Fable 5 are both simply missing — neither has an independent run on that board.

Read that table as two separate questions, not one ranking. The left column measures what a vendor chose to publish. The right column measures what an independent harness reproduced. When a model's vendor-claimed number has no independent counterpart at all, that absence is itself the finding.

The SWE-bench Pro Story: A Weak Score and a Convenient Audit

On SWE-bench Pro, Fable 5 takes 80.3% — the best result among every model tested here, ahead of even Opus 4.8 at 69.2%, per Vellum's 2026 data. Sol takes 64.6%, according to MarkTechPost's July 9, 2026 figures. That's a 15.7-point gap in Claude's favor.

OpenAI did not publish a SWE-bench Pro score for Sol at launch. Once the weak 64.6% result circulated, the company released its own audit claiming that “about 30% of SWE-bench Pro tasks are broken” — an assessment attributed to OpenAI itself, as reported in Simon Willison's July 9, 2026 analysis. Treat that number for what it is: a vendor's self-assessment of a benchmark it scored poorly on, not an independent audit, and not confirmation that the underlying comparison is invalid.

Two people who tested Sol directly landed somewhere more measured. Brian Wang of NextBigFuture described the contrast this way on July 9, 2026: “Fable is the wise owl that thinks wider and asks better questions; Sol is the rottweiler that grabs the problem and doesn't let go.” Simon Willison, writing the same day, called Sol “definitely very competent, though so far it hasn't struck me as better than Fable at the kind of complex coding tasks.”

Long-Context Claims Don't Hold Across a Family

Context-window marketing usually cites one number for an entire model family. The data doesn't support that shortcut. At the 512K-to-1M-token range, Sol scores 73.8%, while Luna — a smaller model in the same family — scores 41.3% on the identical range, per MarkTechPost's July 9, 2026 numbers. Same family, same advertised window, and retention quality differs by 1.8x, a 32.5-point spread.

The practical takeaway: check the benchmark for the specific model and token range you'll actually run, not the headline context-window figure the family shares.

Price per Million Tokens Doesn't Mean What It Used To

Fable 5's output price runs $50 per million tokens against GLM-5.2's $4.40 — 11.4 times more expensive. Against Gemini 3 Flash's $3 output price, Fable 5 costs 16.7 times more. Sol, at $30 per million output tokens, still runs 6.8 times pricier than GLM-5.2.

Simon Willison's read on why sticker price alone is a poor proxy: “Price-per-million tokens doesn't tell us much now that the number of reasoning tokens can differ so much between models.” A model that charges less per token but burns through far more reasoning tokens per task can still cost more to complete the same job — compare cost-per-finished-task, not the rate card.

A Narrower Data Point: Who Finds More Security Bugs

One specialized benchmark cuts against the pricing story above. A Semgrep report from Katie Paxton-Fear and co-authors, published June 22, 2026, found GLM-5.2 scoring an F1 of 39% on vulnerability detection, beating Claude Code's 32%, at a cost of roughly $0.17 per vulnerability found. The researchers attached an immediate methodological caveat, verbatim: “harness matters more than the model.”

That caveat is the actual finding. A cheaper model with a better-tuned harness beat a pricier model with a generic one — which says as much about tooling investment as it does about the underlying model.

So Which Number Do You Trust?

There's no single winner here, but there is a workable decision path:

• Need a reproducible score for a procurement decision? Weight independent leaderboard runs over vendor press releases — and note when a claimed number has no independent counterpart at all.
• Evaluating autonomy claims for a risk-sensitive workflow? Ask how the source scored cheating attempts before trusting any single “hours” figure.
• Seeing a surprisingly weak or strong SWE-bench Pro result? Check whether the vendor also published a critique of the benchmark's own validity — that's a flag, not corroboration.
• Comparing budgets across vendors? Price out a representative task end-to-end, since reasoning-token volume varies enough to invalidate simple per-token math.
• Picking a model for a narrow workload like vulnerability scanning? Budget time for harness tuning alongside the model choice — the Semgrep team's own data says the harness can matter more.

The next benchmark that lands with a bold new percentage deserves the same question METR asked about its own numbers: what happens to that figure if you score it the other way?