Same Model, 23% vs 52%: Why the Agent Harness Decides

The same model scored 23% on SWE-bench Pro in one coding-agent harness and 52% in another, a roundup published this week reports.

The claim comes from a developer survey of harnesses outside the Claude Code / Codex CLI / Gemini CLI trio, and it points at a buying decision most teams never make deliberately: the runtime wrapped around the model, not the model itself.

The spread the roundup reports

The piece leans on what it describes as "an independently maintained catalog of over 100 harnesses that gets rescored weekly," which defines the split cleanly — a model answers, an agent acts, and the harness is the runtime that turns one into the other. That catalog, the roundup says, found that swapping the harness around a model can move its pass rate more than upgrading the model does.

Other figures it cites, all attributed to third-party leaderboards rather than the author's own testing:

• Claude Opus at 77% on Terminal-Bench inside Claude Code but 93% inside Cursor, per a comparison the roundup calls "The Harness Effect."
• The official Terminal-Bench board spreading the same underlying models from 51.82% for Claude Code with Opus 5 down to 20.30% for xAI's Grok Build, with Codex CLI at 37.27%.
• A research harness named "vix" running Claude Opus 4.7 topping the Terminal-Bench 2.0 board at 90.2%, ahead of Codex CLI's 82.2% with GPT-5.5.

Read those as directional. The roundup itself notes the numbers mix boards from different generations, which is exactly why the 77-vs-93 gap and the 51.82 ceiling do not sit on the same scale.

The short list, and what each is for

Harness	Stars (as quoted)	License	Best for
pi	~61,000-101,000 (snapshot-dependent)	MIT core	Context control, token efficiency
mini-swe-agent	~6,900	MIT	Auditing the whole agent in one sitting
Crush	~27,900	FSL-1.1-MIT	Any provider, mid-session model swaps
Plandex	~15,600	-	Large multi-file tasks, reviewable diffs
Amp	Not open source	Commercial	Team threads, background cloud agents

pi is the one the roundup keeps returning to. It was built by Mario Zechner, creator of libGDX and RoboVM, who joined Armin Ronacher's company Earendil in 2026; the project moved to earendil-works/pi in May 2026 with the core staying MIT. Its pitch is subtraction — read, write, edit and bash tools, no web search, no compaction, a system prompt under 1,000 tokens — and on that setup the roundup says pi took 8th place on Terminal-Bench with Opus 4.5. The author flags that result as a single run reported by pi's own creator.

mini-swe-agent is the outlier on effort-to-result: roughly 100 lines of code from the Princeton and Stanford team behind SWE-bench, scoring over 74% on SWE-bench Verified. Plandex keeps a cumulative diff review sandbox so agent changes stay out of your working files until approved. Amp, launched by Sourcegraph in May 2025 and spun out as Amp Inc in December 2025 under co-founder Quinn Slack, sells the opposite trade — polish plus Orbs, per-thread cloud machines that keep working after you close the laptop. Its docs list no pricing.

Is switching worth it?

For most teams, not on benchmark numbers alone — the boards move monthly and the star counts above are snapshots. The switch pays when your bottleneck matches a harness's design: unreviewable diffs on big refactors point to Plandex, a local-model or privacy requirement points to gptme or Goose with local providers, and a mismatch between what your agent injects into context and what you want there points to pi. If your bottleneck is model quality, changing harness will not fix it.

What to watch: whether that weekly-rescored catalog keeps showing harness swings larger than model upgrades once Terminal-Bench 4.0 is the only board in play. That is the number that decides whether "which harness" outranks "which model" as the real procurement question.