Open-Source LLM Routers Tested: Three Barely Read Prompts

A new evaluation of four open-source LLM routers found three of them emit constant or near-constant model tiers, barely reacting to prompt content.

That matters because a router is usually sold as the component that reads your request and picks the right model for it. Under a common measurement protocol, most of the measured gain came from which tier got picked overall, not from per-task targeting.

What the study measured

The paper, posted to arXiv on 28 July 2026 by Kiran N. Kumar and one co-author, runs "a common measurement protocol and hybrid evaluation of four router implementations across RouterBench, BFCL v4, tau2-bench, and WebArena". The scale is specific: 290 frozen tasks against "a locked matrix of 2,610 candidate outcomes". It runs 34 pages with 25 tables.

Freezing the tasks and locking the candidate outcome matrix is the point. Routers are normally benchmarked with different tasks, candidate pools, and execution protocols, which is why cross-router comparisons have not meant much until now.

A fixed-tier baseline matched a router exactly

The headline finding is a baseline result. "Always-Mid matches Aurelio exactly on three benchmarks and within 0.003 on the fourth" — that is, a rule that ignores the prompt entirely and always picks the mid tier reproduced a real router's numbers.

Only one system behaved like a router at all: "only vLLM Semantic Router varies materially with prompt content, and it has the highest observed success rate on none of the four benchmarks." The authors then tested whether that variation buys anything. "For vLLM, task-level superiority tests detect no task-specific advantage over a share-matched content-blind allocation; equivalence is established only on WebArena at the protocol-declared five-percentage-point margin."

Their conclusion is narrower than a dismissal of routing: "observed gains track selected-tier composition more closely than demonstrated task-specific targeting," and the findings are explicitly "scoped to these configurations, candidate pool, and frozen benchmark samples, not to routing paradigms in general."

What to do before you add a router

The paper names the control that was missing: "fixed-tier baselines and selected-tier distributions are therefore necessary controls in router evaluation." Read as buying advice, that inverts the usual evaluation order.

Run always-cheap and always-mid as baselines first, on your own traffic. Then log the distribution of tiers your candidate router actually selects. If its win over always-mid disappears once you match that distribution, you are paying for a routing layer to do what a constant would do — and you still carry its latency and failure surface.

The same discipline is missing from tool-gating claims

Adjacent cost-cutting research is being reported the same way. A roundup of recent tool-gating work relays that Qwen-UI-Agent's "partial-progress score exceeds those of MiniMax M3 and Qwen 3.7 Plus by 17.7 and 18.5 percentage points, respectively, while requiring 58.4% fewer steps per task," and that StateAct "achieves comparable success while operating at ≈ 9× lower cost per task than a screenshot-driven baseline." Those are the source reports' own figures, not independent replications.

That roundup's own caveats are the useful part: Qwen-UI-Agent's results rely on 10,000 concurrent environments for its parallel rollouts, and StateAct assumes access to program state that many closed-source applications do not expose. Its recommendation — that "cost-aware benchmarks should start reporting tool-call count alongside success metrics" — is the tool-side version of the router paper's baseline demand.

What to watch

Two things would move this. First, whether router projects start publishing selected-tier distributions alongside accuracy, which would make the next comparison cheap to audit. Second, whether any router demonstrates a task-specific advantage that survives a share-matched content-blind baseline on more than one benchmark — so far, on this candidate pool, none did.