BM25 Beats Agentic Search at Scale, 50.5 vs 30.7 Accuracy

A new scaling study puts plain BM25 at 50.5 accuracy on its largest corpus tier, ahead of an agentic File-System Agent at 30.7.

The write-up summarizing the paper — BM25 Wins at Scale: A Scaling Study of Retrieval-Augmented Generation Paradigms — argues the gap is not a rounding error. It widens as tokens are added, reaching a margin of nearly twenty points at full scale.

Where the crossover sits

The number that matters for architecture decisions is the crossover, not the headline. Around 10 million corpus tokens, BM25 overtakes the File-System Agent and stays dominant across every larger shared tier. Below that, the comparison is open; above it, the summary's read is blunt — global candidate ranking consistently beats localized, stepwise discovery.

Method	Accuracy at the largest tier
BM25	50.5
File-System Agent	30.7
DenseRAG	29.9

The token bill is the other half

Accuracy is where the argument starts; cost is where it lands. On identical data, the File-System Agent's sequential exploration is reported to consume 39 times more query tokens than a single BM25 pass, and its effectiveness deteriorates as the search space expands. Two things going the wrong way at once — more spend, less accuracy — is what makes this a procurement question rather than a benchmark curiosity.

If you are running agentic retrieval over a corpus in the tens of millions of tokens, that multiplier is a line item you can check today. Pull the query-token count per answered question, compare it against a single lexical pass over the same index, and you have a local version of the study's finding without waiting for independent replication.

What the study does not settle

The summary flags its own limits, and they are real. The work isolates a single reader model and one judging protocol, so the reported superiority may not generalize to other downstream heads or evaluation metrics. It is one paper on one setup, not a closed case — and prior evaluations that mixed benchmarks and fixed corpus scales are exactly what it faults for reporting dense retrievers, graph indexes, and sequential agents as competitive with plain BM25.

Treat the accuracy figures as a strong prior for large corpora, not as a measured property of your own data.

The hybrid the authors actually recommend

This is not an argument for deleting your agents. The authors' own framing, quoted in the summary, is that "agentic reasoning works best after ranked discovery rather than in place of it." The recommended shape follows from that: BM25 as the default retrieval layer for any corpus beyond a few million tokens, with agentic components reserved for post-ranking refinement.

The practical appeal is that it cuts two costs at once — engineering complexity and token spend — while, per the summary, preserving or improving accuracy. A lexical index that a junior engineer can debug on a Monday morning is worth something the leaderboard does not score.

What to watch

Two follow-ups decide how much this generalizes: whether the result holds under a different reader model and judging protocol, and where the crossover moves for corpora that are messier than a clean benchmark set — heavy jargon, multilingual text, or documents whose useful content is in tables. Until then, the defensible position is to make BM25 the layer you must argue your way out of, rather than the fallback you reach for when the agent gets expensive.