Agent Code Search Tested: ripgrep vs Serena vs claude-context
One Flask checkout, four tools, setup from zero to 21 minutes. Grep tied a semantic index at 92% — and the benchmarks come from a tool author.
A developer ran ripgrep, Serena, claude-context and their own tool over the same Flask 3.1.3 checkout. Setup time ranged from nothing to about 21 minutes.
The write-up comes from the author of Semlith, one of the four tools, and they say so up front: "I build one of these tools, Semlith, so treat my bias accordingly." Every number below is their measurement on an M1 MacBook Air with 8 GB, not an independent benchmark.
What each one costs to stand up
| Tool | What it is | Setup on Flask | Moving parts |
|---|---|---|---|
| ripgrep 15.2.0 | Text search, no index | Nothing | One binary |
| Serena 2.1.0 | Language servers behind MCP | 30.9 s to index | Python env plus language servers |
| claude-context 0.1.15 | Embeddings in Milvus | About 21 min (CPU in Docker) | Docker, Milvus, Ollama, npx |
| semlith 0.50.0 | Local index: embeddings, keyword, code graph | 84.3 s to index | One binary |
The claude-context run used Milvus 3.0.3 and Ollama serving nomic-embed-text. Images came to 6.14 GB and the first npx start pulled 418 MB of packages. The author notes Docker on a Mac has no GPU access, so that 21 minutes is CPU embedding and not a fair speed test.
Grep still wins when the question uses the code's words
Asked where Flask decides which error handler runs, grep did best: rg -n -i "error handler" src landed on the function's docstring in src/flask/sansio/app.py. Semlith's top five were all documentation pages, and claude-context had the function at rank 8. The author's conclusion is blunt — "semantic search beats grep" isn't true in general.
The reverse case is just as clear. Ask for "limit how big an uploaded request body can be" and grep returns nothing useful, because the relevant file never says "upload" or "big".
The scale numbers, and their caveats
On SWE-bench Verified (500 real GitHub issues), the author reports Semlith had a correct file in its top 10 for 76.8% of issues against 47.0% for one ranked ripgrep pass — retrieval only, not whether an agent then fixed the bug. On 20 SWE-bench Lite issues, Serena 1.7.0 finished 16 and hit top-5 for 25.0% of those, against 56.2% for Semlith on the same 16; they flag that this run predates Serena's REPL, so 2.1.0 may differ.
The number that cuts against their own product: on 50 questions about one unfamiliar repo, headless Claude Code on Opus scored 92% with grep alone and 92% with Semlith installed.
Two things to check before you wire one in
First, a status bug. claude-context's get_indexing_status reported the codebase fully indexed at 92 seconds with 100 files and 100 chunks; the server's own log finished at 1,266.9 seconds with 86 files and 1,837 chunks. An agent that trusts that status searches a partial index without knowing.
Second, token overhead. Every MCP server ships its tool definitions at session start and the agent pays for them each turn — measured here at 4,544 bytes (~1,136 tokens) for Serena's two tools and 2,668 bytes (~667 tokens) for Semlith's eight, plus 9,292 characters of instructions when Serena's initial_instructions is called.
Where this leaves a buying decision
These are different jobs, and nothing stops you running two. One repo and a strong model with names you mostly know: grep is enough, and the 92% tie says so. An agent that edits by symbol is the case for Serena, which is the only one of the four that writes code back. An existing Milvus or Zilliz deployment is the case for claude-context.
Worth watching: whether a rerun on Serena 2.1.0 moves that 25.0% top-5 figure, and whether anyone reproduces the SWE-bench Verified gap on hardware the tool's author doesn't own.
More from DangMua