SGLang vs vLLM: 4.47x Faster TTFT, But Only With Prefixes

One published 8x H100 benchmark puts SGLang 4.47x ahead of vLLM on median time-to-first-token — but only when prompts share prefixes.

That caveat is the whole decision. The comparison comes from a single author's own test run, not an independent lab, and its own stateless numbers show the two engines are effectively tied. What separates them is agent-shaped traffic: long system prompts, tool declarations, and conversation history resent on every turn.

Stateless traffic: a coin flip

Scenario A in the write-up uses a 2,048-token prompt, 512-token output and 0% prefix overlap on 8x H100 SXM5 80GB running Qwen2.5-72B-Instruct at FP8. At concurrency 64, the reported throughput is 2,410.8 tokens/s for vLLM against 2,480.3 for SGLang, with P99 TTFT at 420ms and 410ms. At concurrency 128 it is 4,120.5 against 4,190.2.

The author's own verdict: performance is "essentially identical," with SGLang holding a 1–3% edge credited to FlashInfer kernel tuning. If your workload is one-shot classification or summarization with no shared preamble, this benchmark gives you no reason to migrate.

Agent loops: where the gap opens

Scenario B re-runs the test at 4,096-token prompts with 75% prefix overlap at concurrency 64 — the shape of a tool-calling loop. The reported results:

Metric	vLLM (APC)	SGLang (RadixTree)
Median TTFT (P50)	380 ms	85 ms
P99 TTFT	1,250 ms	280 ms
KV cache hit rate	41.2%	78.6%
Output throughput	3,120 tok/s	5,430 tok/s

Treat these as the author's figures on their hardware, not a neutral result. The direction, though, follows from a real architectural difference.

Why the cache behaves differently

vLLM's PagedAttention splits the KV cache into fixed blocks — typically 16 or 32 tokens — linked through page tables, with Automatic Prefix Caching matching blocks by cryptographic hash. The write-up's stated limitation is that this organizes cache blocks "linearly and flatly," so branching workflows such as tree search or agent rollbacks miss the cache or get evicted early.

SGLang's RadixAttention instead holds all live KV cache in one radix tree. Two requests sharing a long system prompt but diverging at a tool output split the node at the exact divergence token, sharing the parent's cache and allocating memory only for the branch. Eviction prunes least-recently-used leaves, so hot system prompts near the root stay resident.

Structured output is the second axis

For JSON Schema enforcement, vLLM masks invalid logits from an external state machine each forward step; the write-up reports that at 64+ concurrent requests, CPU overhead from masking a 128k vocabulary drops throughput by more than 40%. SGLang compiles the schema into a finite state machine inside the scheduler and uses jump-forward decoding to inject deterministic strings — static syntax like {"status": "success" — in one step instead of generating them token by token. The claimed gain is up to 2.5x throughput.

How to decide

Measure your own prefix overlap before migrating — it is the variable the entire gap depends on. At the 0% overlap point tested, these numbers say vLLM costs you 1–3%. At the 75% overlap point with JSON-constrained tool calls, the claimed TTFT gap is large enough to be worth a staging replay of your own traffic. Either way, run the concurrency sweep on your model and your GPUs: both engines came out of Berkeley, and neither published these numbers themselves.