Speculative Decoding Pays 4x Until Concurrency Hits 128

Speculative decoding is sold as a free 3x. A production write-up published this week puts a boundary on that number: past roughly 128 concurrent requests, it can cost you throughput instead.

The technique is real and the math behind it is not in dispute. What changes the answer is your concurrency level and how predictable your output is. Here is where the line sits, using the figures the write-up publishes.

The bottleneck is memory bandwidth, not compute

The case for speculative decoding starts with an idle GPU. The write-up works through a 70B model served unquantized in FP16 at batch size 1: static weights occupy 140 GB of VRAM, and every single new token requires the GPU to fetch all 140 GB from HBM into on-chip SRAM and registers.

On a GPU delivering 3 TB/s of memory bandwidth, the author calculates that moving 140 GB takes roughly 46 milliseconds. The conclusion is blunt: "For over 95% of each decoding cycle, expensive compute units idle waiting on memory bus transfers."

A separate architecture breakdown of SGLang and vLLM states the same constraint as the premise for engine design: "Because autoregressive generation is fundamentally memory-bandwidth bound, the core differentiator of any inference engine is how efficiently it allocates, retains, and reuses the KV Cache."

Speculative decoding attacks that idle time directly. A lightweight draft unit proposes several tokens, and the target model verifies all of them in one forward pass — one weight fetch instead of five.

"Lossless" here is a proof, not a marketing word

The quality question has a specific answer. The write-up states that the output distribution of speculative decoding is "provably and strictly 100% identical to the target base model."

The mechanism is modified rejection sampling. For each candidate token the engine computes an acceptance probability of min(1, p/q), samples a uniform random variable, and accepts or rejects. On rejection it resamples from the normalized positive residual distribution, and verification for the rest of that draft branch stops. The write-up attributes the invariance proof to Leviathan et al. (2023), and notes it holds "whether employing greedy decoding or temperature-based stochastic sampling."

That matters for procurement conversations. Unlike quantization, this is not a quality-for-speed trade — the cost shows up as wasted compute when guesses are wrong, not as degraded output.

Four generations, and why EAGLE won

The write-up traces four architectural generations:

Generation	Approach	Reported limitation
Gen 1: Draft-Target	A small dense model drafts for a large target model	The draft model is a full transformer, competing for HBM bandwidth on busy GPUs
Gen 2: Medusa	Parallel MLP prediction heads on the target model's final layer	Heads lack causal attention across draft positions; acceptance collapses beyond 3 tokens
Gen 3: EAGLE-1/2	Feature extrapolation plus context-aware dynamic draft trees	Requires training a lightweight head per target architecture
Gen 4: EAGLE-3	Fuses low-, mid- and high-level hidden representations into the dynamic tree	Requires distributed offline synthetic feature extraction during training

The reported payoff climbs with each generation: Gen 3 pushes acceptance rates "above 80%, yielding >3x speedup", and the write-up credits EAGLE-3 with unlocking 4x~5.6x speedups. It describes EAGLE as having "become the standard speculative backend across both vLLM and SGLang."

The structural change worth understanding is the move from a linear draft chain to a tree. In a chain, a rejection at token 2 invalidates tokens 3 and 4 even when those were correct. EAGLE flattens a multi-branch candidate tree into one sequence and verifies every branch in a single forward pass using a 2D tree-attention mask, which the write-up says keeps average accepted tokens per step between 3.5 and 4.8.

Where it turns negative

This is the section most speedup charts leave out. The write-up names two conditions under which speculative decoding produces a regression.

High concurrency. At concurrency of 16 or below, tensor cores sit underutilized during decoding and speculative verification uses that idle compute — the author reports inter-token latency improving by 60%~75%. At concurrency of 128 or above, batches already saturate the hardware into a compute-bound state, and the write-up reports that adding verification passes "can reduce overall system throughput by 10%~15%."

High-entropy output. Realized speedup is bounded by the empirical acceptance rate. For structured generation — code, JSON, math, translation — the write-up reports acceptance rates above 85% and 4x-plus speedups. For open-ended conversation at sampling temperatures of 1.0 or higher, it reports acceptance dropping toward 50% and speedups of only 1.8x~2.2x. In its decision flow, acceptance below 30% means draft overhead exceeds verification gains.

The workload split is convenient for agent platforms. The SGLang architecture breakdown notes that "over 60% of API endpoints require strict compliance with JSON Schemas, regex patterns, or domain-specific languages" — which is precisely the deterministic output class where acceptance rates are highest.

The flags that turn it on

The write-up publishes deployment commands for a 72B target model. On vLLM, speculative decoding is enabled through --speculative-model pointed at EAGLE head weights the author sizes at roughly 500MB to 1GB, with --num-speculative-tokens 5 and --speculative-draft-tensor-parallel-size 1 because the draft head fits on a single GPU without cross-GPU communication.

On SGLang the equivalent is --speculative-algorithm EAGLE with --speculative-num-steps 5, --speculative-eagle-topk 4 and --speculative-num-draft-tokens 16, which the author says activates dynamic tree exploration at depth 5 across 16 draft nodes. Treat both as illustrative starting configurations, not tuned settings.

One operational note from the same source: the base model stays frozen. EAGLE trains an auxiliary head on frozen intermediate features, which the write-up sizes at 0.5% to 1% of the base model's parameters and "a few hours of commodity GPU training."

What to measure before you enable it

Two numbers decide this, and both are yours, not the benchmark's: your steady-state concurrency and your acceptance rate on real traffic. If you serve a code or JSON workload at low-to-medium concurrency, the reported range is favorable. If you are already batching at 128-plus, measure total throughput with the feature off before you assume it helps — on the figures above, that is the regime where it takes throughput away.