DeepInfer vs vLLM on 8x B200: One Run, a Split Verdict
GPUStack benchmarked DeepInfer-server against vLLM on eight NVIDIA B200 GPUs: 25.2% more throughput one way, 2.09x the time to first token the other.
On eight NVIDIA B200s, DeepInfer-server posted 25.2% higher mean output throughput than vLLM — and took about 2.09x longer to emit a first token.
The numbers come from GPUStack, which built the serving wrapper under test and ran both engines itself. Read them as a vendor's single-run report, not a settled benchmark.
The setup
GPUStack says it integrated DeepSeek's open-source DeepInfer engine with its platform through a lightweight project called DeepInfer-server, then deployed DeepSeek-V4.1-Flash on one server with eight NVIDIA B200 GPUs and compared it against vLLM under a matched DP8+EP8 configuration. Both sides ran 100 requests at a target concurrency of 16, a 32,768-token maximum context, 8,192/1,024 target input/output lengths, GPU memory utilization at 0.85, and speculative decoding disabled. Versions: DeepInfer at commit 9bd9cf2, vLLM at 0.32.0.
The split verdict
| Metric | DeepInfer-server | vLLM |
|---|---|---|
| Mean output throughput (tokens/s) | 1,392.44 | 1,112.20 |
| Mean TTFT (s) | 4.28 | 2.04 |
| P95 TTFT (s) | 16.33 | 8.27 |
| Mean inter-token latency (ms) | 8.74 | 11.98 |
| Mean end-to-end latency (s) | 13.23 | 14.30 |
| P95 end-to-end latency (s) | 25.46 | 20.03 |
| Successful requests | 100 / 100 | 100 / 100 |
GPUStack puts DeepInfer-server's mean inter-token latency about 27.0% lower and its mean end-to-end request latency about 7.5% lower. On the other side of the ledger, it puts DeepInfer-server's P95 request latency at 25.46 seconds against vLLM's 20.03 — about 27.1% higher, so vLLM owns the tail.
Which engine fits which workload
The two engines do not win the same things, so the choice follows the workload rather than a single score. This reading is our analysis of the published figures, not a claim from the report.
- Batch and offline generation — summarization sweeps, evaluation runs, document pipelines — reward total tokens per second. That is where the throughput and inter-token latency gap points to DeepInfer-server.
- Interactive chat and coding assistants live on time to first token and on the slowest requests users actually notice. Both of those favored vLLM in this run.
Read the caveats before you migrate
GPUStack states its own limits plainly: the differences are "between two specific test runs; they are not yet proof of a stable performance advantage." It also describes DeepInfer-server as a single-node, multi-GPU demonstration project that does not reproduce DeepSeek's complete production architecture for KV-cache management, disaggregated prefill and decode, or load balancing, and should not be treated as equivalent to DeepSeek's official online service.
That matters because disaggregated prefill and decode is precisely the technique that usually attacks the TTFT weakness this run exposed — the gap may be a property of the wrapper rather than of the engine.
Why self-hosting math keeps moving
Serving-engine choice sits inside a cost picture that is shifting on the API side too. A survey of official list prices for 66 production models released between April 2024 and September 2026 reports the newest top-tier model at $50 per million output tokens against $8 for an April 2025 flagship, while workhorse models hold at $10 and the budget floor stays near $0.40 to $0.60. The same survey puts cached input at 90 to 98% cheaper than fresh input on most current models.
What to watch
Watch for a second independent run on the same hardware, and for DeepInfer-server gaining disaggregated prefill and decode. Until then, treat these figures as one data point on one node, and re-run the comparison on your own traffic shape before switching engines.
More from DangMua