2026-11-15 18:20 UTC
DANGMUAAI & Developer Tools, Decoded
BackDev Tools

Your LLM Benchmark Might Be Measuring Your Load Client

NVIDIA's AIPerf replaces GenAI-Perf with a multiprocess load client. Three flags, p25-p99 breakdowns and trace replay decide whether your numbers hold.

DangMua EditorialNov 15, 20264 min read
Your LLM Benchmark Might Be Measuring Your Load Client

NVIDIA replaced GenAI-Perf with AIPerf, a multiprocess load client, because a single-process benchmark harness measures Python's GIL as much as it measures your server.

AIPerf shipped on the NVIDIA Developer Blog on September 18, 2026 as the designated successor to GenAI-Perf. If you are sizing a vLLM fleet or buying GPUs against vendor numbers, the change matters less than what it implies about the numbers you already have.

The client was the silent variable

Every inference benchmark has a hidden actor. As an analysis from infrastructure consultancy Netics puts it, if the client is a single-process script — curl loops, a hand-rolled asyncio harness, or GenAI-Perf on top of Perf Analyzer — Python's GIL caps concurrency, and the numbers describe the client's ceiling as much as the server's.

AIPerf's answer is architectural: load generation runs in worker processes, result handling in record-processor services, and the two are coordinated over ZMQ, so one benchmark client can saturate a real server without becoming the bottleneck. GenAI-Perf wrapped an older stack; this is a ground-up rewrite, which is why NVIDIA shipped a migration guide alongside it.

The symptom to recognise: raising concurrency changes nothing, because the client saturated first.

Three flags that decide whether a run is reproducible

  • --streaming is required to measure TTFT and inter-token latency at all. Without it the server batches the full response, so there are no first-token or decode-token events to time.
  • min_tokens and ignore_eos pin output length. Without them the model stops whenever it naturally finishes, output counts drift, and throughput numbers stop being comparable across runs.
  • --random-seed 42 makes Poisson timing and synthetic length draws reproducible, so the same command produces the same request sequence — the difference between a headline number and a test you can re-run after a config change.

Percentiles are the skill, not the tool

AIPerf reports four core metrics — TTFT, inter-token latency, request latency and output token throughput — with p25 to p99 breakdowns plus min, max, average and standard deviation, and optional GPU telemetry through DCGM or pynvml for power draw, utilization and memory.

That last part is the practical win: correlating a latency spike with a memory-pressure event no longer needs a separate profiling session, which turns "the LLM stack feels slow" into "the decode phase is memory-bound."

Netics' argument is that the tool swap is the smaller story. A server with a healthy mean TTFT and an outlier p99 looks fine in aggregate and fails in production, so reading percentiles — not means — is the discipline that survived the rewrite.

Replay beats inventing your own traffic

The quieter capability is trace replay. AIPerf supports 15+ endpoint types, public datasets like ShareGPT, and replay formats from Mooncake, Baseten and WEKA AgentX, alongside constant, Poisson and gamma arrival patterns with tunable burstiness and gradual ramping.

A vendor's p50 chart says nothing about your users' request-rate jitter. A Poisson run at your expected concurrency gives you the p99 your on-call engineers will actually feel.

Before your next capacity decision

  • Re-run existing GenAI-Perf scenarios under AIPerf with --streaming and pinned output lengths — the delta shows where the old client was the limit.
  • Adopt one real load shape, Poisson at your expected request rate, and record p50, p90 and p99 instead of the mean.
  • If your traffic is bursty or long-context, replay a captured trace rather than a synthetic distribution, and keep the repeat command line AIPerf prints as your regression test.

Note that Netics is an AI-infrastructure consultancy writing about its own problem space, and the performance characteristics above come from NVIDIA's announcement rather than independent testing. The habit it recommends — proving the measurement client is not part of the bottleneck — stands regardless of whose tool you run.

More from DangMua