Mercury 3 Hits 1,068 Tokens/Second: What to Test First

Inception Labs released Mercury 3, a diffusion-based LLM the company clocks at 1,068 tokens per second — fast enough to make your inference assumptions wrong.

The launch numbers, as announced: a 1M-token context window, throughput of 1,068 tokens/sec on NVIDIA GPUs, and pricing of $0.20/$0.70 per million tokens, half price during launch. Inception claims the model beats GPT-6 Luna and Claude Haiku 5.5 on benchmarks. Those figures are vendor-reported, from an Inception announcement relayed via PANews on October 20 — no independent evaluation exists yet.

It is on OpenRouter today

Distribution is the part that makes this testable this week rather than next quarter. Mercury 3 is live on OpenRouter, Baseten and Vercel, and behind an OpenAI-compatible API, so, per the launch write-up, "you can try it by changing a base URL." If you already route through OpenRouter, the cost of a side-by-side is an afternoon, not a migration.

Why diffusion decoding is a different animal

Autoregressive models emit one token at a time and cannot revise what they have emitted. Diffusion decoding starts from a fully masked sequence and iteratively denoises every position in parallel across K refinement steps — which means, in the launch write-up's framing, "Position 5 can be revised in step 12 based on what position 500 looks like." The model gets K parallel passes over the whole sequence instead of n serial commits, and that parallelism is where the throughput figure comes from.

The consequence is a different shape of strength, not just more speed.

Workload	Expected behavior	What to verify
Infilling, mid-file code completion, structured-output repair	Structurally advantaged — every position stays refinable	Feed it broken JSON and half-written functions
Constrained generation (JSON schemas, global format rules)	Late constraints can rewrite earlier choices	Multi-constraint schema compliance rate
Open-ended long-form, precise instruction-following	Historically behind autoregressive models, per the diffusion literature	Multi-constraint prompts — the known weak spot

Four measurements before you switch anything

• Throughput at your context length, not the headline. The launch write-up is explicit that 1,068 tok/s was "measured at favorable conditions" and that "the 1M-context + max-speed pairing is almost certainly not simultaneous." Plot tokens/sec at 4K, 32K, 128K and 1M before you quote a number internally.
• Time to first usable token versus full completion. Diffusion refines the whole sequence, so "first token" does not mean what it means in a streaming autoregressive UI. Measure the latency your product actually exposes.
• Edit and infill accuracy on your real tasks. This is the claimed advantage; confirm it on your own repair cases.
• Instruction-following precision. This is the architecture's historical weakness. Inception claims the gap is closed — treat that as a hypothesis to test, not a spec.

The caveats worth pricing in

Two of them are structural rather than rhetorical. First, the serving stack is wrong for this: KV-cache tricks, speculative decoding and essentially every serving framework optimization were built for autoregressive decoding, so expect immature tooling and failure modes nobody has catalogued. Second, prompt engineering may not transfer — techniques tuned for left-to-right generation, chain-of-thought scaffolding included, were shaped by a constraint diffusion does not have.

Price those as engineering time, not as a discount on $0.20 per million tokens.

Context: the pressure is coming from several directions

Mercury 3 lands in a month when the assumption of one general architecture has been loosening. A separate analysis of the same week points at domain-specific models doing the same thing from the other end: Mistral's Timestral time-series family, released October 15, spans 5M to 209M parameters and runs on CPUs, with the 209M model producing a one-day-ahead forecast from three weeks of 15-minute data in roughly 140 milliseconds on an eight-thread server CPU. Different architecture, same thesis: the general-purpose autoregressive API is no longer the only sensible default.

What to watch

Independent benchmark runs are the next real signal, and they should arrive within days. Until then the defensible position is the one the launch write-up itself lands on: run the eval, do not migrate production on launch-day numbers. If your workload is heavy on editing and repair rather than long-form generation, the test is worth scheduling this week — that is where the architecture, not the marketing, predicts a win.