A New Tokenizer Cut One Team’s Inference Cost by 77%

A team running a small model in production says it cut measured inference cost by 77% without touching the base model, the training data or the recipe — by rebuilding the tokenizer.

The claim comes from Dharma-AI in a Hugging Face blog post previewing results from a paper the authors say is still under review, so the numbers may change before publication. The system is an automated grader for Portuguese essays built on Qwen3-4B, which returns a JSON assessment per competency.

Why the tokenizer was the lever

The grader was already fine-tuned and already small at 4 billion parameters, but it inherited the stock Qwen3-4B tokenizer — built for many languages and many kinds of text, then used for exactly one language and one response format. The authors describe two costs from that mismatch: the JSON envelope and competency names the model writes on every request got split into more tokens than needed, and at every generated token the model scored a vocabulary of 151,669 entries, most of which the task never uses.

Their argument for why token count is the wrong target is the part worth borrowing. Prefill reads the whole input in one pass and is compute-bound; decode generates one token at a time and is memory-bandwidth-bound. In their measured case, a generated token cost 7.4 times a token read — a ratio they note varies with hardware, batch size and context length. So a method that compresses input and output equally, they argue, misses most of the available saving. They also point out that in a model the size of Qwen3-4B, the LM head holds one row of parameters per vocabulary entry and accounts for almost 10% of the parameters.

The measured result

To isolate the effect, the team says it trained seven versions of the grader with the same base model, data, recipe and seed, differing only in tokenizer, then ran the same 500 requests on one machine with an RTX 5090 and the vLLM inference server, calculating cost from measured runtime.

The reported outcome: average response length fell from 350 tokens to 101, cost fell 77.0%, and the bench score — their combined measure of grading accuracy and off-topic screening — rose from 0.703 to 0.810.

The useful failure is in the middle of that. Their first vocabulary fused each competency's grade into the surrounding JSON as a single macro token. It was the most compact option and quality dropped: bench score 0.689, below the 0.703 of the production grader. Giving the grade its own token cost two extra tokens per response, raised the score to 0.810 — and measured cheaper anyway, 77.0% reduction against 73.3% for the fused version. Their stated lesson: compression costs you when the compressed span carries the decision the model exists to make.

Does it generalize?

The authors report comparing the method against three reference methods across five public domains — contracts, medical papers, scientific papers, news and Python code — and three model families, Qwen3, Llama-3.1 and Gemma-3. They say it compressed more than the strongest reference in all fifteen combinations, by 1.2 to 9.3 points. On contracts, they report 62.6% token reduction with 113,426 vocabulary entries against 61.8% for the best in-place BPE configuration using 207,803.

Treat all of this as one team's preview of their own unpublished paper, measured on their own hardware. What transfers regardless of whether the headline number holds is the diagnostic: if you serve a fine-tuned small model on a single language and a fixed output format, the stock tokenizer is a cost you have not checked — and the output side of the bill is the side worth measuring first.