Prompt Caching, Batches API, and Model Routing to Cut LLM Costs

LLM spend is an engineering variable, not a fixed bill — one that can be measured and reduced with the same rigor as query latency or memory footprint. That reframing is the starting point for three concrete levers that lower a Python-based Claude API bill without touching which model wins the benchmark chart.

None of them require switching providers. Two are configuration changes to an existing request. The third is a routing decision made before the request goes out.

Where the Money Actually Goes

A long system prompt, a tool list, or a RAG context block gets billed as input on every request, not written once. According to a worked example in a source writeup on prompt caching and cost control in Python, a 20K-token system prompt sent across 10,000 requests adds up to 200 million input tokens — at Opus 4.8 rates, that is $1,000 before the model produces a single output token.

Verbose output compounds the problem. It costs twice: once directly, since output tokens bill at the higher rate, and again on the next turn, when that verbosity gets carried forward as input history. Fixing prompt structure and output length matters before reaching for any of the three levers below.

Lever 1: Cache the Prefix That Doesn't Change

Anthropic's prompt caching lets a request mark a prefix — system instructions, tool definitions, a large document — for reuse. The writeup's numbers: a cache read costs roughly 0.1× the base input price, while a cache write costs 1.25× with a 5-minute TTL or 2× with a 1-hour TTL. At a 5-minute TTL, two requests against the same prefix already break even (1.25× + 0.1× beats 2× paid uncached); a 1-hour TTL needs about three requests to earn back the higher write cost.

A minimal illustrative shape, not the writeup's exact code, looks like this in the Python SDK:

# illustrative example — adapt to your own client wrapper
response = client.messages.create(
    model="claude-opus-4-8",
    system=[
        {
            "type": "text",
            "text": SYSTEM_PROMPT,
            "cache_control": {"type": "ephemeral"},
        }
    ],
    messages=[{"role": "user", "content": user_input}],
)
print(response.usage.cache_read_input_tokens)
print(response.usage.cache_creation_input_tokens)

Two failure modes are worth checking for immediately. First, the minimum cacheable prefix is model-dependent — the writeup notes Opus 4.8 needs at least 4,096 tokens. Below that threshold, cache_control silently does nothing: no error is raised, the request just shows a nonzero cache_creation_input_tokens and no read ever follows. Second, if cache_read_input_tokens stays at zero across requests that look identical, something in the prefix is quietly changing between calls — a datetime.now() baked into the system prompt, a uuid4() near the front, json.dumps(d) called without sort_keys=True, or a tool list assembled from an unordered set.

Lever 2: Move Non-Urgent Calls to the Batches API

Not every call needs a response in two seconds. Nightly report summarization, bulk document classification, backfilling embeddings metadata, and re-scoring an eval set are not latency-sensitive — and that is exactly the workload profile the Message Batches API is priced for.

According to the writeup, the Batches API discounts standard token pricing by 50% in exchange for asynchronous processing: most batches finish within an hour, the hard ceiling is 24 hours, and results stay retrievable for 29 days. The two discounts stack independently — a batch of 10,000 classification calls that all share one large system prompt gets both the 50% batch discount and the cache-read discount on that shared prefix, per the writeup's own example.

Lever 3: Route Easy Calls Through a Cheap Model First

The cheapest tokens are the ones sent to a cheaper model. In the writeup's support-ticket example, if 80% of tickets are confidently triaged by Haiku at $1/$5 per million tokens, and only the remaining 20% escalate to Opus at $5/$25 per million tokens, the blended cost comes out to a fraction of routing everything through Opus — with no quality loss on the easy majority, because the escalation path exists for exactly the cases where the cheap model reports it isn't sure.

The writeup flags one failure mode to guard against directly: a cheap model that is overconfident. Routing only works if the triage step actually escalates uncertain cases instead of guessing past them.

The pressure to have this lever ready isn't abstract. A separate report tracking API pricing across vendors describes a full-blown price war erupting across every major AI provider, one where the premium tier is shrinking fast enough that GPT-4 Turbo — still in production use at some enterprises — is now, in that report's words, 'laughably overpriced' compared to what's currently available. A routing layer that can point traffic at whichever tier actually fits the task is what turns that shift into savings instead of just noise.

Where These Levers Backfire

Each lever has a matching way to lose money. Caching a prefix that changes on every request pays the write premium with zero reads — worse than not caching at all. Sending latency-sensitive traffic through the Batches API breaks the product for users waiting on a synchronous reply. And routing decisions built on a cheap model's confidence score, without an escalation path, just moves the quality problem downstream instead of removing it.

Measure Before You Optimize

Before implementing any of the three levers, pull the usage fields the API already returns on every response. They answer whether a lever will pay off before a line of routing logic gets written.