OpenRouter vs Direct API: What Breaks in Production

A production chatbot answering questions over a private knowledge base at roughly 130,000 chat completions a month forces a decision most teams get wrong: call OpenAI, Anthropic, and Gemini directly, or route through an aggregator like OpenRouter. The token price gap between the options is small. The operational gap is not.

The workload nobody sizes correctly

Model the traffic before comparing vendors. A mid-size storefront running an in-app support assistant might see 40,000 sessions a month, a little over three turns per session, and roughly 2,800 prompt tokens plus 180 completion tokens per turn after retrieval — call it 130,000 chat completions a month. At that volume the per-token cost is real but "boring": it moves by a factor of two or three depending on which model answers, and every comparison article already covers that line. The costs nobody models sit off the invoice entirely — the embedding refresh every time a policy page changes, keeping a vector store warm for sub-100ms retrieval, and the engineer-weeks spent teaching one backend three different retry dialects.

Three vendors, three rate-limit rulebooks

Going direct means three accounts, three key rotations, three quota regimes, and three sets of rate-limit headers whose semantics don't agree. OpenAI meters requests and tokens per minute against a usage tier. Anthropic publishes its own per-model limits and its own overage behavior. Google's Gemini API adds project-level quotas that a traffic burst can trip independently of the other two. None of it is individually hard — there's just three times as much of it, concentrated in the retry path, which by definition only executes once something has already gone wrong.

Three invariants that outlast whichever vendor you pick

Regardless of direct-vs-gateway, a production support chatbot has to hold three properties. A turn settles exactly once: if a dropped connection replays the same user turn, the transcript can't end up with two contradictory answers for someone to reconcile in a billing dispute six months later — client-supplied idempotency keys, keyed by turn id rather than by attempt, are what make a retry safe. Every answer is attributable: which model answered, which knowledge-base revision grounded it, what it cost, how long it took — "a model said so" is not an audit trail. And routing has to respect region: if the knowledge base holds EU customer data, a gateway that quietly moves a request to another region turns a resilience feature into a compliance incident.

Direct vs. gateway vs. cloud-native

Option	Credentials & invoices	Who owns retries/rate limits	Fits when
Direct (OpenAI, Anthropic, Gemini)	One account/key/invoice per vendor	You do, three times, three header conventions	You need a provider-exclusive feature or a committed-use contract
OpenRouter	One key, one invoice, per-request routing preferences	Gateway handles fallback; you still tune per-model limits	Broad catalogue, fast model experimentation
Bedrock / Vertex AI	Folded into a cloud contract you already have	The IAM and quota model your team already operates	You're deep in AWS or GCP and want regional control

Read that table as an operating-cost sheet, not a price list. The direct row costs integration weeks up front and a monthly reconciliation ritual forever — three invoices on three billing periods that don't line up against your own per-turn ledger without extra work. The gateway rows trade a margin for a bill you can tie back to request IDs.

What to watch

The routing decision that saves the most money isn't vendor selection — it's per-request model tiering. Retrieval already carries most of the answer for routine traffic (order status, return windows, shipping), so a fast mid-tier model grounded on good excerpts beats a frontier model grounded on nothing; escalate to a stronger model only when retrieval confidence is low or the question touches money. Before committing to any option, check which regions and vendors a gateway actually supports for your capability — don't assume "resilient failover" and "your compliance boundary" mean the same thing to the vendor.