Cutting inference costs by a third: routing, caching, batching

The way to reduce LLM inference costs is not to argue with the vendor about price. It is to send fewer tokens, send them to the cheapest model that passes your evals, and stop paying real-time rates for work nobody is waiting on. Those four moves, routing, caching, batching and trimming context, are the whole of it. Applied in that order to a product that has been running for a few months, they usually take a third or more off the monthly bill without a user noticing. This article explains each lever, what it costs to apply, and how to make sure the savings do not quietly degrade answers.

One caveat before the detail: you cannot cut what you cannot see. If the bill arrives as a single line, the first job is per-feature cost attribution, which is covered in LLM observability: tracing every request from prompt to cost. Everything below assumes you know which features spend what.

Where the inference bill actually goes

Most production LLM bills are dominated by input tokens, not output. A support agent that stuffs a long system prompt, five retrieved passages and the full history into every call is sending the same few thousand tokens over and over. A classification job on the frontier model is paying for reasoning it does not need. A nightly batch on the real-time endpoint is paying for latency nobody asked for. The levers map onto those waste patterns directly.

Lever 1: route by task, not by habit

Most products started on one model because that is what the first engineer tried. Six months later every call, from extracting a date to drafting a nuanced reply, goes to the same frontier endpoint. Routing means classifying each call type and assigning it the cheapest model that passes the evaluation set for that task. Extraction, classification, reformatting and simple summarisation are candidates for small hosted or open-weight models. Reasoning-heavy and customer-facing generation may stay on the frontier model.

The discipline is that routing decisions are made by the eval suite, not by opinion. We keep a golden set per call type, run every candidate model against it, and only move a call type when the cheaper model matches on the metrics that matter. Our routing layer is model-agnostic across OpenAI, Anthropic, Google and open-weight models, so switching a route is a config change and rolling back is the same change in reverse. The multi-model routing article covers the mechanics.

A route can also carry a fallback for outages and a confidence threshold that escalates hard cases to the larger model. Watch the escalation rate; if it climbs, the small model is being asked to do something it cannot.

Lever 2: prompt caching for the prefix you send every time

Prompt caching lets the provider reuse the computation for a prompt prefix that is identical across calls. The major vendors offer it; Anthropic's prompt caching documentation describes how cached prefixes are priced and how long they persist. The engineering work is almost entirely prompt restructuring: put the stable material first (system instructions, tool definitions, policy text, few-shot examples), and put the volatile material last (the user's message, the retrieved passages, the conversation turn).

The common mistake is a timestamp, a user name or a random ordering near the top of the prompt, which breaks the cache on every call. We audit prompts for anything that changes between requests and move it below the cache boundary. The second mistake is assuming the cache is working; we log cache hit rates per call type and alert when they fall, because a prompt edit that silently reorders a block can double the cost of a feature overnight.

Lever 3: cache retrieval and repeated answers

Prompt caching is the vendor's cache. LLM caching on your side is separate and often bigger. Three layers are worth building. First, embedding cache: the same chunk should never be embedded twice, and the same query string within a short window should not be re-embedded either. Second, retrieval cache: for a given normalised query and permission scope, the ranked passages can be reused for minutes to hours depending on how fast the corpus changes. Third, response cache: for genuinely repeated questions with no user-specific content, the final answer can be served directly, with a short TTL and a fingerprint of the source documents so a policy change invalidates it.

Invalidation is the hard part. Tie retrieval and response caches to the document version, not the clock alone. Never cache user-specific data across users; scope keys include the permission context. And keep a way to bypass the cache for a single request so support can reproduce a live answer.

Lever 4: batch inference for anything nobody is waiting on

Nightly document classification, backfills, embedding a new corpus, generating summaries for a report, scoring last week's conversations for quality: none of these need a response in two seconds. Vendors price asynchronous batch endpoints below real-time rates in exchange for a completion window measured in hours. OpenAI's Batch API guide describes the file-based submission model. Self-hosted deployments get the same effect by scheduling batch jobs on otherwise idle GPU capacity.

The engineering is a queue, a job table with status, idempotent result handling and a retry for items that fail. The product change is honesty about latency: if the summary appears tomorrow morning rather than instantly, say so in the interface.

Lever 5: trim the context you send

Once the other levers are in place, look at what each call actually contains. Conversation history rarely needs to be the full transcript; a rolling summary plus the last few turns performs as well for most assistants. Retrieval that returns ten passages when the answer is in the top three is paying for seven passages of noise and often lowering answer quality at the same time. Tool definitions that are never called in a given route can be dropped from that route.

Trimming is the lever with real quality risk, so it is the one that most needs evals. Reduce the passage count and measure groundedness and recall on the golden set. Shorten history and measure task completion on multi-turn conversations. The method is in how to measure RAG quality. If a trim costs quality, keep the tokens; they are cheaper than the churn.

A worked example

A B2B SaaS company with an in-app copilot came to us with a bill that had tripled since launch and no clear view of why. Tracing showed the copilot's system prompt and tool schema were sent uncached on every turn, retrieval returned eight passages by default, and a nightly "insights" feature ran through the real-time endpoint. We restructured the prompt so the stable prefix cached, cut default retrieval to the reranked top few after evals showed no loss, moved insights to the batch endpoint, and routed the copilot's intent classification to a small model. Quality metrics on the golden set stayed level; the bill dropped substantially in the first full month and finance got a per-feature cost line for the first time. The work ran under a Care Plan, the kind of tuning a running system needs and a build project never gets round to. A comparable engagement is described in in-app copilot for a B2B SaaS.

Team and timeline

Cost reduction work is best done by the people who already maintain the system, with an evaluation suite in hand. For most products it is a four to six week effort: one week on attribution and a baseline, one to two on caching and batching, and two to three on routing and trimming with eval runs between each change. It fits naturally inside a Standard or Enterprise Care Plan, where the monthly hours cover the changes and the cost review becomes a standing agenda item. If the system is new to us, we begin with a short takeover audit. For a product that has never had an evaluation set, LLM application engineering builds one first. Care Plan pricing starts at $1,000 a month on the pricing page.

Before you start: a checklist

• Per-feature cost attribution in place, with tokens and spend per call type
• A golden evaluation set for every call type you intend to route or trim
• Prompts audited for volatile content above the cache boundary
• Cache hit rate logged per call type and alerted on
• A queue and job table ready for batch work
• Invalidation rules for retrieval and response caches tied to document versions
• A rollback path for every routing change
• A monthly cost review with someone from finance in the room

Glossary

• Prompt caching: vendor-side reuse of computation for an identical prompt prefix
• Cache boundary: the point in the prompt after which content may vary between calls
• Model routing: sending each call type to the cheapest model that passes its evals
• Batch inference: asynchronous processing at a lower rate with a completion window
• TTL: time to live; how long a cached item is served before being refreshed
• Golden set: a fixed set of inputs with expected outputs used to measure quality

Related reading

LLM inference costs: how to forecast your monthly bill, Total cost of ownership for AI systems and What a Care Plan should cost and what it should include cover the surrounding budget questions.

Send fewer tokens, to cheaper models, at off-peak rates, and let the eval suite tell you when to stop.