Cutting Agent Cost 8x Without Touching the Prompt
Router models, caching, structured outputs, and the one line of code that saved a client $14k/month.
Cutting Agent Cost 8x Without Touching the Prompt
Router models, caching, structured outputs, and the one line of code that saved a client $14k/month.
There's a predictable arc to every agent deployment. Month one: "it works!" Month three: "usage is growing!" Month five: someone from finance forwards the API invoice with a single question mark. In 2026, with agents finally moving into production at scale — nearly a third of enterprises now run at least one — runaway inference cost has become one of the top reasons projects get killed, right alongside unclear ROI. Which is maddening, because agent cost is one of the most fixable problems in this entire field.
Here's the thing most teams miss: the prompt is usually innocent. The waste lives in how you call the model, not what you say to it. Everything below left our client's prompts byte-for-byte identical — behavior unchanged, evals green — and stacked up to an 8x reduction. Let's go through the levers in order of effort.
Lever 1: Prompt caching — the one line of code
The story from the subtitle. A client ran a support agent handling ~40k conversations a month. Every single API call carried the same enormous prefix: a 6,000-token system prompt, tool definitions, policy documents, few-shot examples — about 9,000 tokens of identical content — followed by a few hundred tokens of actual conversation. They were paying full input price for those 9,000 tokens on every call, tens of thousands of times a month.
The fix was literally one line: adding a cache_control breakpoint to mark the static prefix as cacheable. On Anthropic's API, cached prefix reads bill at 10% of base input price (you pay a one-time write premium — 1.25x for the 5-minute tier, 2x for the 1-hour tier — then 0.1x on every read). OpenAI and Google now do implicit prefix caching with no write premium at all, at 50–90% read discounts. Same model, same prompt, same outputs. The invoice dropped by about $14k/month, and as a bonus, time-to-first-token improved by double digits because the provider skips re-processing the cached prefix.
The 2026 consensus is blunt: prompt caching is the highest-leverage, lowest-risk cost lever in production LLM systems, worth 45–80% off input costs for prefix-heavy workloads — and agents are the most prefix-heavy workload there is, because tool definitions and system instructions repeat on every step of every loop. The catch: caching works on exact prefix matches, so ordering matters. Static content first (system prompt, tools, examples), variable content last (user message, retrieved context). Put a timestamp at the top of your system prompt and you've silently disabled your cache. Audit this — it's the most expensive f-string in your codebase.
Lever 2: Model routing — stop paying frontier prices for non-frontier work
Log a week of production agent traffic and classify what the model is actually doing. In a typical agent loop, most calls are not hard reasoning. They're classification ("which tool does this need?"), extraction ("pull the order ID"), reformatting, summarizing a tool result. Frontier models in mid-2026 — Claude Fable 5, GPT-5.6, Gemini 3.1 Pro — are extraordinary, and using them to parse a date out of an email is like hiring a surgeon to apply band-aids.
Routing sends each call to the cheapest model that passes your evals for that step. The published numbers are consistent: open routing frameworks like RouteLLM report ~95% of frontier-model quality while sending only 14–26% of calls to the expensive model — a 75–85% cost cut on routed traffic. Current-generation small models (Haiku 4.5-class, GPT-5.6's mini tier, Gemini Flash) run at 10–25x cheaper per token than their flagship siblings and handle these auxiliary steps at near-parity.
You don't need an ML-based router to start. A static routing table — this agent step goes to the small model, this one to the flagship — captures most of the win, is fully debuggable, and is exactly what your eval suite (you have one, right?) can validate per-step. Add escalation for uncertainty later: small model first, promote to the flagship when confidence is low or output fails validation.
Lever 3: Structured outputs — the token diet
Two costly behaviors hide in free-text agent outputs. First, verbosity: models narrate ("Certainly! Based on my analysis of the retrieved documents, I can determine that the order status is...") when the next step of your pipeline needed {"status": "shipped"} — a 40-token answer where 8 would do, at output-token prices, which run 3–5x input prices. Second, parse failures: free text that your code can't reliably parse triggers retries, and every retry doubles the cost of that step.
Constrained decoding — JSON schema mode on all major APIs in 2026 — fixes both at once. Intermediate agent steps (tool selection, extraction, validation verdicts) should virtually never be free prose. Teams that make schemas the default report meaningful output-bill reductions and, just as valuable, the near-elimination of retry storms. Prose is for humans; your pipeline's internal chatter should be terse, typed, and machine-readable.
Lever 4: Fix the loop itself
The most expensive agent behavior isn't any single call — it's pathological trajectories. The agent that retries a failing tool eleven times. The one that re-reads the same document each iteration because results aren't carried in state. The runaway loop that burns 400k tokens before someone notices. Three boring controls, in rising order of importance: hard caps on steps and per-conversation token budgets, so a bad trajectory dies at $2 instead of $200; a trajectory-cost dashboard — track cost per completed task, not per call, and alert on outliers; and context pruning between steps, because agents accumulate conversation history like a hoarder and stale tool outputs from step 2 rarely need to ride along, at full price, into step 14. Also: anything non-interactive (nightly enrichment, backfills, report generation) goes to batch APIs at 50% off. Half price for tolerating latency you weren't using anyway.
The order of operations matters
A practical note on sequencing, because teams routinely do this backwards. Start with measurement — you cannot optimize an invoice you can't decompose. One day of tagging API calls by agent, step, and model, flowing into any dashboard, converts "the bill is $21k" into "tool-selection steps on the flagship model are $9k of it." Then take the levers in the order above: caching first (near-zero risk, no behavior change, biggest single win for agent workloads), routing second (needs eval coverage per step before you trust it), structured outputs third (touches code paths, needs regression testing), loop hygiene continuously. Teams that start with routing before they have per-step evals end up either too timid (routing 5% of traffic and saving nothing) or too aggressive (quality regressions that poison the whole cost program politically — one bad week of outputs and leadership hears "the cost cutting broke the agent" forever). Caching has no such failure mode, which is why it goes first even when routing promises bigger numbers on paper.
Stacking to 8x
These levers multiply rather than add. Real numbers from the engagement above: caching took input costs down ~60% across the workload. Routing moved ~70% of calls to a model a tenth the price. Structured outputs and loop hygiene cut total tokens ~30%. Batch moved a fifth of volume to half price. Net: monthly spend fell from ~$21k to ~$2.6k — 8.1x — with prompts untouched and eval pass rates identical. That last clause is the discipline that makes this safe: every lever ships behind the eval suite, and any optimization that drops the score gets reverted, no matter what it saves.
Cost is a feature you engineer, not a bill you receive. And in a year when 2026's model releases keep making intelligence cheaper every quarter, teams with this discipline get to ride the curve down — while everyone else just gets a bigger invoice with a question mark on it.
Want a cost audit on your agent stack? The first pass usually finds 3–5x sitting in plain sight — book a call.
