Cut Your LLM Token Bill by 85% in 2026: Caching, Routing, and Context Discipline

Every team that shipped an LLM feature in 2025 is having the same meeting in 2026: someone pulls up the API bill, someone else says “but tokens got cheaper,” and nobody can explain the gap. Per-token prices genuinely did collapse. Bills went up anyway.

The reason is structural. We stopped sending single prompts and started running agent loops, and an agent loop does not consume tokens linearly — it consumes them quadratically. This post is the engineering fix: where the tokens actually go, and the five levers that reliably take 80-90% off the bill without downgrading your output.

First, the thing nobody tells you about tokens

A token is roughly 4 characters of English, or about ¾ of a word. That heuristic is fine for capacity planning and actively misleading for cost planning, because of three asymmetries:

• Output costs 5x input. Across every major provider, output tokens are priced at 4-6x the input rate. A 200-token answer costs more than a 900-token prompt.
• Non-English text costs 2-5x more. BPE tokenizers are trained predominantly on English. The same sentence in Hindi, Thai, or Japanese fragments into far more tokens — sometimes one token per character. Your Bengaluru support bot and your Boston support bot do not cost the same per conversation.
• You pay for the whole context on every call. Not the delta. The whole thing. This is the one that kills agent workloads.

The quadratic trap in agent loops

Here is the math that explains most surprise bills. In a multi-step agent, each step re-sends the entire accumulated conversation. If S is your static system prompt plus tool definitions, u is tokens added per user/assistant turn, and r is tokens added per tool result:

Total input tokens = N·S + u·N(N+1)/2 + r·N(N-1)/2

Those triangular-number terms are the trap. Plug in a realistic agent — 4,000-token system prompt with tool schemas, 1,200 tokens per turn, 800 tokens per tool result:

A 10-step loop costs 23x a single pass, not 10x. A 20-step loop costs 85x. If you estimated your agent budget by multiplying single-call cost by step count, you are off by an order of magnitude — and that is before retries, which drag their entire failed context into the next attempt.

Published instrumentation of real agent traces backs this up: in one measured run, 30,400 of 48,400 total tokens came from tool results alone, and 40-60% of that was removable with no measurable quality loss.

The 2026 pricing landscape

Verified against provider documentation at the time of writing (per 1M tokens, USD). Note the cached input column — it is the single most under-used number in this table.

Two structural discounts apply nearly everywhere and stack with each other:

• Prompt caching: input drops to 10% of base rate on a cache hit. Anthropic and Google both price cache reads at 0.1x; OpenAI’s cached input runs 0.1x as well. Cache writes cost a premium (1.25x for a 5-minute TTL, 2x for an hour on Anthropic), so caching pays for itself after the first or second read.
• Batch API: 50% off input and output across Anthropic, OpenAI, and Google, in exchange for asynchronous completion.

Prices shift constantly — Gemini’s current rates are documented as holding through December 31, 2026, and OpenAI’s gpt-5.6-sol pricing is explicitly promotional. Pin the numbers in a config file, not in your head.

Lever 1: Prompt caching (the free 25%)

If your system prompt, tool definitions, and few-shot examples are stable across calls — and they almost always are — you are paying full price to re-upload the same bytes thousands of times a day.

The rule: put everything static at the front, everything dynamic at the back. Caching works on prefixes. One dynamic token near the top invalidates the entire cache below it.

import anthropic

client = anthropic.Anthropic()

resp = client.messages.create(
    model="claude-sonnet-5",
    max_tokens=1024,
    system=[
        {
            "type": "text",
            "text": SYSTEM_PROMPT + TOOL_DOCS + FEW_SHOT_EXAMPLES,
            "cache_control": {"type": "ephemeral"},   # <- everything above this is cached
        },
    ],
    messages=[{"role": "user", "content": user_query}],   # dynamic, never cached
)

u = resp.usage
print(f"write={u.cache_creation_input_tokens} "
      f"read={u.cache_read_input_tokens} "
      f"fresh={u.input_tokens} out={u.output_tokens}")

That last print statement matters more than the cache itself. If cache_read_input_tokens is zero on your second identical call, your prefix is not actually stable — usually because a timestamp, a session ID, or a re-ordered JSON key snuck into the top of the prompt. Log this ratio in production; a falling cache hit rate is a cost regression you can catch before the invoice does.

For multi-turn agents, mark the cache breakpoint at the end of the last completed turn so each new step reads everything before it from cache.

Lever 2: Route by difficulty, not by habit

Most production traffic is not hard. Classification, extraction, routing, formatting, short factual lookups — these run fine on a model that costs 20-50x less. Teams default to the flagship for everything because it is one line of config, and pay for that convenience on every single call.

The published research on this is unusually encouraging: routing frameworks retain roughly 95% of frontier-model quality while sending only 14-26% of calls to the expensive model.

You do not need a router framework to start. A cheap classifier in front of your expensive model captures most of the win:

TIERS = {
    "simple":  {"model": "claude-haiku-4-5",  "in": 1.00, "out": 5.00},
    "complex": {"model": "claude-opus-5",     "in": 5.00, "out": 25.00},
}

ROUTER_PROMPT = """Classify this request as 'simple' or 'complex'.
simple  = lookup, extraction, classification, formatting, rephrasing
complex = multi-step reasoning, code generation, ambiguous requirements
Reply with one word."""

def route(query: str) -> str:
    verdict = client.messages.create(
        model="claude-haiku-4-5",
        max_tokens=5,
        system=ROUTER_PROMPT,
        messages=[{"role": "user", "content": query}],
    ).content[0].text.strip().lower()
    return TIERS.get(verdict, TIERS["complex"])["model"]   # fail toward quality

Two design notes that matter in production. Fail toward quality — when the router is uncertain or returns garbage, send it to the strong model; a mis-routed hard query costs you a user, a mis-routed easy query costs you a fraction of a cent. And log every routing decision alongside the eventual outcome, so you can tune the boundary on evidence instead of vibes.

Lever 3: Context discipline in agent loops

This is where the quadratic curve gets broken. Four patterns, in rough order of payoff:

Filter tool payloads before they enter context

The single biggest source of agent bloat is raw JSON from tool calls. An API returns 3,000 tokens of response envelope, pagination metadata, and null fields; the agent needs six of them. Extract deterministically in code — not by asking the model to summarize, which costs tokens to save tokens.

def compact_tool_result(raw: dict, fields: list[str]) -> str:
    """Deterministic extraction. No LLM call, no token cost."""
    rows = raw.get("results", raw.get("data", []))
    return "\n".join(
        " | ".join(f"{f}={r.get(f)}" for f in fields if r.get(f) is not None)
        for r in rows[:20]        # hard cap: agents rarely need row 21
    )

Compact on a schedule

Every 10-15 tool calls, replace the accumulated transcript with a structured summary of state — decisions made, facts established, current objective — and drop the raw history. Measured savings land around 22%, and more importantly it converts a quadratic curve into a sawtooth.

Split coordinator from specialists

The highest-leverage architectural change. An orchestrator holds the plan and delegates each sub-task to a subagent with a fresh, minimal context; only the subagent's compact result returns to the coordinator. Benchmarks put this at ~54% average token reduction, with the coordinator itself consuming under 10% of total tokens.

Cap everything

Hard limits on steps, retries, and max_tokens. An unbounded retry loop is the only bug in this list that can produce a five-figure invoice overnight. Set a per-task token budget and abort when it is exceeded — a failed task is cheaper than an infinite one.

Lever 4: Semantic caching for repetitive traffic

If you run a support bot, a docs assistant, or anything customer-facing, a large share of your queries are paraphrases of each other. "How do I reset my password" and "password reset help" are the same question and should cost one inference, not two.

Exact-match caching catches almost none of these. Semantic caching embeds the query, does a vector similarity search against previous queries, and returns the stored response above a similarity threshold. Reported savings on high-repetition workloads run to ~70%, with cache hits returning in milliseconds instead of seconds — so this is a latency win as much as a cost win.

import numpy as np, redis
from redis.commands.search.query import Query

r = redis.Redis()
THRESHOLD = 0.92   # tune on your own traffic; too low returns wrong answers

def cached_completion(query: str):
    vec = embed(query)                       # your embedding model
    hit = r.ft("qcache").search(
        Query("(*)=>[KNN 1 @v $bv AS score]")
            .return_fields("response", "score")
            .dialect(2),
        {"bv": np.asarray(vec, dtype=np.float32).tobytes()},
    )
    if hit.docs and (1 - float(hit.docs[0].score)) >= THRESHOLD:
        return hit.docs[0].response, True     # cache hit, $0

    answer = call_llm(query)
    r.hset(f"q:{hash(query)}", mapping={
        "v": np.asarray(vec, dtype=np.float32).tobytes(),
        "response": answer,
    })
    return answer, False

The threshold is the whole game. Set it too loose and you serve confidently wrong answers to slightly different questions — which is far more expensive than the tokens you saved. Start at 0.95, measure false-hit rate against a labelled set, and only then loosen it. Never semantically cache anything personalized or account-specific.

Lever 5: Batch anything that is not interactive

Nightly summarization, bulk classification, embedding backfills, eval runs, data enrichment — none of these need a synchronous response, and all of them are eligible for 50% off both input and output. This is the highest ratio of savings to engineering effort on the entire list, and most teams simply never revisit which of their workloads are actually interactive.

Putting it together: a worked example

A support agent handling 100,000 requests/month. Per request: 4,000 tokens of system prompt and tool schemas, 6,000 tokens of retrieved context, 2,000 tokens of conversation history, 400 tokens of output. Everything on Claude Opus 5, no optimization:

None of these steps changes the model's ceiling on the hard queries. The flagship still handles the 30% that need it, with the same context quality — the retrieval trim in step three is a reranking improvement, which typically raises answer quality by cutting distractor passages.

Note the ordering. Caching first, because it is a configuration change with no quality risk. Routing second, because it is the largest single lever. Context trimming third, because it requires evaluation work. Semantic caching last, because it carries the most correctness risk.

Instrument before you optimize

Every number above is worthless if you cannot see your own breakdown. Minimum viable instrumentation — log these per request, tagged by feature, user tier, and model:

from dataclasses import dataclass, asdict

PRICES = {  # per 1M tokens: (input, cached_input, output)
    "claude-opus-5":    (5.00, 0.50, 25.00),
    "claude-haiku-4-5": (1.00, 0.10,  5.00),
}

@dataclass
class Spend:
    model: str; feature: str
    fresh_in: int; cached_in: int; out: int

    def usd(self) -> float:
        pin, pcache, pout = PRICES[self.model]
        return (self.fresh_in * pin + self.cached_in * pcache
                + self.out * pout) / 1e6

    def cache_hit_rate(self) -> float:
        total = self.fresh_in + self.cached_in
        return self.cached_in / total if total else 0.0

Then watch three ratios: cache hit rate (should be climbing), share of calls hitting the flagship model (should be falling), and tokens per completed task (the only metric that catches agent loops going quadratic). Cost per request hides regressions; cost per completed task does not.

The checklist

1. Log fresh input, cached input, and output tokens per request, tagged by feature. Nothing else on this list matters without it.
2. Move every static token to the front of the prompt and enable prompt caching. Verify the hit rate is non-zero in production.
3. Set max_tokens on every call and ask for brevity in the prompt — output is your most expensive token class.
4. Classify your traffic and route the easy majority to a cheap model. Fail toward quality on uncertainty.
5. Filter tool payloads deterministically in code before they enter context.
6. Cap agent steps, retries, and per-task token budget. Compact the transcript every 10-15 calls.
7. Move non-interactive workloads to the Batch API for a flat 50%.
8. Add semantic caching only where traffic genuinely repeats, with a threshold you have measured.
9. Re-check provider pricing quarterly. It changes, and the changes have been in your favour.

The teams whose bills fell in 2026 were not the ones who found a cheaper model. They were the ones who stopped sending the same 4,000 tokens ten thousand times a day.