A cheaper model can cost more when routing drops the prompt cache

I explored a smart router that sends the easy questions to a cheaper model. In theory it trims the bill. In practice the prompt cache does not move with the request, and a lost cache hit can cost more than the smaller model saves.

Sending easy questions to a cheap model is the obvious way to cut an LLM bill. The classifier is the easy part. The saving is the part that did not show up.

Why routing to a cheaper model can cost more. Steps: 1) Query arrives: Classified by keyword, embedding, or a small LLM classifier; 2) Route to the cheap model: Easy questions go to a smaller, cheaper model; 3) Cache is keyed per model: The warm prompt cache stays on the model you left; 4) Prefix recomputed cold: System prompt, tools and history reprocessed at full input price; 5) The cheap route costs more: The cold recompute can outweigh the smaller per-token rate; 6) Fix: pin the conversation: Sticky routing keeps one model so the cache stays warm — The whole trap in one path: route to a cheaper model, lose the warm cache, recompute the prefix cold, pay more. Sticky routing is the way out.

The router is the easy part

Keywords, an embedding match, or a small classifier scores how hard a query is, with escalation words to force the strong model. Easy goes cheap, hard goes capable. On paper it is a big lever: most traffic is routine and runs at the same felt quality on a mid-tier model.

The catch: the cache does not move with the request

Prompt caching only pays when the next request lands on the same model. Route a turn elsewhere and the whole stable prefix, the system prompt, the tools, the history, is recomputed cold at full input price on the new model. On a long prefix that dwarfs the smaller model's per-token saving: the router shaves the output tokens and hands it back on the input. And a cheaper headline rate is not cheaper if the switch forfeits a cache hit you already had, which is what most routers misprice.

The fix others use: pin the conversation

Cache-aware, or sticky, routing: classify the first turn, then keep the conversation on that model so its cache stays warm. LiteLLM ships it as session_affinity, OpenRouter as provider sticky routing. Done this way the two compound: LiteLLM's benchmark puts routing plus caching 37 to 69 percent below caching one model alone, and a router with caching off at about 4x more expensive than a single fixed model. The trade cuts both ways: the tier is pinned to the first turn, so a chat that opens hard then eases off overpays all the way, and one that starts simple then turns thorny stays stuck on the cheap model, unless an escalation trigger forces a re-route and eats one cold recompute to do it.

What to explore next

Measure it on my own traffic: naive per-turn routing against sticky routing and a single cached model, real numbers not the benchmark's.
Find the break-even, how long a stable prefix and how high a cache-hit rate have to be before routing starts to lose money.
Weigh the classifier's own cost and latency (keyword vs embedding vs small LLM) against the saving it unlocks.
Design the escalation re-route: what signal trips it, and when one cold recompute is worth the stronger model.
Price routing on cached-input rates, not headline rates, inside the router itself.
Test a semantic response cache alongside it, to skip the model call entirely for repeat or paraphrased queries.

The bottom line

Still a seed, and I have not measured my own numbers, so this is a trap spotted in theory. The rule already looks clear: price the decision against cached rates, not headline rates, and keep each conversation on one model. My static per-agent pins sidestep it entirely, each agent stays on one model so the cache never falls out. Dynamic per-query routing is the same cost question on hard mode.