# A cheaper model can cost more when routing drops the prompt cache

> I explored a smart router that sends the easy questions to a cheaper model. In theory it trims the bill. In practice the prompt cache does not move with the request, and a lost cache hit can cost more than the smaller model saves.

Sending easy questions to a cheap model is the obvious way to cut an LLM bill. The classifier is the easy part. The saving is the part that did not show up.

![Why routing to a cheaper model can cost more. Steps: 1) Query arrives: Classified by keyword, embedding, or a small LLM classifier; 2) Route to the cheap model: Easy questions go to a smaller, cheaper model; 3) Cache is keyed per model: The warm prompt cache stays on the model you left; 4) Prefix recomputed cold: System prompt, tools and history reprocessed at full input price; 5) The cheap route costs more: The cold recompute can outweigh the smaller per-token rate; 6) Fix: pin the conversation: Sticky routing keeps one model so the cache stays warm](/garden-assets/experience/atom_notes/smart-router-prompt-cache/smart-router-prompt-cache-flow.svg?v=21ee5409c8)

The whole trap in one path: route to a cheaper model, lose the warm cache, recompute the prefix cold, pay more. Sticky routing is the way out.

## The router is the easy part

Keywords, an embedding match, or a small classifier scores how hard a query is, with escalation words to force the strong model. Easy goes cheap, hard goes capable. On paper it is a big lever: most traffic is routine and runs at the same felt quality on a mid-tier model.

## The catch: the cache does not move with the request

Prompt caching only pays when the next request lands on the same model. Route a turn elsewhere and the whole stable prefix, the system prompt, the tools, the history, is recomputed cold at full input price on the new model. On a long prefix that dwarfs the smaller model's per-token saving: the router shaves the output tokens and hands it back on the input. And a cheaper headline rate is not cheaper if the switch forfeits a cache hit you already had, which is what most routers [misprice](https://llmgateway.io/blog/cache-aware-llm-routing).

## The fix others use: pin the conversation

Cache-aware, or sticky, routing: classify the first turn, then keep the conversation on that model so its cache stays warm. LiteLLM ships it as [`session_affinity`](https://docs.litellm.ai/docs/proxy/auto_routing), OpenRouter as provider sticky routing. Done this way the two compound: LiteLLM's [benchmark](https://docs.litellm.ai/blog/auto-router-prompt-caching-benchmark) puts routing plus caching 37 to 69 percent below caching one model alone, and a router with caching off at about 4x more expensive than a single fixed model. The trade cuts both ways: the tier is pinned to the first turn, so a chat that opens hard then eases off overpays all the way, and one that starts simple then turns thorny stays stuck on the cheap model, unless an escalation trigger forces a re-route and eats one cold recompute to do it.

## What to explore next

* [ ] Measure it on my own traffic: naive per-turn routing against sticky routing and a single cached model, real numbers not the benchmark's.
* [ ] Find the break-even, how long a stable prefix and how high a cache-hit rate have to be before routing starts to lose money.
* [ ] Weigh the classifier's own cost and latency (keyword vs embedding vs small LLM) against the saving it unlocks.
* [ ] Design the escalation re-route: what signal trips it, and when one cold recompute is worth the stronger model.
* [ ] Price routing on cached-input rates, not headline rates, inside the router itself.
* [ ] Test a semantic response cache alongside it, to skip the model call entirely for repeat or paraphrased queries.

## The bottom line

Still a seed, and I have not measured my own numbers, so this is a trap spotted in theory. The rule already looks clear: price the decision against cached rates, not headline rates, and keep each conversation on one model. My [static per-agent pins](/garden/model-per-agent) sidestep it entirely, each agent stays on one model so the cache never falls out. Dynamic per-query routing is the same cost question on hard mode.