Model Routing vs. Prompt Caching: Which Saves More on LLM Costs?

Two cost reduction strategies dominate every serious LLM architecture conversation right now: model routing and prompt caching. Both can slash your API bill significantly. But they work in fundamentally different ways, and the wrong choice for your workload can mean leaving 80% of the savings on the table.

This post breaks down exactly how each strategy works, shows real cost math for both, and gives you a decision framework to know which one (or both) to reach for first.

What Is Model Routing?

Model routing means sending each request to the cheapest model that can handle it adequately — rather than sending everything to your most capable (and most expensive) model. A simple classification or FAQ lookup doesn't need GPT-4o. A cheap, fast model handles it just as well at a fraction of the cost.

Routing decisions are typically made by:

Task type: classification, extraction, summarization, generation, reasoning
Query complexity: keyword heuristics, a small classifier model, or even a cheap LLM pre-screening step
User tier: free users get Haiku; paid users get Sonnet or GPT-4o
Confidence threshold: retry on a stronger model if the cheap model returns low confidence

The cost savings come from the massive price gap between frontier and smaller models. In 2026, GPT-4o mini costs roughly 94% less per token than GPT-4o. Claude Haiku 4 costs 87% less than Claude Sonnet 4. Route even 50% of your traffic to smaller models and your bill compresses substantially.

What Is Prompt Caching?

Prompt caching is a provider-level feature that lets you mark a portion of your prompt as cacheable. On subsequent requests that share the same cached prefix, the provider skips reprocessing those tokens and charges you a deeply discounted rate — typically 75-90% less than normal input pricing.

Anthropic's prompt caching (available on Claude models) charges cached tokens at 10% of the standard input rate after a minimum cache lifetime. OpenAI's equivalent feature (available on GPT-4o and o-series models) charges cached tokens at 50% of the standard input rate. Both providers require a minimum prefix length to qualify for caching.

Caching is most effective when:

You have a long, repeated system prompt (product context, personas, instructions)
Many users see the same documents or RAG context in their prompt
You're running multi-turn conversations where history is included each time
You have high-volume support bots with identical preamble across tickets

The Numbers: Model Routing Savings

Let's model a customer support application running on Claude Sonnet 4, handling 100,000 requests per month. Each request averages 800 input tokens and 400 output tokens.

Baseline (all Sonnet 4):

Input: 100K * 800 = 80M tokens * $3.00/M = $240/mo
Output: 100K * 400 = 40M tokens * $15.00/M = $600/mo
Total: $840/mo

Now suppose analysis shows 60% of requests are simple FAQ lookups and classification tasks that Claude Haiku 4 handles just as well. Route those to Haiku ($0.80/M input, $4.00/M output). The remaining 40% of complex requests stay on Sonnet.

After routing (60% Haiku, 40% Sonnet):

Haiku input: 60K * 800 = 48M tokens * $0.80/M = $38.40/mo
Haiku output: 60K * 400 = 24M tokens * $4.00/M = $96/mo
Sonnet input: 40K * 800 = 32M tokens * $3.00/M = $96/mo
Sonnet output: 40K * 400 = 16M tokens * $15.00/M = $240/mo
New total: $470.40/mo
Savings: $369.60/mo (~44%)

That's squarely in the 40-60% savings range that well-implemented model routing typically delivers. Push the Haiku split higher (to 70-80%) if your query mix allows, and savings approach 55-60%.

The Numbers: Prompt Caching Savings

Now model a documentation assistant built on Claude Sonnet 4. Each request includes a 10,000-token system prompt containing product docs, a 200-token user question, and generates a 500-token answer. Volume: 50,000 requests per month.

Baseline (no caching):

Input: 50K * 10,200 = 510M tokens * $3.00/M = $1,530/mo
Output: 50K * 500 = 25M tokens * $15.00/M = $375/mo
Total: $1,905/mo

Enable prompt caching on the 10,000-token system prompt. Anthropic charges cached tokens at $0.30/M (10% of $3.00). The 200-token user question and the output are unchanged.

After caching (assuming 95% cache hit rate on system prompt):

Cached input: 50K * 0.95 * 10,000 = 475M tokens * $0.30/M = $142.50/mo
Non-cached input: 50K * 0.05 * 10,000 + 50K * 200 = 25M + 10M = 35M tokens * $3.00/M = $105/mo
Output (unchanged): $375/mo
New total: $622.50/mo
Savings: $1,282.50/mo (~67%)

Notice that caching saved more in absolute dollars here — but only because the system prompt was enormous (10,000 tokens) and the hit rate was high. The savings on cached tokens approached 90%; overall bill savings were 67% because output costs are unaffected.

Decision Framework: Which Strategy Wins?

Factor	Routing Wins	Caching Wins
Query diversity	High diversity, unpredictable inputs	Low diversity, many similar requests
System prompt size	Short or varied prompts (<1K tokens)	Long, stable prompts (>2K tokens)
Task complexity mix	Mixed — some tasks need big models, most don't	Uniform — all requests need same capability
Output token volume	High output volume (routing reduces output cost too)	Input-heavy workloads (caching only affects input)
Implementation effort	Moderate (need routing logic, quality checks)	Low (add cache_control flag to prompt)
Primary cost driver	Per-request model cost is the main lever	Repeated long context is the main driver

A simple way to think about it: routing wins when your requests vary; caching wins when your prompts repeat.

If you have a chatbot where every user asks completely different questions with a short system prompt, caching helps little. Route the easy queries to a cheap model. If you have a RAG assistant where 90% of every prompt is the same retrieved document, caching that document is nearly free money — no routing logic required.

When Routing Wins Clearly

Model routing is the right primary strategy when:

Your workload spans multiple task types with different capability requirements (e.g., classification + summarization + complex reasoning in the same pipeline)
A meaningful fraction of requests (30%+) are simple enough for a small model
Output tokens are a large share of your cost — routing to a cheaper model cuts both input and output simultaneously
You're on a provider like OpenAI where cached token discounts are only 50% (less dramatic than Anthropic's 90%)

When Caching Wins Clearly

Prompt caching is the right primary strategy when:

Your system prompts are long (3,000+ tokens) and stable across requests
You're building a RAG system where the retrieved context is frequently the same (or very similar) across users
Your use case inherently needs a frontier model — you can't route down — but the context is highly repetitive
You're on Anthropic's API where cached tokens cost just 10% of normal input price
Implementation time is limited — adding a cache_control flag takes minutes; building a router takes days

Using Both Together

The highest-leverage architectures combine both strategies. They're not mutually exclusive — they attack different cost components.

Here's what a combined approach looks like for a high-volume SaaS assistant:

Add prompt caching to your shared context. Any system prompt, document context, or instruction block that's stable across requests gets the cache_control flag. This is almost always worth doing — it takes 10 minutes and the savings are immediate.
Layer routing on top of that. Once caching is reducing your input costs on big-model requests, analyze your query mix. Identify the task types a smaller model can handle. Route those, keeping caching enabled on your Haiku/mini calls too if they share the same system prompt.
Measure the combined effect. With both strategies active, the savings compound. Caching cuts the input cost of the requests that stay on expensive models. Routing eliminates the expensive model cost for the requests that don't need it.

Going back to our documentation assistant example: if we also implement routing to shift 40% of the simpler queries to Claude Haiku 4 (with caching on those calls too), the combined savings can exceed 75-80% versus the unoptimized baseline.

Strategy	Typical Savings	Implementation Effort	Best For
Model Routing only	40–60%	Moderate	Diverse queries, mixed task types
Prompt Caching only	50–90% on input tokens	Low	Repeated long context, RAG
Both combined	65–85% total	Moderate	High-volume production apps

The Trap: Optimizing Blind

Both strategies require knowing your actual cost structure before you can apply them effectively. Teams that implement routing without knowing their query distribution often over-route to cheap models and degrade quality. Teams that implement caching without knowing their prompt structure often cache the wrong things and see minimal impact.

The questions you need to answer before choosing:

What percentage of my requests are truly simple vs. complex?
How large is my system prompt, and does it change between requests?
What is my cache hit rate likely to be, given my request patterns?
Are input or output tokens driving more of my cost?
Which provider am I on, and what are their specific cache discount rates?

Without this data, you're guessing. A tool like LLMeter gives you the visibility you need — breaking down costs by provider, model, input vs. output token split, and request patterns, without requiring any code changes or proxy layer. Once you can see which models are carrying what load and where your token spend is concentrated, both routing and caching decisions become obvious rather than speculative.

A Practical Starting Point

If you're not sure where to begin, here's the fastest path to meaningful savings:

Measure first. Instrument your API calls (or use LLMeter) to get a clear picture of your cost breakdown by model and token type. Two minutes of setup, immediate insight.
Check your system prompt length. If it's over 2,000 tokens and stable, enable prompt caching today. This is the lowest-effort, highest-certainty win available.
Audit your query types. If 30%+ of requests are classifiable as "simple," start building a routing layer. Even a rule-based classifier (query length, keyword presence) beats no routing at all.
Set a quality baseline before routing. Capture output quality metrics on your current model so you can verify the smaller model isn't degrading user experience.
Combine once both are working. Once caching is live and routing is validated, layer them together and measure the combined effect.

LLM cost optimization is not a one-time project — it's an ongoing practice as your usage grows and providers update pricing. The teams that control their bills in the long run are the ones who can see exactly where every dollar is going and act on that data systematically.