You run a production LLM service. Your bills keep climbing because every request hits your most capable—and expensive—model. Smart routing promises to cut those costs by 10X by sending simple queries to cheaper models. But the math only works if you understand the actual mechanics: how routing decisions happen, what you trade away, and when the overhead cancels out your savings. This article breaks down the engineering reality behind model routing systems and what determines whether they help or hurt your bottom line.
How routing systems classify query complexity
A routing system sits between your application and your LLM fleet. It examines each incoming request and decides which model should handle it. The core mechanism: classify the query's complexity, then dispatch to the smallest model capable of answering well.
Classification happens through one of three strategies. The simplest approach uses heuristic rules—character count, keyword presence, or structured query patterns. A request asking for a definition might route to a 7B parameter model, while a multi-step reasoning task routes to 70B. This costs almost nothing in latency but relies on brittle pattern matching.
The second strategy uses a small classifier model trained to predict query difficulty. You label a dataset of queries as "simple," "medium," or "complex" based on which model tier answered them successfully. A tiny BERT-style model (under 100M parameters) learns these patterns and runs inference in single-digit milliseconds. This adds minimal latency but requires training data and periodic retraining as your query distribution shifts.
The third approach: LLM-based routing. You send the query to a fast, cheap model that predicts which tier should handle it. This sounds circular—you're using an LLM to decide which LLM to use—but a small model making a routing decision costs far less than a large model doing the full task. The trade-off: you now make two API calls per request, doubling your latency floor.
The router's job isn't perfection—it's making the right call often enough that your average cost-per-query drops faster than your accuracy degrades.
Think of routing like triaging patients in an emergency room. Not every case needs the head surgeon. A nurse (the router) assesses severity and sends simple cases to a resident (small model), moderate cases to an attending (medium model), and only critical cases to the specialist (large model). The system works if triage accuracy is high and the surgeon's time actually costs more than the overhead of triage itself.
The real cost savings formula
Cost savings from routing depend on three variables multiplied together: the pricing spread between model tiers, your request distribution across complexity levels, and your routing accuracy. Miss on any dimension and your 10X claim evaporates.
Start with pricing spreads. If GPT-4 costs $30 per million tokens and GPT-3.5 costs $0.50 per million tokens, the spread is 60X. If Claude Opus costs $15 and Claude Haiku costs $0.25, the spread is also 60X. But if your "large" model costs $5 and your "small" model costs $3, the spread is only 1.67X—routing can't save much when prices bunch together.
Request distribution matters more than most teams expect. Suppose 70% of your queries are genuinely simple and 30% need your best model. If you route perfectly, you save on 70% of requests. But if your distribution is inverted—70% complex, 30% simple—you only save on the minority. The 10X marketing claim assumes most of your traffic is overpaying for capability it doesn't need. Measure your actual split before building anything.
Routing accuracy is where the formula gets painful. Say your router correctly classifies simple queries 90% of the time but misroutes 10%. Those misrouted queries hit the small model, produce garbage, and force a retry on the large model. You now pay for the small model call (wasted cost), the large model retry, and the latency penalty of two round trips. If your router's precision drops to 80%, you might spend more than routing nothing.
Here's a concrete example. You handle 10 million queries per month. 60% are simple ($0.50 per million tokens × 6 million = $3,000), 40% are complex ($30 per million tokens × 4 million = $120,000). Total baseline cost: $123,000. With perfect routing, you save $72,000 (simple queries that would've hit the expensive model). But if your router has 85% precision and triggers retries on 15% of simple queries, you add back $18,000 in wasted calls and pay $10,800 in retry costs. Your actual savings: $43,200—a 35% reduction, not 10X.
The math only works when pricing spreads are wide, most requests are overpaying, and routing accuracy stays above 90%. Otherwise you're optimizing in the wrong layer.
Latency overhead and accuracy penalties
Routing adds latency. Every millisecond matters if you're building a chatbot or autocomplete feature. For batch workloads like document summarization, latency tolerance is higher. The question: does the overhead cancel out the benefit?
A heuristic router adds near-zero latency—maybe 1 to 5 milliseconds for pattern matching in your application layer. A small classifier model adds 10 to 50 milliseconds depending on where it runs (local inference vs. API call). LLM-based routing adds a full round trip: 200 to 800 milliseconds for the routing decision, then another round trip for the actual model call. If your p99 latency budget is 2 seconds, that's manageable. If you promise 300ms responses, LLM-based routing breaks your service-level agreement (SLA).
Accuracy penalties show up in two ways. First: false negatives, where the router sends a hard query to a small model. The small model either fails outright or returns a plausible-sounding wrong answer. In production, you need a confidence threshold—if the small model's output scores below a certain certainty level, automatically escalate to the large model. This requires your small model to output calibrated confidence scores, which many don't.
Second: false positives, where the router sends an easy query to an expensive model. This wastes money but doesn't hurt user experience. The asymmetry matters: false negatives damage quality, false positives just leak budget. Conservative routers tolerate more false positives to avoid quality degradation.
Routing makes sense in three scenarios. One: your request volume is high enough that even small per-query savings compound. If you serve 100 requests a day, building routing infrastructure is overkill. If you serve 10 million, a 30% cost reduction is meaningful.
Two: your workload naturally splits into distinct tiers. A support chatbot handling frequently asked questions (FAQs) plus complex technical questions is a perfect fit. A creative writing assistant where every query needs nuance is not.
Three: your latency budget has slack. Batch jobs, background processing, and async workflows tolerate routing overhead better than real-time features.
Routing hurts performance when your queries are uniformly complex, when your latency budget is tight, or when your pricing spread is too narrow to justify the infrastructure cost. Many teams would save more money by caching, prompt engineering, or negotiating volume discounts than by building a routing layer.
Building a routing layer
Start with a routing decision log. Every request should record which model the router chose, what the input query looked like, whether the output succeeded or failed, and the cost of the call. This log is your ground truth for measuring routing accuracy and iterating on your classifier.
For classification strategies, begin with heuristics unless you have labeled data ready. A simple rule set based on query length, presence of keywords like "explain," "summarize," or "analyze," and structured patterns (SQL, JSON, code) covers 60 to 70% of cases with zero training. Track misroutes in your log and add rules iteratively. This approach scales further than most teams expect before hitting diminishing returns.
If you move to a learned classifier, train on historical data where you know both the query and which model answered it successfully. Label each query with the smallest model tier that produced a satisfactory result. Use a lightweight model like DistilBERT or a small sentence transformer—speed matters more than perfect accuracy because false positives are cheap. Retrain monthly as your query distribution shifts.
LLM-based routing works when your queries are highly variable and you can tolerate the latency. The prompt for your routing LLM should include examples of simple vs. complex queries and ask it to classify the current request. Make the output structured (JSON with a "tier" field) so parsing is deterministic. Use the cheapest fast model you can find—this is not the place to spend on capability.
Fallback mechanisms prevent catastrophic failures. If the small model returns low confidence or empty output, automatically retry with the next tier up. Set a confidence threshold (e.g., 0.7) based on your accuracy/cost trade-off. If the routing model itself fails (network error, timeout), default to your most capable model—availability beats optimization.
Monitor three metrics continuously. Routing accuracy: what percentage of queries sent to each tier would've been correctly answered by that tier? Measure this by sampling outputs and either human-labeling quality or using a separate eval model. Cost per query by tier: track actual spend to confirm your assumptions about pricing spreads and request distribution. Retry rate: how often do requests escalate from small to large models? A retry rate above 15% suggests your router is too aggressive.
Here's a comparison of routing strategies:
| Strategy | Latency overhead | Training required | Accuracy ceiling | Best for |
|---|---|---|---|---|
| Heuristic rules | < 5ms | None | 70–80% | Structured queries, low volume |
| Small classifier | 10–50ms | Moderate | 85–92% | High volume, stable distributions |
| LLM-based routing | 200–800ms | Minimal | 90–95% | Variable queries, async workloads |
Build observability before you build optimization. Instrument your existing single-model setup to measure query complexity, token counts, and cost per request. Once you know your actual distribution, you can decide if routing is worth the engineering effort. Many teams discover their queries are more uniform than expected—routing solves a problem they don't have.
What this means for builders
If you're evaluating routing for cost savings, start by measuring your current request distribution. Export a week of query logs and manually classify 200 to 500 samples into "could be answered by a small model" vs. "needs large model." If fewer than 50% fall into the small bucket, routing won't save enough to justify the work.
For teams that do see potential, begin with heuristic routing. Ship rules-based logic in your application layer and monitor routing decisions for two weeks. This proves out cost savings with minimal latency impact before you invest in machine learning (ML) infrastructure. If heuristics plateau below your accuracy target, only then train a classifier.
Avoid building custom routing infrastructure until you've exhausted prompt engineering and caching. A well-crafted system prompt can often make a smaller model perform better than a larger model with a vague prompt. Response caching with Redis or Memcached eliminates repeat queries entirely—zero cost, zero latency. Both deliver better return on investment (ROI) than routing for most early-stage products.
When you do build routing, treat the router as critical infrastructure. A broken router that sends everything to the expensive model costs you unexpected budget burn. A broken router that sends everything to the cheap model degrades quality silently. Implement circuit breakers, alerting on retry rate spikes, and automatic failover to single-model mode if routing accuracy drops below a threshold.
Finally, remember that model pricing changes faster than your routing logic. Providers drop prices, introduce new tiers, or retire models. Your routing system needs regular recalibration. What saved money six months ago might cost more today if pricing spreads compressed or your query distribution shifted. Build this as a maintained system, not a set-and-forget script.
Conclusion
Smart model routing can genuinely cut LLM costs, but the 10X figure depends on wide pricing spreads, favorable request distributions, and high routing accuracy. The engineering challenge isn't building the router—it's measuring whether your workload benefits from routing in the first place and maintaining accuracy as models and queries evolve. For many products, simpler interventions like caching, prompt tuning, or batching deliver better cost savings with less operational complexity. Routing earns its place when you have proven query diversity, latency tolerance, and the instrumentation to catch when it stops working. Build the measurement layer first, the routing layer second.