This week's edition of The Pulse from The Pragmatic Engineer opened with an interesting question: did Anthropic's restrictive policies on its newest model, Fable, just hand market share to rivals like Codex? The reporting is anecdotal, but the underlying pattern is consistent — when one frontier provider gets noticeably stricter about what its model will do, the developers who actually ship products start sending more of their traffic elsewhere.
The interesting part is not the inter-vendor competition. The interesting part is the infrastructure question it forces: how do you actually route inference traffic across providers, in a way that survives a provider changing its policies, pricing, or availability overnight?
The pattern has a name in some teams already — smart model routing — and the production version is more substantial than "swap in a different API key." This article walks through what smart model routing means, why it became necessary now, the engineering shape of it, and the failure modes nobody mentions in the launch announcements.
Why smart model routing became necessary now
Until roughly the last twelve months, most production LLM (large language model) systems were single-provider by default. You picked OpenAI, Anthropic, or Google early, built around their SDK, and the cost of switching was high enough that you treated the choice as semi-permanent. Vendor diversity was an architectural ideal more than an operating practice.
Three changes broke that pattern at roughly the same time.
Pricing on equivalent capability diverged sharply. The same task — say, summarising a long document — can now cost 2–10x more on one provider than another, depending on which generation of model each provider has just released. Pricing windows open and close on monthly cycles. Lock-in to one vendor is now a continuous tax.
Usage policies started diverging visibly. Different providers now refuse different categories of request, with different sensitivities, and the refusal rate on each shifts with each model release. A workflow that worked perfectly against last month's Fable model may now bounce on safety filters for reasons your users do not understand and you cannot debug.
Availability stopped being a solved problem. Frontier-model APIs have always had outages. The newer regulatory environment — with at least one government-ordered suspension of a frontier model already on the record — turns availability into something you have to plan for, not assume.
The combination is straightforward. If pricing, policy, and uptime can all move under you without notice, the only durable design is one that can move with them.
What smart model routing actually is
The naive version of routing is if (primary.unavailable) fall_back_to(secondary). That is not smart routing. That is a try-catch with extra steps.
Smart model routing is the practice of treating model selection as a per-request decision, made at runtime based on the properties of the task. The router does not know one canonical model. It knows a capability map — which models are good at which kinds of work — and uses the incoming request to pick the cheapest model that can plausibly handle it.
Think of it like a delivery dispatcher rather than a single courier on retainer. The dispatcher does not always send the fastest vehicle. They look at the package, the distance, the deadline, and the cost, and they pick the right one for the job. The fleet exists so the dispatcher has options.
A working smart router has four pieces.
A request classifier. Something cheap and deterministic — a small classifier model, or even a regex-and-heuristics layer — that tags the incoming request with what kind of work it is. Code generation? Summarisation? Extraction? Conversation? The classifier does not have to be perfect; it has to be fast and cheap.
A capability map. A configuration table that lists each available model with its strengths, costs, latency profile, and current health. This is the source of truth that the router reasons against. It has to be updateable without a code deploy, because frontier-model pricing and capability shift faster than your release cycle.
A routing policy. The actual decision logic. Sometimes it is as simple as "for this task type, prefer model A, fall back to B, fall back to C." Sometimes it is multi-criteria — minimise cost subject to latency budget and quality floor. The point is the policy is explicit and changeable, not buried in if branches across the codebase.
A verification layer. This is the piece teams forget. When you swap providers mid-flight, the responses are not interchangeable, even for nominally identical tasks. Different models format JSON differently, refuse different categories, hallucinate different facts. The router needs a thin verification pass — schema check, refusal-pattern detector, sometimes a second model as a judge — that decides whether the response is acceptable or needs a retry on a different model.
A minimal capability map
The actual contents of the capability map matter more than the routing code around it. Here is a shape that holds up in practice.
| Field | Purpose | Updated by |
|---|---|---|
| Model ID | Canonical name (e.g. provider-x/model-name-version) | Engineering |
| Task fit | Tags for which task types this model handles well | Eval pipeline |
| Cost per 1k tokens | Input and output token pricing | Billing integration |
| P95 latency | Observed, not advertised | Observability layer |
| Refusal rate | Percentage of recent requests refused, per task type | Verification layer |
| Health | Last successful call timestamp, current error rate | Health checker |
| Policy notes | Known restrictions (e.g. "no medical advice") | Engineering |
The pattern is that nothing in this table is hard-coded. Every column is sourced from something the system actually observes — your eval pipeline, your billing data, your traces. The router reads the table on every request, so updating the table updates routing behaviour immediately.
The router is just glue. The capability map is the actual product.
The engineering shape
Once the four pieces are in place, the request lifecycle looks like this.
| Step | Component | Reads | Writes |
|---|---|---|---|
| 1 | Classifier | Incoming request | Task tag |
| 2 | Router policy | Task tag, capability map | Chosen model |
| 3 | Provider adapter | Chosen model, normalised input | Provider-specific call |
| 4 | Verifier | Provider response | Pass or fail |
| 5a | Response | Verified output | Caller |
| 5b | Fallback adapter | Next model in policy | Re-runs steps 3 and 4 |
Three properties of this shape matter.
It is declarative at the edges and procedural in the middle. The classifier and the capability map are data. The policy is configuration. The router is the only piece with meaningful logic, and it is small.
It is observable per request. Every call records which classification fired, which model was selected, what it cost, how long it took, and whether the verifier accepted it. This is the data that feeds back into the capability map, closing the loop.
It is independent of any single provider's SDK. Each provider sits behind a thin adapter that normalises requests and responses. Adding a new provider is writing a new adapter, not rewriting the call site.
Failure modes nobody mentions
Smart routing is not free. There are three failure modes that turn up after the second or third month in production.
Capability drift between providers. Two models that scored similarly on your eval suite three months ago may diverge silently as the providers ship updates. If you only re-run your eval suite quarterly, you can be routing traffic to a model that has quietly become worse for your specific tasks. The eval pipeline has to run continuously, not occasionally.
Cost optimisation gone wrong. A naive cost-minimising router will happily route everything to the cheapest model, including the requests that needed the expensive one. The cheapest model returns a plausible-looking wrong answer, the verifier accepts it because it is well-formed, and a quality regression ships to production. The fix is to define quality floors per task type explicitly, and not let the cost optimiser drop below them.
Hidden state in prompts. Some workflows accumulate state across calls — conversation history, tool results, retrieved documents. When the router switches providers mid-conversation, that state has to be re-formatted for the new model's conventions. Get this wrong and you get a model that suddenly seems to forget what it just said. The fix is to keep state in a provider-agnostic format and let the adapter handle translation.
What this means for builders
Smart model routing is no longer a nice-to-have. It is the production answer to a market where pricing, policy, and availability all change faster than the deploy cycle. Three concrete things to do.
Treat your provider list as a first-class config, not a code constant. If switching providers requires a code change, you do not have routing — you have a hardcoded choice with extra steps. Move the provider list, the capability map, and the routing policy into something you can update without a deploy.
Build the verifier before the second provider. The most common failure with smart routing is shipping a router that calls multiple providers without checking whether their responses are interchangeable for your task. The verifier is what makes "we support multiple providers" honest. Build it first; add the second provider when it passes.
Run continuous evals against every model in your map. A capability map that is updated once a quarter is a stale capability map. The minimum viable practice is a small, representative eval suite that runs against every model in the map every day, with results feeding back into the routing policy.
Conclusion
The Anthropic/Codex story is a market-share story on the surface and an infrastructure story underneath. The teams that win when providers tighten their policies, change their pricing, or get suspended by regulators are not the teams who picked the right vendor — they are the teams who built routing infrastructure that does not particularly care which vendor is on top this month.
That is the shift smart model routing represents. The unit of architectural commitment used to be the provider. It is now the routing layer. The provider is a parameter.