Get started with Routing
Enterprise Tier
Routing is not one mechanism, and which one is reached for depends on which concern is in play and who holds it. Four concerns are in play: platform, how fast a credential may consume; availability, what serves a request when a provider fails; cost, what a class of work is worth spending on; and business, which requests deserve the better model. They are held by different people, they change on different timescales, and they do not substitute for one another. A control picked for the wrong concern rarely fails loudly, it simply never fires: a rate limit does not make a request cheaper, and a traffic split does not know which requests matter.
Persona: Platform operator working in the Admin Console, alongside the developers and site reliability engineers who own the applications sending the requests.
Estimated time: 5 minutes to read.
The four concerns
Platform concerns
Keeping shared capacity intact and keeping model names stable. A rate limit bounds how fast one credential may consume, in tokens, requests, or parallel calls. A model-name override holds one logical model name steady on an API key while the provider model behind it changes.
Availability concerns
Keeping requests succeeding when a provider does not. A fallback chain attached to an API key moves a request to the next backend when the primary returns a recoverable failure, and returns an error only once every backend in the chain has failed.
Cost concerns
What a class of work is worth spending on. Traffic splitting sends a set share of requests to a cheaper backend today. Budget-based fallback, which switches a request to a cheaper model once a spend limit is reached, is planned for the 2026 Q3 release.
Business concerns
Which requests deserve the better model. Attribute-based routing matches a class of traffic to the model that suits it, from what the calling application declares that request to be, so that overnight batch work and a premium customer tier are not served alike.
The division between the first two cards and the last two is a division of audience as much as of mechanism. Availability and traffic shaping are expressed in retries, weights, and ceilings, which is the vocabulary of the team that owns the calling code, and the fallback and traffic-splitting guides are written for that team accordingly. Deciding that overnight batch work does not need the most expensive model, or that one customer tier does, is a judgment about the value of the work rather than the health of a backend. Nobody writing the calling code is well placed to make that judgment, and the answer should not require every application team to ship a release, which is why budget enforcement and attribute-based routing are platform operator controls, even though the traffic split that also lowers cost is set by a developer on the key.
What each mechanism decides
Eight mechanisms bear on the routing of a request. They do not all decide the same kind of thing, and they do not all ship at the same time. This table is the section's record of what each one decides and when it is available; the guides underneath describe each mechanism in full.
| Mechanism | What it decides | Concern | Persona | Availability |
|---|---|---|---|---|
| Model alias | Which model a name in the catalog resolves to, for every caller of that name | Platform | Platform operator | Available now |
| Model-name override (logical model name) | Which provider model a logical model name resolves to, on the one API key that carries the routing configuration | Platform | Developer | Available now |
| Advanced routing rules | Which route entry serves a request when several sit on one API key, by weight, as in a weighted A/B or canary progression | Platform | Developer | Available now |
| Rate limiting | Nothing about which model serves the request: a rate limit is flow control on how fast a credential may consume | Platform | Platform operator | Available now |
| Availability fallback | Which backend serves the request after the primary fails, from an ordered chain attached to an API key | Availability | Developer | Available now |
| Traffic splitting | What share of requests each backend receives, by a weight set against the API key | Cost | Developer | Available now |
| Budget-based fallback | Whether a request is switched to a cheaper model because a spend limit has been reached | Cost | Platform operator | 2026 Q3 |
| Attribute-based routing (smart routing) | Which model serves a request, from what the calling application declares that request to be | Business | Platform operator | 2026 Q3 |
Not every row is a choice of model. A rate limit never redirects a request, it refuses one once a ceiling is hit, and a budget in the current release reports on spend rather than acting on it. Both live in Cost and quota management and are linked from this page because they are routinely mistaken for routing.
Match the situation to the control
| Situation | Control that answers it | Persona | Availability |
|---|---|---|---|
| A provider goes down and requests should continue on an equivalent model somewhere else | Availability fallback | Developer | Available now |
| Two models should be compared head to head under real traffic before one is chosen | Traffic splitting | Developer | Available now |
| A new model version should take a small share of traffic first, then progressively more | Advanced routing rules, as a weighted canary on a logical model name | Developer | Available now |
| A cheaper model should serve part of the traffic without changing anything in the calling code | Traffic splitting | Developer | Available now |
| Requests have to stay inside a data-residency boundary | A fallback chain composed only of in-region providers | Developer | Available now |
| A model version changes and the old name is spread across application code | A stable name that outlives the model behind it | Platform operator | Available now |
| A retry loop or a runaway agent is sending far more requests than a healthy client | A rate limit on requests per minute | Platform operator | Available now |
| One team's traffic is crowding out another sharing the same deployment | A rate limit at the group scope | Platform operator | Available now |
| A group should not have the expensive model at all | The project's model set, which is not a routing control | Platform operator | Available now |
| Overnight batch work is billed at interactive rates | Attribute-based routing | Platform operator | 2026 Q3 |
| Traffic that generates revenue should be served by a better model than traffic that does not | Attribute-based routing | Platform operator | 2026 Q3 |
| A monthly spend ceiling should hold whatever the request rate | A budget | Platform operator | Monitoring now, enforcement 2026 Q3 |
Where the choice is between a routing control and a cost control for the same workload, see Choose the right cost control for each workload.
Smart routing and traffic splitting are not the same control
The two are easy to conflate, and conflating them is the most common reason a routing decision is made in the wrong place. Smart routing is the name used in this documentation for matching a class of request to the model that should serve it; it is not an official product term. It covers two mechanisms with different availability: a model alias, available now, and attribute-based routing, planned for 2026 Q3. The comparison below concerns the attribute-based half, which is the one traffic splitting is mistaken for.
| Traffic splitting | Smart routing | |
|---|---|---|
| What selects the backend | Chance, against a weight set on the API key's routing configuration | What the calling application declares the request to be |
| What the request has to carry | Nothing | An attribute declared by the caller |
| Concern it answers | Cost and evaluation | Business value of the work |
| Availability | Available now | 2026 Q3 |
A traffic split sends a percentage of requests elsewhere whatever those requests are, so it cannot tell a premium customer from a nightly batch job. Smart routing sends a request elsewhere because of what that request is, so it cannot be used to hold a fixed percentage. A traffic split composes with a fallback chain: the split decides which backend serves the request, and the chain catches it if that backend fails.
Choosing which model serves a request
Smart routing
Business concerns. Model aliases, available now, and attribute-based routing, coming in 2026 Q3, worked through ten scenarios.
Improve resilience with fallbacks
Availability concerns. The ordered chain that moves a request when the primary model fails. Available today.
Reduce cost with traffic splitting
Cost concerns. Weighted shares across backends, for cost work and head-to-head comparison. Available today.
Apply advanced routing rules
Platform concerns. Logical model names, canary progressions, and the routing behavior that applies without configuration.
Routing policy reference
The objects a routing policy is built from: attributes, conditions, evaluation order, scope precedence, modes, and the decision record.
Adjacent controls in cost and quota management