Apply advanced routing rules

Advanced routing covers what fallback policies and traffic splitting do not: decisions that depend on the request itself rather than on the static configuration of the chain, and decoupling application code from the exact provider model identifier in use today. It is a collection of capabilities rather than one feature. Model-name override is configured explicitly, letting the application speak a stable logical name while the gateway resolves it to whichever provider model is currently appropriate. Session affinity, dynamic backend selection, and request and response transformation across provider APIs apply transparently to every request without configuration.

---

Persona: Developer working in the Developer Console.

Estimated time: 15 to 30 minutes, depending on whether model-name overrides and A/B or canary patterns are configured during the same session.

When this guide applies

Advanced routing is the right level to engage with when one or more of the following are true:

Situation	Capability that addresses it
Application code keeps changing whenever the upstream provider releases a new model version	Model-name override: expose a stable logical model name and resolve it to the provider's identifier in the gateway
Two model versions need to be compared head-to-head under real production traffic	A/B with traffic splitting: combine model-name overrides with weight-based distribution
A new model version needs to be rolled out gradually to limit blast radius	Canary deployment: start with a small weight on the new model and shift over time
Multi-turn conversations or agent workflows are involved	Session affinity: handled automatically; no configuration is required
Backend load varies, and latency matters	Dynamic backend selection: handled automatically; no configuration is required
The application speaks OpenAI but should be free to route to any provider	Request and response transformation: handled automatically; no configuration is required

For pure resilience or pure traffic distribution by weight, the simpler guides, Improve resilience with fallbacks and Reduce cost with traffic splitting, are the right starting points. Advanced routing is layered on top of those patterns rather than replacing them.

What is available today, and what is not

Advanced routing is a collection of capabilities rather than a single feature, and they do not all ship at the same time.

Capability	Availability
Model-name override, configured in Step 1	Available today
A/B and canary over a logical name, configured in Step 2	Available today
Session affinity and request and response transformation, described in Step 3	Available today, applied without configuration
Dynamic backend selection, described in Step 3	Available where an InferencePool and an Endpoint Picker are configured on the data plane, as covered in Load-balance across regions
Attribute-based routing (also referred to as smart routing), described under Route by compliance, cost, or latency policy	2026 Q3

Everything available today chooses a backend from the routing configuration itself: a static mapping from a logical name to a provider model identifier, a weight, or live backend metrics. None of it selects a model from what the calling application says an individual request is. Resolving one logical name differently according to a property the caller declares about each request, such as a tenant tag, a priority header, or a traffic classification, is attribute-based routing. That mechanism is scheduled for the 2026 Q3 release and is documented ahead of it in Smart routing so that the design can be reviewed and planned against. The scenarios in this guide rest only on the capabilities marked available today.

Outcomes

By the end of this guide:

• At least one logical model name has been defined that decouples the application from a specific provider model identifier.
• The override has been combined with a traffic split to run an A/B between two model versions, or to operate a canary rollout for a new model version.
• The behaviors that operate automatically (session affinity, dynamic backend selection, and cross-provider transformation) are understood, even though they require no configuration.

Prerequisites

This guide builds on the routing configuration patterns established in the earlier dev guides. Specifically:

• A working API key with a routing configuration attached, as set up in Route Requests Across Providers.
• Familiarity with traffic splitting weights, as covered in Reduce Cost with Traffic Splitting. Model-version experiments are implemented as traffic splits over logical model names.
• At least two enabled models in the Admin Console, or one model with multiple versions available for promotion.

Step 1: define a logical model name

The most direct piece of advanced routing to configure is the model-name override, exposed in the routing configuration as the modelNameOverride field: a mapping from a logical name the application speaks to the specific provider model identifier the gateway dispatches against. Without this, every provider model version bump becomes an application change; with it, version changes become a routing-configuration change inside Agent Router.

The override is attached to a route entry on the API key's routing configuration. The choice of logical name belongs to the application team and should be stable across versions. A few useful conventions:

Logical name	Resolves to	When to use this pattern
my-gpt4	gpt-4o-2024-08-06	A simple alias that hides the specific dated version from the application
my-claude	claude-sonnet-4-20250514	An alias that hides a cross-provider model identifier behind a stable name
stable-chat	gpt-4o-mini-2024-07-18	A long-lived alias used by parts of the system that prefer predictability over capability
next-gen-chat	gpt-4o-2024-11-20	A name reserved for whichever version is currently being evaluated for promotion

Configure the override:

1. Open the detail page for the API key whose routing configuration should expose the logical name.
2. Open the routing configuration and add a route entry.
3. Set the logical name in the route entry (for example, my-gpt4).
4. Set the resolved provider model identifier (for example, gpt-4o-2024-08-06).
5. Save the configuration.

The application now requests my-gpt4 in the model field of its OpenAI-compatible payload, and the gateway forwards the request to the configured provider model. The application code is unaware of the underlying version.

Step 2: combine overrides with traffic splitting

Logical names are most useful when they are paired with the traffic-splitting mechanics covered in the previous guide. Two routes can resolve the same logical name to two different provider models, and a weighted split distributes requests between them. The application speaks one stable name; the gateway runs an A/B in the background.

A typical A/B setup against the logical name chat-model:

Route	Logical name	Resolves to	Weight
A	chat-model	gpt-4o-2024-08-06	50
B	chat-model	gpt-4o-2024-11-20	50

Configure:

1. In the routing configuration for the chosen API key, add two route entries with the same logical name and different resolved models.
2. Switch the routing strategy to Traffic Splitting if it is not already set.
3. Assign weights to each route entry (50/50 for an A/B; a heavier weight on the current version for an evaluation that should not perturb production much).
4. Save and confirm the Enable toggle is on.

Quality and performance comparisons are then made through Request Logs and Usage Analytics, both of which expose the resolved model per request even though the application only ever saw chat-model.

A canary deployment is structurally identical to an A/B but with intentionally lopsided weights. A common progression:

Stage	Current version weight	New version weight
Initial canary	95	5
Hold-and-observe	80	20
Expand	50	50
Cutover	0	100

The weights are adjusted in the Console at each stage; the application requires no change at any point. Once the cutover is complete, the route pointing at the old version can be removed entirely.

Route by compliance, cost, or latency policy (coming in 2026 Q3)

The routing pattern in this section rests on attribute-based routing, which is scheduled for the 2026 Q3 release. It is described here ahead of that release, alongside the split it is a counterpart to, and it cannot be configured on the routing configurations covered in Step 1 and Step 2 today.

Weighted splits distribute one logical name across backends by chance. The same logical name is instead resolved deterministically by keying the routing decision on attributes of the request, a policy condition rather than a weight. This is an application of the attribute-based dispatch already described: a routing rule inspects request attributes (for example, a tenant tag, a priority header, or a traffic classification) and selects the route entry whose condition matches, so that chat-model resolves to a different backend depending on the kind of request that arrived.

Three policy conditions cover the common cases:

Condition on the request	Resolves chat-model to	Rationale
Tagged as regulated or residency-bound	An in-region, compliant provider	Keeps regulated traffic on a backend that satisfies data-residency and retention constraints
Tagged as low-priority or bulk	A cheaper backend	Reserves premium capacity for traffic that needs it and lowers cost on the rest
Tagged as interactive	A lower-latency backend	Protects the responsiveness of user-facing requests

The route entries take the same shape as in the preceding split, the same logical name mapped to different resolved models, except that each entry carries a match condition instead of a weight, and the gateway dispatches to the first entry whose condition the request satisfies. Where cost is the driver, this attribute-based approach complements the proportional split in Reduce cost with traffic splitting: the split distributes by chance, whereas a policy condition routes by a known property of the request. Compliance-driven routing is most often enforced on the operator side, where the residency and no-retention guarantees are configured for the backend itself; see Configure data residency.

Step 3: understand the behavior that requires no configuration

A meaningful portion of Agent Router's advanced routing happens behind the scenes. The behaviors below apply to every request and do not need to be enabled or tuned. This step is short on configuration but useful to read once, because the behavior shapes how the gateway responds under load and across providers.

Dynamic backend selection

When more than one backend is eligible to serve a request (for example, two replicas of a self-hosted model behind a load-balanced endpoint), the gateway evaluates live backend metrics and chooses the best target. The mechanism is an InferencePool paired with an Endpoint Picker Provider: rather than relying solely on static weights or ordered fallback lists, the Endpoint Picker evaluates live metrics for each candidate and routes to the backend with the best capacity and cache affinity, reducing latency and improving throughput. Three signals contribute to the decision:

Signal	What it captures
KV-cache usage	Memory pressure on each backend. Heavily loaded backends are deprioritized.
Queue depth	Number of pending requests on each backend. Less-loaded backends are preferred.
Prefix cache scoring	How well a backend's cache matches the request's prompt prefix. Better matches reduce latency.

Dynamic selection operates only within the eligible set defined by the routing configuration. A fallback policy that limits requests to a specific provider still constrains dynamic selection to that provider; the policy boundary always takes precedence.

Session affinity

Multi-turn conversations, agent loops, and Model Context Protocol (MCP) sessions benefit from being processed by the same gateway instance throughout their lifetime, because state and cache accumulate locally. The gateway architecture guarantees this affinity:

• The proxy component and the external processor are deployed as a sidecar pair, so once a session is established, subsequent requests in that session route to the same processor.
• MCP sessions use encoded multi-backend session identifiers that pin the session to whichever combination of backends it was established against.

The affinity matters in three cases: multi-turn conversations where context accumulates on a specific backend, stateful agent interactions that maintain tool state across calls, and MCP sessions that manage connections to multiple tool servers.

No configuration is required to obtain this behavior; it is a property of how the data plane is deployed.

Request and response transformation

The gateway exposes an OpenAI-compatible request surface but routes to a wide range of provider APIs that do not natively speak OpenAI. The translation between the two happens transparently in the gateway's processing pipeline:

Transformation	Behavior
Header mutations	Provider-specific authentication headers are set or replaced
Body mutations	JSON fields are added or rewritten, for example, injecting a default max_tokens if the provider requires one
Path rewriting	The OpenAI path is rewritten to the provider's native endpoint, for example, /v1/chat/completions becomes the Anthropic Messages endpoint at /anthropic/v1/messages
Model field rewriting	The model field is rewritten to the provider-specific model identifier, picking up any logical-name override from Step 1
Response normalization	The provider's response is translated back into the OpenAI-compatible shape the calling application expects

The result is that an application written against the OpenAI SDK can route through the gateway to OpenAI, Anthropic, Google, Azure OpenAI, Mistral, or any other supported provider with no per-provider integration code.

Example transformation flow

When an OpenAI Chat Completions request is routed to Anthropic Claude, the fields are transformed in sequence:

1. Path: /v1/chat/completions is rewritten to the Anthropic Messages endpoint.
2. Body: the messages array is converted to the Anthropic message format, and max_tokens is injected if absent.
3. Headers: Authorization: Bearer is replaced with the provider-specific auth header.
4. Model: the model field is verified against the provider model identifier.
5. Response: the Anthropic response is translated back into the OpenAI Chat Completions format.

The calling application receives a response in the exact format it requested, regardless of which provider served it.

Scenarios

The steps above cover each mechanism on its own. The situations below are the ones those mechanisms are usually assembled for, worked from the problem through to the behavior it produces. Each scenario states the problem, the setup it needs, what happens as a result, and what to watch for.

An A/B has run and a decision has to be made

The scenario. Two route entries have been splitting chat-model 50/50 between two model versions for a week. The result now has to be acted on, and neither answer, keeping the incumbent or moving to the new version, should cost an application release.

The setup. The split from Step 2 is already in place. Concluding it is three edits.

1. Read the per-resolved-model breakdown in Usage Analytics for the window the split has been running, and open individual requests worth inspecting in Request Logs.
2. Set the weight on the winning route entry to 100 and the weight on the other to 0, then save.
3. Once Request Logs confirm the losing entry is receiving nothing, remove it.

What happens. The application sends chat-model throughout and never learns that the experiment started, ran, or ended. The new distribution takes effect from the moment of save, and no in-flight request is affected.

What to watch for. Two things.

What is compared is requests, not conversations. Weights distribute each request independently, so consecutive turns of one exchange can be served by different route entries. A comparison drawn at the level of a whole conversation is not something a split produces.

Latency figures carry more than the model. Provider-side response time varies with conditions that have nothing to do with the model under test, and it varies independently for each of the two backends over the life of the split. Quality signals separate more cleanly than timing ones.

An application has to move to a different provider without a release

The scenario. An application written against the OpenAI SDK has to be served by an Anthropic model. Its code cannot change in the window available, and it names a logical model name rather than a provider model identifier, which is what makes the move possible at all.

The setup

1. Confirm the destination model is enabled in the Admin Console, as the prerequisites require.
2. Add a second route entry on the same logical name resolving to the destination model, and give it a small weight, so the first requests to reach the new provider are a minority of traffic.
3. Raise the weight in the stages set out in Step 2 until the destination model carries all of it, then remove the route entry pointing at the original model.

What happens. The application keeps sending an OpenAI Chat Completions request to the same endpoint with the same SDK. For the share of requests resolving to the Anthropic model, the gateway rewrites the path, converts the messages array, replaces the authentication header, rewrites the model field, and translates the response back into the OpenAI shape, as set out in Step 3. On the standard chat-completions message path, nothing in the application distinguishes the two.

What to watch for. Two things.

Transformation makes the call succeed; it does not make the two models behave the same. Prompt handling, refusal behavior, and token accounting still differ between providers. The weighted stage exists so that difference is observed on a minority of traffic before it reaches all of it.

Only the standard message path is insulated this way. Structured outputs are not translated across providers, and embeddings, image, and rerank calls are accepted in passthrough mode only, so a request on any of those paths has to already match the destination provider's native format. An application that uses them needs those calls examined before the weight is raised, not only its chat completions. See the known limitations in AI capabilities.

One logical name has to mean two different things

The scenario. Two applications both send stable-chat. One of them now needs that name to resolve to a different model, and editing either application's code is the thing the logical name existed to avoid.

The setup. This one is not a route entry. It is the API key.

1. Issue a second API key for the application whose resolution should change.
2. Attach a routing configuration to that key resolving stable-chat to the model that application should use. Routing is configured per key, so the two keys hold two independent resolutions of the same name.
3. Move that application onto the new key.

What happens. Both applications keep sending stable-chat, and the key presented with the request decides what the name resolves to. Request Logs are filtered per key, so the two applications also separate cleanly in the record.

What to watch for. Two things.

A routing configuration chooses between route entries by weight, not by who is calling. A second route entry on the shared key splits both applications across both models, which is not the outcome wanted. The separation available today is the API key, which is why one key per distinct routing configuration is the working convention.

Resolving a name from a property of the request is a different mechanism. Serving one name differently according to something the calling application declares about each request, rather than according to which key it holds, is attribute-based routing: see Smart routing, scheduled for 2026 Q3.

A response names a model the application never sent

The scenario. A developer debugging a failing test finds a model value in the response body that the application did not send, and raises it as a defect.

The setup. None. This is the expected result of an override.

What happens. The model field in the response body reflects the backend that actually produced the answer, not the logical name the request carried. Where a split is in place, the same request repeated can resolve differently, because each request is distributed independently. Which entry served any individual request is on the record: open Monitoring → Request Logs, filter by the API key, and read the Resolved model field on the request.

What to watch for. Two things.

A test that asserts on the response's model field breaks the first time an override is introduced. The stable value is the one the application sends. The resolved value is deliberately free to change without an application release, which is the point of the override.

A model from a fallback position is a different finding. Where the same routing configuration also carries a fallback chain, a resolved model below the primary means the chain was walked because a backend failed, not that the override or the split chose it. See Improve resilience with fallbacks for reading a failover in the request timeline.

---

Where to go next

Use your own provider credentials

Introduce BYOK credentials alongside the routes defined in this guide.

Reduce cost with traffic splitting

Distribute traffic by weight, the mechanism behind A/B and canary routing.

Monitor traffic and usage

Evaluate the results of an A/B or canary by examining the per-resolved-model breakdown in usage analytics.

Export telemetry to an observability stack

Push per-route metrics into an existing observability platform so quality comparisons can run alongside the rest of the application's data.