Improve resilience with fallbacks

A fallback policy is an ordered list of backends. The gateway tries the first; if the call fails with a recoverable error, it walks to the next and tries again, until a backend succeeds or the chain is exhausted. The calling application sees one request that either returns a response or a final error, with the failover invisible to it and no retry or branching logic in the application itself. That removes the single point of failure left by calling one provider directly, which has no answer to a regional incident, capacity exhaustion, rate-limit throttling, or an unexpectedly fast model deprecation.

---

Persona: Developer working in the Developer Console.

Estimated time: 10 to 20 minutes, plus a short period of intentional failure exercise.

Availability

The fallback policy this guide configures is available in the current release. Two nearby mechanisms that also change which model serves a request are not, and they are named here so that the distinction is clear from this page alone.

Capability	Availability
Fallback policy: an ordered chain of backends, walked when a backend returns a recoverable failure	Available today
Fallback evidence in Request Logs: the resolved model names the backend that served each request	Available today
Budget-based fallback: moving a request to a cheaper model because a spend limit has been reached	Planned for the 2026 Q3 release, described in Enforce a budget cap
Attribute-based routing: choosing a model from what the calling application declares the request to be	Planned for the 2026 Q3 release, described in Smart routing

Nothing described on this page depends on either of the planned mechanisms.

When this guide applies

Fallback chains are the right tool when the goal is resilience: keeping requests successful in the face of provider issues, without distributing traffic deliberately across providers under normal conditions. The chain is walked only on failure; a healthy primary provider serves 100 % of traffic, and the secondary backends are effectively idle until something goes wrong.

For other routing intents, different patterns apply:

Intent	Pattern	Where it is covered
Distribute traffic by weight (cost, evaluation, gradual rollout)	Traffic splitting	Reduce cost with traffic splitting
Route based on request attributes (tenant, header, task type)	Advanced routing rules	Apply advanced routing rules
Enforce data residency or compliance boundaries	Fallback policy with carefully chosen providers	This guide (see Routing under compliance constraints below)

Fallback and traffic splitting can be combined. A weighted split selects the primary backend for a given request, and an additional fallback chain takes over only when the chosen backend fails. The patterns are not mutually exclusive.

Outcomes

By the end of this guide:

• A fallback policy with at least two backends from different providers is attached to a working API key.
• The chain has been tested under at least one simulated failure, with the gateway falling through to the secondary backend.
• Request Logs show, for each request, which backend actually handled it and whether a fallback event occurred.

Prerequisites

This guide builds directly on Route requests across providers. Specifically, the following should already be in place:

• An API key created in the Console that routes to at least one backend.
• At least two enabled models in the Admin Console, ideally from different upstream providers. A chain with two backends from the same provider still works mechanically, but it provides no protection against a provider-wide outage, which is the most common failure the chain is designed to absorb.
• A terminal with curl (or a Python environment with the openai package) for the exercise step.

Step 1: choose the chain composition

A good fallback chain is not "a primary model plus whatever else is enabled". The composition of the chain determines what kinds of failures it can absorb and how surprised the calling application is when failover occurs.

Three considerations matter when choosing backends:

• Provider independence. The chain protects against provider failures only to the extent that its backends fail independently. A chain consisting of two OpenAI models still goes down when OpenAI does. A chain that alternates providers (for example, an OpenAI primary with an Anthropic or Google secondary) absorbs single-provider outages without disruption. Differing wire formats are not an obstacle: the original request body is retained and re-translated for each backend attempted, so an Anthropic model sits behind an OpenAI one without the calling application changing format.
• Quality consistency. When the chain falls through, the application gets a response from a different model. If the secondary model produces meaningfully worse output, the failover is visible to the end user even though the request technically succeeded. Where quality consistency matters, choose secondaries that are close to the primary in capability: frontier model behind frontier model, not frontier model behind a budget option.
• Cost profile. Some chains are deliberately built so that the secondary is cheaper than the primary. This is acceptable when degraded-but-cheaper output is preferable to no output at all, and it has the side effect of reducing cost exposure during long-running incidents on the primary provider.

A typical first chain pairs a primary frontier model with one secondary from a different provider:

Position	Example backend	Role
Priority 0 (Primary)	gpt-4o	Serves 100 % of traffic under normal conditions
Priority 1 (Fallback)	claude-sonnet-4-20250514	Used when the primary returns a recoverable failure

Additional backends can be added at priority 2 and beyond. The marginal value drops quickly (two well-chosen backends absorb the vast majority of realistic incidents), but a third backstop is reasonable for workloads with strict SLA commitments.

Step 2: configure the fallback policy

The chain is attached to the API key created in the previous guide, so a change made here affects only the traffic presented with that key.

1. From the API Keys page in the Console, click Configure on the row for the key the configuration should apply to.
2. Expand the Manual Availability Fallback section and set its Enable toggle on.
3. Optionally adjust Max Attempts (between 1 and 10) and Retry Delay (seconds) (between 1 and 60). Both sit above the policy cards and apply to the section rather than to an individual policy.
4. Under Fallback Policy 1, add the primary model. It takes the Primary badge and heads the chain.
5. Click Add Model and choose the secondary backend. It is added below the primary and takes a Fallback badge.
6. Optionally repeat to add further backups, each appended to the end of the chain.
7. Save the configuration.

The Console exposes drag-and-drop reordering and a remove control on each row, which is useful when chains are being tuned over time, for example when a secondary provider is being promoted to primary after performing well during an incident.

Fallback policy field reference

The Manual Availability Fallback section exposes the following controls for a fallback policy:

• Model scope. The picker offers the models assigned to the project the API key belongs to, whatever project the project switcher is on. A model assigned to a different project is not listed, and a policy naming one is rejected on save with an error identifying the model. A model missing from the list is therefore a question of the project grant rather than of the switcher, covered in Add providers and models to projects.
• Position. Each row carries a Primary or Fallback badge marking its place in the chain. The primary receives all traffic under normal conditions, and each fallback is reached only once the backends above it have failed. These positions are the priority levels described in Operations and reliability, where the primary is priority 0. The error from the last attempted backend is returned to the caller only when every backend in the chain has failed.
• No weight control. Fallback rows carry no weight setting. Weight is the mechanism used by traffic splitting, where several backends share one position; in a fallback chain each position holds a single backend, so there is nothing to weight.
• Enable toggle. The section carries its own toggle, so a policy can be switched off without being deleted and independently of the other sections on the key. While it is off, the chain is not enforced and requests go directly to the model named in the API call.
• Max Attempts. Between 1 and 10 attempts.
• Retry Delay (seconds). Between 1 and 60 seconds. Both this and Max Attempts are set once for the section rather than per policy.
• Automatically add same model from other providers. A checkbox on the policy card. When set, the same model is populated from the other providers that offer it, which is how a chain such as claude-sonnet-4-5 on anthropic followed by vertexanthropic/claude-sonnet-4-5 on vertexanthropic is built without selecting each entry by hand.
• Reorder. Drag a row to change a backend's position in the chain, then save.
• Remove. The remove control on a row drops that backend; the remaining backends keep their relative order.

More than one policy. Up to two fallback policies can be created on a key. Within a policy, models are tried sequentially; if every model in the first policy fails, the second policy is tried in turn. Add New Policy creates the second, and the delete control on a policy card removes it.

Precedence over automatic fallback. Automatic Availability Fallback is a separate section on the same page, with its own Enable toggle and no configuration of its own, currently covering Anthropic models. Where both sections are enabled and name the same model, the policy configured here takes precedence and the request follows this chain.

tip

Keep one API key per logical chain composition rather than swapping the configuration on a shared key. Per-key routing is Agent Router's natural separation boundary, and Request Logs are filtered per key, so a separate key per chain keeps the analytics tidy and the rollback path obvious.

Step 3: test the failover path

A fallback chain that has never failed over is effectively unverified. The simplest way to exercise the chain is to introduce a failure on the primary backend and observe whether the gateway successfully walks to the secondary. Two approaches are practical, depending on how much disruption is acceptable in the environment under test.

• Misconfigure the primary temporarily. Change the primary backend in the fallback policy to a model that is enabled in the catalog but lacks a working provider connection, then send a request. The gateway records the primary failure and falls through to the secondary, which should succeed. Restore the original primary after the test.
• Use a non-production API key. Repeat the exercise against a key that is not serving production traffic, so that the experiment is fully isolated from real users.

Send a test request after each change. PROXY_URL stands for the proxy endpoint from the Console Dashboard, scheme included and ending in /v1:

curl PROXY_URL/chat/completions \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o",
    "messages": [{"role": "user", "content": "Confirm which backend served this request."}]
  }'

The request should still succeed. The model field in the response body reflects the backend that actually produced the answer, which, during a deliberate failover exercise, should be the secondary rather than the primary.

note

Not every provider failure is a recoverable one. The gateway walks the chain on transient errors (5xx responses, timeouts, connection failures, or rate-limit responses) but not on client-side errors such as malformed requests or authentication failures. Those are returned to the caller directly, because retrying them against a different backend does not change the outcome.

Step 4: verify the failover in Request Logs

The exercise above is only useful if its result is observable after the fact. Request Logs records, per request, which backend actually served the response and whether a fallback event occurred.

1. In the Console, open Monitoring → Request Logs.
2. Filter by the API key used in the test.
3. Locate the request issued during the exercise.
4. Expand the row to see the detail panel.
5. Confirm the Resolved model field matches the secondary backend, not the primary. If a fallback event is recorded, it appears in the request timeline as a separate attempted-and-failed entry on the primary, followed by a successful entry on the secondary.

The expected pattern in normal operation is the opposite: every request resolves to the primary, and no fallback events appear. A sudden cluster of fallback events in production is a strong leading indicator that something is happening on the primary provider's side, often visible in Request Logs before the provider's own status page acknowledges the incident.

For richer filtering (by time range, by status, or by resolved model), see Monitor traffic and usage.

What the chain covers on streaming requests

Streaming narrows the window in which the chain can act, and an application that streams should be built with that in mind.

The chain is walked up to the first response byte. A backend that refuses the request, times out before responding, or answers with a recoverable status is replaced by the next one, and a streaming client never learns that the substitution happened: the stream it eventually receives is the secondary's, in the format the request used.

Once the first chunk has been delivered, the chain is spent. A backend that accepts the request and then fails partway through generation cannot be replaced, because a second backend's generation cannot be spliced onto a partial answer. The client sees a stream that ends without its terminating event, and recovery is the application's to perform by re-issuing the request. The chain then protects that retry in the ordinary way.

Two further limits belong to the same picture:

• A dropped client connection is not resumable. Nothing is buffered for replay, so a client that disconnects mid-stream re-sends the request rather than reconnecting to the generation in progress.
• Enforcing output guardrails remove streaming for the request. The response is buffered for evaluation before any of it is delivered, which is unrelated to fallback but changes what the caller experiences on the same request path.

A streaming client should therefore treat a missing terminator as a failure and retry. The mechanics, including the per-format terminating events and a worked client example, are in Streaming behavior.

Fallback on the Responses API

/v1/responses is routable to OpenAI and Azure OpenAI backends only. A chain attached to a key that serves Responses API traffic fails on any hop resolving to another provider, which removes the protection at the moment it is needed. Chains that cross the wider provider catalog require Chat Completions or Anthropic Messages. See Provider support for the Responses API.

Routing under compliance constraints

Fallback chains are also the natural mechanism for policy-driven routing where compliance rather than resilience is the dominant concern. Because the chain contains only the backends explicitly added to it, the chain itself is the enforceable boundary: a chain that contains only EU-hosted providers cannot, by construction, route to a US-hosted provider, regardless of what the calling application requests.

Three common patterns use this property directly:

• Data residency. Build the chain from providers in the required region only (for example, Azure EU as the primary with GCP EU as the secondary) so failover never leaves the region.
• Approved providers. Limit the chain to providers that have been reviewed and approved for the use case in question, and assign that chain to a key dedicated to that use case.
• Output consistency. Use the same logical model class across positions in the chain to keep response quality steady during failover, even when the upstream provider changes.

There is no "compliance mode" to enable. The boundary is enforced by the composition of the chain itself.

Scenarios

The situations below are the ones that bring developers to a fallback chain most often. Each is worked against the two-backend chain from Step 1, and each states the problem, the setup it needs, the resulting behavior, and what to watch for.

The primary provider fails during the working day and the application has no retry logic

The scenario. A developer owns a customer-facing feature that calls one model through Agent Router. That provider starts returning 5xx responses and throttling mid-afternoon. The application has no retry or branching logic of its own, and adding some means a release.

The setup. The chain from Step 2, attached to the key the feature uses: gpt-4o as the primary, and claude-sonnet-4-20250514 from a different provider as the fallback below it. The application is not changed.

What happens. The gateway attempts the primary, records the failure, and serves the request from the secondary. The application receives one response rather than an error it has to handle. The model field in the response body names the backend that produced the answer, so a caller that reads it can tell that the substitution happened.

What to watch for. Two things.

Throttling counts as a failure worth moving on. Rate-limit responses sit in the recoverable set alongside 5xx responses, timeouts, and connection failures, so a primary that is throttling moves traffic to the secondary rather than passing the throttle back to the caller.

Not every failure is one the chain covers. A malformed request or an authentication failure is returned to the caller directly and the chain is not walked, because a second backend answers the same way. Telling the two cases apart is the next scenario.

A request failed and it is not clear whether the chain was walked

The scenario. A developer is looking at a failed request from an application whose key carries a fallback policy. Whether the chain was attempted and exhausted, or never entered at all, decides what has to be fixed and which provider to raise it with.

The setup. None. The resolved model is recorded for every request.

What happens. The error class decides whether the chain is walked at all. Transient errors move the request to the next backend, and an error reaches the caller only once every backend in the chain has failed. Client-side errors are returned directly and the chain is not walked.

In Monitoring → Request Logs, filtering by the key and expanding the failed request shows the Resolved model field. A resolved model other than the primary is evidence that the chain was walked, because a later backend is reached only after an earlier one has failed. That field does not, on its own, separate a chain that was walked and exhausted from one that was never entered.

What to watch for. Two things.

An exhausted chain returns the last backend's error, not the primary's. Reading that error as the primary provider's failure sends the investigation to the wrong provider.

A policy that is switched off produces the same result as no policy at all. While the Enable toggle is off the chain is not enforced, and requests go directly to the model named in the API call, so confirming the toggle is part of reading the result.

A secondary proved itself during an incident and should take over as primary

The scenario. A provider outage put a week of traffic on the secondary, which handled it without complaint. The developer who owns the key wants that backend as the primary and the former primary behind it, with the key holding a working chain throughout.

The setup. Three steps:

1. Open the key's detail page and the Manual Availability Fallback section.
2. Drag the promoted backend to the top of the chain. It takes the Primary badge, and the former primary drops to the row below it and takes a Fallback badge.
3. Save the configuration.

A backend is retired rather than reordered with the remove control on its row; the backends that remain keep their relative priority order.

What happens. From the moment the configuration is saved, the promoted backend serves traffic under normal conditions, and the demoted one is reached only when the new primary returns a recoverable failure. No change is required in the calling application.

What to watch for. A reordered chain is a chain that has not been exercised in its new order. The failover exercise in Step 3 applies again after the reorder, because the position that is now the fallback has never been reached from the position that is now the primary.

The chain has to be stood down without being taken apart

The scenario. A developer needs a key to stop falling through for a while, either to reproduce a fault against the primary alone or to rule the chain out of an unrelated investigation. Deleting the policy and rebuilding it afterward is not an acceptable answer, because the composition and its order then have to be reconstructed from memory.

The setup. Set the Enable toggle on the Manual Availability Fallback section to off.

What happens. The chain is not enforced while the policy is switched off, and requests go directly to the model named in the API call. The policy itself, its backends and their order, is left in place, so setting the toggle back on restores the same chain.

What to watch for. The key's own chain is not walked while the toggle is off. A request to a failing primary fails unless something outside the key absorbs it: any model-level auto-fallback an operator has configured against the model itself still applies, as described in Provision models and providers. That makes a switched-off policy a deliberate and short-lived state on any key serving real traffic, and a reason to run this kind of work on a non-production key, as Step 3 does.

---

Where to go next

The guide assumes the fallback chain configured above remains attached to the key. Subsequent guides build on this configuration rather than replacing it.

Reduce cost with traffic splitting

Distribute traffic across backends by weight rather than priority.

Apply advanced routing rules

Combine fallback semantics with attribute-based dispatch.

Bring Your Own Key (BYOK) credentials in the chain

Introduce BYOK credentials into one or more positions in the fallback chain.

Monitor traffic and usage

Extend the verification step into ongoing dashboards, with cost and latency tracked per resolved model.