Improve resilience with fallbacks

A fallback policy names, per model, the ordered list of models the gateway tries when that model fails with a recoverable error. The calling application sees one request and one response, with no retry or branching logic of its own. This guide configures the policy on one API key in the Developer Console, the most specific of the three levels a request resolves.

---

Persona: Developer working in the Developer Console.

Estimated time: 10 to 20 minutes, plus a short period of intentional failure exercise.

Where a key policy sits

Since the 0.6.0 release, a request resolves its fallback list from the most specific level that defines one for the requested model:

1. The API key override, when the key carries a policy for that model.
2. The project override otherwise.
3. The organization default otherwise.

The winning level supplies the whole list; lists are not merged across levels. A key policy that is switched off is skipped, not treated as empty, so the project or organization list applies to that key until the policy is switched back on. An explicit empty list (Don't fall back in the Admin Console, an empty fallback_models through the API) is what stops a key or project inheriting a broader list.

The key level has three authoring surfaces:

Surface	Who	Levels it can author
Developer Console, the key's Fallback settings (this guide)	The key's owner	API key
Admin Console, Policies → Fallback, Add override (Manage fallback in the Console)	Organization administrators and project owners	Organization, project, API key
Policy API, UpsertFallbackPolicy with a target_scope_type of CUSTOMER, PROJECT, or API_KEY (Set up fallback)	Automation	Organization, project, API key

All three write the one key-scoped policy a key can hold, and the gateway resolves it the same way whichever surface wrote it. The two Consoles do not list each other's key policies, though: a policy saved here is enforced but is not shown on the Admin Console's Fallback page, and saving this section replaces whatever the Admin Console or the API set on the key. Author a given key's policy from one surface.

When this guide applies

Fallback is the right tool when the goal is resilience: keeping requests successful through provider issues, without distributing traffic deliberately across providers under normal conditions. The list is walked only on failure; a healthy primary serves 100% of traffic, and the fallback models are idle until something goes wrong.

For other routing intents, different patterns apply:

Intent	Pattern	Where it is covered
Distribute traffic by weight (cost, evaluation, gradual rollout)	Traffic splitting	Reduce cost with traffic splitting
Route based on request attributes (tenant, header, task type)	Advanced routing rules	Apply advanced routing rules
Enforce data residency or compliance boundaries	Fallback policy with carefully chosen providers	This guide (see Routing under compliance constraints below)

Fallback and traffic splitting can be combined. A weighted split selects the backend for a given request, and the fallback policy takes over only when the chosen backend fails.

Outcomes

By the end of this guide:

• A fallback policy with at least two models from different providers is attached to a working API key.
• The policy has been tested under at least one simulated failure, with the gateway falling through to the fallback model.
• Request Logs show, for each request, which model actually handled it and whether a fallback event occurred.

Prerequisites

This guide builds directly on Route requests across providers. Specifically, the following should already be in place:

• An API key created in the Console that routes to at least one backend.
• At least two enabled models in the Admin Console, ideally from different upstream providers. A policy whose fallback model is on the same provider as the protected model still works mechanically, but it provides no protection against a provider-wide outage, which is the most common failure the policy is designed to absorb.
• A terminal with curl (or a Python environment with the openai package) for the exercise step.

Step 1: choose the chain composition

A good fallback list is not "the requested model plus whatever else is enabled". Its composition determines what kinds of failures it can absorb and how surprised the calling application is when failover occurs. Three considerations matter:

• Provider independence. The list protects against provider failures only to the extent that its models fail independently. Two OpenAI models still go down when OpenAI does. A list that alternates providers (for example, an OpenAI primary with an Anthropic or Google fallback) absorbs single-provider outages. Differing wire formats are not an obstacle: the original request body is retained and re-translated for each model attempted, so an Anthropic model sits behind an OpenAI one without the calling application changing format.
• Quality consistency. When the list is walked, the application gets a response from a different model. If the fallback produces meaningfully worse output, the failover is visible to the end user even though the request succeeded. Where quality consistency matters, choose fallbacks close to the primary in capability: frontier model behind frontier model.
• Cost profile. Some lists are deliberately built so that the fallback is cheaper than the primary. This is acceptable when degraded-but-cheaper output is preferable to no output, and it reduces cost exposure during long incidents on the primary provider.

A typical first policy pairs the requested model with one fallback from a different provider:

Position	Example model	Role
Protected model	gpt-4o	The model the application names; serves 100% of traffic under normal conditions
Fallback 1	claude-sonnet-4-20250514	Tried when the protected model returns a recoverable failure

Further fallbacks can be added below. The marginal value drops quickly (two well-chosen models absorb the vast majority of realistic incidents), but a third backstop is reasonable for workloads with strict SLA commitments. The attempt budget caps the list: with the default of 3 attempts, the protected model plus two fallbacks is the most that is tried.

Step 2: configure the fallback policy

The policy is attached to the API key created in the previous guide, so a change made here affects only traffic presented with that key.

1. From the API Keys page in the Console, click Configure on the row for the key. The key's settings page opens on its Fallback section.
2. Set the Model-specific fallback policies toggle on.
3. Click Add model policy and choose the model to protect: the one the application names in its requests. It heads a new policy card.
4. Expand the card and click Add fallback model for each backup, in the order they should be tried. Where the same model is offered by other providers, those are listed under Suggested, and Add or Add all appends them.
5. Under Retry settings, keep max retries at 3 and the delay at 5 seconds, or adjust them. Both apply to every policy on the key.
6. Under Retry on, keep the five conditions or remove the ones that should not walk the list.
7. Click Save. The Console confirms with "Fallback configuration saved!".

Fallback policy field reference

The Fallback section of a key's settings page exposes the following controls:

• Model scope. The picker offers the models assigned to the project the API key belongs to, whatever project the project switcher is on. A model assigned to a different project is not listed, and a policy naming one is rejected on save with an error identifying the model. A model missing from the list is therefore a question of the project grant rather than of the switcher, covered in Add providers and models to projects.
• One policy per protected model. Each card protects the model at its head: only requests that name that model use the card's list. Up to five model policies can be created per key with Add model policy; a request for a model with no policy on the key resolves the project or organization level instead. The trash control on a card removes the policy and its whole list.
• Order. Fallback rows are numbered and tried in that order after the protected model. A row is reordered by dragging it, and the Remove control on a row drops that model while the others keep their relative order. The protected model stays at the head: a different first model is a different policy, not a reorder.
• Enable toggle. The Model-specific fallback policies toggle switches every policy on the key off without deleting any. While it is off, the key's policies are skipped: a request resolves the project or organization level for its model, and goes directly to the provider only when neither defines a list.
• max retries. The attempt budget for a request, counting the requested model as the first attempt, so the default of 3 leaves room for two fallback models and the gateway stops after the third attempt even when more are listed. Between 1 and 10. It is one attempt per model, not several attempts against the same model; runtime walk semantics are in Fallback and streaming. The same number is max_attempts in the policy API, which accepts 1 to 10 since the 0.7.0 release (earlier releases accepted any positive value). The Admin Console shows the budget as Up to N attempts, including the requested model, sets 3 on any level it creates, and does not edit it: the budget of a key is changed here or through the API, and the budget of a project or organization level through the API.
• sec delay. The wait before each retry, up to the value entered (the gateway picks a random wait up to it), between 1 and 60 seconds. Since the 0.7.0 release this is the backoff before the next attempt; in earlier releases the same value was the time each attempt had to start responding. Both retry settings are stored once per key rather than per policy.
• Retry on. Which failures walk the list, as removable tags with an Add condition control. Five conditions are available (timeouts, rate limits, server errors, connection failures, and provider overload), and all five are selected by default, which is the set the gateway has always retried. A provider authentication failure or a rejected payload never walks the list. Through the policy API, upserting any other retry_on token, including auth-error or the retired provider-auth and payload-rejected, is rejected with INVALID_ARGUMENT. Deselecting one stops the list being walked for it; the failure is returned to the calling application instead. At least one condition must remain: to stop retrying altogether, switch the section off or set max retries to 1. Requires a data plane on 0.3.0 or later; an older one keeps the full set until it is upgraded. Retry on applies to every policy on the key and is independent of the ordered list: changing one does not change the other.
• Passthrough requests. A request that carries the caller's own provider credential falls back within one provider only: the gateway keeps the steps of the first provider in the list that supports passthrough and drops the rest. The Passthrough and fallback notice at the top of the section says so, and the data-plane side is under Things a fallback cannot do.

Precedence over automatic fallback. Automatic fallback is a separate section on the same page, with its own toggle and no configuration of its own, currently covering Anthropic models. Where both sections are enabled and name the same model, the policy configured here takes precedence.

tip

Keep one API key per logical list composition rather than swapping the configuration on a shared key. Per-key routing is Agent Router's natural separation boundary, and Request Logs are filtered per key, so a separate key per composition keeps the analytics tidy and the rollback path obvious.

Step 3: test the failover path

A fallback policy that has never failed over is effectively unverified. The simplest way to exercise it is to introduce a failure on the protected model and observe whether the gateway walks to the fallback. Two approaches are practical, depending on how much disruption is acceptable in the environment under test.

• Protect a model that cannot answer. On a test key, add a policy whose protected model is enabled in the catalog but lacks a working provider connection, with the real fallback below it, then send a request naming that model. The gateway records the failure and falls through to the fallback, which should succeed.
• Use a non-production API key. Run the exercise against a key that is not serving production traffic, so that the experiment is fully isolated from real users.

Send a test request after each change. PROXY_URL stands for the proxy endpoint from the Console Dashboard, scheme included and ending in /v1:

curl PROXY_URL/chat/completions \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o",
    "messages": [{"role": "user", "content": "Confirm which backend served this request."}]
  }'

The request should still succeed. The model field in the response body reflects the model that actually produced the answer, which, during a deliberate failover exercise, should be the fallback rather than the protected model.

note

The gateway walks the list when a provider returns 429, 500, 503, 504, or 529, or when the attempt fails with a connection failure, gateway error, other 5xx, or deadline exceeded. A provider-origin 401, 413, or 422 is returned to the caller without walking the list. A gateway-origin 401 for an invalid Agent Router API key is produced before any provider is contacted, so the list is not entered. Statuses outside that set (for example a gateway-rejected 400 for a malformed body) are returned to the caller without walking the list. See Retry on in the field reference.

Step 4: verify the failover in Request Logs

The exercise above is only useful if its result is observable after the fact. Request Logs records, per request, which model actually served the response and whether a fallback event occurred.

1. In the Console, open Monitoring → Request Logs.
2. Filter by the API key used in the test.
3. Locate the request issued during the exercise.
4. Expand the row to see the detail panel.
5. Confirm the Resolved model field matches the fallback, not the protected model. If a fallback event is recorded, it appears in the request timeline as a separate attempted-and-failed entry on the protected model, followed by a successful entry on the fallback.

The expected pattern in normal operation is the opposite: every request resolves to the protected model, and no fallback events appear. A sudden cluster of fallback events in production is a strong leading indicator that something is happening on the primary provider's side, often visible in Request Logs before the provider's own status page acknowledges the incident.

For richer filtering (by time range, by status, or by resolved model), see Monitor traffic and usage.

What the list covers on streaming requests

Streaming narrows the window in which fallback can act, and an application that streams should be built with that in mind.

The list is walked up to the first response byte. A model that refuses the request, times out before responding, or answers with a recoverable status is replaced by the next one, and a streaming client never learns that the substitution happened: the stream it eventually receives is the fallback's, in the format the request used.

Once the first chunk has been delivered, the list is spent. A model that accepts the request and then fails partway through generation cannot be replaced, because a second model's generation cannot be spliced onto a partial answer. The client sees a stream that ends without its terminating event, and recovery is the application's to perform by re-issuing the request. The policy then protects that retry in the ordinary way.

Two further limits belong to the same picture:

• A dropped client connection is not resumable. Nothing is buffered for replay, so a client that disconnects mid-stream re-sends the request rather than reconnecting to the generation in progress.
• Enforcing output guardrails removes streaming for the request. The response is buffered for evaluation before any of it is delivered, which is unrelated to fallback but changes what the caller experiences on the same request path.

A streaming client should therefore treat a missing terminator as a failure and retry. The mechanics, including the per-format terminating events and a worked client example, are in Streaming behavior.

Fallback on the Responses API

/v1/responses is routable to OpenAI and Azure OpenAI backends only. A policy on a key that serves Responses API traffic fails on any step resolving to another provider, which removes the protection at the moment it is needed. Lists that cross the wider provider catalog require Chat Completions or Anthropic Messages. See Provider support for the Responses API.

Routing under compliance constraints

A fallback list is also the natural mechanism for policy-driven routing where compliance rather than resilience is the dominant concern. Because the list contains only the models explicitly added to it, the list itself is the enforceable boundary: a list that contains only EU-hosted providers cannot, by construction, route to a US-hosted provider, regardless of what the calling application requests.

Three common patterns use this property directly:

• Data residency. Build the list from providers in the required region only (for example, Azure EU as the protected model with GCP EU as the fallback) so failover never leaves the region.
• Approved providers. Limit the list to providers that have been reviewed and approved for the use case in question, and assign that policy to a key dedicated to that use case.
• Output consistency. Use the same logical model class across positions in the list to keep response quality steady during failover, even when the upstream provider changes.

There is no "compliance mode" to enable. The boundary is enforced by the composition of the list. Where a broader level already defines a list for the model, a key that must never fall back at all needs an explicit Don't fall back override from the Admin Console or the API, since a key with no policy of its own inherits the broader list.

Scenarios

The situations below are the ones that bring developers to a fallback policy most often. Each is worked against the two-model policy from Step 1, and each states the problem, the setup it needs, the resulting behavior, and what to watch for.

The primary provider fails during the working day and the application has no retry logic

The scenario. A developer owns a customer-facing feature that calls one model through Agent Router. That provider starts returning 5xx responses and throttling mid-afternoon. The application has no retry or branching logic of its own, and adding some means a release.

The setup. The policy from Step 2, attached to the key the feature uses: gpt-4o protected, with claude-sonnet-4-20250514 from a different provider as its fallback. The application is not changed.

What happens. The gateway attempts gpt-4o, records the failure, and serves the request from the fallback. The application receives one response rather than an error it has to handle. The model field in the response body names the model that produced the answer, so a caller that reads it can tell that the substitution happened.

What to watch for.

Throttling counts as a failure worth moving on. Rate-limit responses (429) sit in the recoverable set alongside the other statuses and connection failures listed under Retry on, so a primary that is throttling moves traffic to the fallback rather than passing the throttle back to the caller.

Not every failure is one the list covers. A gateway-rejected request (for example a malformed body that never leaves the gateway, or an invalid Agent Router API key) is returned to the caller and the list is not entered. A provider-origin 401, 413, or 422 is not in the recoverable set either, so it is returned without walking the list. Telling the two cases apart is the next scenario.

A request failed and it is not clear whether the list was walked

The scenario. A developer is looking at a failed request from an application whose key carries a fallback policy. Whether the list was attempted and exhausted, or never entered at all, decides what has to be fixed and which provider to raise it with.

The setup. None. The resolved model is recorded for every request.

What happens. The error class decides whether the list is walked at all. Recoverable provider failures (the statuses and connection classes under Retry on) move the request to the next model, and an error reaches the caller only once every model in the list has failed or the attempt budget is spent. A failure produced by the gateway before any provider is contacted is returned directly and the list is not entered.

In Monitoring → Request Logs, filtering by the key and expanding the failed request shows the Resolved model field. A resolved model other than the protected one is evidence that the list was walked, because a later model is reached only after an earlier one has failed. That field does not, on its own, separate a list that was walked and exhausted from one that was never entered.

What to watch for.

An exhausted list returns the last model's error, not the primary's. Reading that error as the primary provider's failure sends the investigation to the wrong provider.

A key whose policy is switched off may still have fallen back. While the toggle is off the key's own policy is skipped, but a project or organization list for the same model still applies. Which level supplied the list is answered by Test a request on the Admin Console's Fallback page or by the API's effective-policy read (F7).

A fallback proved itself during an incident and should become the primary

The scenario. A provider outage put a week of traffic on the fallback, which handled it without complaint. The developer who owns the key wants that model as the primary and the former primary behind it, with the key holding a working policy throughout.

The setup. A policy belongs to the model a request names, so the fallback becomes the primary when the application asks for it. Two steps:

1. On the key's settings page, click Add model policy, choose the model being promoted, and add the former primary under it as its fallback. Save. The key now holds two policies, one per protected model.
2. Change the model the application names in its requests to the promoted one. Where the application's model name is better left alone, a model alias maps the old name to the new model without a change in the calling code.

The old policy can stay on the key, so a request that still names the former primary keeps its protection, or be removed with the trash control on its card.

What happens. From the moment the application names the promoted model, that model serves traffic under normal conditions, and the demoted one is reached only when the new primary returns a recoverable failure.

What to watch for. A new policy is a policy that has not been exercised. The failover exercise in Step 3 applies again, because the model that is now the fallback has never been reached from the model that is now the primary.

The policy has to be stood down without being taken apart

The scenario. A developer needs a key to stop falling through for a while, either to reproduce a fault against the primary alone or to rule the policy out of an unrelated investigation. Deleting the policy and rebuilding it afterward is not an acceptable answer, because the composition and its order then have to be reconstructed from memory.

The setup. Set the Model-specific fallback policies toggle on the key to off.

What happens. The key's policies are skipped while the toggle is off and left in place, so setting the toggle back on restores the same lists.

What to watch for. Switching the key off does not switch fallback off. A request for the model resolves the project or organization level instead, and goes directly to the provider only when neither defines a list; any model-level auto-fallback an operator has configured against the model itself still applies, as described in Provision models and providers. A key that must get the primary's error and nothing else needs an explicit Don't fall back override for that model, set from the Admin Console or the API. Either way this is a deliberate and short-lived state on any key serving real traffic, and a reason to run this kind of work on a non-production key, as Step 3 does.

---

Where to go next

The guide assumes the fallback policy configured above remains attached to the key. Subsequent guides build on this configuration rather than replacing it.

Manage fallback in the Console

The organization and project levels, key overrides, and Test a request in the Admin Console.

Reduce cost with traffic splitting

Distribute traffic across backends by weight rather than priority.

Apply advanced routing rules

Combine fallback semantics with attribute-based dispatch.

Bring Your Own Key (BYOK) credentials in the list

Introduce BYOK credentials into one or more positions in the fallback list.

Monitor traffic and usage

Extend the verification step into ongoing dashboards, with cost and latency tracked per resolved model.