Fallback: keep a model serving when it fails
Enterprise Tier
A fallback keeps requests answered when a model fails. Nine scenarios cover the ones people set most often, worked against one company so the examples build.
Available in Release 1. This page describes the shipped fallback-policy behavior for Enterprise deployments.
The public contract calls this a Fallback Policy. The generated API still contains the compatibility names FallbackChainPolicy and chain; new configuration uses grouped entries, with one protected model and its ordered fallback_models per entry. See the fallback policy implementation and the scope and upsert contract.
Release 1 contract
- A policy entry protects one
(model, provider)pair. Itsfallback_modelsare tried in order only after the protected model fails with a configured retryable category. - New policy resolution is API key > project > organization/customer. The winning scope supplies the complete entry; entries are not merged with a broader scope.
- An empty
fallback_modelslist explicitly disables inherited fallback for that protected model. - Tags are not a new fallback-policy authoring scope. Existing tag-targeted rows remain a legacy compatibility path and may still appear in effective-policy results.
- A complete-set upsert is safe to replay. It replaces the managed set, so an entry omitted from the next desired set is removed rather than left behind.
- Fallback targets are not resolved recursively. The configured entry is the complete attempt sequence.
What a fallback is
A fallback belongs to a model. You pick the model you want protected, then list what to use instead when it fails.
| Model | the one you are protecting. This is what the fallback is for |
| Fallback model | what to use instead. List more than one and they are tried in order |
| Fall back on | which failures move to the next model: a timeout, a rate limit, a server error |
One model, one fallback. Ten models that need protecting means ten fallbacks, each named by its own model.
Where it applies is separate. Set a fallback for the whole organization and every project gets it, including projects created later. Set one on a single project when only that project needs it, or on a single API key when one caller must behave differently. The most specific canonical scope wins, and it is the whole answer. Narrowing changes who it reaches; it does not change what the fallback is.
The order is the point. Requests use the primary until it fails. A fallback model never takes traffic while the model is healthy. This is not a traffic split, which sends a set share of requests elsewhere all the time.
Fallback is not failover. Fallback moves a request to a different model. Failover moves traffic between a project's member gateways, and is set up with your DNS. They are different mechanisms and neither replaces the other.
The company
Acme reaches the same models two ways:
| Provider | Models |
|---|---|
openai | gpt-5.6-sol and the rest, direct |
openai-azure | the same models, through Azure |
Every model has an entry on both, and normal traffic goes to openai. Acme's catalog also carries gpt-5.6-luna, which costs a fraction of gpt-5.6-sol.
Why there are two is up to you: a second vendor for the same model, the same vendor bought through a cloud, the same model in another region, or a cheaper model standing in for a better one. Nothing below depends on which, and the two do not have to be in different places.
Find your situation
| You want to | Go to |
|---|---|
| Keep serving when a model fails | F1 |
| Use the same fallback for every model in a family | F2 |
| Fall back to a different model, not the same one elsewhere | F3 |
| Try more than one fallback model | F4 |
| Narrow one to a single caller | F5 |
| Choose which failures count as a failure | F6 |
| Check what is actually in effect | F7 |
| Set it up from a pipeline | F8 |
| Stay in one region unless you cannot | F9 |
Things you cannot do: listed at the end.
F1: Keep serving when a model fails
You want to. When gpt-5.6-sol on openai refuses a request, serve it from openai-azure rather than returning an error.
Set it up. Create a fallback for gpt-5.6-sol. The model goes first, fallback models below it in the order they should be tried:
| Model | Provider | |
|---|---|---|
| Model | gpt-5.6-sol | openai |
| Fallback model | gpt-5.6-sol | openai-azure |
Fall back on: timeout, rate limit, server error.
Models are named as they appear in your catalog. There are no ids to look up. Set it on the organization and every project gets it; set it on one project when only that project needs it.
The fallback models are the attempt count. The model is tried first, then each fallback model in turn, so adding one adds an attempt. Nothing is retried twice: a failed model is left behind, not tried again.
What happens. Requests go to openai. When one fails in a way you have listed, the same request is retried against openai-azure and the caller sees a normal response. Nothing goes to Azure while openai is healthy.
Watch out for. A fallback is per request, not a switch. The next request starts at the model again. If openai is failing steadily, every request pays the failed attempt first, so a long outage makes every response slower.
F2: Use the same fallback for every model in a family
You want to. Every model in the catalog has an entry on both providers, and each should fall back to its own equivalent.
Set it up. One fallback per model, all in the same project:
| Model | Provider | |
|---|---|---|
| Model | gpt-5.6-sol | openai |
| Fallback model | gpt-5.6-sol | openai-azure |
| Model | Provider | |
|---|---|---|
| Model | gpt-5.6-terra | openai |
| Fallback model | gpt-5.6-terra | openai-azure |
| Model | Provider | |
|---|---|---|
| Model | gpt-5.6-luna | openai |
| Fallback model | gpt-5.6-luna | openai-azure |
Ten models means ten fallbacks. Each is named by the model it protects, so they do not collide.
What happens. A request names a model; the fallback for that model decides where it goes. Models with no fallback are unaffected.
Watch out for. Adding a model to the catalog does not give it a fallback. Each one is created separately, which is why this is worth driving from a pipeline rather than by hand.
F3: Fall back to a different model, not the same one elsewhere
You want to. gpt-5.6-sol is refusing requests and there is no second account to reach it through. Serving on the cheaper gpt-5.6-luna is better than serving nothing.
Set it up. The fallback model is a different model on the same provider:
| Model | Provider | |
|---|---|---|
| Model | gpt-5.6-sol | openai |
| Fallback model | gpt-5.6-luna | openai |
Fall back on: timeout, rate limit, server error.
What happens. A failed request is retried on the cheaper model and the caller gets an answer. Nothing goes to gpt-5.6-luna while gpt-5.6-sol is working.
Watch out for. The caller is not told they got a different model. A fallback returns a normal response, and the response names the model that served it, so an application that assumes it always gets gpt-5.6-sol will not notice. Where that matters, the request log is the record of which model answered.
F4: Try more than one fallback model
You want to. If both accounts are failing, try the cheaper model before giving up.
Set it up. List the fallback models in the order they should be tried:
| Model | Provider | |
|---|---|---|
| Model | gpt-5.6-sol | openai |
| Fallback model 1 | gpt-5.6-sol | openai-azure |
| Fallback model 2 | gpt-5.6-luna | openai |
Fall back on: timeout, rate limit, server error.
What happens. Each is tried in turn until one answers. The last is reached only when everything above it has failed.
Watch out for. Every attempt costs time before the caller hears anything. A model plus two fallback models means the slowest possible failure is three failures end to end. Keep the list as short as the outcome needs, and set the caller's own timeout with the whole list in mind.
F5: Narrow one to a single caller
You want to. The payments project falls back gpt-5.6-sol to Azure, but one batch job must never be served by a different account. It should get an error instead.
Set it up. Create a second fallback for the same model, scoped to that job's API key, with no fallback model:
| Set at | Fallback for gpt-5.6-sol |
|---|---|
the payments project | falls back to gpt-5.6-sol on openai-azure |
| the batch job's API key | no fallback model |
What happens. That key gets an error when gpt-5.6-sol fails, rather than a different account. Every other key in the project falls back as normal, including keys made tomorrow.
Watch out for. The narrower one replaces the wider one, it does not add to it. A key's own fallback does not also get the project's. Whichever is most specific is the whole answer, and because a key sits in a project and a project in an organization, there is never a tie.
F6: Choose which failures count as a failure
You want to. Fall back when the provider is struggling, and not when the request itself is wrong.
Set it up. Choose which failures move a request on:
| Condition | What it covers | Falls back? |
|---|---|---|
timeout | the provider did not answer in time | ✅ |
rate-limit | the provider refused for load (HTTP 429) | ✅ |
server-error | the provider failed (HTTP 500 or 503) | ✅ |
| a provider out of capacity | throttling arrives as HTTP 429, unavailable as 503, so rate-limit and server-error cover it | ✅ |
| a malformed request | your application sent something invalid | ❌ |
| a content refusal | the request was refused for what it contained | ❌ |
| a bad credential | the provider rejected the account | ❌ |
What happens. A slow or overloaded provider moves the request to the fallback model. A malformed request, or one refused for its content, returns its error to the caller.
Watch out for. Do not make authentication errors fall back. If a bad credential counts as a failure, a broken provider looks like healthy fallback: every request quietly serves from the fallback model and nothing reports a problem. The account keeps working, the bill moves, and the credential stays broken.
F7: Check what is actually in effect
You want to. Confirm a key falls back the way you think it does, before trusting it in production.
Set it up. Nothing to create. Read the effective fallback for the key:
| What you get back | |
|---|---|
| Model | gpt-5.6-sol on openai |
| Fallback model | gpt-5.6-sol on openai-azure |
| Fall back on | timeout, rate limit, server error |
| Where it came from | the payments project |
What happens. You see the fallback models in order and which fallback supplied them: the organization's, the project's, or the key's own.
Watch out for. What is configured and what applies are different questions. A key with its own fallback shows that one, not the project's, which is the fastest way to find out why one key behaves differently from its neighbours.
The effective-policy response also identifies the source scope and policy ID for the winning entry. Use that read-back when a project or key appears not to inherit the organization policy you expected.
F8: Set it up from a pipeline
You want to. Manage fallback as code, applied on every pipeline run, without duplicating anything.
Set it up. Submit the complete desired set of grouped entries in your pipeline. A representative payload is:
{
"entries": [
{
"model": {"model": "gpt-5.6-sol", "provider": "openai"},
"fallback_models": [
{"model": "gpt-5.6-sol", "provider": "openai-azure"}
]
},
{
"model": {"model": "gpt-5.6-luna", "provider": "openai"},
"fallback_models": []
}
]
}
| Pipeline run | Result |
|---|---|
| first run, two entries | both protected-model entries are created |
| second run, same input | the same policy set is retained; no duplicates are created |
| second run, a fallback model changed | only that protected model's ordered entry changes |
| a model dropped from your config | its entry is removed from the managed set |
What happens. Re-running with the same complete input converges to the same result. Replacing the complete set also removes entries no longer present, which prevents stale fallback routes from surviving a configuration change.
Watch out for. The request is the desired set, not a patch. Omitting an entry removes it from the managed target; use an explicit empty fallback_models list when you want to keep the protected model but disable its inherited fallback.
F9: Stay in one region unless you cannot
You want to. The same model runs in two regions, on a provider account in each. Requests should be served in the primary region, and cross to the second only when there is no alternative.
Set it up. The fallback model is the same model on the other region's account:
| Model | Provider | |
|---|---|---|
| Model | claude-sonnet-5 | bedrock-primary-region |
| Fallback model | claude-sonnet-5 | bedrock-second-region |
Fall back on: timeout, rate limit, server error.
Repeat it for every model that needs it, one fallback each.
What happens. Steady-state traffic is served in the primary region. A request crosses only when it has failed there, and the next request starts in the primary region again.
Watch out for. Two things.
A fallback is a decision about where a request may be served. The fallback model runs somewhere else: another region, another account, sometimes another contract. If some traffic must never leave the primary region, that model needs no fallback at all rather than a carefully chosen one.
Two regions is two provider accounts, not one provider with a setting. Each region is its own entry in the catalog with its own credentials, so a model has to exist in both before it can be named in a fallback.
Things you cannot do
Set a fallback on a model in the catalog, or through a project setting. A fallback is its own thing, created where policies are. A project setting named global-fallback is unrelated despite the name. It holds no model mappings.
Fall back to a model the project does not have. The fallback model has to be in the project's catalog. A fallback chooses among the models a project already has; it does not grant access to a new one.
Fall back across data planes. A fallback picks a different model. Moving traffic between a project's member gateways is failover, set up with your DNS.
Split traffic between two models permanently. That is a traffic split, and it is a different policy. A fallback only moves a request that failed.
Match a specific error message. Fall back on takes categories (timeout, rate limit, server error), not individual status codes or strings.
Have two fallback policies apply at once. One applies to a key (the most specific one) and it replaces the others rather than combining with them.
Know that a fallback fired from the response. A retried request returns a normal response. Use the request log to see which model served it.