Gateway behavior

The gateway provides consistent, predictable behavior across all AI providers. It normalizes provider errors into a single format, attaches correlation IDs to every request, and emits structured observability data regardless of which upstream provider handles the request. Streaming is the exception to that uniformity, because a failure after the first chunk cannot be normalized or retried; those limits are set out under Streaming behavior.

---

Error handling

Every AI provider has its own error schema, status code semantics, and message vocabulary. When a request routes through the gateway (whether to a primary provider, a fallback target, or one leg of a traffic split), every error response is normalized into a single OpenAI-compatible format before being returned to the client. A single error-handling path is sufficient on the application side even as providers change behind the scenes.

Error response format

All errors, whether they originate at the gateway itself or at an upstream provider, are returned with the following JSON structure:

{
  "error": {
    "message": "A human-readable description of the error",
    "type": "error_type",
    "param": null,
    "code": "error_code"
  }
}

Field	Description
message	A human-readable explanation. Useful for logging and surfacing context during development.
type	A machine-readable error category (rate_limit_error, timeout_error, invalid_request_error, and similar). Use this field for programmatic error handling.
param	The specific request parameter that caused the error, when applicable. Often null for provider-level and gateway-level errors.
code	A specific error code within the error type. Distinguishes subtypes (rate_limit_exceeded vs. model_not_found, for example).

Error origin

Errors originate at two distinct points in the request lifecycle:

• Gateway errors are produced by the gateway before the request reaches a provider. These include authentication failures, malformed request bodies, unknown model names, and policy violations. HTTP status codes are typically in the 4xx range.
• Provider errors are returned by an upstream provider and normalized by the gateway before being forwarded to the client. These include rate limits, model overload conditions, and provider outages. The HTTP status code reflects the nature of the provider failure.

In both cases, the response body uses the same JSON format and the same X-Request-ID correlation header is present.

HTTP status codes

HTTP Status	Meaning	Typical cause
400	Bad Request	Malformed request body, unsupported parameters, missing required fields, or a model id that is not in the catalog
401	Unauthorized	Invalid or missing API key
403	Forbidden	API key lacks permission for the requested model or endpoint
404	Not Found	Unknown endpoint path, or a model that is in the catalog but has no route on this data plane
429	Too Many Requests	Rate limit exceeded at the client level or by the upstream provider
500	Internal Server Error	Unexpected error within the gateway
502	Bad Gateway	The upstream provider returned an invalid or unparseable response
503	Service Unavailable	The upstream provider is temporarily unavailable or returning server errors, or the gateway itself cannot validate credentials (see the note below)
504	Gateway Timeout	The upstream provider did not respond. No request deadline is imposed on inference routes, so this reflects a connection-level failure or the provider's own timeout, see Timeouts

Retryable vs. non-retryable errors

Status	Retryable?	Recommended action
400	No	Inspect the message and param fields; fix the request before retrying
401	No	Verify and rotate the API key
403	No	Confirm the model or endpoint is enabled for the API key
404	No	Verify the model id with GET /v1/models for the same API key (Models API), and confirm the model is enabled for that key
429	Yes	Retry with exponential backoff; honor any Retry-After header if present
500	Maybe	Retry once; if the error persists, investigate using Request Logs
502	Yes	Retry; the provider response was malformed but may succeed on a subsequent attempt
503	Yes	Retry with backoff; the provider is temporarily unavailable, or the gateway replica cannot currently validate credentials
504	Yes	Retry. If it recurs consistently, raise the client-side timeout: the gateway sets none of its own

A 503 that is not the provider's fault

A gateway replica that cannot validate credentials answers every request with a 503 of its own, before any provider is contacted:

{
  "error": {
    "message": "This gateway replica is temporarily unable to validate credentials. Retry the request.",
    "type": "service_unavailable",
    "param": null,
    "code": "service_unavailable"
  }
}

The status is deliberate: the client's API key is not at fault, so a 401 would be misleading. On a self-hosted data plane, a 503 on every request with this message points at the data plane credential rather than at a provider. See Data plane credentials.

tip

When fallback policies are configured, the gateway walks the chain for the recoverable set that applies to the key. For Console-configured keys that set is provider responses with HTTP status 401, 413, 422, 429, 500, 503, 504, or 529, plus connection failures, gateway errors, other 5xx responses, and deadline exceeded. The walk is transparent to the calling application. The application only receives an error response if every provider in the chain is exhausted or if the failure is outside that set (including a gateway-origin rejection before any provider is contacted). The table above is client guidance for what an application should retry on its own; it is not the same list as the gateway's fallback walk. A gateway-origin 401 for an invalid Agent Router API key should not be retried by the client; a provider-origin 401 can still move the request to the next backend in the chain.

Error examples

Upstream rate limit (provider returns 429):

{
  "error": {
    "message": "Rate limit exceeded. Please retry after 30 seconds.",
    "type": "rate_limit_error",
    "param": null,
    "code": "rate_limit_exceeded"
  }
}

Provider timeout (no response within the timeout window):

{
  "error": {
    "message": "Upstream provider did not respond in time.",
    "type": "timeout_error",
    "param": null,
    "code": "timeout"
  }
}

Invalid or missing API key:

{
  "error": {
    "message": "Invalid API key. Please check your credentials.",
    "type": "authentication_error",
    "param": null,
    "code": "invalid_api_key"
  }
}

Model not present in the catalog. This is checked before the request is proxied anywhere, so it returns 400, not 404:

{
  "error": {
    "message": "Model 'example-model' does not exist or is not enabled. Call /v1/models for the list of available models.",
    "type": "invalid_request_error",
    "param": null,
    "code": "model_not_found"
  }
}

A model that is enabled but not yet routable

Three different conditions all present as a missing model, and the code field is what distinguishes them. Only the first is a client mistake.

Code	Status	Meaning
model_not_found	400	The model id is not in the catalog, or it is disabled. Check the spelling against GET /v1/models.
model_not_ready	503	The model's route exists but is not active yet, because a configuration change is still reaching this data plane. The response carries Retry-After: 30.
model_not_routed	404	The model is enabled but no route is deployed for it on this data plane.

model_not_ready is a wait, not a failure. It is returned with a Retry-After header for exactly that reason, and most SDKs retry it without any application code:

{
  "error": {
    "message": "The route for model 'example-model' is provisioned but not yet active on this gateway - it is still propagating. Retry shortly.",
    "type": "service_unavailable_error",
    "param": null,
    "code": "model_not_ready"
  }
}

model_not_routed immediately after enabling a model usually means the same thing and clears on its own. If it persists for more than a minute, the model has no route on this data plane and the cause is configuration rather than timing: confirm the model is enabled in the catalog and assigned to the project the API key belongs to.

{
  "error": {
    "message": "Model 'example-model' is enabled but has no active route on this gateway. If the model was added recently its route may still be propagating; otherwise no route is deployed for it on this data plane.",
    "type": "not_found_error",
    "param": null,
    "code": "model_not_routed"
  }
}

The gateway's GET /v1/models returns the configuration this data plane has received, not the management-plane catalog. A model enabled a moment ago is therefore absent from it until the change lands, and a request for it returns model_not_found. The Console shows the model straight away, because the Console reads the catalog. The two surfaces disagreeing is the normal appearance of a change in flight. See Configuration propagation.

---

Streaming behavior

Streaming changes what the gateway can and cannot do about a failure. Once a provider has answered with 200 and the first bytes have been forwarded, the response status is already committed: it cannot be changed, and the request cannot be sent to a different backend. Everything below follows from that one constraint.

What is supported

Capability	State	Detail
Streaming on gateway-managed APIs	Supported	Chat Completions, Completions, Responses, Anthropic Messages, and audio speech. See Gateway APIs for the per-endpoint list
Token-by-token delivery over SSE	Supported	Chunks are forwarded as the provider emits them, on both HTTP/1.1 and HTTP/2
Zero buffering on the streaming path	Supported	Nothing accumulates in the gateway between the provider and the client, see Streaming latency
Stream translation across providers	Supported	The wire format follows the API the client called, not the provider that served it. Chat Completions and Messages translate to every provider; the Responses API is restricted to OpenAI and Azure OpenAI
Error normalization before the stream opens	Supported	A provider that fails at the request stage returns the standard error envelope
Draining in-flight streams during a data plane upgrade	Supported	300 seconds by default, see Upgrades and pod replacement
Fallback after the first response byte	Not supported	Failover is decided when the provider's response headers arrive, see Fallback and streaming
Error normalization after the stream opens	Not supported	The status is committed; the client sees the provider's own event stream or a truncated one
Replay or buffering for a disconnected client	Not supported	See Client disconnects
A gateway-side deadline on generation length	Not configured	No request timeout is set on inference routes, see Timeouts
A custom error body on timeout or failure	Not supported	The error envelope is fixed

Fallback and streaming

Fallback policies are evaluated up to the first response byte. The decision point is the arrival of the provider's response headers: a provider that refuses the request, times out before responding, or answers with a retryable status is replaced by the next backend in the chain, and the client is unaware that anything happened.

Each attempt is translated independently. The original request body is retained and re-translated for whichever backend is tried next, and route header and body mutations are reapplied, so a chain may span providers whose wire formats differ. A chain running an Anthropic backend behind an OpenAI one, or the reverse, works without the calling application changing format.

A provider that accepts the request and then fails partway through generation is past the decision point. The chain is not walked, no second backend is tried, and no tokens are re-sent. Any resilience for a stream that dies mid-generation belongs to the client application.

The boundary is the same one that governs error normalization, and for the same reason: until the response headers arrive, the response is still buffered and the gateway holds every option. Once a 200 has been forwarded, the status is committed.

Mid-stream provider failures

Three things can happen once a stream is open, and they are distinguishable on the client side:

• The provider emits its own error event. The event is forwarded to the client. Its contents are the provider's, not the gateway's, so it does not follow the error envelope and its shape varies by provider.
• The provider closes the connection. The client sees the stream end with no terminating event: no data: [DONE] on Chat Completions and Completions, no response.completed event on the Responses API, no message_stop on Anthropic Messages.
• The stream completes normally. The terminator arrives.

The practical consequence is that the terminator, not the HTTP status, is what tells a client the response is complete. A stream that ends without one was truncated, and the status code is 200 in every case.

stream = client.chat.completions.create(
    model="claude-sonnet-4-6",
    messages=[{"role": "user", "content": "Hello"}],
    stream=True,
)

completed = False
chunks = []
try:
    for chunk in stream:
        if chunk.choices and chunk.choices[0].finish_reason is not None:
            completed = True
        chunks.append(chunk)
except Exception as exc:
    # The connection dropped partway through generation.
    print(f"stream interrupted after {len(chunks)} chunks: {exc}")

if not completed:
    # Truncated. Treat as a failure and retry the whole request; partial
    # output cannot be resumed. Capture x-request-id first, see Request tracking.
    ...

A guardrail violation is the one failure that looks mid-stream but is not a truncation, and the check above does not apply to it. Because output-stage enforcement buffers the response before any of it is delivered, a blocked response never becomes a partial stream: the client receives a structured 403 and the standard error envelope in place of the stream, rather than a connection that dies partway through. See Guardrails and streaming.

Streaming latency

The streaming path holds nothing back. Once a provider has answered 200 to a streaming request, response chunks are relayed as they arrive rather than accumulated, so the gateway adds no buffering delay of its own and delivery latency is dominated by the provider's own generation speed. Nothing on the gateway makes a model produce tokens faster.

What is controllable is the work the gateway performs around the stream, and there are two levers:

Lever	Effect on streaming	Where it is set
Guardrail mode on the output stage	Monitor mode evaluates asynchronously and leaves the stream intact. Enforce mode buffers the whole response before delivering any of it, which removes streaming for that request	Per rule, see Guardrails and streaming
Guardrail evaluation timeout	Caps how long any single evaluation may add. Input-stage evaluation runs before the request is forwarded, so it is added to time-to-first-token even when the stream itself is untouched	Per guardrail, in milliseconds, see Guardrails reference

Neither lever is a latency budget for the generation itself. No gateway-side deadline is imposed on inference routes at all, as Timeouts sets out.

Streaming latency is measured rather than estimated. Two instruments are recorded per request and exported over OTLP, and both are specific to streaming:

Metric	Meaning
gen_ai.server.time_to_first_token	Time from request-header receipt to the first token. Recorded on streaming responses only
gen_ai.server.time_per_output_token	Mean time per output token after the first, recorded at end of stream

Both carry model, provider, and tenancy dimensions, so a slow provider or a slow model is separable from a slow gateway, and alerts can be scoped per model. A First Token Stream Event is also stamped on the span for the request. Full definitions, dimensions, and the known gaps are in OTel metrics.

Timeouts

No request timeout and no backend-request timeout are set on inference routes. A long generation is not cut off by a gateway-side deadline, which is what allows reasoning models and long completions to run to completion.

Two consequences follow. The first is that the client's own timeout is the effective one, so an HTTP client whose default is 30 or 60 seconds will abandon a long generation that the gateway would have served. Raise it, or stream. The second is that neither the timeout window nor the error body returned on failure is configurable per route or per key.

A 504 therefore reflects a connection-level failure or a provider-side deadline rather than a gateway request deadline.

Client disconnects

Nothing is buffered for a client that has gone away. When the connection drops, the gateway does not hold the response, does not continue delivery on reconnect, and does not replay the tokens already sent. There is no session to resume against and no resume token.

An application that must survive client disconnects has to persist chunks as they arrive and re-issue the request from the beginning.

Token accounting for an interrupted stream is not reliable. Usage is totalled and recorded once, at the end of the stream, so a stream that never reaches its terminator may report incomplete token counts or none at all. Treat Request Logs as authoritative for whether a request happened, and as approximate for what an interrupted one consumed.

Upgrades and pod replacement

In-flight streams are drained rather than dropped when a gateway pod is replaced, whether by an upgrade, a rollout, or rescheduling. The gateway stops accepting new connections, keeps serving open ones, and terminates once they finish or the drain window expires.

Setting	Default	Effect
drainTimeout	300s	Total time open connections are given before the pod is terminated. The 300-second default is sized for long-running LLM streams
minDrainDuration	5s	Floor observed even when no connections are active, so load balancers stop sending new traffic before the pod goes away

A stream still running when drainTimeout expires is cut off, and the client sees a truncated stream. Where generations routinely run longer than five minutes, the drain window should be raised to match before an upgrade rather than during one.

Guardrails and streaming

An output-stage guardrail in enforce mode buffers the whole response before delivering any of it, because a rule cannot judge content it has not seen in full. Streaming is lost for that request, and the caller waits for the complete generation. This holds for both rule actions: a redact rule has to rewrite content it has already seen, so it buffers exactly as a block rule does. Output-stage rules in monitor mode evaluate asynchronously and leave streaming intact.

Two consequences follow for a streaming client:

• A block is a clean error, not a broken stream. Because the response was buffered, nothing has been delivered when the violation is found. The caller receives HTTP 403 with the standard error envelope and a correlation ID, and never sees a half-written answer. This is the one case where a streaming request legitimately returns a non-200 status after the provider succeeded.
• Input-stage enforcement delays the stream without removing it. Input rules run before the request is forwarded, so their evaluation time is added to time-to-first-token, but the response still streams. See Streaming latency.

Mode and stage are per rule, so a single guardrail can enforce on input while monitoring on output, which keeps prompts policed and responses streaming. See Streaming behavior in the guardrails reference.

---

Health and status visibility

There is no consumer-facing status page and no aggregate health API. What exists is one per-gateway probe and two after-the-fact surfaces.

The membership probe

Each project gateway answers a probe whose meaning is ready member of this project:

curl -s https://<gateway-hostname>/healthz

The published path is /healthz; the original /healthz/membership returns the same body and is not deprecated.

Three properties make it usable as a client-side pre-flight, and one limits it:

• It is answered by the data plane itself, with no management-plane round trip, so it keeps answering during a management-plane outage.
• GET and HEAD are exempt from gateway API-key authentication, so no credential is needed to call it. Every other method and path still requires a key.
• It is served on the same hostname and standard HTTPS port that clients use, so it exercises DNS, TLS, and the listener.
• It reports nothing about upstream providers. A green probe on a gateway whose provider credentials are missing is expected and correct.

The probe is the same signal DNS health checks target in a multi-gateway deployment. Its full contract, including what each state means during a detach, is in Gateway sets and DNS-level failover.

Checking whether a request will actually work

The probe answers whether the gateway is serving. It does not answer whether a given model is callable with a given key. That question is answered by GET /v1/models, which returns exactly the models the calling key can reach on that data plane. An empty or short list is the pre-flight signal that configuration, not the gateway, is the problem.

Observing downtime after the fact

Question	Surface
Did this request fail, and where	Request Logs, searchable by X-Request-ID
How often are requests failing, over time	The gateway's OTel metrics, exported to the observability stack
Was the data plane itself unhealthy	tare doctor on a self-hosted data plane, see Reading tare doctor output

Alerting on gateway availability is built on the exported metrics rather than on a Tetrate-hosted status signal. Operators who need a consumer-facing status page publish one from those metrics.

---

Request tracking

The gateway attaches correlation IDs to every request. These IDs link the HTTP response the application receives to the detailed record stored in Request Logs and to the spans emitted to the OpenTelemetry backend.

Correlation headers

Header	Direction	Description
X-Request-ID	Response	A gateway-generated UUID attached to every response. The primary identifier for looking up the request in Request Logs and OTel traces.
X-Client-Request-ID	Response	Echoed back from the X-Request-ID header sent by the client, if present. Allows correlation between gateway records and application-side request identifiers.

When a client sends an X-Request-ID header, the gateway preserves it as X-Client-Request-ID in the response and generates its own X-Request-ID. Both IDs appear in Request Logs and OpenTelemetry traces.

Sending a correlation ID

PROXY_URL stands for the proxy endpoint from the Console Dashboard, scheme included and ending in /v1.

curl -i PROXY_URL/chat/completions \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -H "X-Request-ID: my-session-abc-123" \
  -d '{
    "model": "gpt-4o",
    "messages": [{"role": "user", "content": "Hello"}]
  }'

Response headers:

HTTP/2 200
x-request-id: 8f14e45f-ceea-467f-a8f0-6b2a1c3d4e5f
x-client-request-id: my-session-abc-123
content-type: application/json

The x-request-id value (8f14e45f-...) is the gateway's canonical identifier. The x-client-request-id (my-session-abc-123) is the original client ID, preserved for joining gateway records with application-side logs.

Data captured per request

Every request processed by the gateway produces a record containing:

Field	Description
Request ID	The gateway-assigned X-Request-ID UUID
Client Request ID	The echoed X-Client-Request-ID, if provided by the client
Timestamp	When the request was received by the gateway
Model	The model identifier specified in the request
Provider	The upstream provider that served the response
HTTP Status	The final HTTP status code returned to the client
Latency	Total round-trip time from gateway receipt to completed response
Token usage	Prompt tokens, completion tokens, and total tokens
Request / response body	Full payloads, subject to the deployment's data retention settings

This data is searchable in Request Logs and is also available as OpenTelemetry span attributes in the tracing backend.

Correlating in observability tools

• Request Logs. Search by X-Request-ID (gateway-assigned) or X-Client-Request-ID (the application's ID) to retrieve the complete request record: provider used, latency breakdown, token counts, and raw request and response payloads.
• OTel traces. Both IDs are emitted as span attributes on every trace. Search for them in the tracing backend (Jaeger, Grafana Tempo, Honeycomb, Datadog, and others) to view the full execution path including any fallback retries.

tip

Custom application-defined headers (agent-session-id, user-id, workflow-run-id, and similar) are forwarded as OpenTelemetry span attributes. This allows traces to be grouped or filtered by any application-level concept (conversation ID, agent run, team, deployment) without affecting how the gateway routes requests.

Debugging workflow

A typical end-to-end debugging workflow using correlation IDs:

1. Capture the response header. Log the x-request-id value from every response in the application. For error responses, also capture the response body.
2. Search Request Logs. Paste the x-request-id value into the search field in Request Logs to retrieve the full request record: which provider was used, total latency, token counts, fallback attempts, and raw payloads.
3. Inspect the trace. If OTel export is configured, search for the same x-request-id as a span attribute in the tracing backend to see provider-level timing and any retry hops.
4. Join with application logs. If a custom X-Request-ID was sent, use the echoed x-client-request-id in the gateway record to join against application logs and reconstruct the full request context.

---

Related

• Gateway APIs: inference wire formats and endpoint reference
• Configuration propagation: how long a configuration change takes to reach the data plane
• Console messages and states: what a banner, badge, or empty state in the Console means
• OTel Metrics: metric names, types, and labels exported by the gateway
• Audit Log Events: event schema for administrative actions

---

Where to go next

Gateway APIs

Inference wire formats and endpoint reference.

Monitor traffic and usage

Find requests in Request Logs by correlation ID.