Gateway behavior

The gateway behaves the same way whichever AI provider serves the request: provider errors are normalized into a single format, correlation IDs are attached to every request, and structured observability data is emitted regardless of the upstream. Streaming is the exception to that uniformity, because a failure after the first chunk cannot be normalized or retried; those limits are set out under Streaming behavior.

---

Error handling

Every AI provider has its own error schema, status code semantics, and message vocabulary. When a request routes through the gateway (whether to a primary provider, a fallback target, or one leg of a traffic split), every error response is normalized into a single OpenAI-compatible format before being returned to the client. A single error-handling path is sufficient on the application side even as providers change behind the scenes.

Error response format

All errors, whether they originate at the gateway itself or at an upstream provider, are returned with the following JSON structure:

{
  "error": {
    "message": "A human-readable description of the error",
    "type": "error_type",
    "param": null,
    "code": "error_code"
  }
}

Field	Description
message	A human-readable explanation. Useful for logging and surfacing context during development.
type	A machine-readable error category (rate_limit_error, timeout_error, invalid_request_error, and similar). Use this field for programmatic error handling.
param	The specific request parameter that caused the error, when applicable. Often null for provider-level and gateway-level errors.
code	A specific error code within the error type. Distinguishes subtypes (rate_limit_exceeded vs. model_not_found, for example).

Error origin

Errors originate at two distinct points in the request lifecycle:

• Gateway errors are produced by the gateway before the request reaches a provider. These include authentication failures, malformed request bodies, unknown model names, and policy violations. HTTP status codes are typically in the 4xx range.
• Provider errors are returned by an upstream provider and normalized by the gateway before being forwarded to the client. These include rate limits, model overload conditions, and provider outages. The HTTP status code reflects the nature of the provider failure.

In both cases, the response body uses the same JSON format and the same X-Request-ID correlation header is present.

HTTP status codes

HTTP Status	Meaning	Typical cause
400	Bad Request	Malformed request body, unsupported parameters, missing required fields, or a model id that is not in the catalog
401	Unauthorized	Invalid or missing API key
403	Forbidden	API key lacks permission for the requested model or endpoint
404	Not Found	Unknown endpoint path, or a model that is in the catalog but has no route on this data plane
429	Too Many Requests	Rate limit exceeded at the client level or by the upstream provider
500	Internal Server Error	Unexpected error within the gateway
502	Bad Gateway	The upstream provider returned an invalid or unparseable response
503	Service Unavailable	The upstream provider is temporarily unavailable or returning server errors, or the gateway itself cannot validate credentials (see the note below)
504	Gateway Timeout	The upstream provider did not respond. No request deadline is imposed on inference routes, so this reflects a connection-level failure or the provider's own timeout, see Timeouts

Retryable vs. non-retryable errors

Status	x-should-retry	Retryable?	Recommended action
400	false	No	Inspect the message and param fields; fix the request before retrying
401	false	No	Verify and rotate the API key
403	false	No	Confirm the model or endpoint is enabled for the API key
404	false	No	Verify the model id with GET /v1/models for the same API key (Models API), and confirm the model is enabled for that key
429	true	Yes	Retry with exponential backoff; honor any Retry-After header if present. Two exceptions below
500	true	Maybe	Retry once; if the error persists, investigate using Request Logs
502	true	Yes	Retry; the provider response was malformed but may succeed on a subsequent attempt
503	true	Yes	Retry with backoff; the provider is temporarily unavailable, or the gateway replica cannot currently validate credentials
504	true	Yes	Retry. If it recurs consistently, raise the client-side timeout: the gateway sets none of its own

The x-should-retry header

Errors the gateway composes before a response starts streaming carry an x-should-retry header whose value is the literal string true or false. The official OpenAI and Anthropic SDKs, in both TypeScript and Python, read that header before applying their own status-code rules, so where it is present it decides the retry rather than the status does. An application that implements its own retry logic should do the same, and treat an absent header as "apply the status rule".

Three properties are worth knowing:

• It is a pre-stream signal only. It appears on error responses, never on a 2xx, and never once a stream has begun. Past the first byte of the body neither SDK retries anything regardless of what any header said, which is why the mid-stream cases under Streaming behavior are the application's to handle.
• The default restates the status. For most errors the value is what the SDKs would have concluded on their own. Publishing it makes the advice a contract rather than a per-client inference, and it survives an intermediary that rewrites the status.
• Two responses deliberately diverge from their status, and both are 429:

Case	Header	Why
Budget exhausted	false	A period budget refills on a billing period, not in seconds. Left to the bare status, both SDKs retry twice within about a second and the same budget rejects all three attempts
Rate limit exceeded	absent	Produced earlier in the proxy than the gateway's own error path, so no header is attached and the client falls back to the status rule. That is the right outcome: a rate limit does refill on its own

Distinguish the two by the body. A budget block carries x-tars-budget-action: block.

A 503 that is not the provider's fault

A gateway replica that cannot validate credentials answers every request with a 503 of its own, before any provider is contacted:

{
  "error": {
    "message": "This gateway replica is temporarily unable to validate credentials. Retry the request.",
    "type": "service_unavailable",
    "param": null,
    "code": "service_unavailable"
  }
}

The status is deliberate: the client's API key is not at fault, so a 401 would be misleading. On a self-hosted data plane, a 503 on every request with this message points at the data plane credential rather than at a provider. See Data plane credentials.

tip

When fallback policies are configured, the gateway walks the chain for the recoverable set that applies to the key. That set defaults to provider responses with HTTP status 429, 500, 503, 504, or 529, plus connection failures, gateway errors, other 5xx responses, and deadline exceeded, and is configurable per key under Retry on in the Fallback section. A provider-origin 401, 413, or 422 is returned to the caller without walking the chain. The walk is transparent to the calling application. The application only receives an error response if every provider in the chain is exhausted or if the failure is outside that set (including a gateway-origin rejection before any provider is contacted). The table above is client guidance for what an application should retry on its own; it is not the same list as the gateway's fallback walk.

These are two independent layers. A provider 429 is where they most visibly differ: the gateway walks the chain when a backend throttles, and where the chain is exhausted the same 429 reaches the client with x-should-retry guidance to back off and retry. A client retry and a chain walk are not the same action, so the two verdicts are consistent rather than contradictory.

Error examples

Upstream rate limit (provider returns 429):

{
  "error": {
    "message": "Rate limit exceeded. Please retry after 30 seconds.",
    "type": "rate_limit_error",
    "param": null,
    "code": "rate_limit_exceeded"
  }
}

Provider timeout (no response within the timeout window):

{
  "error": {
    "message": "Upstream provider did not respond in time.",
    "type": "timeout_error",
    "param": null,
    "code": "timeout"
  }
}

Invalid or missing API key:

{
  "error": {
    "message": "Invalid API key. Please check your credentials.",
    "type": "authentication_error",
    "param": null,
    "code": "invalid_api_key"
  }
}

Model not present in the catalog. This is checked before the request is proxied anywhere, so it returns 400, not 404:

{
  "error": {
    "message": "Model 'example-model' does not exist or is not enabled. Call /v1/models for the list of available models.",
    "type": "invalid_request_error",
    "param": null,
    "code": "model_not_found"
  }
}

A key rejected on the wrong hostname

A request whose API key belongs to a project that the hostname does not serve is refused with 403. Since 0.3.0 the refusal carries machine-readable fields alongside the standard envelope. On a data plane on 0.7.0 or later the message is one fixed sentence, whatever the reason for the refusal, and it names neither the hostname nor the project:

{
  "error": {
    "message": "This API key's project is not served at this hostname. Check which endpoints serve your project, and quote reference_id to support if you believe this is wrong.",
    "type": "authorization_error",
    "param": null,
    "code": "forbidden",
    "category": "hostname_not_selected",
    "reference_id": "req_4f2a91c7",
    "your_hostnames": ["proxy.example.com"]
  }
}

The added fields:

Field	Description
category	hostname_not_selected. A stable value a client can branch on. Clients that match on the message text break on 0.7.0, because the text changed and no longer differs by reason.
reference_id	An identifier to quote to support, present when one was assigned to the request.
your_hostnames	Deprecated in 0.7.0, removed in 0.8.0. The hostnames this data plane serves for the calling key's own project. It is scoped to the one data plane that answered and can be empty.

A client that receives this 403 retries against another endpoint serving its project. The endpoints serving a project are listed in the Developer Console, so record them there rather than reading your_hostnames, which a data plane on 0.8.0 no longer sends.

A model that is enabled but not yet routable

Four different conditions all present as a missing model, and the code field is what distinguishes them. Only the first is a client mistake.

Code	Status	Meaning
model_not_found	400	The model id is not in the catalog, or it is disabled. Check the spelling against GET /v1/models.
model_not_available	400	The model is listed by GET /v1/models but its catalog entry is still propagating to the serving path. Retry; the window closes on its own. Returned by data planes on 0.4.0 or later; earlier data planes report this case as model_not_found.
model_not_ready	503	The model's route exists but is not active yet, because a configuration change is still reaching this data plane. The response carries Retry-After: 30.
model_not_routed	404	The model is enabled but no route is deployed for it on this data plane.

model_not_ready is a wait, not a failure. It is returned with a Retry-After header for exactly that reason, and most SDKs retry it without any application code:

{
  "error": {
    "message": "The route for model 'example-model' is provisioned but not yet active on this gateway - it is still propagating. Retry shortly.",
    "type": "service_unavailable_error",
    "param": null,
    "code": "model_not_ready"
  }
}

model_not_routed immediately after enabling a model usually means the same thing and clears on its own. If it persists for more than a minute, the model has no route on this data plane and the cause is configuration rather than timing: confirm the model is enabled in the catalog and assigned to the project the API key belongs to. Look at the model's own provider when doing so: since 0.5.0 an unreadable credential on one provider no longer stalls route deployment for the others (the affected provider stays on its last deployed configuration until its credential is readable again), so a persisting model_not_routed is not explained by a broken credential elsewhere.

{
  "error": {
    "message": "Model 'example-model' is enabled but has no active route on this gateway. If the model was added recently its route may still be propagating; otherwise no route is deployed for it on this data plane.",
    "type": "not_found_error",
    "param": null,
    "code": "model_not_routed"
  }
}

The gateway's GET /v1/models returns the configuration this data plane has received, not the management-plane catalog. A model enabled a moment ago is therefore absent from it until the change lands, and a request for it returns model_not_found. The Console shows the model straight away, because the Console reads the catalog. The two surfaces disagreeing is the normal appearance of a change in flight. Since 0.4.0 the catalog is published to the gateway earlier, so a just-enabled model appears in GET /v1/models sooner; a request for it answers model_not_available while its entry propagates and model_not_ready while its route deploys, and model_not_found is left meaning the model is genuinely not in the catalog. See Configuration propagation.

---

Request normalization

Translating a request between API dialects sometimes means removing what the target model cannot accept. Since 0.3.0 the gateway does this predictably and says what it did: everything it removes is named in the request log under normalized_dropped, and dropped tools are additionally named in a response header. The two cases below are the ones that matter in practice; both apply only when the route's upstream is OpenAI-shaped.

Headers removed from every request

Separately from dialect translation, the gateway removes a set of request headers on arrival, before anything reads them. They are the headers only the gateway itself legitimately writes, so a client-sent copy can never be genuine and is never forwarded to a model provider or an MCP server.

Data plane	Removed on arrival
0.6.0 and later	Every x-ai-eg-* header: Envoy AI Gateway's internal routing signals. A client sending x-ai-eg-model could influence which route was picked, and client-sent values now have no effect
0.7.0 and later	The identity headers the gateway stamps from the verified key, such as x-tars-user and x-router-project-id, and every x-internal-traces-* header, the per-user trace attribution the gateway sets. The removal applies to every request, including /mcp and the OAuth endpoints

Authorization and x-api-key are handled as before. x-tars-supports-reasoning, the per-request reasoning override, is read by the gateway and then removed, so no filter behind it sees it either.

Two consequences follow for a caller that sends one of these headers for an upstream to read:

• The upstream no longer receives it. Move the value to a header the gateway forwards before upgrading the data plane to 0.7.0.
• An identity provider token carried in a gateway auth policy's extraction header is removed after verification and no longer reaches a model provider or MCP server. The extraction header has to be one the gateway can remove after verifying the token, which is why the management plane refuses an extraction header it cannot strip (x-api-key, cookie, or proxy-authorization, for example) and warns about a stored policy that names one. Upgrade the management plane before the data planes: a data plane upgraded first removes x-tars-idp-token before the gateway can verify it where a gateway auth policy already exists. See Upgrade a self-hosted data plane.

Extended thinking toward a model that does not reason

On /v1/messages, extended thinking is not forwarded to an OpenAI-compatible model the catalog does not report as reasoning-capable. Both carriers are removed before the request leaves the gateway: the thinking field, which would otherwise be translated into a reasoning_effort an upstream without reasoning rejects, and the thinking and redacted_thinking blocks in earlier assistant turns, which an OpenAI-compatible upstream rejects outright since it accepts only text there. Conversation history acquires those blocks on a mid-session model switch, so this is what keeps a session that started on a reasoning model working after switching to a model that does not reason.

Nothing is removed on other routes: an Anthropic, Bedrock, Vertex Anthropic, Gemini, or Vertex upstream understands thinking natively or replays it, and a reasoning-capable model of any provider receives its thinking untouched. The capability comes from the catalog, and a catalog entry with no capabilities recorded reads as not reasoning-capable; sending x-tars-supports-reasoning: true on a request overrides that for one call when an OpenAI-compatible backend does reason and the catalog does not say so.

Anthropic server tools toward an OpenAI-backed model

Anthropic server tools (web search, bash, the text editor family, and newer ones such as tool search) have no OpenAI equivalent: Anthropic runs them itself, whereas an OpenAI tool definition is a contract for the client to execute, so there is nothing to translate them into. A request that asks an OpenAI-backed model for one still succeeds, the tool is dropped, and the response carries an x-tars-dropped-fields header naming each dropped tool. Ordinary function tools alongside them are unaffected. The header is the thing to check when an answer looks like it skipped a tool, for example a confident answer to a question that needed a web search. Routing the same request to an Anthropic or Vertex Anthropic model forwards the tool unchanged.

---

Streaming behavior

Streaming changes what the gateway can and cannot do about a failure. Once a provider has answered with 200 and the first bytes have been forwarded, the response status is already committed: it cannot be changed, and the request cannot be sent to a different backend. Everything below follows from that one constraint.

What is supported

Capability	State	Detail
Streaming on gateway-managed APIs	Supported	Chat Completions, Responses, Anthropic Messages, and audio speech. See Gateway APIs for the per-endpoint list
Token-by-token delivery over SSE	Supported	Chunks are forwarded as the provider emits them, on both HTTP/1.1 and HTTP/2
Zero buffering on the streaming path	Supported	Nothing accumulates in the gateway between the provider and the client, see Streaming latency
Stream translation across providers	Supported	The wire format follows the API the client called, not the provider that served it. Chat Completions and Messages translate to every provider; the Responses API is restricted to OpenAI and Azure OpenAI
Error normalization before the stream opens	Supported	A provider that fails at the request stage returns the standard error envelope
Draining in-flight streams during a data plane upgrade	Supported	300 seconds by default, see Upgrades and pod replacement
Fallback after the first response byte	Not supported	Failover is decided when the provider's response headers arrive, see Fallback and streaming
Error normalization after the stream opens	Not supported	The status is committed; the client sees the provider's own event stream or a truncated one
Replay or buffering for a disconnected client	Not supported	See Client disconnects
Provider-native stream resume	Not supported	Provider endpoints for reattaching to an existing response, such as the Responses API retrieve path, are not gateway inference endpoints. See Client disconnects
A gateway-side deadline on generation length	Not configured	No request timeout is set on inference routes, see Timeouts
A custom error body on timeout or failure	Not supported	The error envelope is fixed

Fallback and streaming

Fallback policies are evaluated up to the first response byte. The decision point is the arrival of the provider's response headers: a provider that refuses the request, times out before responding, or answers with a retryable status is replaced by the next backend in the chain, and the client is unaware that anything happened.

Each attempt is translated independently. The original request body is retained and re-translated for whichever backend is tried next, and route header and body mutations are reapplied, so a chain may span providers whose wire formats differ. A chain running an Anthropic backend behind an OpenAI one, or the reverse, works without the calling application changing format.

A provider that accepts the request and then fails partway through generation is past the decision point. The chain is not walked, no second backend is tried, and no tokens are re-sent. Any resilience for a stream that dies mid-generation belongs to the client application.

The boundary is the same one that governs error normalization, and for the same reason: until the response headers arrive, the response is still buffered and the gateway holds every option. Once a 200 has been forwarded, the status is committed.

How many attempts each backend gets

On a recoverable failure, the gateway makes one attempt per backend, then moves to the next backend in the fallback chain. There is no same-target retry. The recoverable set (Retry on) and the backend order (the chain) are independent: either may be changed without changing the other. Both are configured under Fallback policy field reference.

Fallback is decided before the first response byte, so no partial generation is replayed. There is no idempotency opt-in. A request that already reached a provider and then failed with a recoverable error can still be sent to the next backend. Side-effecting work is handled in the client.

Mid-stream provider failures

Three things can happen once a stream is open, and they are distinguishable on the client side:

• The provider emits its own error event. The event is forwarded to the client. Its contents are the provider's, not the gateway's, so it does not follow the error envelope and its shape varies by provider.
• The provider closes the connection. The client sees the stream end with no terminating event: no data: [DONE] on Chat Completions, no response.completed event on the Responses API, no message_stop on Anthropic Messages.
• The stream completes normally. The terminator arrives.

The practical consequence is that the terminator, not the HTTP status, is what tells a client the response is complete. A stream that ends without one was truncated, and the status code is 200 in every case.

stream = client.chat.completions.create(
    model="claude-sonnet-4-6",
    messages=[{"role": "user", "content": "Hello"}],
    stream=True,
)

completed = False
chunks = []
try:
    for chunk in stream:
        if chunk.choices and chunk.choices[0].finish_reason is not None:
            completed = True
        chunks.append(chunk)
except Exception as exc:
    # The connection dropped partway through generation.
    print(f"stream interrupted after {len(chunks)} chunks: {exc}")

if not completed:
    # Truncated. Treat as a failure and retry the whole request; partial
    # output cannot be resumed. Capture x-request-id first, see Request tracking.
    ...

A guardrail violation is the one failure that looks mid-stream but is not a truncation, and the check above does not apply to it. Because output-stage enforcement buffers the response before any of it is delivered, a blocked response never becomes a partial stream: the client receives a structured 403 and the standard error envelope in place of the stream, rather than a connection that dies partway through. See Guardrails and streaming.

Streaming latency

The streaming path holds nothing back. Once a provider has answered 200 to a streaming request, response chunks are relayed as they arrive rather than accumulated, so the gateway adds no buffering delay of its own and delivery latency is dominated by the provider's own generation speed. Nothing on the gateway makes a model produce tokens faster.

What is controllable is the work the gateway performs around the stream, and there are two levers:

Lever	Effect on streaming	Where it is set
Guardrail mode on the output stage	Monitor mode evaluates asynchronously and leaves the stream intact. Enforce mode buffers the whole response before delivering any of it, which removes streaming for that request	Per rule, see Guardrails and streaming
Guardrail evaluation timeout	Caps how long any single evaluation may add. Input-stage evaluation runs before the request is forwarded, so it is added to time-to-first-token even when the stream itself is untouched	Per guardrail, in milliseconds, see Guardrails reference

Neither lever is a latency budget for the generation itself. No gateway-side deadline is imposed on inference routes at all, as Timeouts sets out.

Streaming latency is measured rather than estimated. Two instruments are recorded per request and exposed on the Prometheus scrape endpoint, from which an external collector can forward them as OTLP, and both are specific to streaming:

Metric	Meaning
gen_ai.server.time_to_first_token	Time from request-header receipt to the first token. Recorded on streaming responses only
gen_ai.server.time_per_output_token	Mean time per output token after the first, recorded at end of stream

Both carry model, provider, and tenancy dimensions, so a slow provider or a slow model is separable from a slow gateway, and alerts can be scoped per model. A First Token Stream Event is also stamped on the span for the request. Full definitions, dimensions, and the known gaps are in OTel metrics.

Timeouts

No request timeout and no backend-request timeout are set on inference routes. A long generation is not cut off by a gateway-side deadline, which is what allows reasoning models and long completions to run to completion.

Two consequences follow. The first is that the client's own timeout is the effective one, so an HTTP client whose default is 30 or 60 seconds abandons a long generation that the gateway would have served. Raise it, or stream. The second is that neither the timeout window nor the error body returned on failure is configurable per route or per key.

A 504 therefore reflects a connection-level failure or a provider-side deadline rather than a gateway request deadline.

Client disconnects

Nothing is buffered for a client that has gone away. When the connection drops, the gateway does not hold the response, does not continue delivery on reconnect, and does not replay the tokens already sent. There is no session to resume against and no resume token.

Provider-native resume is not available through the gateway either. Where a provider offers its own mechanism for reattaching to a response in progress, such as the retrieve path of the OpenAI Responses API, that path is not one of the gateway inference endpoints and a request to it is rejected with a 404 before any provider is contacted. Resuming a stream is therefore always a matter of re-issuing the original request.

An application that must survive client disconnects has to persist chunks as they arrive and re-issue the request from the beginning.

Token accounting for an interrupted stream is not reliable. Usage is totalled and recorded once, at the end of the stream, so a stream that never reaches its terminator may report incomplete token counts or none at all. Treat Request Logs as authoritative for whether a request happened, and as approximate for what an interrupted one consumed.

Upgrades and pod replacement

In-flight streams are drained rather than dropped when a gateway pod is replaced, whether by an upgrade, a rollout, or rescheduling. The gateway stops accepting new connections, keeps serving open ones, and terminates once they finish or the drain window expires.

Setting	Default	Effect
drainTimeout	300s	Total time open connections are given before the pod is terminated. The 300-second default is sized for long-running LLM streams
minDrainDuration	5s	Floor observed even when no connections are active, so load balancers stop sending new traffic before the pod goes away

A stream that finishes inside the drain window is delivered in full. The terminating pod serves it through to its terminator and records it as a completed 200, and the calling application sees an ordinary response: no error, no truncation, and nothing to retry. This is measured rather than inferred. A streaming completion was opened through the gateway and its serving pod deleted two seconds later; the pod went on to deliver the remaining four minutes of that response and closed it normally.

A stream still running when drainTimeout expires is cut off, and the client sees a truncated stream. Where generations routinely run longer than five minutes, the drain window should be raised to match before an upgrade rather than during one.

Guardrails and streaming

An output-stage guardrail in enforce mode buffers the whole response before delivering any of it, because a rule cannot judge content it has not seen in full. Streaming is lost for that request, and the caller waits for the complete generation. This holds for both rule actions: a redact rule has to rewrite content it has already seen, so it buffers exactly as a block rule does. Output-stage rules in monitor mode evaluate asynchronously and leave streaming intact.

Two consequences follow for a streaming client:

• A block is a clean error, not a broken stream. Because the response was buffered, nothing has been delivered when the violation is found. The caller receives HTTP 403 with the standard error envelope and a correlation ID, and never sees a half-written answer. This is the one case where a streaming request legitimately returns a non-200 status after the provider succeeded.
• Input-stage enforcement delays the stream without removing it. Input rules run before the request is forwarded, so their evaluation time is added to time-to-first-token, but the response still streams. See Streaming latency.

Mode and stage are per rule, so a single guardrail can enforce on input while monitoring on output, which keeps prompts policed and responses streaming. See Streaming behavior in the guardrails reference.

---

Health and status visibility

Since 0.3.0 a gateway endpoint answers status questions directly: an unauthenticated status document at its root, and an authenticated per-provider reachability endpoint at /v1/status. Both are answered by the data plane itself, with no management-plane round trip, so they keep working during a management-plane outage, and both accept only GET and HEAD. Alongside them sit the data plane's readiness probe and the after-the-fact surfaces described below. There is still no aggregate, deployment-wide health API.

Checking whether the gateway is serving

The root of a gateway endpoint answers with a JSON status document, with no API key needed, so the base URL already in an SDK configuration can be opened in a browser to tell an outage at the gateway from a problem with a key or request:

curl -s https://<gateway-hostname>/

{
  "status": "serving",
  "message": "This gateway is up and has routes deployed. That does not report the health of the model providers it forwards to.",
  "hostname": "proxy.example.com",
  "data_plane": "proxy.example.com",
  "checked_at": "2026-09-10T12:00:00Z"
}

Field	Description
status	serving when routes are deployed, not_serving when the gateway is up with nothing configured to serve, and unknown when the gateway cannot read which of those is true. On a data plane on 0.7.0 or later the state is about the hostname called: a hostname with nothing deployed reports not_serving even while other hostnames on the same data plane serve. Earlier data planes could report serving for such a hostname by reading another hostname's routes, and requests then answered 404.
message	A human-readable explanation of the state. Every state other than serving carries the sentence "This is not a problem with your API key or your request."
hostname	The hostname the request reached, confirming which endpoint answered.
data_plane	Deprecated. Carries the same value as hostname and never named a data plane: one hostname can be served by several data planes. Read hostname instead. /v1/status names the pod that answered as replica.
checked_at	When the reported state was read, in RFC 3339 UTC.

No response at all means the data plane itself is down. The root is unauthenticated, so the document names no model, provider, or project: it reports state only. GET /ok, and any method other than GET or HEAD on the root, still return the bare 200 they always did.

Per-provider reachability

GET /v1/status takes the same API key used for inference and adds per-provider reachability, so a failing provider can be identified without contacting support:

curl -s https://<gateway-hostname>/v1/status -H "Authorization: Bearer $TARS_API_KEY"

{
  "status": "serving",
  "message": "This gateway is up and has routes deployed. That does not report the health of the model providers it forwards to.",
  "hostname": "proxy.example.com",
  "data_plane": "proxy.example.com",
  "checked_at": "2026-09-10T12:00:00Z",
  "replica": "tars-egress-5c9d7c6b9-x2x7l",
  "observation_window": "5m0s",
  "observed_only": true,
  "providers": [
    { "name": "anthropic", "reachable": true, "observed_requests": 128, "failures": 0 },
    { "name": "openai", "reachable": null, "observed_requests": 0, "failures": 0 }
  ]
}

Reachability is derived from the outcomes of requests this gateway actually forwarded, never from probing providers, and the response states its own limits. On a data plane on 0.7.0 or later the figures cover only requests from the calling key's own project: a provider that another project on the same data plane calls is not listed, and its traffic cannot make a provider read reachable while every request from the caller's project fails. Expect lower counts than on earlier data planes, and an empty providers list on a pod that has served none of the project's requests.

Field	Description
replica	The gateway pod that answered. Each pod observes only its own traffic, so two replicas behind one load balancer can answer differently.
observation_window	How far back the figures reach: a sliding five-minute window. The counters also reset when configuration is pushed.
observed_only	Always true: the figures record forwarded requests, not probes.
providers[].reachable	true when observed traffic is coming back, false when every observed request in the window failed, and null when no traffic was observed, meaning no traffic seen, not healthy.
providers[].observed_requests, failures	Requests forwarded to the provider in the window and how many failed. Only 5xx outcomes count as failures.
providers[].last_failure_at, last_failure_code	The most recent failure in the window and its status code, present when there was one.

Gateway and provider health in the Admin Console

Since 0.3.0 the same distinction is visible to operators without touching an endpoint: the Admin Console's data plane list carries a Serving column with a Gateway row and a Providers row side by side. The gateway row reads healthy when requests are reaching an upstream at all, whatever the upstream then returns; the providers row names any provider that has answered only with server errors. Before the split, a data plane forwarding every request normally to a provider that was answering nothing but errors looked the same as a data plane that was down, so a provider outage arrived as a support question about the gateway. Either row reads as unreported rather than healthy when the data plane has not sent the signal, so an old or silent data plane is never shown as green.

The readiness probe

A data plane answers a readiness probe on every hostname it declares:

curl -s https://<declared-hostname>/healthz/membership

{"status":"ready","dataplane":"dp-prod-use1"}

/healthz/membership carries the readiness answer on every deployment. Bare /healthz carries the same answer once the operator turns on the published contract with the global.publishHealthzContract Helm value, which is off by default; until then bare /healthz answers a controller liveness 200 with {"status":"ok"} on any host. Green is 200 with "status":"ready". Red is any other status, concretely 404, or a connection failure.

On a data plane on 0.7.0 or later the probe answers for the whole data plane, not for one project on it:

• It answers 200 on every declared hostname whether the data plane carries no project, one project, or many. Removing one project from the data plane leaves the probe green for the others.
• A request whose Host is not a declared hostname gets 404. A cloud load balancer that probes by IP address gets 404 here and belongs on the isolated liveness listener on port 10081 instead.
• A data plane whose chart declares no hostname answers on the address stored for it through the admin API, on its gateway URLs, and on the hostnames its routes carry. A chart that declares an empty list answers 404 everywhere, which is how a data plane is drained deliberately.

A data plane before 0.7.0 answers one probe per project gateway on that gateway's address, with {"status":"ready","gateway":"<gateway-uuid>"}, and the probe turns red when the project is detached from the gateway.

Three properties make the probe usable as a client-side pre-flight, and two limit it:

• It is answered by the data plane itself, with no management-plane round trip, so it keeps answering during a management-plane outage.
• GET and HEAD are exempt from gateway API-key authentication, so no credential is needed to call it. Every other method and path still requires a key.
• It is served on the same hostname and standard HTTPS port (443) that clients use, so it exercises DNS, TLS, and the listener.
• It says nothing about whether a given project is served on the data plane, or holds its current configuration. That is read from the project's Data Planes tab in the Admin Console.
• It reports nothing about upstream providers. A green probe on a data plane whose provider credentials are missing is expected and correct.

The probe is the signal DNS health checks target when several data planes serve one hostname. The setup is in Serve a project from two data planes.

Checking whether a request will actually work

The probe answers whether the gateway is serving. It does not answer whether a given model is callable with a given key. That question is answered by GET /v1/models, which returns exactly the models the calling key can reach on that data plane. An empty or short list is the pre-flight signal that configuration, not the gateway, is the problem.

Observing downtime after the fact

Question	Surface
Did this request fail, and where	Request Logs, searchable by X-Request-ID
How often are requests failing, over time	The gateway's OTel metrics, exported to the observability stack
Was the data plane itself unhealthy	tare doctor on a self-hosted data plane, see Reading tare doctor output

Alerting on gateway availability is built on the exported metrics rather than on a Tetrate-hosted status signal. Operators who need a consumer-facing status page publish one from those metrics.

---

Request tracking

The gateway attaches correlation IDs to every request. These IDs link the HTTP response the application receives to the detailed record stored in Request Logs and to the spans emitted to the OpenTelemetry backend.

Correlation headers

Header	Direction	Description
X-Request-ID	Response	A gateway-generated UUID attached to every response. The primary identifier for looking up the request in Request Logs and OTel traces.
X-Client-Request-ID	Response	Echoed back from the X-Request-ID header sent by the client, if present. Allows correlation between gateway records and application-side request identifiers.

When a client sends an X-Request-ID header, the gateway preserves it as X-Client-Request-ID in the response and generates its own X-Request-ID. Both IDs are kept in Request Logs. Only X-Request-ID reaches the trace stream, as the span attribute request.id.

Attribute routing headers

A data plane on 0.7.0 or later with attribute-based routing enabled names the routing decision on every response a routing policy governs, in monitor mode as in enforce. A request no policy governs carries none of these headers.

Header	Direction	Description
X-Tars-Metadata	Request	The attributes the calling application declares, as comma-separated key=value entries: at most 8 entries and 2 KiB in all, keys matching [a-z][a-z0-9_]{0,31}, values matching [A-Za-z0-9._:-]{1,64}. An entry outside the format, an entry past the eighth, or a header over 2 KiB is dropped and counted on router_attribute_routing_header_dropped_total; the request is routed on what remains and is never refused for it. The full grammar is in the routing policy reference
x-tars-attribute-routing-outcome	Response	matched, default_no_attributes, default_no_match, or default_rules_skipped
x-tars-attribute-routing-policy	Response	The identifier of the policy that applied
x-tars-attribute-routing-rule	Response	The identifier of the rule that matched. Present on matched only

The headers say what routing decided, not what served: a fallback or a traffic split can still move the request afterwards, and x-model-effective names the model that answered. The request log detail shows the same policy and rule, labelled Routing rule or Routing default.

Sending a correlation ID

PROXY_URL stands for the proxy endpoint from the Console Dashboard, scheme included and ending in /v1.

curl -i PROXY_URL/chat/completions \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -H "X-Request-ID: my-session-abc-123" \
  -d '{
    "model": "gpt-4o",
    "messages": [{"role": "user", "content": "Hello"}]
  }'

Response headers:

HTTP/2 200
x-request-id: 8f14e45f-ceea-467f-a8f0-6b2a1c3d4e5f
x-client-request-id: my-session-abc-123
content-type: application/json

The x-request-id value (8f14e45f-...) is the gateway's canonical identifier. The x-client-request-id (my-session-abc-123) is the original client ID, preserved for joining gateway records with application-side logs.

Data captured per request

Every request processed by the gateway produces a record containing:

Field	Description
Request ID	The gateway-assigned X-Request-ID UUID
Client Request ID	The echoed X-Client-Request-ID, if provided by the client
Timestamp	When the request was received by the gateway
Model	The model identifier specified in the request
Provider	The upstream provider that served the response
HTTP Status	The final HTTP status code returned to the client
Latency	Total round-trip time from gateway receipt to completed response
Token usage	Prompt tokens, completion tokens, and total tokens
Request / response body	Full payloads, subject to the deployment's data retention settings

This data is searchable in Request Logs. The OpenTelemetry span for the request carries a subset of it, listed under Span attributes.

Correlating in observability tools

• Request Logs. Search by X-Request-ID (gateway-assigned) or X-Client-Request-ID (the application's ID) to retrieve the complete request record: provider used, latency breakdown, token counts, and raw request and response payloads.
• OTel traces. X-Request-ID is emitted as the span attribute request.id. X-Client-Request-ID is not a span attribute. Search for request.id in the tracing backend (Jaeger, Grafana Tempo, Honeycomb, Datadog, and others) to find the span for the request.

Each request produces one span, even when a fallback chain tries several backends. Fallback attempts do not create child spans or span events. To see which backends were tried, the request is opened in Request Logs, where the upstream backend field and the fallback indicators record the attempt chain.

note

Only headers listed in the data plane's span header mapping become span attributes. The shipped mapping copies x-request-id to request.id and the x-tars-* identity headers to tars.* attributes, on every endpoint span except image generation, which carries no mapped header attributes. The full list is under Header-mapped attributes. Other application headers (user-id, workflow-run-id, and similar) are not copied onto spans, but every inbound header is kept in the Request Logs record.

The same applies to agent-session-id: spans carry no session.id attribute. The session ID is available as session.id on access logs and as the raw agent-session-id header in Request Logs.

Debugging workflow

A typical end-to-end debugging workflow using correlation IDs:

1. Capture the response header. Log the x-request-id value from every response in the application. For error responses, also capture the response body.
2. Search Request Logs. Paste the x-request-id value into the search field in Request Logs to retrieve the full request record: which provider was used, total latency, token counts, fallback attempts, and raw payloads.
3. Inspect the trace. If OTel export is configured, search for the same x-request-id as the span attribute request.id in the tracing backend to see the end-to-end timing of the request. The span does not show fallback attempts; those are read from the Request Logs record in the previous step.
4. Join with application logs. If a custom X-Request-ID was sent, use the echoed x-client-request-id in the gateway record to join against application logs and reconstruct the full request context.

---

Related

• Gateway APIs: inference wire formats and endpoint reference
• Configuration propagation: how long a configuration change takes to reach the data plane
• Console messages and states: what a banner, badge, or empty state in the Console means
• OTel metrics: metric names, types, and labels exported by the gateway
• Audit log events: event schema for administrative actions

---

Where to go next

Gateway APIs

Inference wire formats and endpoint reference.

Monitor traffic and usage

Find requests in Request Logs by correlation ID.