OpenTelemetry traces and metrics
The gateway emits two complementary streams of observability data. Trace data flows over OTLP to any OpenTelemetry-compatible backend; metrics are exposed by each data plane component on a Prometheus-compatible scrape endpoint, where they can be pulled by an existing metrics agent and forwarded as OTLP metrics by a collector if the destination requires it. This page documents the structure of both streams: the span model used for traces, the attributes and events actually stamped on those spans, and the metric families exposed by the gateway. For the configuration mechanics of trace export and the supported authentication modes, see Export Telemetry to an Observability Stack.
Every span name, attribute, event, and metric instrument on this page corresponds to a specific construct in the shipped data plane, based on Envoy AI Gateway v0.7.0 with Tetrate patches applied. Instrumentation changes between data plane releases. Where a dashboard or alert rule depends on an exact string, confirm it against the running deployment: spans through a test request to the configured backend, metrics through the scrape endpoint described under Metric names on the Prometheus endpoint.
Trace structure
Spans are emitted by a single component: the ai-gateway extproc, the external-processing service that Envoy calls for each AI request. Each request produces one flat span. There is no nesting.
Three consequences follow, and each matters when dashboards or alert rules are being authored against this data:
- There are no gateway, routing, or inference child spans. The extproc span covers the request end to end. A fallback walk across several backends does not add spans, and the individual attempts are not separately visible in the trace stream.
- There are no Envoy proxy spans. The
EnvoyProxyresource rendered by the data plane does not configure a tracing provider, so the Envoy hop is not instrumented. The extproc span is the whole of the gateway's contribution to the trace. - Client trace context is honoured. Incoming request headers are run through the configured propagator (W3C
traceparentby default) before the span is started, so the extproc span attaches to the caller's trace as a child rather than starting a new one. Where a client is instrumented, the gateway span appears inside the application's existing trace.
The span carries the OTel span kind Internal for all inference endpoints, and Client for MCP spans.
Span names
The span name identifies the endpoint that served the request. Names are fixed strings supplied by the per-endpoint recorder, not derived from the request path:
| Span name | Endpoint |
|---|---|
ChatCompletion | Chat completions |
Completion | Legacy text completions |
Responses | Responses API |
CreateEmbeddings | Embeddings |
ImagesResponse | Image generation |
AudioSpeech | Speech synthesis |
Transcription | Audio transcription |
Translation | Audio translation |
Rerank | Rerank (Cohere schema) |
Message | Anthropic Messages, including token-counting requests, which share the Messages recorder |
MCP traffic is named from the JSON-RPC method rather than the endpoint: Initialize, ListTools, CallTool, ListPrompts, GetPrompt, ListResources, ReadResource, Subscribe, Unsubscribe, ListResourceTemplates, SetLoggingLevel, Complete, and Ping. Any method without a mapping is used verbatim as the span name.
Resource attributes
Resource attributes are set from the environment by the data plane chart and are identical on every span:
| Attribute | Value |
|---|---|
service.name | ai-gateway-extproc |
service.layer | ENVOY_AI_GATEWAY |
job_name | envoy-ai-gateway |
Span attributes
Attributes follow the OpenInference semantic conventions, the open standard for large-language-model trace attributes. This is what allows an LLM-aware backend, such as Arize Phoenix or any other OpenInference-compatible viewer, to render the span as a model invocation rather than a generic HTTP span.
Identity and parameters:
| Attribute | Type | Description |
|---|---|---|
openinference.span.kind | string | LLM for inference endpoints, EMBEDDING for embeddings, RERANKER for rerank |
llm.system | string | openai, anthropic, or cohere, reflecting the request schema. Not set on embedding spans, where the convention excludes it |
llm.model_name | string | The model name. Set from the request at span start, then overwritten from the response model where the provider returns one |
llm.invocation_parameters | string | JSON of the request parameters excluding messages and tools, which have their own attributes |
embedding.model_name | string | The embeddings equivalent of llm.model_name, on CreateEmbeddings spans |
embedding.invocation_parameters | string | JSON of model, encoding_format, dimensions, and user, on CreateEmbeddings spans |
Input and output payloads:
| Attribute | Type | Description |
|---|---|---|
input.value | string | The raw request body as JSON, or __REDACTED__ when input capture is disabled |
input.mime_type | string | application/json, set alongside a captured input.value. On Responses spans it is set unconditionally |
output.value | string | The serialised response body, or __REDACTED__ when output capture is disabled |
output.mime_type | string | application/json. Omitted when output capture is disabled |
output.audio_duration | double | Audio duration reported by a transcription response, where present |
output.language | string | Language detected by a transcription response, where present |
Token counts. These are treated as metadata and are still recorded when prompt and response capture is disabled:
| Attribute | Type | Description |
|---|---|---|
llm.token_count.prompt | integer | Prompt (input) tokens |
llm.token_count.completion | integer | Completion (output) tokens |
llm.token_count.total | integer | Combined prompt and completion tokens |
llm.token_count.prompt_details.cache_read | integer | Prompt tokens served from the provider's prompt cache |
llm.token_count.prompt_details.cache_creation | integer | Prompt tokens written to the provider's prompt cache |
llm.token_count.prompt_details.audio | integer | Audio tokens in the prompt |
llm.token_count.completion_details.reasoning | integer | Tokens spent on reasoning or chain-of-thought |
llm.token_count.completion_details.audio | integer | Audio tokens in the completion |
Each count is recorded only when the provider reports a value greater than zero, so absence of an attribute indicates absence of usage rather than a gap in instrumentation.
Message content. These are indexed families rather than single keys, and they are present only when the active logging mode permits prompt and response capture. See Configuring Request Logs for the controls:
| Attribute family | Description |
|---|---|
llm.input_messages.{i}.message.role, .message.content | Role and content per input message |
llm.input_messages.{i}.message.contents.{j}.message_content.{text,type,image.image.url} | Multi-part input content, including images |
llm.output_messages.{i}.message.role, .message.content | Role and content per output message |
llm.output_messages.{i}.message.tool_calls.{j}.tool_call.{id,function.name,function.arguments} | Tool calls returned by the model |
llm.tools.{i}.tool.json_schema | JSON schema of each tool offered in the request |
llm.prompts.{i}.prompt.text, llm.choices.{i}.completion.text | Prompt and choice text on legacy Completion spans |
embedding.embeddings.{i}.embedding.text, .embedding.vector | Embedding input text and output vector on CreateEmbeddings spans |
Because indexed message attributes scale with conversation length, the default OTel attribute count limit of 128 is lifted when message capture is enabled, and retained otherwise.
MCP spans carry their own set instead, including mcp.protocol.version, mcp.transport, mcp.request.id, mcp.method.name, and, according to the method, mcp.tool.name, mcp.prompt.name, mcp.resource.uri, mcp.session.id, and mcp.client.{name,title,version}.
Header-mapped attributes
The data plane configures a header-to-attribute mapping on the extproc. Each listed request header, when present, is copied onto the span under the mapped attribute name:
| Attribute | Source header | Description |
|---|---|---|
request.id | x-request-id | The per-request correlation ID |
tars.user | x-tars-user | The identity that issued the request |
tars.customer | x-tars-customer | The owning customer |
tars.workspace | x-tars-workspace | The owning workspace |
tars.router | x-tars-router | The router that served the request |
Two further internal mappings (tars.signature and bellhop.user.id) are configured alongside these and are not intended as a query surface.
This mapping is applied to every endpoint span except ImagesResponse, whose tracer is constructed without a header mapping. Image-generation spans therefore carry no request.id and no tars.* attributes, which is worth accounting for in any dashboard that groups by workspace or customer.
Span events
The event set is small: lifecycle markers on streaming responses, the standard OTel error event, and one event specific to MCP.
| Event | Recorded when |
|---|---|
First Token Stream Event | The first chunk of a streaming response is observed. Emitted on ChatCompletion, Completion, Responses, and Message spans |
Response Completed Event | A streaming Responses API request reaches its completion event. Emitted on Responses spans only |
exception | The upstream returned a non-success status. Carries exception.type (BadRequestError, AuthenticationError, PermissionDeniedError, NotFoundError, RateLimitError, InternalServerError, or Error, mapped from the status code) and exception.message. The span status is also set to error |
route to backend | An MCP request is routed to a backend. Carries mcp.backend.name and mcp.session.id. Emitted on MCP spans only |
There is no first-byte, request-received, routing-resolved, or backend-attempted event. Non-streaming requests emit no lifecycle event at all: the span's start and end timestamps are the only timing signal.
Fields not present on spans
The following are absent from the trace stream and cannot be queried, alerted on, or charted from span data. Several are available elsewhere, as noted:
| Expected field | Reality |
|---|---|
gateway.* (request_id, api_key_id, requested_model, resolved_model, resolved_provider, fallback_attempts) | No gateway.* namespace is emitted. Correlation is available as request.id; the model appears as llm.model_name |
llm.usage.* | Not emitted. Token counts use the llm.token_count.* keys above |
llm.provider | Not emitted. The OpenInference embedding spec excludes it, and no recorder sets it. The nearest equivalent is llm.system, which names the request schema rather than the resolved upstream |
gateway.latency_ms, gateway.time_to_first_token_ms | Not emitted as attributes. Latency is derivable from span duration, and both are available as metrics (gen_ai.server.request.duration, gen_ai.server.time_to_first_token) |
gen_ai.* | These are metric attributes only. No gen_ai.* attribute is set on any span |
http.method, http.target, http.status_code | Not set by the extproc recorders. The endpoint is identifiable from the span name, and failures from the exception event |
| Fallback attempt detail | Not represented in the trace stream. Fallback behaviour is observable through metrics and Request Logs |
For the export configuration that delivers these spans to a backend, see Export Telemetry to an Observability Stack.
Metric families
Metrics are registered by the ai-gateway extproc, the same component that emits spans. Two delivery paths exist and are independent of each other:
- Prometheus scrape. The extproc admin listener serves
/metricsin Prometheus exposition format, on port 1064 by default. The Prometheus reader is always registered, irrespective of the configured OTLP metrics exporter, so this endpoint is available whenever the extproc is running. - OTLP push. When metric export is enabled, the same instruments are also pushed over OTLP to the configured collector.
Envoy's own connection and HTTP statistics are a separate stream, published by the proxy's Prometheus stats sink and documented by Envoy rather than here.
The instrument set is small and follows the OpenTelemetry semantic conventions for generative AI metrics. Every instrument is a histogram or a counter; no gauges are registered, so there is no in-flight or active-request metric to chart.
Inference metrics
Four instruments cover all inference traffic. All four are histograms:
| Metric | Unit | Records |
|---|---|---|
gen_ai.client.token.usage | token | Token counts, one observation per token category. Partitioned by gen_ai.token.type |
gen_ai.server.request.duration | s | Total request duration, measured from receipt of request headers in the extproc to the end of response-body processing. Recorded for every request, successful or not |
gen_ai.server.time_to_first_token | s | Time from request-header receipt to the first token of the response. Streaming responses only |
gen_ai.server.time_per_output_token | s | Mean time per output token after the first, calculated as (request_duration - time_to_first_token) / (output_tokens - 1). Recorded once at end of stream, and only when more than one output token was produced |
Because gen_ai.server.request.duration spans the extproc's whole involvement, it does not separate gateway-internal processing from time spent waiting on the upstream provider. There is no backend-latency instrument, so that split is not available from the metric stream.
gen_ai.client.token.usage records a separate observation per token category rather than a single total. Summing across all values of gen_ai.token.type double counts, because cached and cache-creation input tokens overlap with the input total as the provider reports them:
gen_ai.token.type | Meaning |
|---|---|
input | Prompt tokens |
output | Completion tokens |
cached_input | Prompt tokens served from the provider's cache |
cache_creation_input | Prompt tokens written to the provider's cache |
reasoning | Tokens spent on reasoning |
The cached_input and cache_creation_input values are not part of the upstream specification yet; they are permitted as custom values and are used here pending standardisation.
MCP metrics
MCP traffic is instrumented separately:
| Metric | Type | Records |
|---|---|---|
mcp.request.duration | Histogram | Duration of an MCP request. Carries error.type on the error path |
mcp.initialization.duration | Histogram | Duration of MCP session initialisation |
mcp.method.count | Counter | MCP method invocations. Carries mcp.method.name and status (success, failed, or error) |
mcp.capabilities.negotiated | Counter | Capabilities agreed during initialisation. Carries capability.type (tools, resources, prompts, sampling, roots, experimental, elicitation, completions, logging) and capability.side (client or server) |
mcp.progress.notifications | Counter | Progress notifications relayed |
MCP instruments additionally carry mcp.backend, identifying the upstream server that handled the request.
mcp.initialization.duration is registered with the unit token despite measuring seconds. The value is a duration; only the declared unit is wrong. This matters because the Prometheus exporter appends the unit to the metric name, so the scraped series is named for tokens rather than seconds. Confirm the rendered name against a live scrape before writing a query against it.
Metric names on the Prometheus endpoint
The names above are the OTLP instrument names. The Prometheus exporter rewrites them on the way out: dots become underscores, the unit is appended as a suffix, and histograms expand into _bucket, _sum, and _count series. gen_ai.server.request.duration is therefore not the string a PromQL query uses.
Rather than reproduce the mapping here, where it would drift, query the live endpoint for the exact series names. The extproc runs as a sidecar container in the Envoy proxy pods, so the admin listener is reached from inside one of those pods:
kubectl port-forward -n NAMESPACE POD_NAME 1064:1064
curl -s localhost:1064/metrics | grep -E '^# (HELP|TYPE)'
Substitute the namespace and pod of the gateway being inspected; the proxy namespace is set at install time and is not fixed. The # TYPE lines give the exposed series name and instrument type for each metric.
Metric attributes
Every inference metric carries the same five base attributes, plus the header-mapped set below. Two further attributes are conditional.
| Attribute | Cardinality | Description |
|---|---|---|
gen_ai.operation.name | Low | The endpoint that served the request: chat, completion, embeddings, messages, image_generation, responses, speech, transcription, translation, or rerank |
gen_ai.provider.name | Low | The backend's API schema: openai, azure.openai, aws.bedrock, aws.anthropic, gcp.vertex_ai, gcp.anthropic, anthropic, or cohere. Any other schema falls back to the configured backend name |
gen_ai.original.model | Medium | The model named in the incoming request body, before any virtualisation is applied |
gen_ai.request.model | Medium | The model sent upstream after resolution |
gen_ai.response.model | Medium | The model the provider reports as having generated the response, which is often a dated build of the requested model |
gen_ai.token.type | Low | Token category. Present on gen_ai.client.token.usage only |
error.type | Low | Present on gen_ai.server.request.duration only, and only for failed requests. The value is always the _OTHER placeholder; the gateway does not yet classify error types on this metric |
The three model attributes are the requested-versus-resolved distinction: gen_ai.original.model is what the caller asked for, gen_ai.request.model is what routing selected, and gen_ai.response.model is what actually answered. Any of the three reports unknown when the value could not be determined, which is the expected reading for requests that failed before model resolution.
Because error.type is the only failure dimension and it carries a single placeholder value, the metric stream distinguishes failed from successful requests but not one failure mode from another. Error classification comes from Request Logs, or from the exception.type attribute on the corresponding span.
Header-mapped attributes
As with spans, request headers are copied onto metrics under mapped attribute names. The metric mapping is not the same as the span mapping, which is worth attention when building queries that span both:
| Attribute | Source header | On spans | On metrics |
|---|---|---|---|
tars.user | x-tars-user | Yes | Yes |
tars.customer | x-tars-customer | Yes | Yes |
tars.workspace | x-tars-workspace | Yes | Yes |
tars.router | x-tars-router | Yes | Yes |
router.project.id | x-router-project-id | No | Yes |
router.gateway.id | x-router-gateway-id | No | Yes |
request.id | x-request-id | Yes | No |
Two consequences follow:
- There is no per-request join between metrics and traces.
request.idis a span attribute only, by design: it would be an unbounded label on a metric. Per-request investigation belongs to the trace stream and Request Logs. - Chargeback and per-tenant reporting are available from metrics, at customer, workspace, project, and router granularity. There is no API key dimension on any metric, so per-key attribution is not obtainable from the metric stream; Request Logs and the in-Console Usage Analytics surface are the sources for that.
Sampling
The data plane does not set a sampler, so the OpenTelemetry SDK default applies: parentbased_always_on. The consequences differ according to whether the caller is instrumented, and the second case is a common source of confusion:
- Unparented requests, where the client sends no
traceparentheader, are always sampled. Every such request produces a span. - Parented requests, where the client does send trace context, inherit the caller's sampling decision. If the calling application samples at 1 %, then 99 % of its gateway requests arrive marked as not sampled, and the gateway records no span for them.
The second case means trace volume is partly outside the gateway's control. An instrumented application that samples aggressively will appear to be missing from the trace stream, and no gateway-side setting recovers those spans: the decision was made upstream and propagated. Where a service's gateway traffic is expected in the trace stream but absent, the caller's own sampling configuration is the first thing to check.
Metrics are not sampled. Every request is recorded in every applicable instrument regardless of the trace sampling decision, so the metric stream is a complete record where the trace stream may not be.
This shapes which stream answers which question:
| Question | Stream |
|---|---|
| What did this specific request send and receive? | Traces, or Request Logs |
| Why did this request fail, and with what error type? | Traces (exception event), or Request Logs |
| What is the p95 latency for this model? | Metrics |
| How many tokens did this workspace consume? | Metrics |
| Did every request get counted for billing? | Metrics or Request Logs, never traces |
Because the gateway contributes a single flat span, the trace stream does not break latency down by stage and does not show which backends a fallback chain attempted. Neither is available from metrics either; fallback behaviour is visible only in Request Logs.
Related
- Gateway Behavior: correlation IDs and per-request data
- Audit Log Events: event schema for administrative actions
- Export Telemetry to an Observability Stack: configuration walkthrough for trace and metric export
Where to go next