Gateway APIs

Agent Router supports three inference API formats: the OpenAI Chat Completions API, the OpenAI Responses API, and the Anthropic Messages API, plus an OpenAI-compatible Models endpoint for discovering which models are routable for a given API key. Gateway features (routing, fallback policies, traffic splitting, cost tracking, and observability) apply equally across the three formats. The formats differ in one respect that affects backend choice rather than features: Chat Completions and Messages translate to every supported provider, while the Responses API reaches OpenAI and Azure OpenAI backends only, as set out under Provider support for the Responses API. Applications send requests in one format, and the gateway handles provider translation transparently, normalizing responses and errors back to the format that was requested. For new projects with no existing SDK preference, Chat Completions offers the widest ecosystem compatibility.

Management APIs live elsewhere

These are the gateway inference endpoints that applications and SDKs hit. The OpenAPI catalog does not enumerate /v1/chat/completions and the other paths on this page. For provisioning keys, clients, models, and other control-plane operations, use the Management API reference.

Endpoint support

The table below is the full list of gateway inference paths applications call. Supported endpoints share the same model-based routing, fallback policies, and traffic splitting. A path the gateway does not route returns 404 with Unsupported endpoint.

The Streaming column gives the request field that opts a call into server-sent events. Where it reads "Not streamed", the endpoint returns a single complete response whatever the request asks for, and a streaming flag on it has no effect. Embeddings, images, and rerank are accepted in passthrough mode only; see Known limitations.

Endpoint	Path	Streaming	Routable backends	Status
Chat Completions	`/v1/chat/completions`	`stream: true`	All providers	Supported
Completions	`/v1/completions`	`stream: true`	All providers	Supported
Responses	`/v1/responses`	`stream: true`	OpenAI and Azure OpenAI only	Supported
Messages	`/v1/messages`	`stream: true`	All providers	Supported
Embeddings	`/v1/embeddings`	Not streamed	Passthrough	Supported
Images	`/v1/images/generations`	Not streamed	Passthrough	Supported
Audio speech	`/v1/audio/speech`	`stream_format: "sse"`	OpenAI-compatible TTS backends	Supported
Rerank	`/v1/rerank`	Not streamed	Passthrough	Supported
Models	`/v1/models`	Not streamed	n/a	Supported
Audio transcriptions	`/v1/audio/transcriptions`	n/a	n/a	Unsupported

Audio speech uses a different field

Text-to-speech is the one streaming endpoint that is not opted in with stream. It streams when stream_format is set to sse, and stream: true on /v1/audio/speech is ignored.

Speech-to-text is not supported

POST /v1/audio/transcriptions returns 404 with Unsupported endpoint. Speech-to-text (Whisper-style) models cannot be called through the gateway. Under /v1/audio/*, only /v1/audio/speech is listed as supported above.

In the examples below, replace PROXY_URL with the proxy endpoint shown on the Console Dashboard (for example, https://proxy.poc.tetrate.ai/v1) and YOUR_API_KEY with a key from API Keys.

This page covers the wire formats: paths, request shapes, and the SSE event shape each format emits. What the gateway does when a stream fails partway through, what a client should treat as a truncated response, and how streaming interacts with guardrails, fallback, and latency are covered in Streaming behavior.

Chat Completions API (`/v1/chat/completions`)

The most widely supported format, compatible with OpenAI and most third-party SDKs.

Non-streaming

curl PROXY_URL/chat/completions \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o",
    "messages": [{"role": "user", "content": "Hello, world!"}]
  }'

from openai import OpenAI

client = OpenAI(
    base_url="PROXY_URL",
    api_key="YOUR_API_KEY",
)

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello, world!"}],
)

print(response.choices[0].message.content)

Streaming

curl PROXY_URL/chat/completions \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o",
    "messages": [{"role": "user", "content": "Hello, world!"}],
    "stream": true
  }'

from openai import OpenAI

client = OpenAI(
    base_url="PROXY_URL",
    api_key="YOUR_API_KEY",
)

stream = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello, world!"}],
    stream=True,
)

for chunk in stream:
    content = chunk.choices[0].delta.content
    if content:
        print(content, end="", flush=True)

SSE format

Chat Completions streaming uses data-only SSE. Each event is a data: line containing a JSON object, terminated by data: [DONE]:

data: {"id":"chatcmpl-...","object":"chat.completion.chunk","choices":[{"delta":{"content":"Hello"}}]}

data: {"id":"chatcmpl-...","object":"chat.completion.chunk","choices":[{"delta":{"content":" world"}}]}

data: [DONE]

To receive token usage in the stream, add "stream_options": {"include_usage": true} to the request. Usage appears in the final chunk before [DONE]:

{"prompt_tokens": 10, "completion_tokens": 5, "total_tokens": 15}

Responses API (`/v1/responses`)

The newer OpenAI Responses API provides a simplified interface with semantic streaming events.

OpenAI and Azure OpenAI backends only

Unlike Chat Completions and Messages, this format is not translated to every provider. A /v1/responses request is routable to OpenAI and Azure OpenAI backends only, streaming and non-streaming alike. See Provider support for the Responses API.

Non-streaming

curl PROXY_URL/responses \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o",
    "input": "Hello, world!"
  }'

from openai import OpenAI

client = OpenAI(
    base_url="PROXY_URL",
    api_key="YOUR_API_KEY",
)

response = client.responses.create(
    model="gpt-4o",
    input="Hello, world!",
)

print(response.output_text)

Streaming

curl PROXY_URL/responses \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o",
    "input": "Hello, world!",
    "stream": true
  }'

from openai import OpenAI

client = OpenAI(
    base_url="PROXY_URL",
    api_key="YOUR_API_KEY",
)

stream = client.responses.create(
    model="gpt-4o",
    input="Hello, world!",
    stream=True,
)

for event in stream:
    if event.type == "response.output_text.delta":
        print(event.delta, end="", flush=True)

SSE format

Responses API streaming uses semantic event: plus data: lines. Each event has a named type describing what happened:

event: response.created
data: {"id":"resp_...","object":"response","status":"in_progress"}

event: response.output_item.added
data: {"item":{"id":"msg_...","type":"message","role":"assistant"}}

event: response.output_text.delta
data: {"delta":"Hello"}

event: response.output_text.delta
data: {"delta":" world"}

event: response.output_text.done
data: {"text":"Hello world"}

event: response.completed
data: {"id":"resp_...","status":"completed","usage":{"input_tokens":10,"output_tokens":5}}

Differences from Chat Completions

Aspect	Chat Completions	Responses API
Input field	`messages` array	`input` (string or array)
Usage fields	`prompt_tokens` / `completion_tokens`	`input_tokens` / `output_tokens`
SSE format	Data-only (`data: {...}`) with `data: [DONE]` sentinel	Semantic events (`event: response.created`, etc.)
Stream usage	Opt-in via `stream_options.include_usage`	Always in `response.completed` event
Response access	`response.choices[0].message.content`	`response.output_text`
Provider support	Every supported provider	OpenAI and Azure OpenAI only

Provider support for the Responses API

The gateway translates Chat Completions and Anthropic Messages requests to any provider in the catalog. The Responses API is the exception: it is translated to OpenAI and to Azure OpenAI, and to nothing else. A request that resolves to any other backend, an Anthropic, Bedrock, Vertex AI, or Mistral model for example, is rejected rather than translated. The restriction is a property of the format, not of streaming, so it applies to stream: true and to ordinary request-response calls alike.

Two consequences are worth planning around:

Model selection is narrower on this format. GET /v1/models lists every model routable for the key across all formats, so it is not a filter for Responses API compatibility. Confirm that the intended model is served by OpenAI or Azure OpenAI before adopting the format.
Fallback chains must stay inside the same two providers. A chain attached to a key used for /v1/responses traffic fails on any hop that resolves elsewhere, which defeats the purpose of the chain at the moment it is needed. Cross-provider resilience across the wider catalog requires Chat Completions or Messages. See Improve resilience with fallbacks.

Where a single application needs both semantic streaming events and the full provider catalog, Anthropic Messages is the format that offers both.

Anthropic Messages API (`/v1/messages`)

For applications built with the Anthropic SDK. The gateway accepts standard Authorization: Bearer headers; the Anthropic-native x-api-key header is not required.

Non-streaming

curl PROXY_URL/messages \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -H "anthropic-version: 2023-06-01" \
  -d '{
    "model": "claude-sonnet-4-20250514",
    "max_tokens": 1024,
    "messages": [{"role": "user", "content": "Hello, world!"}]
  }'

from anthropic import Anthropic

client = Anthropic(
    base_url="PROXY_URL",
    auth_token="YOUR_API_KEY",
)

response = client.messages.create(
    model="claude-sonnet-4-20250514",
    max_tokens=1024,
    messages=[{"role": "user", "content": "Hello, world!"}],
)

print(response.content[0].text)

Streaming

curl PROXY_URL/messages \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -H "anthropic-version: 2023-06-01" \
  -d '{
    "model": "claude-sonnet-4-20250514",
    "max_tokens": 1024,
    "messages": [{"role": "user", "content": "Hello, world!"}],
    "stream": true
  }'

from anthropic import Anthropic

client = Anthropic(
    base_url="PROXY_URL",
    auth_token="YOUR_API_KEY",
)

with client.messages.stream(
    model="claude-sonnet-4-20250514",
    max_tokens=1024,
    messages=[{"role": "user", "content": "Hello, world!"}],
) as stream:
    for text in stream.text_stream:
        print(text, end="", flush=True)

SSE format

Anthropic Messages streaming uses semantic event: plus data: lines with block-level granularity:

event: message_start
data: {"type":"message_start","message":{"id":"msg_...","role":"assistant","model":"claude-sonnet-4-20250514"}}

event: content_block_start
data: {"type":"content_block_start","index":0,"content_block":{"type":"text","text":""}}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"Hello"}}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":" world"}}

event: content_block_stop
data: {"type":"content_block_stop","index":0}

event: message_delta
data: {"type":"message_delta","delta":{"stop_reason":"end_turn"},"usage":{"output_tokens":5}}

event: message_stop
data: {"type":"message_stop"}

note

When using the gateway, authenticate with Authorization: Bearer YOUR_API_KEY instead of the Anthropic-native x-api-key header. The gateway translates the auth header before forwarding to the provider.

Models API (`/v1/models`)

List the models available to the API key in use. The endpoint is OpenAI-compatible and requires authentication. Use each returned id as the model value in Chat Completions, Responses, Messages, and other inference calls.

The response follows the OpenAI list shape (object: "list" with a data array) and extends each model object with pricing and capability fields used for routing and cost awareness.

Request

curl PROXY_URL/models \
  -H "Authorization: Bearer YOUR_API_KEY"

from openai import OpenAI

client = OpenAI(
    base_url="PROXY_URL",
    api_key="YOUR_API_KEY",
)

models = client.models.list()
for model in models.data:
    print(model.id)

Response

The example below is truncated. A live response returns every model routable for the key.

{
  "object": "list",
  "data": [
    {
      "id": "claude-sonnet-4-6",
      "object": "model",
      "created": 1771393580,
      "owned_by": "system",
      "input_price": "0.000003",
      "caching_price": "0.00000375",
      "cached_price": "0.0000003",
      "output_price": "0.000015",
      "max_output_tokens": 64000,
      "context_window": 1000000,
      "supports_caching": true,
      "supports_vision": true,
      "supports_computer_use": true,
      "supports_reasoning": true
    },
    {
      "id": "gpt-4o-mini",
      "object": "model",
      "created": 1773126109,
      "owned_by": "system",
      "input_price": "0.00000015",
      "caching_price": "0",
      "cached_price": "0.000000075",
      "output_price": "0.0000006",
      "max_output_tokens": 16384,
      "context_window": 128000,
      "supports_caching": false,
      "supports_vision": true,
      "supports_computer_use": true,
      "supports_reasoning": false
    }
  ]
}

note

GET /v1/models returns the models routable for the calling key. For the unauthenticated public catalog with richer metadata, see https://router.tetrate.ai/api/public/models. For management-plane catalog CRUD, use GET /v1/catalog/models (documented in the API reference).

Provider translation

The gateway automatically translates between the canonical (OpenAI-compatible) schema and 25+ provider-specific APIs. Applications send requests in one format, and the gateway handles all conversions transparently.

Translated elements

Request body: field names, structure, and defaults adjusted per provider
Path: endpoint paths mapped to provider conventions
Headers: authentication and provider-specific headers set automatically
Response format: provider responses normalized back to the format that was requested

For example, an OpenAI Chat Completions request that routes to Anthropic Claude is translated to the Anthropic Messages format before being forwarded, and the response is translated back to Chat Completions format. The application never sees the difference.

No configuration is needed; translation is built into the gateway. For how errors are normalized across providers, see Gateway Behavior.

Protocols

The gateway supports REST over HTTPS for all inference traffic. This is the only protocol needed to use any endpoint.

Streaming adds no second protocol. Every streaming endpoint delivers server-sent events (SSE) over the same HTTPS connection the request arrived on, and both HTTP/1.1 and HTTP/2 carry it. No protocol negotiation, upgrade handshake, or client-side configuration is involved: a client that can hold an HTTPS response open can consume a stream, and the HTTP version is whatever the client and the listener agree on.

Protocol	Inference	Telemetry
HTTPS, request and response	Supported	n/a
HTTPS with SSE, on HTTP/1.1 and HTTP/2	Supported, for the streaming endpoints listed above	n/a
gRPC	Not supported	Supported, for OpenTelemetry (OTLP) export
WebSocket	Not supported	Not supported

gRPC carries OTLP telemetry export and is used between data plane components; it is not an inference transport. See Export telemetry to an observability stack for the configuration surface.

Choosing an API format

Use case	Recommended format
Widest SDK and tool compatibility	Chat Completions
New OpenAI projects with the simplified interface	Responses API
Anthropic Claude-native applications	Anthropic Messages
Agent frameworks (LangChain, CrewAI, and similar)	Chat Completions
Code assistants (Cursor, Cline, Aider)	Chat Completions
Streaming with semantic events	Anthropic Messages, or the Responses API on OpenAI and Azure OpenAI backends
Streaming across the full provider catalog	Chat Completions or Anthropic Messages

All three formats support the same gateway features. The choice is driven by SDK preference, provider ecosystem alignment, and the provider restriction on the Responses API.

Integrate the Gateway with an App: developer-side integration patterns for each SDK
Gateway Behavior: request handling, error semantics, and routing resolution rules

Where to go next

Management API reference

Control-plane OpenAPI catalog: clients, keys, catalog, and related services.

Gateway behavior

How requests are handled, normalized, and routed.

Endpoint support​

Chat Completions API (/v1/chat/completions)​

Non-streaming​

Streaming​

SSE format​

Responses API (/v1/responses)​

Non-streaming​

Streaming​

SSE format​

Differences from Chat Completions​

Provider support for the Responses API​

Anthropic Messages API (/v1/messages)​

Non-streaming​

Streaming​

SSE format​

Models API (/v1/models)​

Request​

Response​

Provider translation​

Translated elements​

Protocols​

Choosing an API format​

Related​

Endpoint support

Chat Completions API (`/v1/chat/completions`)

Non-streaming

Streaming

SSE format

Responses API (`/v1/responses`)

Non-streaming

Streaming

SSE format

Differences from Chat Completions

Provider support for the Responses API

Anthropic Messages API (`/v1/messages`)

Non-streaming

Streaming

SSE format

Models API (`/v1/models`)

Request

Response

Provider translation

Translated elements

Protocols

Choosing an API format

Related