Key concepts
A handful of terms show up again and again across these docs. Some are common to any AI gateway; others are specific to how Tetrate Agent Router is built and run.
Read it end to end the first time through. When you just need a quick definition later, the Glossary in the Reference section gives the one-line version.
Structural concepts
These are the building blocks: the components of Agent Router itself and how they're deployed.
Data plane and management plane
Agent Router splits into two halves that talk over a small, well-defined interface.
The data plane handles the request path. It's a customer-managed Kubernetes deployment containing the Controller and a proxy component, and every AI request and response flows through it. That's the key point: your traffic, payloads, prompts, completions, and credentials never leave your network. The data plane receives application requests, evaluates routing policy, attaches the right credentials, calls the upstream AI provider, and returns the response.
The management plane is the Tetrate-hosted control surface. It stores routing rules, policies, model and provider definitions, user records, API key metadata, audit history, and analytics rollups. The data plane pulls its configuration from here and pushes telemetry back, but no request payloads ever cross the boundary. You keep full data sovereignty without having to operate the control-plane stack yourself.
The Developer Console and the Admin Dashboard
Two web applications front Agent Router, each scoped to a different audience.
The Developer Console is the developer-facing application. It's where you issue API keys, author routing policies, assemble MCP profiles, configure integrations, and test prompts in the Playground. Developers spend most of their time here.
The Admin Dashboard is the platform-operator application. It's where you provision models and providers, manage users, set budgets and rate limits, review audit logs, and configure SSO. Most of Agent Router's governance happens here.
The two applications share the same underlying system. They're two views of the same data, each scoped to the responsibilities of its audience.
Projects
A project is the logical boundary that owns configuration and access in Agent Router. Everything a team works with day to day belongs to a project: the models and providers granted to it, its API keys, its MCP servers and profiles, its routing and policy, its budgets, and its members. It is also the unit of attribution, so usage and audit records are tied to the project the traffic ran under. If you have used the concept of a workspace or a tenant in other tools, a project plays a similar role here: it is the container that keeps one team's or one application's setup separate from the next.
Projects, gateways, and data planes
The project is one of three objects that work together, and the clearest way to understand a project is to see how it relates to the other two. They divide into a logical layer and a physical layer:
- A project is logical. It is the configuration and access boundary described above. It decides what should be possible (which models, which members, which policy), but it runs no software of its own.
- A gateway is physical. It is a single inference endpoint with one customer-facing URL: the thing an application actually calls. Each gateway serves exactly one project.
- A data plane is physical too. It is the layer, deployed in a Kubernetes cluster you control, that provisions and manages gateways. It is the capacity a gateway runs on.
These three connect through a small set of fixed relationships:
| Objects | How they relate |
|---|---|
| Project and gateway | One-to-one. A project maps to exactly one gateway, and a gateway serves exactly one project. |
| Data plane and gateways | One-to-many. A single data plane provisions and manages many gateways, one for each project placed on it. |
| Management plane and data planes | One-to-many. The single management plane manages many data planes, which can sit in different clouds and regions. |
Read together, the picture is straightforward: the one management plane holds every project's configuration, each data plane is a pool of capacity in some location, and each gateway is one project's live endpoint running on one of those data planes. A project comes to life by requesting a gateway from a data plane. The data plane provisions the runtime, the management plane maps the project to that gateway and publishes its desired configuration, and once the gateway reports ready its URL appears on the project. Until then, a project is just configuration with nowhere to send traffic.
Keeping the logical project separate from the physical gateway and data plane is deliberate, and it is what makes the model flexible. Because a project's identity (its URL, its keys, its configuration) is decoupled from the capacity that serves it, a gateway's runtime can be upgraded in place or moved to a different data plane without the project's configuration or its URL changing, and without dropping in-flight requests. The same decoupling lets you place projects wherever they need to run: several projects can share one data plane, each with its own gateway, or be spread across data planes by region or cloud (for example, one project served from an AWS data plane and another from an Azure data plane).
Why projects matter
The project is the isolation boundary, and that isolation is enforced in two places. In the applications, every view is scoped to the selected project: it lists only the models, MCP servers, MCP profiles, and request logs that belong to that project, and switching projects with the project switcher re-scopes the whole app. At the gateway, cross-project requests are refused at request time. A key issued for one project does not work on another project's gateway (it is rejected with a 403), and a model a project has not been granted cannot be called through it (a 404). This runtime enforcement is what lets several teams or applications share one platform, or even one data plane, while staying fully separated from one another.
That boundary is why projects are the organising idea for running Agent Router at scale. They give you a hard line for security and access, clean per-team attribution for cost and audit, and the freedom to move or grow the underlying infrastructure without disturbing the applications that depend on it. A default project is created for each customer during onboarding, and administrators can add more afterwards: one per team, application, or environment (for example, separate production and staging projects). The operator tasks for creating projects, provisioning their gateways, and placing them on data planes are collected under Project and data plane management, and the fleet-level arrangement of data planes is described in Operating multiple data planes.
Routing concepts
Routing is the gateway's central job: for each incoming AI request, deciding which backend should serve it. A few concepts describe how that decision gets made.
Backends, models, and providers
A provider is an upstream AI service the gateway can talk to: OpenAI, Anthropic, Google, Azure OpenAI, Mistral, and many others, plus self-hosted endpoints and customer-specific deployments. A model is a specific offering from a provider, such as gpt-4o or claude-sonnet-4-6. A backend, in routing terms, is a model-on-a-provider combination paired with the credentials needed to reach it. Every routing decision ultimately resolves to a choice of backend.
Routing chains and policies
Each API key carries a routing chain: an ordered or weighted list of backends the gateway considers when a request arrives. The chain encodes the consumer's preferences, fallbacks, and constraints, and one or more routing policies attached to the key shape how it behaves.
Fallback policy
A fallback policy is an ordered list of backends. When a request arrives, the gateway tries the first backend in the chain; if that call fails with a recoverable error (a 5xx, a rate-limit response, or a connection timeout), it moves to the next backend and tries again, continuing until one succeeds or the chain runs out. The fallback decision happens inside the gateway, so your application sees a single request that either succeeds or fails. It never has to implement retry logic of its own.
Fallback policies are how you manage provider risk: a primary commercial model with one or more secondary models behind it, or a commercial model with a self-hosted open-weights model as a last-resort backstop.
Traffic splitting
A traffic splitting policy distributes requests across two or more backends by percentage rather than by priority. A 70/30 split, for example, sends 70 % of requests to the first backend and 30 % to the second. Use it to evaluate one model against another under real production traffic, to manage cost by routing a portion of traffic to a cheaper backend, or to roll out a new provider gradually.
Splitting and fallback combine: a split picks the primary backend for a given request, and a fallback chain takes over if that backend fails.
Advanced routing rules
When neither weighted splitting nor priority-ordered fallback is enough, advanced routing rules dispatch requests based on attributes of the request itself: the model the caller asked for, custom headers, request metadata, or other signals. Use them to encode policies such as "route requests from this tenant to this provider" or "send anything tagged code-completion to a model tuned for that task".
Credentials and identity
Identity in Agent Router is layered. Applications authenticate to the gateway with API keys; human operators authenticate to the Console and Admin Dashboard with single sign-on; and the gateway authenticates to upstream providers using credentials that are either centrally managed or supplied by the consumer.
API keys
An API key is the credential an application presents when it calls the gateway. Each key ties to a consumer (a person or a service), a routing chain, and any policies attached to that chain. Keys are issued from the Admin Dashboard or claimed by developers in the Console, depending on your onboarding model. Revocation, rotation, and per-key budgets are all managed centrally.
Bring your own key (BYOK)
Bring Your Own Key, or BYOK, is the model under which a consumer's own upstream provider credentials are used in place of centrally managed ones. A team that holds its own OpenAI account, for example, can register those credentials with Agent Router and have its requests routed through them rather than through the centrally pooled credentials.
BYOK is a first-class part of the routing model: BYOK and centrally managed backends can sit in the same routing chain, and policy, not application code, decides between them. Reach for BYOK when billing, compliance, or contractual relationships require a specific team's traffic to land on a specific provider account.
Single sign-on and role mapping
Agent Router integrates with corporate identity providers through single sign-on (SSO) over OIDC. Microsoft Entra ID, Okta, Auth0, Keycloak, and any OIDC-compliant provider can handle authentication for the Console and the Admin Dashboard. Once SSO is configured, role assignment is driven from the identity provider itself: app role or group claims in the OIDC token map to Agent Router roles on every login, so directory changes propagate without manual work in Agent Router. The Configuring SSO: Mapping Roles and Groups guide covers the mapping mechanics in detail.
Agent infrastructure
Beyond model routing, Agent Router gives you a control surface for the agent ecosystem, particularly the Model Context Protocol servers that AI clients consume.
Model context protocol (MCP)
The Model Context Protocol (MCP) is an emerging standard for exposing tools, data sources, and context to AI clients. MCP servers act as adapters: one wraps a file system, another a ticketing API, another a documentation source, and so on. AI clients such as Claude Code, Cursor, and VS Code consume those servers to extend their own capabilities.
MCP profile
An MCP profile is a single endpoint that aggregates multiple MCP servers behind one interface. Instead of configuring each AI client with a separate connection to every server it needs, you assemble a profile centrally, govern it with Agent Router's identity and access model, and expose it as one URL that clients connect to once. Profiles get rid of the per-agent fan-out that becomes unmanageable as the MCP ecosystem grows, and they bring MCP traffic under the same audit, access-control, and observability regime as standard model traffic.
Extensibility and observability
Two more concepts shape how Agent Router plugs into your surrounding software estate: the extension model inside the gateway, and the telemetry interface that exports operational data to external systems.
Dynamic modules
The gateway is built on a proxy component, and Agent Router–specific behaviour is delivered through dynamic modules: high-performance proxy extensions, written in Rust or Go, compiled to shared libraries that run inline in the filter chain. Dynamic modules implement the routing logic, credential handling, and provider-specific request translation that set Agent Router apart from a generic HTTP gateway. You don't configure them directly, but they're why the gateway behaves consistently across providers and why request-path overhead stays low.
OpenTelemetry export
Agent Router speaks OpenTelemetry (OTEL), the open observability standard. Request-level metrics, traces, and logs are exported in OTEL format to whatever observability stack you already run: Grafana, Datadog, New Relic, Honeycomb, or any other OTEL-compatible backend. The platform operator owns the export configuration and applies it centrally, so individual applications don't need separate instrumentation to show up in your existing dashboards.
Audit logging
Alongside operational telemetry, Agent Router keeps an audit log of administrative actions: model and provider changes, user and role assignments, SSO configuration changes, API key issuance and revocation, and other state-modifying events in the Admin Dashboard. You can query the audit log from the Admin Dashboard, and it's the primary record reviewed during compliance investigations or post-incident analysis. The audit event schema is documented in the Reference section.
Where to go next
Architecture overview
How these concepts assemble into the running system, with diagrams of the request path between application, data plane, management plane, and upstream provider.
Solution capabilities
Structured outputs, multimodal inputs, and function calling, passed through to any provider or translated across providers where mappings exist.
Glossary
The same vocabulary in one-line form for quick lookup.