Key concepts
A handful of terms show up again and again across these docs. Some are common to any AI gateway; others are specific to how Tetrate Agent Router is built and run. This page is meant to be read end to end once; for a quick definition later, the Glossary in the Reference section gives the one-line version.
Structural concepts
Data plane and management plane
Agent Router splits into two halves that talk over a small, well-defined interface.
The data plane handles the request path. It contains the Controller and a proxy component (the AI gateway), and every AI request and response flows through it. The data plane receives application requests, evaluates routing policy, attaches the right credentials, calls the upstream AI provider, and returns the response. Where it runs depends on the deployment model: in Agent Router Service and Agent Router Enterprise Fully Managed, Tetrate operates it; in Agent Router Enterprise Self-Hosted Data Plane, it is deployed in a customer-controlled Kubernetes cluster, so traffic, payloads, prompts, responses, and credentials stay inside the customer's network. The three paths are compared in Deployment models.
The management plane is the Tetrate-hosted control surface in every deployment model. It stores routing rules, policies, model and provider definitions, user records, API key metadata, audit history, and analytics rollups. The data plane pulls its configuration from here and pushes telemetry back, but no request payloads ever cross the boundary. A centrally managed control surface is available without operating that stack in-house.
The Developer Console and the Admin Console
Two web applications front Agent Router, each scoped to a different audience.
The Developer Console is the developer-facing application. It is where API keys are issued, per-key policies configured, MCP profiles assembled, integrations configured, and prompts tested in the Playground. Developers spend most of their time here. Routing policies are authored in the Admin Console.
The Admin Console is the platform-operator application. It is where models and providers are provisioned, users managed, budgets and rate limits set, audit logs reviewed, and SSO configured. Most of Agent Router's governance happens here.
The two applications are two views of the same underlying data.
Projects
A project is the logical boundary that owns configuration and access in Agent Router. Everything a team works with day to day belongs to a project: the models and providers granted to it, its API keys, its MCP servers and profiles, its routing and policy, its budgets, and its members. It is also the unit of attribution, so usage and audit records are tied to the project the traffic ran under. A project plays the role a workspace or a tenant plays in other tools: it keeps one team's or one application's setup separate from the next.
Projects, data planes, and hostnames
The project is one of three objects that work together, divided into a logical layer and a physical layer:
- A project is logical. It is the configuration and access boundary described above. It decides what should be possible (which models, which members, which policy), but it runs no software and owns no hostname.
- A data plane is physical. It is the request-path deployment an application's traffic reaches, Tetrate-operated in Agent Router Service and Agent Router Enterprise Fully Managed, or deployed in a customer-controlled Kubernetes cluster in Agent Router Enterprise Self-Hosted Data Plane. A platform administrator labels it with each project it serves (a project label), and it serves the configuration of every project labelled on it.
- A hostname is the address an application calls. Since 0.7.0 a data plane lists its hostnames in its own Helm values and reports them to the management plane (a declared hostname). DNS, certificates, and load balancers in front of it stay the operator's.
These three connect through a small set of relationships:
| Objects | How they relate |
|---|---|
| Project and data plane | Many-to-many, through project labels. One data plane serves several projects, and one project can be served by several data planes, for example one per region, or two behind one DNS name for high availability (see Serve a project from two data planes). A project with no label on any data plane is fully configured and callable nowhere. |
| Data plane and hostnames | One-to-many. A data plane declares one or more hostnames, and the first is the address the consoles print. Two data planes can declare the same hostname, which is how a pair behind one DNS name works. |
| Management plane and data planes | One-to-many. The single management plane manages many data planes, which can sit in different clouds and regions. |
Read together: the one management plane holds every project's configuration, each data plane is a pool of capacity in some location that serves the projects labelled on it, and each hostname is a door into one or more data planes. A project comes to life when a platform administrator labels a data plane with it. The data plane pulls the project's configuration on its next update, and the Developer Console then lists the data plane's addresses as the project's endpoints. Until then, a project is configuration with nowhere to send traffic, and the consoles say so rather than printing an address.
Keeping the logical project separate from the physical data plane is deliberate. Because a request resolves to its project from its API key and never from the hostname it arrives on, several projects can share one data plane and one hostname without seeing each other's models, MCP servers, or traffic. The same separation lets a project move or grow: label another data plane with it, in another region or cloud, and the project's keys and configuration work there unchanged.
Before 0.7.0, a project reached a data plane through a gateway: a single customer-facing URL mapped to exactly one project, with each gateway carrying its own hostname. Gateways still exist in the admin API, and a data plane before 0.7.0 still serves through them. See gateway in the glossary.
Why projects matter
The project is the isolation boundary, and that isolation is enforced in two places. In the applications, every view is scoped to the selected project: it lists only the models, MCP servers, MCP profiles, and request logs that belong to that project, and switching projects with the project switcher re-scopes the whole app. At the gateway, cross-project requests are refused at request time. A key issued for one project is rejected with a 403 on a hostname that does not serve its project, and a model a project has not been granted cannot be called with its key (a 404, or 400 model_not_found on a hostname shared with other projects). This runtime enforcement is what lets several teams or applications share one Agent Router deployment, or even one data plane, while staying fully separated from one another.
A default project is created for each customer during onboarding, and administrators can add more afterward: one per team, application, or environment (for example, separate production and staging projects). The operator tasks for creating projects and labelling data planes with them are collected under Project and data plane management, and the fleet-level arrangement of data planes is described in Operating multiple data planes.
Routing concepts
Routing is the gateway's central job: for each incoming AI request, deciding which backend should serve it.
Backends, models, and providers
A provider is an upstream AI service the gateway can talk to: OpenAI, Anthropic, Google, Azure OpenAI, Mistral, and many others, plus self-hosted endpoints and customer-specific deployments. A model is a specific offering from a provider, such as gpt-4o or claude-sonnet-4-6. A backend, in routing terms, is a model-on-a-provider combination paired with the credentials needed to reach it. Every routing decision ultimately resolves to a choice of backend.
Routing chains and policies
Each API key carries a routing chain: an ordered or weighted list of backends the gateway considers when a request arrives. The chain encodes the consumer's preferences, fallbacks, and constraints. The key's own fallback and traffic-splitting policies shape how it behaves, and at most one routing policy applies to a key.
Fallback policy
A fallback policy is an ordered list of backends. When a request arrives, the gateway tries the first backend in the chain; if that call fails with a recoverable error, it moves to the next backend and tries again, continuing until one succeeds or the chain runs out. The recoverable set defaults to provider responses with HTTP status 401, 413, 422, 429, 500, 503, 504, or 529, plus connection failures, gateway errors, other 5xx responses, and deadline exceeded. It is configurable per key under Retry on in the Fallback section, so a key may be narrower than this default. The fallback decision happens inside the gateway, so the calling application sees a single request that either succeeds or fails. It never has to implement retry logic of its own.
Fallback policies are how provider risk is managed: a primary commercial model with one or more secondary models behind it, or a commercial model with a self-hosted open-weights model as a last-resort backstop.
Traffic splitting
A traffic splitting policy distributes requests across two or more backends by percentage rather than by priority. A 70/30 split, for example, sends 70% of requests to the first backend and 30% to the second. A split is used to evaluate one model against another under real production traffic, to manage cost by routing a portion of traffic to a cheaper backend, or to roll out a new provider gradually.
Splitting and fallback combine: a split picks the primary backend for a given request, and a fallback chain takes over if that backend fails.
Advanced routing rules
When neither weighted splitting nor priority-ordered fallback is enough, advanced routing rules dispatch requests based on attributes of the request itself: the model the caller asked for, custom headers, request metadata, or other signals. They encode policies such as "route requests from this tenant to this provider" or "send anything tagged code-completion to a model tuned for that task".
Policy and cost concepts
Routing decides where a request goes. A second set of concepts decides what a request is allowed to do: what it may cost, how fast it may arrive, and what it may contain.
Policies
A policy is an admin-managed rule that decides what happens to a model request. Each one carries a type, which control it is; a scope, which API keys it governs; a configuration belonging to that type; and a mode, monitor or active. Five controls are built on that single shape, each pairing a trigger with an action.
| Type | Fires when | Action |
|---|---|---|
| Budget | a budget is exhausted | block, or degrade to a cheaper model |
| Availability fallback | the primary model fails, on error or timeout | move to the next model in the chain |
| Traffic split | every request | route by weight |
| Rate limit | a token cap would be exceeded | block (HTTP 429) |
| Guardrail | a content or safety check matches | block, or redact |
Fallback policy and traffic splitting, described above, are two of those five. Setting a rule once and having it apply to a whole class of keys, rather than configuring the same control on hundreds of keys by hand, is the reason the model exists. Policies covers the rest of the machinery: the six scopes a policy can take, how two policies covering one key combine, and what monitor mode does.
Budgets and budget policies
A budget is an amount of money for one period, attached to a subject that says whose spend the amount covers: a team, each user on a team, or one API key. The period is daily, weekly, or monthly, chosen when the budget is created and editable afterward. A budget is money, never tokens or request counts.
A budget measures; stopping is opt-in. It tracks spend against its limit and reports on it. Spend is evaluated in the management plane against aggregated request logs rather than inline at the gateway.
Enforcement is the budget's action. By default a budget only watches: it raises an alert and changes nothing about what the caller can do next. Choosing Hard stop or Degrade gracefully (on a single API key) takes the budget's threshold as its trigger and acts on the request path once the threshold is crossed, either blocking the caller or degrading to a cheaper model and continuing to serve. One surface owns the number and the report; the action owns the consequence.
Capping how fast a credential may consume, rather than how much it may spend over the period, is a rate limit instead. See Budgets.
The three stages of cost control
Cost control arrives in three stages, and what separates them is what happens when a number is reached.
| Stage | What it does | Status |
|---|---|---|
| Analytics | Reports what is being spent, by team, by model, by key, and over time. Nothing is limited | Available now |
| Budget monitoring | Tracks spend against a daily, weekly, or monthly limit and raises an alert as the limit is approached or crossed. Requests continue to be served | Available now |
| Enforcement | A budget action acts at the limit, refusing the caller or degrading to a cheaper model | Available now |
Budgets are one of three cost controls, and the narrowest in what they can address. The stages are worked through in Cost and quota management, and Choose the right cost control for each workload covers which instrument fits a given workload.
Credentials and identity
Identity in Agent Router is layered. Applications authenticate to the gateway with API keys; human operators authenticate to the Console and Admin Console with single sign-on; and the gateway authenticates to upstream providers using credentials that are either centrally managed or supplied by the consumer.
API keys
An API key is the credential an application presents when it calls the gateway. Each key ties to a Console user, with applications registered as clients, plus a routing chain and any policies attached to that chain. Keys are issued from the Admin Console or claimed by developers in the Console, depending on the onboarding model in use. Revocation, rotation, and per-key budgets are all managed centrally.
Bring Your Own Key (BYOK)
Bring Your Own Key, or BYOK, is the model under which a consumer's own upstream provider credentials are used in place of centrally managed ones. A team that holds its own OpenAI account, for example, can register those credentials with Agent Router and have its requests routed through them rather than through the centrally pooled credentials.
BYOK is a first-class part of the routing model: BYOK and centrally managed backends can sit in the same routing chain, and policy, not application code, decides between them. BYOK fits when billing, compliance, or contractual relationships require a specific team's traffic to land on a specific provider account.
Single sign-on and role mapping
Agent Router integrates with corporate identity providers through single sign-on (SSO) over OIDC. Microsoft Entra ID, Google Workspace, Auth0, Okta, and any OIDC-compliant provider can handle authentication for the Console and the Admin Console. Once SSO is configured, role assignment is driven from the identity provider itself: app role or group claims in the OIDC token map to Agent Router roles on every login, so directory changes propagate without manual work in Agent Router. The SSO role mapping guide covers the mapping mechanics in detail.
Agent infrastructure
Beyond model routing, Agent Router provides a control surface for the agent ecosystem, particularly the Model Context Protocol servers that AI clients consume.
Model context protocol (MCP)
The Model Context Protocol (MCP) is an emerging standard for exposing tools, data sources, and context to AI clients. MCP servers act as adapters: one wraps a file system, another a ticketing API, another a documentation source, and so on. AI clients such as Claude Code, Cursor, and VS Code consume those servers to extend their own capabilities.
MCP profile
An MCP profile is a single endpoint that aggregates multiple MCP servers behind one interface. Instead of configuring each AI client with a separate connection to every server it needs, a profile is assembled centrally, governed with Agent Router's identity and access model, and exposed as one URL that clients connect to once. Profiles remove the per-agent fan-out that becomes unmanageable as the MCP ecosystem grows, and they bring MCP traffic under the same audit, access-control, and observability regime as standard model traffic.
Extensibility and observability
Three more concepts shape how Agent Router plugs into the surrounding software estate: the extension model inside the gateway, the telemetry interface that exports operational data to external systems, and the audit log of administrative actions.
Dynamic modules
The gateway is built on a proxy component, and Agent Router-specific behavior is delivered through dynamic modules: high-performance proxy extensions, written in Rust or Go, compiled to shared libraries that run inline in the filter chain. Dynamic modules implement the routing logic, credential handling, and provider-specific request translation that set Agent Router apart from a generic HTTP gateway. They are not configured directly, but they are why the gateway behaves consistently across providers and why request-path overhead stays low.
OpenTelemetry export
Agent Router speaks OpenTelemetry (OTel), the open observability standard. Traces are exported over OTLP, and metrics are exposed on a Prometheus scrape endpoint that an external collector can forward as OTLP, reaching whatever observability stack is already in place: Grafana, Datadog, New Relic, Honeycomb, or any other OTel-compatible backend. The platform operator owns the export configuration and applies it centrally, so individual applications need no separate instrumentation to show up in existing dashboards.
Audit logging
Alongside operational telemetry, Agent Router keeps an audit log of administrative actions: model and provider changes, user and role assignments, SSO configuration changes, API key issuance and revocation, and other state-modifying events in the Admin Console. The audit log can be queried from the Admin Console, and it is the primary record reviewed during compliance investigations or post-incident analysis. The audit event schema is documented in Audit log events.
Where to go next
Architecture overview
How these concepts assemble into the running system, with diagrams of the request path between application, data plane, management plane, and upstream provider.
AI capabilities
Structured outputs, multimodal inputs, and function calling, passed through to any provider or translated across providers where mappings exist.
Glossary
The same vocabulary in one-line form for quick lookup.