Key concepts
A handful of terms show up again and again across these docs. Some are common to any AI gateway; others are specific to how Tetrate Agent Router is built and run.
Read it end to end the first time through. For a quick definition later, the Glossary in the Reference section gives the one-line version.
Structural concepts
These are the building blocks: the components of Agent Router itself and how they're deployed.
Data plane and management plane
Agent Router splits into two halves that talk over a small, well-defined interface.
The data plane handles the request path. It contains the Controller and a proxy component (the AI gateway), and every AI request and response flows through it. The data plane receives application requests, evaluates routing policy, attaches the right credentials, calls the upstream AI provider, and returns the response. Where it runs depends on the deployment model: in Agent Router Service and Agent Router Enterprise Fully Managed, Tetrate operates it; in Agent Router Enterprise Self-Hosted Data Plane, it is deployed in a customer-controlled Kubernetes cluster, so traffic, payloads, prompts, completions, and credentials stay inside the customer's network. The three paths are compared in Deployment models.
The management plane is the Tetrate-hosted control surface in every deployment model. It stores routing rules, policies, model and provider definitions, user records, API key metadata, audit history, and analytics rollups. The data plane pulls its configuration from here and pushes telemetry back, but no request payloads ever cross the boundary. A centrally managed control surface is available without operating that stack in-house.
The Developer Console and the Admin Console
Two web applications front Agent Router, each scoped to a different audience.
The Developer Console is the developer-facing application. It's where API keys are issued, routing policies authored, MCP profiles assembled, integrations configured, and prompts tested in the Playground. Developers spend most of their time here.
The Admin Console is the platform-operator application. It's where models and providers are provisioned, users managed, budgets and rate limits set, audit logs reviewed, and SSO configured. Most of Agent Router's governance happens here.
The two applications share the same underlying system. They're two views of the same data, each scoped to the responsibilities of its audience.
Projects
A project is the logical boundary that owns configuration and access in Agent Router. Everything a team works with day to day belongs to a project: the models and providers granted to it, its API keys, its MCP servers and profiles, its routing and policy, its budgets, and its members. It is also the unit of attribution, so usage and audit records are tied to the project the traffic ran under. For anyone familiar with the concept of a workspace or a tenant in other tools, a project plays a similar role here: it is the container that keeps one team's or one application's setup separate from the next.
Projects, gateways, and data planes
The project is one of three objects that work together, and the clearest way to understand a project is to see how it relates to the other two. They divide into a logical layer and a physical layer:
- A project is logical. It is the configuration and access boundary described above. It decides what should be possible (which models, which members, which policy), but it runs no software of its own.
- A gateway is physical. It is a single inference endpoint with one customer-facing URL: the thing an application actually calls. Each gateway serves exactly one project.
- A data plane is physical too. It is the layer that provisions and manages gateways, Tetrate-operated in Agent Router Service and Agent Router Enterprise Fully Managed, or deployed in a customer-controlled Kubernetes cluster in Agent Router Enterprise Self-Hosted Data Plane. It is the capacity a gateway runs on.
These three connect through a small set of fixed relationships:
| Objects | How they relate |
|---|---|
| Project and gateway | Default one gateway; each gateway serves exactly one project. Selecting a data plane auto-provisions the project's first gateway. On Agent Router Enterprise with multi-gateway projects, a project may optionally attach additional like-configured gateways for high availability (see Gateway sets and DNS-level failover). A gateway never serves more than one project. |
| Data plane and gateways | One-to-many. A single data plane provisions and manages many gateways. Typically that is one gateway per project placed on it; when a project uses optional HA members, those members usually run on separate data planes. |
| Management plane and data planes | One-to-many. The single management plane manages many data planes, which can sit in different clouds and regions. |
Read together, the picture is straightforward: the one management plane holds every project's configuration, each data plane is a pool of capacity in some location, and each gateway is a project's live endpoint running on one of those data planes. A project comes to life by requesting a gateway from a data plane. The data plane provisions the runtime, the management plane maps the project to that gateway and publishes its desired configuration, and once the gateway reports ready its URL appears on the project. Until then, a project is just configuration with nowhere to send traffic. Most deployments stop there; optional additional members are an Enterprise HA concern, not the default path.
Keeping the logical project separate from the physical gateway and data plane is deliberate, and it is what makes the model flexible. Because a project's identity (its URL, its keys, its configuration) is decoupled from the capacity that serves it, a gateway's runtime can be upgraded in place or moved to a different data plane without the project's configuration or its URL changing, and without dropping in-flight requests. The same decoupling lets projects be placed wherever they need to run: several projects can share one data plane, each with its own gateway, or be spread across data planes by region or cloud (for example, one project served from an AWS data plane and another from an Azure data plane).
Why projects matter
The project is the isolation boundary, and that isolation is enforced in two places. In the applications, every view is scoped to the selected project: it lists only the models, MCP servers, MCP profiles, and request logs that belong to that project, and switching projects with the project switcher re-scopes the whole app. At the gateway, cross-project requests are refused at request time. A key issued for one project does not work on another project's gateway (it is rejected with a 403), and a model a project has not been granted cannot be called through it (a 404). This runtime enforcement is what lets several teams or applications share one Agent Router deployment, or even one data plane, while staying fully separated from one another.
That boundary is why projects are the organizing idea for running Agent Router at scale. They provide a hard line for security and access, clean per-team attribution for cost and audit, and the freedom to move or grow the underlying infrastructure without disturbing the applications that depend on it. A default project is created for each customer during onboarding, and administrators can add more afterward: one per team, application, or environment (for example, separate production and staging projects). The operator tasks for creating projects, provisioning their gateways, and placing them on data planes are collected under Project and data plane management, and the fleet-level arrangement of data planes is described in Operating multiple data planes.
Routing concepts
Routing is the gateway's central job: for each incoming AI request, deciding which backend should serve it. A few concepts describe how that decision gets made.
Backends, models, and providers
A provider is an upstream AI service the gateway can talk to: OpenAI, Anthropic, Google, Azure OpenAI, Mistral, and many others, plus self-hosted endpoints and customer-specific deployments. A model is a specific offering from a provider, such as gpt-4o or claude-sonnet-4-6. A backend, in routing terms, is a model-on-a-provider combination paired with the credentials needed to reach it. Every routing decision ultimately resolves to a choice of backend.
Routing chains and policies
Each API key carries a routing chain: an ordered or weighted list of backends the gateway considers when a request arrives. The chain encodes the consumer's preferences, fallbacks, and constraints, and one or more routing policies attached to the key shape how it behaves.
Fallback policy
A fallback policy is an ordered list of backends. When a request arrives, the gateway tries the first backend in the chain; if that call fails with a recoverable error, it moves to the next backend and tries again, continuing until one succeeds or the chain runs out. The recoverable set defaults to provider responses with HTTP status 401, 413, 422, 429, 500, 503, 504, or 529, plus connection failures, gateway errors, other 5xx responses, and deadline exceeded. It is configurable per key under Retry on in the Fallback section, so a key may be narrower than this default. The fallback decision happens inside the gateway, so the calling application sees a single request that either succeeds or fails. It never has to implement retry logic of its own.
Fallback policies are how provider risk is managed: a primary commercial model with one or more secondary models behind it, or a commercial model with a self-hosted open-weights model as a last-resort backstop.
Traffic splitting
A traffic splitting policy distributes requests across two or more backends by percentage rather than by priority. A 70/30 split, for example, sends 70 % of requests to the first backend and 30 % to the second. Use it to evaluate one model against another under real production traffic, to manage cost by routing a portion of traffic to a cheaper backend, or to roll out a new provider gradually.
Splitting and fallback combine: a split picks the primary backend for a given request, and a fallback chain takes over if that backend fails.
Advanced routing rules
When neither weighted splitting nor priority-ordered fallback is enough, advanced routing rules dispatch requests based on attributes of the request itself: the model the caller asked for, custom headers, request metadata, or other signals. Use them to encode policies such as "route requests from this tenant to this provider" or "send anything tagged code-completion to a model tuned for that task".
Policy and cost concepts
Routing decides where a request goes. A second set of concepts decides what a request is allowed to do: what it may cost, how fast it may arrive, and what it may contain.
Policies
A policy is an admin-managed rule that decides what happens to a model request. Each one carries a type, which control it is; a scope, which API keys it governs; a configuration belonging to that type; and a mode, monitor or active. Five controls are built on that single shape, each pairing a trigger with an action.
| Type | Fires when | Action |
|---|---|---|
| Budget | a budget is exhausted | block, or degrade to a cheaper model |
| Availability fallback | the primary model fails, on error or timeout | move to the next model in the chain |
| Traffic split | every request | route by weight |
| Rate limit | a token cap would be exceeded | block, or degrade |
| Guardrail | a content or safety check matches | block, or redact |
Fallback policy and traffic splitting, described above, are two of those five. Setting a rule once and having it apply to a whole class of keys, rather than configuring the same control on hundreds of keys by hand, is the reason the model exists. Policies covers the rest of the machinery: the six scopes a policy can take, how two policies covering one key combine, and what monitor mode does.
Budgets and budget policies
A budget is an amount of money for one period, attached to a subject that says whose spend the amount covers: a team, each user on a team, or one API key. The period is daily, weekly, or monthly, chosen when the budget is created and editable afterward. A budget is money, never tokens or request counts.
A budget measures; it never enforces. It tracks spend against its limit and reports on it, and that is the whole of what it does. Spend is evaluated in the management plane against aggregated request logs rather than inline at the gateway, so crossing the limit changes nothing about what a caller can do next.
Enforcement is a separate object. A budget policy takes the budget's threshold as its trigger and acts on the request path once the threshold is crossed, either blocking the caller or degrading to a cheaper model and continuing to serve. A budget is the accounting layer and a policy is the enforcement layer: one surface owns the number and the report, the other owns the consequence.
Budgets ship today, and budget policies are planned for the 2026 Q3 release. Capping how fast a credential may consume, rather than how much it may spend over the period, is a rate limit instead. See Budgets.
The three stages of cost control
Cost control arrives in three stages, and what separates them is what happens when a number is reached.
| Stage | What it does | Status |
|---|---|---|
| Analytics | Reports what is being spent, by team, by model, by key, and over time. Nothing is limited | Available now |
| Budget monitoring | Tracks spend against a daily, weekly, or monthly limit and raises an alert as the limit is approached or crossed. Requests continue to be served | Available now |
| Enforcement | A policy acts at the limit, refusing the caller or degrading to a cheaper model | Planned for the 2026 Q3 release |
Budgets are one of three cost controls, and the narrowest in what they can address. The stages are worked through in Cost and quota management, and Choose the right cost control for each workload covers which instrument fits a given workload.
Credentials and identity
Identity in Agent Router is layered. Applications authenticate to the gateway with API keys; human operators authenticate to the Console and Admin Console with single sign-on; and the gateway authenticates to upstream providers using credentials that are either centrally managed or supplied by the consumer.
API keys
An API key is the credential an application presents when it calls the gateway. Each key ties to a consumer (a person or a service), a routing chain, and any policies attached to that chain. Keys are issued from the Admin Console or claimed by developers in the Console, depending on the onboarding model in use. Revocation, rotation, and per-key budgets are all managed centrally.
Bring Your Own Key (BYOK)
Bring Your Own Key, or BYOK, is the model under which a consumer's own upstream provider credentials are used in place of centrally managed ones. A team that holds its own OpenAI account, for example, can register those credentials with Agent Router and have its requests routed through them rather than through the centrally pooled credentials.
BYOK is a first-class part of the routing model: BYOK and centrally managed backends can sit in the same routing chain, and policy, not application code, decides between them. Reach for BYOK when billing, compliance, or contractual relationships require a specific team's traffic to land on a specific provider account.
Single sign-on and role mapping
Agent Router integrates with corporate identity providers through single sign-on (SSO) over OIDC. Microsoft Entra ID, Okta, Auth0, Keycloak, and any OIDC-compliant provider can handle authentication for the Console and the Admin Console. Once SSO is configured, role assignment is driven from the identity provider itself: app role or group claims in the OIDC token map to Agent Router roles on every login, so directory changes propagate without manual work in Agent Router. The Configuring SSO: Mapping Roles and Groups guide covers the mapping mechanics in detail.
Agent infrastructure
Beyond model routing, Agent Router provides a control surface for the agent ecosystem, particularly the Model Context Protocol servers that AI clients consume.
Model context protocol (MCP)
The Model Context Protocol (MCP) is an emerging standard for exposing tools, data sources, and context to AI clients. MCP servers act as adapters: one wraps a file system, another a ticketing API, another a documentation source, and so on. AI clients such as Claude Code, Cursor, and VS Code consume those servers to extend their own capabilities.
MCP profile
An MCP profile is a single endpoint that aggregates multiple MCP servers behind one interface. Instead of configuring each AI client with a separate connection to every server it needs, a profile is assembled centrally, governed with Agent Router's identity and access model, and exposed as one URL that clients connect to once. Profiles get rid of the per-agent fan-out that becomes unmanageable as the MCP ecosystem grows, and they bring MCP traffic under the same audit, access-control, and observability regime as standard model traffic.
Extensibility and observability
Two more concepts shape how Agent Router plugs into the surrounding software estate: the extension model inside the gateway, and the telemetry interface that exports operational data to external systems.
Dynamic modules
The gateway is built on a proxy component, and Agent Router-specific behavior is delivered through dynamic modules: high-performance proxy extensions, written in Rust or Go, compiled to shared libraries that run inline in the filter chain. Dynamic modules implement the routing logic, credential handling, and provider-specific request translation that set Agent Router apart from a generic HTTP gateway. They are not configured directly, but they're why the gateway behaves consistently across providers and why request-path overhead stays low.
OpenTelemetry export
Agent Router speaks OpenTelemetry (OTEL), the open observability standard. Request-level metrics, traces, and logs are exported in OTEL format to whatever observability stack is already in place: Grafana, Datadog, New Relic, Honeycomb, or any other OTEL-compatible backend. The platform operator owns the export configuration and applies it centrally, so individual applications don't need separate instrumentation to show up in existing dashboards.
Audit logging
Alongside operational telemetry, Agent Router keeps an audit log of administrative actions: model and provider changes, user and role assignments, SSO configuration changes, API key issuance and revocation, and other state-modifying events in the Admin Console. The audit log can be queried from the Admin Console, and it's the primary record reviewed during compliance investigations or post-incident analysis. The audit event schema is documented in the Reference section.
Where to go next
Architecture overview
How these concepts assemble into the running system, with diagrams of the request path between application, data plane, management plane, and upstream provider.
Solution capabilities
Structured outputs, multimodal inputs, and function calling, passed through to any provider or translated across providers where mappings exist.
Glossary
The same vocabulary in one-line form for quick lookup.