Architecture overview
Tetrate Agent Router is an AI gateway that provides unified, policy-controlled access to large language models across multiple providers. All AI traffic flows through a customer-managed data plane, keeping prompt and response data inside the customer's own infrastructure.
This page builds the mental model of how Agent Router is put together, starting from first principles and defining each term as it appears. No prior familiarity with Tetrate Agent Router is assumed. The deeper topic pages linked at the end expand on each area once the overall shape is clear.
The material here is conceptual: it explains what the pieces are and why they are arranged as they are, not how to install or configure them. Readers who want to start using Agent Router can move straight to the Console quickstart; readers responsible for standing up the infrastructure will find the installation flow under Get started.
New to the vocabulary? The Key concepts page introduces the platform's terms in narrative form, and the Glossary is a quick lookup for any single word. This page defines terms inline as they come up.
What an AI gateway is
A large language model (LLM) is an AI model that takes a piece of text (a prompt) and returns generated text (a response). Models are hosted by providers: companies and services such as OpenAI, Anthropic, Google, Azure, Amazon, and Perplexity, as well as models an organisation runs itself (self-hosted models). Each provider exposes its own API, its own credentials, and its own endpoints.
Without a gateway, every application wires itself directly to each provider it uses. That means scattered API keys, per-provider integration code, no single place to see what is being spent, and no consistent way to apply security or routing rules. Swapping or adding a provider becomes an application change.
An AI gateway is a single entry point that sits between the applications making requests and the providers serving them. Applications send every request to the gateway instead of to a provider directly. The gateway decides which provider and model should handle each request, attaches the right credentials, applies policy (spending limits, content rules, access control), records what happened, and forwards the request. To the application, it looks like one endpoint that speaks a familiar API; behind it, the gateway can reach 200+ models across every supported provider. Adding, swapping, or removing a provider becomes a configuration change rather than an application rewrite.
The effect is easiest to see in a concrete case. Suppose an application asks for a general-purpose chat model. The gateway can send that request to one provider today and a cheaper or higher-quality provider tomorrow, retry against a second provider if the first is overloaded, redact sensitive text before it is forwarded, and refuse the request outright if it would push a team past its monthly budget. None of that logic lives in the application, and none of it requires an application change: the request keeps arriving at one endpoint, while the decisions move into configuration held centrally. The same central position is what makes consolidated spend reporting and a single audit trail possible, because every request is observed in one place.
Tetrate Agent Router is that gateway, plus the applications and services used to configure and operate it. It ships in two forms that share the same core: Agent Router Service, a self-serve tier for individual developers, and Agent Router Enterprise, which adds team-level cost attribution, guardrails, single sign-on, and a data plane the customer runs in its own infrastructure. This documentation describes the Enterprise architecture.
Component overview
The diagram below shows how the main applications sit above a shared backend layer, which in turn connects to the AI gateway routing traffic to upstream providers.
Three layers appear in the diagram, from top to bottom:
- The two applications are the human-facing surfaces. The Developer Console (the Console) is where developers issue credentials, define routing rules, and watch their own traffic. The Admin Dashboard is where operators provision models and providers, manage users, and set the policies developers work within. Both are web applications served from the same backend, and both are read-and-write surfaces onto the same configuration. What differs is scope: the Console shows a developer only their own keys, routes, and usage, while the Admin Dashboard sees every user and every setting across the deployment.
- The shared backend stores all configuration (routing rules, policies, provider settings, user accounts) and serves it to the rest of the system. It is the single source of truth: a change saved in either application is recorded here first, and every other component reads from it rather than holding its own copy. The Controller shown here is the component that carries that configuration down to the gateway.
- The AI Gateway is the component that every request actually passes through. It routes each request to a provider and applies policy inline, meaning as part of handling the request rather than in a separate step. Because policy runs inside the request path, a blocked or redacted prompt is stopped before it reaches a provider, not flagged after the fact.
The split-plane model
Those layers do not all run in the same place. Agent Router divides into two halves, called planes, that run in different environments and have different jobs.
- The management plane is hosted and operated by Tetrate. It holds the configuration and exposes the two web applications. Think of it as the control room: it is where decisions are recorded, but no live AI traffic passes through it. Because it is fully managed, upgrades, availability, and scaling of the control surfaces are Tetrate's responsibility, not the customer's.
- The data plane handles all AI traffic — every prompt and every response passes through it, and nowhere else. In fully hosted deployments (Service or Enterprise Fully Managed), Tetrate operates the data plane. In Self-Hosted Data Plane, you deploy it inside your own Kubernetes cluster (Kubernetes is the industry-standard system for running containerised software).
Where the data plane runs is what distinguishes the deployment models. In fully hosted options, Tetrate runs the data plane as well — the fastest way to start. In Enterprise Self-Hosted Data Plane, you run the data plane in your cluster while Tetrate continues to host the management plane; that is what keeps traffic inside your boundary. All three paths are compared in Deployment models.
The separation is the central idea of the architecture. In Self-Hosted Data Plane deployments, prompt and response data never leaves your environment because the data plane runs in infrastructure you control; the management plane only ever holds configuration, never traffic. In fully hosted deployments, Tetrate operates both planes.
The two planes still have to stay in sync, and they do so over a single connection with a deliberate direction. The data plane reaches out to the management plane to pull down the latest configuration: this is outbound from the customer's environment. The management plane never initiates a connection into the customer's environment. This "outbound-only" design means the customer's firewall needs no inbound openings for Agent Router, which is simpler to approve and reduces the ways in which the environment can be attacked.
Split-plane model, management plane, data plane, and the outbound-only connection are covered in depth in Planes and core components.
Core components of the data plane
The data plane is not a single program. It is a small set of components that run together inside the customer's cluster.
| Component | What it does, in plain terms |
|---|---|
| Agent Router (the AI Gateway) | The proxy every request passes through. A proxy is a server that receives a request, does something with it, and forwards it on. This one chooses the provider, attaches credentials, applies policy, and forwards the request. It is built on Envoy, a widely used open-source proxy. |
| Controller | The bridge between the two planes. It repeatedly asks the management plane for the latest configuration and translates it into settings the gateway understands, keeping routing rules, policies, and credentials current. |
| Dynamic modules | Add-on logic that runs inside the gateway rather than as a separate service, so it adds no extra network hop. Modules are compiled shared libraries (written in Rust or Go) loaded into the gateway's filter chain, and they provide capabilities such as routing decisions, cost controls, and guardrails. |
| Redis | A fast in-memory datastore the gateway uses to count requests, so that rate limits are enforced consistently even when several copies of the gateway run at once. |
These components run in dedicated Kubernetes namespaces within the cluster, keeping Agent Router's resources separate from the customer's own workloads. Because the gateway is horizontally scalable, more copies can be added to handle load; the shared Redis counter is what lets those copies enforce a single rate limit between them rather than each counting in isolation.
An optional external service, Azure Content Safety, can be connected to the gateway for advanced content analysis; it is the one component that lives outside the cluster, and it is only involved when that level of checking is switched on.
How a request flows end to end
Bringing the pieces together, here is the path of a single request once Agent Router is running:
- An application sends a request to the gateway's endpoint, using an API shape it already supports (Agent Router is OpenAI-compatible, so existing code often works unchanged). The request carries an API key, a credential that identifies the caller.
- The gateway authenticates the key and identifies which user or team the request belongs to.
- Routing rules decide which provider and model should serve the request. If a fallback policy is configured (an ordered list of backends to try), the gateway knows what to try next should the first choice fail.
- Guardrails inspect the prompt. A guardrail is a content rule, for example removing personally identifiable information or blocking a disallowed request, enforced inline before the prompt is forwarded.
- Rate limits are checked, so that no key exceeds its allotted tokens or requests. A token is the unit of text LLMs measure work in; roughly a few characters each.
- The gateway attaches the provider credential and forwards the request. If that provider returns an error or is unreachable, the gateway walks to the next backend in the fallback chain, within the same request, with no action needed from the application.
- The provider's response returns through the gateway, where guardrails can inspect it again, and details of the exchange are recorded as telemetry (operational data such as latency, token counts, and cost) before the response is returned to the application. When the caller requests a streamed response, the gateway relays it token by token as the provider produces it, rather than waiting for the full text.
Every step above happens inside the customer's data plane. The management plane supplied the rules beforehand but sees none of this traffic.
Two behaviours make this flow easier to operate. First, errors from different providers are normalized into a consistent shape, so an application handles one error format regardless of which backend was reached. Second, every request is tagged with a correlation identifier returned in the response, which ties an application-side failure to the exact exchange recorded in telemetry and makes a single request traceable end to end. Both are detailed in Gateway behavior.
Key terms
The vocabulary introduced above, collected for reference:
| Term | Meaning |
|---|---|
| Large language model (LLM) | An AI model that turns a text prompt into a generated text response. |
| Provider | A service that hosts models (OpenAI, Anthropic, Google, Azure, Amazon, Perplexity), or a model the organisation self-hosts. |
| Prompt / response | The text sent to a model, and the text it returns. |
| AI gateway | The single entry point requests pass through, which routes and applies policy before reaching a provider. |
| Management plane | The Tetrate-hosted half that stores configuration and serves the web applications. Holds no traffic. |
| Data plane | The customer-hosted half, in Kubernetes, that all AI traffic flows through. |
| Controller | The data-plane component that pulls configuration from the management plane and applies it to the gateway. |
| Routing | Choosing which provider and model serves a given request. |
| Fallback policy | An ordered list of backends the gateway tries in turn when one fails. |
| Guardrail | An inline content rule (for example PII redaction or blocking) applied to prompts and responses. |
| Rate limit | A cap on the tokens, requests, or concurrency a key may consume. |
| Telemetry | Operational data (latency, token counts, cost, logs) recorded about traffic and exported to monitoring tools. |
| Bring Your Own Key (BYOK) | Using a caller's own provider credentials, so that spend is settled with the provider directly. See Use your own provider credentials. |
| Model Context Protocol (MCP) | A standard for connecting AI agents to external tools, which the gateway can aggregate behind one endpoint. See Aggregate MCP servers into a profile. |
Explore the architecture
Planes and core components
The split-plane model, the operating responsibilities of each plane, and the data-plane components that route traffic.
Data flows
The configuration, request, and observability paths, and how they stay isolated from one another.
Guardrails
How the Guardrails dynamic module inspects and enforces policy inline at the gateway.
Deployment models
The hosted SaaS option and the dedicated Enterprise option.
Network and security
Private connectivity, network isolation, transport security, and WAF policies.
Operations and reliability
Control-plane resilience, multi-region deployment, scaling, and per-request fallback.
What's next