Planes and core components
Tetrate Agent Router is built on a split-plane model.
- A management plane hosted by Tetrate holds configuration and exposes the web applications.
- A data plane deployed in the customer's Kubernetes cluster handles all AI traffic.
The two planes are connected by a single outbound link, and all prompt and response data stays inside the customer's environment.
This page explains how Tetrate Agent Router is put together and defines each term as it appears. No prior familiarity with the platform is assumed. The starting point is a single idea that shapes everything else: the parts of the platform that decide what should happen are kept separate from the parts that carry live traffic, and only the latter run inside the customer's own infrastructure.
For a first-principles walk-through of what an AI gateway is and how a request travels end to end, start with the Architecture overview. This page goes deeper on the two planes and the components inside the data plane.
System architecture
A plane is a group of software components that share a single job. Tetrate Agent Router is organised into two planes, each with a distinct responsibility and each running in a different place.
- The management plane is the part hosted and operated by Tetrate. It holds configuration (routing rules, policies, model and provider definitions, user records) and exposes the two web applications used to manage the platform. It is the control room: decisions are recorded here, but no live AI traffic passes through it.
- The data plane is the part deployed inside the customer's own Kubernetes cluster. Kubernetes is the industry-standard system for running containerised software across a group of machines; a cluster is one such group. The data plane handles all AI traffic: every prompt sent to a model, and every response returned, passes through it and nowhere else.
The two planes communicate over a single outbound Hypertext Transfer Protocol Secure (HTTPS) connection initiated by the data plane. Outbound means the connection is opened from inside the customer's environment toward Tetrate, never the other way around. No inbound connections from the internet to the customer environment are required.
The diagram traces two independent journeys. Configuration travels down the left: the management plane hands updates to the Controller, which applies them to the Agent Router. Live traffic travels along the bottom: client applications send requests to the Agent Router, which forwards them to the upstream AI providers and returns the responses. The two journeys never mix, which is the point of the split.
Why the architecture is split this way
The separation exists for one main reason: prompt and response data never leaves the customer's environment. Because the data plane runs in infrastructure the customer controls, the sensitive content of requests (which can include proprietary code, customer records, or internal documents) stays inside the customer's network boundary. The management plane only ever holds configuration and receives telemetry (operational data such as latency and token counts); it never sees the content of a request.
The outbound-only link reinforces this. A firewall configured to allow a connection out is far easier for a security team to approve than one that must accept connections coming in, and refusing all inbound connections removes an entire class of attack. The result is data sovereignty (the customer keeps full control of its data) without the customer having to build and operate the control-plane software itself.
Operating model
The split-plane model divides responsibilities cleanly between a centralized management environment controlled by Tetrate and a customer-controlled data environment where all AI traffic flows. In practice this means each side owns a different set of concerns, and each side can be changed, scaled, or recovered without disturbing the other.
Management plane
Hosted and operated by Tetrate. Contains the Developer Console, the Admin Dashboard, and the central backend that stores routing rules, policies, and configuration. Customers interact with these applications to configure the platform but do not manage the underlying infrastructure, so there are no control-plane servers to patch, back up, or keep highly available on the customer side.
Data plane
Deployed within the customer's own Kubernetes cluster. All prompt and response traffic flows exclusively through this environment, so sensitive data never leaves the customer's infrastructure boundary. The data plane receives configuration from the management plane but operates independently: it holds the most recent configuration locally and continues routing requests normally even if connectivity to the management plane is temporarily interrupted. A brief loss of the link pauses configuration updates, not live traffic.
The two applications in the management plane are two views of the same system, each scoped to a different audience.
| Application | Audience | What it is used for |
|---|---|---|
| Developer Console | Developers | Issue API keys, author routing configurations, assemble Model Context Protocol (MCP) profiles, and monitor their own traffic. |
| Admin Dashboard | Plaform operators (platform / IT) | Provision models and providers, manage users, set budgets and rate limits, review audit logs, and configure single sign-on. |
The three journeys that cross these planes (configuration polling, the live request path, and telemetry export) are traced in detail on the Data flows page.
Operating multiple data planes
Tetrate Agent Router Enterprise supports running more than one data plane from a single management plane. Each data plane is an independent deployment in its own Kubernetes cluster, and the data planes can sit in any combination of clouds, regions, or on-premises environments while the one management plane keeps their configuration in sync.
More than one data plane is useful whenever a single deployment cannot satisfy every requirement at once. A common pattern is one data plane per region, placing traffic handling close to the applications it serves to reduce latency and to keep requests within a required geography for data residency. Another is one data plane per cloud, so that models offered natively by a provider are reached over that provider's own network: an AWS data plane serving Amazon Bedrock models, for example, alongside an Azure data plane serving Azure OpenAI models. Data planes are also split per business unit or per environment (production, staging) when separate teams need to operate without affecting one another.
Within this model, each data plane provisions and manages one or more gateways, where a gateway is a single inference endpoint (one customer-facing URL) mapped to exactly one project. A project is the logical boundary that owns a set of models, API keys, and MCP profiles, together with the members allowed to use them, and is defined in full under Projects. Because a gateway serves only its project, isolation is enforced at request time: a key issued for one project is refused on another project's gateway, and a model a project has not been granted cannot be called through it. Several projects can share one data plane, each with its own gateway, or be spread across data planes as the region and cloud patterns above require.
Holding configuration centrally in the management plane while each data plane runs independently is what makes a fleet of data planes practical to operate. Data planes can be added as demand grows, and a gateway's runtime can be upgraded in place or moved to a different data plane without changing its customer-facing URL or dropping in-flight requests. The operator-facing procedures are covered in Run multiple platform instances and Load balance across regional deployments.
Core components of the data plane
The data plane is not a single program. It is a small set of components that run together inside the customer's cluster. At the center is the AI Gateway, which intercepts and manages all traffic between client applications and upstream AI providers. A gateway is a single entry point that requests pass through on their way somewhere else, and this one is built as a proxy: a server that receives a request, does something with it (here: choosing a provider, attaching credentials, applying policy), and forwards it on.
Controller
Acts as the bridge between the data plane and the management plane. It polls the backend for configuration updates and translates them into a gateway configuration, keeping routing rules, policies, and provider credentials in sync without requiring inbound connections. Polling means the Controller repeatedly asks for updates on a schedule, so it is always the side that reaches out.
Modules
The AI gateway is extended with high-performance native modules for cost controls, guardrails, and custom extensions. A dynamic module is add-on logic loaded into the proxy at runtime, so it runs inline in the proxy filter chain (the ordered sequence of steps each request passes through) and adds processing without an extra network hop.
Agent Router
The proxy that routes every request to a provider and enforces policy inline.
The Agent Router itself (the proxy every request passes through) is the gateway the other components extend and support. It is built on an Cloud Native Computing Foundation (CNCF) backed open source foundation, co-created and maintained by Tetrate. This open source component is a widely used open-source proxy, which is why the platform can route to many providers reliably and with low overhead. The Controller keeps the gateway configured, the dynamic modules add cost controls and guardrails inline, and Redis lets several copies of the gateway share one view of rate-limit usage.
Why these pieces run inline rather than as separate services: every extra service a request must call adds a network hop, and every hop adds latency and a new thing that can fail. Running cost controls and guardrails as dynamic modules inside the proxy keeps that logic on the request path itself, so policy is enforced without leaving the gateway.
The components divide across the two planes as follows. The management plane components are operated by Tetrate; the data plane components run inside the customer's cluster.
| Component | Plane | Role |
|---|---|---|
| Developer Console, Admin Dashboard, central backend | Management | Configuration storage and the web applications operators and developers use |
| Agent Router (AI Gateway) | Data | The proxy that routes every request to a provider and enforces policy inline |
| Controller | Data | Polls the backend and translates configuration into gateway configuration |
| Modules | Data | Inline gateway extensions for cost controls, guardrails, logs, telemetry and more |
Key terms
The vocabulary introduced above, collected for reference:
| Term | Meaning |
|---|---|
| Plane | A group of software components that share one responsibility. The platform has two: management and data. |
| Management plane | The Tetrate-hosted half that stores configuration and serves the web applications. Holds no live traffic. |
| Data plane | The customer-hosted half, in Kubernetes, that all AI traffic flows through. |
| Kubernetes | The industry-standard system for running containerised software across a group of machines (a cluster). |
| Controller | The data-plane component that polls the management plane for configuration and applies it to the gateway. |
| Gateway | The single entry point every request passes through before reaching a provider. |
| Proxy | A server that receives a request, acts on it, and forwards it on. The gateway is built as one, on Envoy. |
| Dynamic module | Add-on logic loaded into the proxy at runtime, running inline in the filter chain with no extra network hop. |
| Redis | A fast in-memory datastore the gateway uses as a shared counter for rate limiting across instances. |
| Rate limiting | Capping how much an API key may consume, enforced consistently across all gateway copies. |
| Outbound-only connection | A link opened from the customer's environment toward Tetrate, never the reverse, so no inbound firewall openings are needed. |
Where to go next
- Data flows: the three traffic paths that cross these planes. See Data flows.
- Guardrails: how the Guardrails dynamic module inspects traffic inline. See Guardrails.
Where to go next