Operations and Reliability
Tetrate Agent Router routes every AI request through a customer-managed data plane. Because that data plane is the single path all traffic takes, its reliability is the reliability of the whole service. This page explains the mechanisms that keep it running: how it survives losing contact with the management plane, how it spreads across regions, how it heals and scales itself, how it is deployed reproducibly, and how it absorbs provider failures one request at a time.
This page assumes no prior familiarity with Tetrate Agent Router and defines each term as it appears. Every mechanism described here follows from one architectural choice: the platform is split into two halves that run in different places and have different jobs. That split is worth restating before the mechanisms make sense.
The split-plane model is introduced in the Architecture overview and covered in depth in Planes and core components. This page builds on both and focuses on what the split means for staying available.
The split that reliability follows from
Tetrate Agent Router divides into two halves, called planes, that run in separate environments:
- The management plane is hosted and operated by Tetrate. It stores all configuration (routing rules, policies, provider settings, user accounts) and exposes the two web applications used to manage the platform. No live AI traffic passes through it. This is sometimes also called the control plane, in the sense that it is where control decisions are recorded.
- The data plane is deployed inside the customer's own Kubernetes cluster. (Kubernetes is the industry-standard system for running containerised software, meaning applications packaged with everything they need to run, across a group of machines.) One or more data planes handle all AI traffic: every prompt sent to a model and every response returned passes through a data plane and nowhere else.
The two planes stay in sync over a single connection with a deliberate direction. The data plane reaches out to the management plane to pull down the latest configuration; the management plane never initiates a connection into the customer's environment. This is described more fully in Planes and core components.
Two consequences of this split drive everything below. First, the component that keeps configuration current, the Controller, pulls configuration on a schedule rather than waiting to be pushed to. Second, because the data plane holds its own copy of the configuration and does the actual routing, it can keep working even when it cannot reach the management plane at all.
Control plane resilience
Configuration polling is the mechanism the data plane uses to stay current. The Controller (the data-plane component that bridges the two planes) repeatedly asks the management plane, on a fixed schedule, whether the configuration has changed, and applies any changes it finds. "Polling" simply means asking on a schedule rather than being notified.
This design matters most when the connection between the planes is interrupted, for example by a network problem or a management-plane maintenance window. In that situation:
- The Controller continues to poll on its normal schedule, retrying the connection rather than giving up.
- All configuration already applied to the data plane remains in effect. The gateway keeps routing requests exactly as before, because it is acting on its own local copy of the rules, not on a live lookup against the management plane.
- When connectivity is restored, the Controller picks up any configuration changes made in the meantime and applies them.
The practical effect is that a management-plane interruption does not stop AI traffic. The configuration cannot be changed while the connection is down, but existing traffic continues to be served. This is the direct payoff of the split: the plane that carries traffic does not depend, moment to moment, on the plane that stores configuration.
The same polling connection doubles as a health signal in the other direction. Because a healthy data plane polls the management plane on a known schedule, the management plane can treat a data plane that stops polling as unreachable, and operators can see that state. Configuration polling is therefore both the mechanism that keeps a data plane current and the way the management plane knows a data plane is alive.
Multi-region deployment
A single management plane can manage many data planes at once. Each data plane is an independent Kubernetes deployment, and each can be placed in any cloud, region, or availability zone. Two of those terms are worth defining:
- A region is a geographically separate location operated by a cloud provider (for example a European or a North American region). Regions are designed to fail independently of one another.
- An availability zone is an isolated section within a region, with its own power and networking, so that a fault in one zone does not take down the others in the same region.
Because the data planes are independent of one another, they can be arranged to fit the organisation's needs rather than a fixed template: one data plane per region, one per business unit, or one per cloud provider are all valid. A common resilient arrangement runs several regional data planes at the same time, all serving traffic, so that the loss of one region degrades the service rather than stopping it. Sizing each region to carry its own share of traffic plus enough spare capacity to absorb a failed peer is a planning task covered in Sizing and scale.
Distributing traffic across several regional deployments of the same model, and draining a region for maintenance without dropping requests, are covered in Load balance across regional deployments. Planning recovery targets and topology for a production deployment is covered in Plan high availability and disaster recovery.
Health checks
For a deployment to heal itself, the system running it has to be able to tell healthy components from unhealthy ones. The data plane exposes two kinds of health signal, at two different levels.
At the cluster level, data plane components expose Kubernetes liveness and readiness probes. A probe is a small check Kubernetes runs against a component on a schedule:
- A liveness probe answers the question "is this component still working?" If it fails, Kubernetes restarts the component automatically, on the assumption that it has become stuck.
- A readiness probe answers the question "is this component ready to receive traffic right now?" If it fails, Kubernetes stops sending new requests to that component until it recovers, without restarting it. This is what prevents traffic from being sent to a component that is still starting up.
Together these probes let Kubernetes (the orchestrator, meaning the system that decides where and whether each component runs) detect and recover from unhealthy workloads without operator intervention.
At the platform level, the management plane monitors data plane health through the configuration polling described above: a data plane that stops checking in is flagged as unreachable. The two levels are complementary. Kubernetes probes keep an individual data plane healthy from the inside; the polling signal tells the management plane whether a whole data plane is reachable from the outside.
Scaling
Traffic to an AI gateway is not steady. It concentrates into working hours, time zones, and bursts, and the peak can sit well above the monthly average. A deployment sized only for the average would be overwhelmed at peak, so the data plane is built to add and remove capacity as load changes.
The data plane scales horizontally, meaning capacity is added by running more copies (called replicas) of a component behind a load balancer, rather than by making a single copy larger. This is the mechanism that also provides redundancy: running more than one replica of each component is what lets the data plane survive the loss of an individual replica or machine.
Data plane components are configured with the Kubernetes Horizontal Pod Autoscaler (HPA), a built-in Kubernetes feature that adds or removes replicas automatically in response to load. (A pod is Kubernetes' unit of running software, so "horizontal pod autoscaler" is literally the thing that scales the number of running copies out and in.) The HPA can be tuned to match a given traffic pattern and resource budget.
The management plane is scaled and managed by Tetrate, so this scaling concern applies to the customer-managed data plane only. Deciding how many replicas a deployment needs (measuring the throughput of one replica, dividing peak concurrency by it, then adding headroom for failover and upgrades) is the subject of Sizing and scale.
Infrastructure as code
Infrastructure as code is the practice of describing a deployment in text files kept under version control, rather than by clicking through consoles or running ad-hoc commands. The benefit is reproducibility: the same files produce the same deployment every time, changes are reviewed and tracked like any other code, and a lost environment can be rebuilt from its description. This is what makes a tested disaster-recovery procedure possible in the first place, because the configuration to restore is the committed source, not something reconstructed by hand.
Tetrate Agent Router supports this in two layers:
- Helm charts for installing and managing the data plane components on Kubernetes. A Helm chart is a packaged, parameterised template for a set of Kubernetes resources; Helm is the standard package manager for Kubernetes. The chart describes what the data plane should look like inside the cluster.
- Terraform modules for provisioning the underlying cloud infrastructure the data plane runs on (networks, subnets, access-control roles, and similar), where that infrastructure needs to be created as well. Terraform is a widely used infrastructure-as-code tool that creates and updates cloud resources from a declared description.
Used together, these give a fully reproducible, version-controlled deployment: Terraform stands up the cloud foundation, and Helm installs the data plane onto it. Because both are declarative descriptions in version control, the committed files serve as the backup from which a region can be rebuilt.
Resilience without full cutover
The mechanisms above keep a data plane and its regions healthy. A separate mechanism handles the failures that come from outside the data plane: the AI providers it forwards requests to. Providers have bad hours (regional incidents, capacity exhaustion, rate-limit throttling, outright outages), and the data plane absorbs these one request at a time rather than by switching an entire cluster or region over.
The tool for this is a fallback policy: an ordered list of backends (a backend being a specific model at a specific provider) to try for a given request. The gateway tries the first backend; if that call fails with a recoverable error, it moves to the next backend in the list and tries again, continuing until one succeeds or the list is exhausted. The calling application sees a single request that either returns a response or a final error; the failover happens inside the gateway and is invisible to the caller. Developers configure these chains as described in Improve resilience with fallbacks.
Under the hood, each backend in a fallback chain is assigned a priority level in the AI gateway's AIGatewayRoute resource (the configuration object that defines how a route's requests are dispatched). Priority 0 is the primary and serves all traffic under normal conditions; priority 1 is the first fallback, priority 2 the second, and so on. When a request to the priority-0 backend fails (a rate-limit response, a server error, or a connection failure), the gateway retries at priority 1, then priority 2, and so on, all within the same HTTP request lifecycle. This is per-request failover: the retry happens inside the handling of one request, before any response is returned, rather than by taking a backend out of service globally.
Two properties follow from this per-request model, and both matter for how the service behaves under provider trouble:
- There is no provider-level health tracking behind the chain. Each request independently discovers failures and walks its own priority chain. A backend is not marked "down" for everyone; it is simply skipped, per request, whenever it fails for that request.
- Intermittent errors are absorbed transparently, without any retry or branching logic in the application. A provider that fails one request in fifty costs those requests a walk down the chain, not an operator intervention.
This is a different layer from the regional and cluster resilience above, and the layers stack. Health-aware balancing inside a regional pool absorbs a single unhealthy region silently; a fallback chain absorbs the case where a whole backend or pool is unavailable; and neither requires a manual, cluster-wide cutover. The relationship between in-pool balancing and chain-level fallback is covered in Load balance across regional deployments.
Key terms
The vocabulary introduced above, collected for reference:
| Term | Meaning |
|---|---|
| Management plane (control plane) | The Tetrate-hosted half that stores configuration and serves the web applications. Carries no traffic. |
| Data plane | The customer-hosted half, in Kubernetes, that all AI traffic flows through. |
| Configuration polling | The Controller asking the management plane for configuration on a fixed schedule, rather than being pushed to. Doubles as a health signal. |
| Kubernetes | The industry-standard system for running containerised software across a group of machines. |
| Liveness probe | A Kubernetes check for "is this component still working?" A failure triggers a restart. |
| Readiness probe | A Kubernetes check for "is this component ready for traffic?" A failure removes it from rotation without a restart. |
| Horizontal Pod Autoscaler (HPA) | The Kubernetes feature that adds or removes replicas automatically as load changes. |
| Helm chart | A packaged, parameterised template for a set of Kubernetes resources; used to install the data plane. |
| Terraform | An infrastructure-as-code tool that creates and updates cloud resources from a declared description. |
| Infrastructure as code | Describing a deployment in version-controlled text files so it is reproducible and reviewable. |
| Region / availability zone | A geographically separate cloud location, and an isolated section within a region, both designed to fail independently. |
| Fallback policy (chain) | An ordered list of backends the gateway tries in turn when one fails. |
| Priority level | A backend's fixed position in a fallback chain; priority 0 is primary, higher numbers are fallbacks. |
AIGatewayRoute | The gateway resource that defines how a route's requests are dispatched, including the priority-ordered backends. |
| Per-request failover | Retrying down the priority chain inside a single request, rather than taking a backend out of service globally. |
Where to go next
The operational guides build on this foundation:
- Plan high availability and disaster recovery covers topology and recovery planning.
- Load balance across regional deployments covers spreading traffic across multiple data planes.
- Improve resilience with fallbacks is the developer-side configuration of the fallback chain.
- Sizing and scale covers how many replicas and regions a deployment needs.
Where to go next