Data flows
Tetrate Agent Router moves three different kinds of data, and each kind travels its own route. The three routes are configuration polling (how the gateway learns its rules), the prompt and response request path (the live AI traffic itself), and the observability pipeline (the operational data recorded about that traffic). Keeping them separate means a configuration update, a live AI request, and a telemetry export never share a channel, which makes the system easier to secure, to reason about, and to operate.
This page explains each of those three flows from first principles, defining every term as it appears. No prior familiarity with Tetrate Agent Router is assumed. A data flow is simply a category of information that moves between two points in a predictable direction: from where it is produced to where it is consumed. Understanding the three flows, and why they are deliberately kept apart, is the clearest way to understand how the platform protects sensitive content while still staying centrally configurable and fully observable.
These flows cross the two halves of the platform (the data plane and the management plane) that are introduced in Planes and core components. That page is useful background, but the essentials are restated here so this page can be read on its own.
The two places data moves between
Before the flows themselves, two terms are worth fixing in place, because every flow is described in relation to them.
- The data plane is the part of the platform deployed inside the customer's own Kubernetes cluster (Kubernetes is the industry-standard system for running containerised software). All live AI traffic passes through the data plane, and nowhere else. Its central component is the AI gateway, the server that every request travels through on its way to a provider.
- The management plane is hosted and operated by Tetrate. It stores all configuration (routing rules, policies, provider settings, user accounts) and exposes the two web applications used to manage the platform. No live AI traffic ever passes through the management plane.
A provider is a service that hosts large language models, for example OpenAI, Anthropic, Google, Azure, Amazon, or Perplexity. A large language model (LLM) is an AI model that takes a piece of text (a prompt) and returns generated text (a response).
The diagram below shows all three flows spanning both planes at once.
Configuration flow: how the gateway learns its rules
The configuration flow carries the rules the gateway needs in order to do its job: which providers exist, which models are available, how requests should be routed, what spending and content policies apply, and which credentials to use. Those rules are decided and stored in the management plane, but they have to reach the gateway in the data plane before they can take effect.
This is handled by the Controller, a component that runs alongside the gateway in the data plane and acts as the bridge between the two planes. The Controller performs configuration polling: on a regular schedule it reaches out to the management plane, asks for the latest configuration, and receives whatever has changed. Polling means the data plane repeatedly asks for updates on its own timer, rather than waiting for the management plane to push them. Once it has the update, the Controller translates it into settings the gateway understands and applies them, keeping routing rules, policies, and credentials current.
The direction of this connection is deliberate and important. It is outbound only, meaning the data plane always initiates the connection out to the management plane, and the management plane never initiates a connection back into the customer's environment. In practical terms, the customer's firewall (the network barrier that controls what may enter and leave their environment) needs no inbound openings for Tetrate Agent Router at all. A firewall that only has to allow outbound connections is simpler to approve, and it reduces attack surface: the number of ways an outside party could try to reach into the environment. Because inbound access is never required, there is simply nothing there to attack.
One consequence of this design is worth noting: the data plane keeps working even if the connection to the management plane is briefly interrupted. It continues routing requests using the configuration it already holds, and it picks up any changes on the next successful poll.
The Controller and its role as the bridge between the planes are described further in Planes and core components.
Request flow: the live prompt and response traffic
The request flow is the live AI traffic itself, and it is the only one of the three flows that carries the actual content of prompts and responses. Requests originate from client applications (the programs an organisation runs that need to call an AI model), pass through the gateway, and are forwarded to the selected provider. Responses travel the same route in reverse.
Along the way, the gateway does several things to each request in turn, all inside the data plane:
- It authenticates the request and identifies which caller it belongs to, using the API key the request carries. An API key is a credential that names and authorises the caller.
- It applies routing decisions: choosing which provider and model should serve the request, based on the configuration delivered by the configuration flow.
- It enforces rate limiting. A rate limit is a cap on how much a given key may consume in a period, measured in requests or in tokens (a token is the unit of text that LLMs measure work in, roughly a few characters each). Rate limiting protects both the provider and the customer's budget from a single caller sending far more traffic than intended.
- It applies guardrails. A guardrail is a content rule enforced inline as part of handling the request, for example removing personally identifiable information from a prompt or blocking a disallowed request. Guardrails can inspect the response on the way back as well.
- It attaches the correct provider credential and forwards the request to the chosen provider. If that provider fails or is unreachable, the gateway can walk to the next backend in a configured fallback order, within the same request, with no action needed from the application.
The defining property of this flow is where it stays. Prompt and response content never leaves the customer's environment except to reach the provider that was selected to serve the request. It never passes through the management plane. Because the gateway runs in infrastructure the customer controls, sensitive content (which can include proprietary code, customer records, or internal documents) stays inside the customer's network boundary. This is the central reason the platform separates the planes at all.
What happens to a request inside the gateway is covered step by step in Guardrails, and the gateway's normalised errors and per-request identifiers are documented in Gateway behavior.
Observability flow: the operational record of traffic
The observability flow carries the operational record of what the platform is doing, so that it can be watched, measured, and alerted on. It carries no prompt or response content by default; it carries facts about the traffic. Three kinds of data travel this flow:
- Telemetry is the general term for operational data recorded about traffic, such as latency (how long a request took), token counts, cost, and the outcome of each request.
- Metrics are numeric measurements aggregated over time, such as total requests, error rates, and latency percentiles. Metrics are the raw material for dashboards and alerting rules.
- Structured logs are individual event records in a consistent, machine-readable format, so that tools can search and correlate them rather than parsing free text.
The flow is assembled from two stages. First, a Grafana Alloy agent running locally in the data plane collects the observability data. Grafana Alloy is a widely used open-source collector: an agent that scrapes (periodically pulls) data from the components that produce it, in this case the gateway, the Controller, and the Redis instance the gateway uses for rate-limit counting. Second, the collected data is forwarded to an OpenTelemetry (OTEL) Collector. OpenTelemetry is an open, vendor-neutral standard for describing and transporting telemetry; the OTEL Collector is a routing hub that receives telemetry in that standard format and forwards it onward.
From the OTEL Collector, the telemetry is routed to whichever monitoring backends the organisation already runs. A monitoring backend is a platform that stores telemetry and turns it into dashboards, searches, and alerts. Common examples are Grafana (open-source dashboards and visualisation), Datadog (a hosted monitoring platform), and New Relic (a hosted observability platform). Because the platform emits telemetry in the open OpenTelemetry format rather than a proprietary one, it slots into the tooling an observability team already operates, instead of forcing a second, separate console.
The point of this two-stage design (a local collector that scrapes, feeding a collector that routes) is that the platform does not decide where telemetry ultimately lives. It emits telemetry in an open format and lets the organisation's existing stack absorb it, which is why storage, retention, and alerting rules stay with the destination platform rather than inside Tetrate Agent Router.
The operational detail of reading this data lives in Monitor traffic and usage (the in-Console surfaces) and Export telemetry to an observability stack (sending it to an external backend). The exact metrics and span attributes are listed in OpenTelemetry metrics.
The three flows at a glance
The flows differ in direction, in what crosses them, and in who consumes the result. The request flow is the only one that carries prompt and response content, and it never leaves the customer environment except to reach the selected provider. Reading the table by column shows why the separation matters: no single channel mixes configuration, live content, and operational data, so each can be secured and scaled on its own terms.
| Flow | Direction | Carries | Consumed by | Why it is separate |
|---|---|---|---|---|
| Configuration | Data plane to management plane, outbound only | Policy, routing, and credential updates | The Controller, which applies them to the gateway | Keeps the gateway in sync with no inbound connections into the customer environment |
| Request | Client to gateway to provider, and back | Prompts and responses (sensitive content) | The client application, via the provider | Isolates live content so it stays inside the customer boundary, never touching the management plane |
| Observability | Data plane to the organisation's monitoring backends | Telemetry, metrics, structured logs | Grafana, Datadog, New Relic, and similar | Lets existing monitoring tools absorb operational data without carrying live traffic |
The configuration and observability flows are covered in operational detail in Monitor traffic and usage and Export telemetry to an observability stack.
Where to go next
- Planes and core components: the planes these flows cross and the components that produce them. See Planes and core components.
- Guardrails: what happens to a request inside the gateway on the request flow. See Guardrails.
Where to go next