Network and Security
Tetrate Agent Router is designed so that sensitive data stays inside the customer's own network and so that the customer's firewall never has to be opened to the internet on the platform's behalf. This page explains the network controls that make that possible: where the software runs, how it reaches AI providers privately, how network isolation is enforced, how traffic is encrypted, and how a Web Application Firewall can inspect requests inline.
This page assumes no prior networking background beyond a working familiarity with the platform's overall shape. Each control is introduced by describing what it is, why it matters, and how it fits the outbound-only design that keeps data inside the customer's environment. Terms are defined inline the first time they appear.
For the architectural foundation these controls sit on (the split-plane model and the outbound-only connection), see Planes and core components. For a quick lookup of any single term, see the Glossary.
The design goal: data stays in the customer's environment
Before the individual controls, it helps to state the property they all serve. Tetrate Agent Router divides into two halves, called planes. The management plane is hosted by Tetrate and holds configuration (routing rules, policies, provider settings) and the web applications. The data plane runs inside the customer's own infrastructure and is the only part that live AI traffic passes through. Because every prompt and every response travels through the data plane and nowhere else, the sensitive content of requests (which can include proprietary code, customer records, or internal documents) stays inside the customer's network boundary.
Two networking terms recur throughout this page:
- Ingress is traffic entering an environment from outside it. An inbound connection is one that some external party opens toward the environment.
- Egress is traffic leaving an environment toward the outside. An outbound connection is one the environment itself opens toward something external.
The controls below are shaped by a single rule: the data plane opens outbound connections when it needs to, but nothing on the internet is allowed to open an inbound connection into it. This is the outbound-only design, and it is what lets the platform run inside a locked-down network with no new holes in the firewall.
Intranet deployment
An intranet is a private network internal to an organisation, reachable by its own systems and staff but not exposed to the public internet. Running the data plane on an intranet is the most restrictive and most common enterprise posture, and Tetrate Agent Router supports it directly.
The data plane (the Controller and the Agent Router gateway) runs entirely inside the customer's Kubernetes cluster. Kubernetes is the industry-standard system for running containerised software; for the purposes of this page it can be read as "the customer's own compute environment". The only outbound connection the data plane requires is to the management plane, which the Controller uses to poll for configuration updates. That connection is a single outbound HTTPS link initiated from inside the customer environment. HTTPS is HTTP secured with encryption, described under Transport security below.
Two consequences follow from this arrangement:
- No inbound traffic from the internet to the data plane is required. Because the management plane never opens a connection into the customer environment, the customer's firewall needs no inbound rules for the platform. That is simpler to get approved by a security team, and it removes an entire class of attack surface (there is no exposed listener for an attacker on the internet to reach).
- If every client application that calls the gateway is itself internal (in the same VPC, the same cluster, or reachable over a corporate network), no public ingress is needed at all. A VPC, or virtual private cloud, is a logically isolated private network inside a cloud provider's infrastructure that the customer controls. When callers and gateway share such a network, the gateway can operate fully privately and still receive its configuration over the one outbound link.
The single outbound configuration link, and the fact that the data plane keeps routing even if that link is briefly interrupted, are covered in Planes and core components and Operations and reliability.
Private provider connectivity
The data plane still has to reach the AI providers that actually serve requests. By default that traffic travels over the public internet. When the data plane runs inside a cloud environment, each provider can instead be configured to be reached over private connectivity, so that model traffic stays on the cloud's internal backbone and never crosses the public internet.
Private connectivity is a cloud feature that exposes a service (here, an AI provider's endpoint) as a private address reachable only from inside the customer's own VPC, rather than as a public address reachable by anyone. Each major cloud offers its own version of this feature:
| Cloud | Private connectivity option | How to configure |
|---|---|---|
| Azure | Azure Private Link | Set the provider endpoint URL to the private-link FQDN for the Azure OpenAI resource |
| AWS | AWS PrivateLink / VPC Endpoints | Set the provider endpoint URL to the VPC endpoint DNS name for the AI service |
| GCP | Private Service Connect | Set the provider endpoint URL to the Private Service Connect endpoint for the AI service |
In each case the mechanism is the same in spirit: the cloud maps the provider's service to a private endpoint inside the customer's network, and the gateway dials that private endpoint instead of the public one. The value entered in the Admin Dashboard is a FQDN (fully qualified domain name, the complete host name of an endpoint, for example my-resource.privatelink.openai.azure.com) or a private DNS name that resolves only inside the customer's cloud network.
To configure a provider with a private endpoint:
- Establish the private connectivity in the cloud environment first (Azure Private Link, AWS PrivateLink or a VPC endpoint, or GCP Private Service Connect, depending on the cloud). The Admin Dashboard configures only the address the gateway dials; it does not provision the underlying network.
- In the Admin Dashboard, open Providers Management and add or edit the provider.
- Set the endpoint URL to the private endpoint address instead of the provider's public URL.
- Save and verify that the connection reports healthy. A private endpoint that resolves only inside the cloud network will fail verification if the data plane cannot reach it, which surfaces a misconfigured route or DNS record before any traffic depends on it.
The gateway routes requests to whatever endpoint URL is configured for a provider, so private endpoints work transparently with every routing feature: fallback policies, traffic splitting, and Bring Your Own Key (BYOK, the pattern where a caller supplies their own provider credentials). Private and public entries can coexist in the same deployment, so one provider can be reached privately while another, not subject to the same constraint, uses its public address.
The step-by-step operator flow for configuring private endpoints across several clouds and regions is in Connect provider subscriptions across clouds and regions.
Private-network enforcement
Private connectivity makes it possible to reach a provider privately. Enforcement is what makes it guaranteed that traffic cannot leak to a public provider. In Tetrate Agent Router, that enforcement reuses the same mechanism that controls routing: the fallback policy.
A fallback policy is an ordered list of backends (provider-and-model combinations) attached to an API key. Under normal operation the gateway sends requests to the first backend in the list; if that backend fails with a recoverable error, the gateway walks to the next one, and so on down the chain. The important property for network isolation is this: the gateway can only route to backends that appear in the chain. Nothing outside the chain is a candidate, regardless of what the calling application asks for.
That property turns the fallback chain into an enforceable network boundary. A team that requires all traffic to stay on private networks configures its fallback chains exclusively with private-endpoint providers. Because a public provider was never added to the chain, the gateway has no path to one; it cannot fall back to a provider that is not listed. There is no separate "private only" mode to enable and no flag that could be left off by mistake. The boundary is the composition of the chain itself.
How to design and attach a fallback chain, and how the same property is used for data-residency and approved-provider constraints, is covered in Improve resilience with fallbacks. Region-level residency guarantees are described in Compliance.
Transport security
The controls above govern where traffic goes. Transport security governs whether that traffic can be read or tampered with on the way. Every hop the gateway participates in can be encrypted, and where identity matters, mutually authenticated.
Two related terms are used here:
- TLS (Transport Layer Security) is the standard protocol that encrypts a network connection and verifies the identity of the server being connected to. It is the "S" in HTTPS. TLS protects data in transit from being read or altered by anything on the network path.
- mTLS (mutual TLS) extends TLS so that both ends of the connection present a certificate and prove their identity to each other, not just the server. This prevents an unauthorised client from connecting even if it can reach the endpoint on the network.
The gateway supports encrypted transport at every hop:
| Hop | Protection | What it achieves |
|---|---|---|
| Client to gateway | mTLS can be configured between client applications and the Agent Router | Mutual authentication at the ingress boundary, so only clients presenting a valid certificate are accepted |
| Gateway to self-hosted upstreams | mTLS is supported for connections to self-hosted or on-premises model endpoints | End-to-end mutual authentication between the gateway and an organisation's own model servers |
| Gateway to public providers | Traffic to public AI providers (OpenAI, Anthropic, Google, and others) is encrypted via TLS | Confidentiality and integrity of requests and responses as they travel to and from the provider |
The pattern reflects the trust model at each hop. Between components the customer controls (client to gateway, gateway to self-hosted models), full mutual authentication is available, so each side can insist on the other's identity. Toward public providers, where the provider's own certificate establishes the server's identity, TLS encrypts the connection.
WAF policies
The final layer inspects the content of requests for known attack patterns before they reach an AI provider. This is the job of a Web Application Firewall (WAF): a security control that examines incoming HTTP requests and blocks or logs those matching signatures of common web attacks, such as SQL injection (attempts to smuggle malicious database commands through an input field) or cross-site scripting (XSS, attempts to inject malicious script into a response).
Tetrate Agent Router provides WAF protection at the data plane through the gateway's own extensibility, rather than as a separate appliance in front of it. Two open-source pieces make this work:
- Envoy Gateway is the management layer for Envoy, the open-source proxy that the Agent Router gateway is built on. It exposes standard Kubernetes resources for extending the proxy's behaviour.
- A WebAssembly (Wasm) filter is a small, sandboxed program that runs inside the proxy as part of processing each request. WebAssembly is a portable binary format that lets such a program run safely and efficiently inline, without being a separate network service. The specific filter used here is the Coraza WAF Wasm filter, an open-source WAF engine.
Together these allow standard OWASP Core Rule Set (CRS) policies to be applied directly at the gateway. The OWASP CRS is a widely used, community-maintained set of WAF rules that detect common attack categories; OWASP (the Open Worldwide Application Security Project) is the organisation that publishes it. Support is delivered through Envoy Gateway Wasm extensibility combined with the Coraza filter above.
To enable WAF protection:
- Deploy the Coraza Wasm filter alongside the data plane using the Envoy Gateway
EnvoyExtensionPolicyresource. - Configure the desired rule sets, for example the OWASP CRS rules covering SQL injection, XSS, and other common attack vectors.
- The filter then inspects requests inline before they reach the AI provider, blocking or logging matches according to the configured policy.
Because the WAF runs as a Wasm filter inside the proxy rather than as a separate service in the request path, it adds no additional network hops, and it can be applied globally across the data plane or scoped to individual routes.
Key terms
The vocabulary introduced above, collected for reference:
| Term | Meaning |
|---|---|
| Intranet | A private network internal to an organisation, not exposed to the public internet. |
| Ingress / egress | Traffic entering an environment (ingress) and traffic leaving it (egress). |
| Inbound / outbound connection | A connection opened toward an environment from outside (inbound) or opened by the environment toward the outside (outbound). |
| Outbound-only design | The data plane opens connections outward but accepts no inbound connections from the internet, so no firewall openings are needed. |
| VPC (virtual private cloud) | A logically isolated private network inside a cloud provider that the customer controls. |
| Private connectivity | A cloud feature exposing a service as a private address reachable only inside the customer's network (Azure Private Link, AWS PrivateLink, GCP Private Service Connect). |
| FQDN (fully qualified domain name) | The complete host name of an endpoint, used as the provider endpoint URL. |
| Fallback policy / chain | An ordered list of backends attached to an API key; the gateway can route only to backends present in the chain. |
| TLS (Transport Layer Security) | The standard protocol that encrypts a connection and verifies the server's identity. |
| mTLS (mutual TLS) | TLS in which both ends present certificates and authenticate each other. |
| WAF (Web Application Firewall) | A control that inspects HTTP requests and blocks or logs known attack patterns. |
| WebAssembly (Wasm) filter | A sandboxed program that runs inline inside the proxy, adding no separate network hop. |
| OWASP Core Rule Set (CRS) | A community-maintained set of WAF rules for common attack categories. |
| Envoy Gateway | The management layer for Envoy, the proxy the gateway is built on, used to deploy the Wasm filter. |
Where to go next
- Deployment models: the Enterprise model that enables these private-connectivity options. See Deployment models.
- Guardrails: the content-safety layer that complements network security on the request path. See Guardrails.
Where to go next