Guardrails
A guardrail is a safety rule that inspects AI traffic as it passes through the gateway and can change or stop that traffic before it reaches a model, or before a model's reply reaches the application. In Tetrate Agent Router, guardrails run as a dynamic module inside the gateway rather than as safety code written into each application. Every request that crosses the gateway passes through the module, which reads the prompt, decides whether it satisfies policy, and acts on the verdict before the request reaches a provider. The same inspection applies to the response on the way back.
This page explains how that inspection works from first principles, defining each term as it appears. No prior familiarity with Tetrate Agent Router, with proxies, or with the internals of an AI gateway is assumed. The operator and developer guides linked at the end cover how guardrails are configured and used once the underlying mechanism is clear.
For the wider context this page sits in, the Architecture overview explains what the gateway is and where it runs, and Planes and core components describes the data-plane components (including the dynamic modules) in one place. The Glossary is a quick lookup for any single term.
What a guardrail is, and why it runs at the gateway
A large language model (LLM) takes a piece of text (a prompt) and returns generated text (a response). Prompts and responses are not always safe to pass along untouched. A prompt might contain Personally Identifiable Information (PII), meaning data that identifies a person, such as a name, an email address, a phone number, or a payment-card number. A prompt might also be an attack. A response might reproduce sensitive data, or return content the organisation does not permit.
A guardrail is a content rule that addresses these risks. It inspects a prompt or a response and can act on it: remove sensitive text, block the request entirely, or record that it happened. Placing that rule at the gateway (the single point every AI request already passes through) rather than inside each application means one rule protects every application at once, and applications need no safety code of their own.
The rest of this page explains the machinery that makes that inspection possible, and then how a single request moves through it.
Proxies, filters, and dynamic modules
Understanding guardrails means understanding a little about how the gateway is built.
A proxy is a server that sits between a client and a destination: it receives a request, does something with it, and forwards it on. The Agent Router gateway is a proxy. It receives each AI request, selects a provider, applies policy, and forwards the request to that provider. It is built on Envoy, a widely used open-source proxy.
A proxy like this handles a request through a filter chain: an ordered pipeline of small processing stages called filters. Each filter inspects or modifies the request in turn, and the request travels through the chain in sequence before being routed to its destination. A filter chain is how a proxy composes many independent behaviours (authentication, routing, rate limiting, safety checks) into one coherent path without any single filter needing to know about the others.
Adding a new behaviour to that pipeline has historically been awkward. One option was writing the logic in C++ and recompiling the entire proxy, which is slow to develop and hard to maintain. Another was using WebAssembly (Wasm), a portable format for running compiled code in a sandbox, or Lua scripts, both of which run more slowly than native code. A dynamic module is a modern alternative. It is add-on logic written as a high-performance extension (often in the Rust or Go programming languages) and loaded into the proxy at runtime as a shared library, a compiled file (for example a .so file) that a program loads and runs as part of itself rather than as a separate process. Because the module runs inside the proxy, it adds no extra network hop and executes at close to native speed.
Guardrails are one such dynamic module. Cost controls are another. The key property is inline processing: the module runs as part of handling the request, inside the same filter chain, rather than as a separate service the gateway has to call over the network and wait for.
The Guardrails module is one of the data-plane components. Its place alongside the Controller, Redis, and the optional Azure Content Safety service is described in Planes and core components.
How the Guardrails dynamic module works
The module operates in four stages within the filter chain: interception, evaluation, enforcement, and egress inspection. The first three apply to the prompt on its way to the model. The fourth applies the same logic to the response on its way back.
Stage 1: interception (the prompt)
When an application sends a prompt to an AI model, the request reaches the gateway first, not the provider. The Guardrails module intercepts the HTTP payload (the payload is the actual content the request carries, in this case the text of the prompt) before it is forwarded to the downstream provider, whether that provider is an external service such as OpenAI or a model the organisation runs itself.
Interception is what makes everything that follows possible: the module holds the prompt while it decides what to do, and nothing is forwarded until that decision is made.
Stage 2: evaluation
Once the module holds the payload, it analyses the content against the safety and security policies an operator has configured. Evaluation happens in one of two ways, and a single policy can combine both.
- Locally: the module runs lightweight checks directly in memory, inside the gateway. These include blocked-keyword lists, regular-expression (regex) patterns that recognise structured data such as Social Security numbers or credit-card numbers, and basic signatures for common attacks. Local checks add minimal latency because no network call is involved and the data never leaves the gateway.
- Remotely, by callout: for analysis that local pattern matching cannot perform, the module makes a callout, meaning it pauses the request and calls an external service, then resumes once the service replies. Agent Router can send the payload to Azure Content Safety, an external machine-learning content-analysis service, to check for categories such as toxicity, hate speech, or sophisticated attacks that simple patterns miss. A callout is more capable but slower than a local check, because it involves a network round trip.
Two categories of threat are worth naming here, because they drive what evaluation looks for.
- Prompt injection is text that tries to subvert the model rather than being unsafe in itself: instructions smuggled into the prompt (or into content the model is given to work with) that make the model ignore its own instructions, leak data, or act incorrectly.
- A jailbreak is a direct form of prompt injection in the user's own prompt, the familiar "ignore previous instructions" family of attacks and role-play framings intended to escape the model's configured behaviour.
Stage 3: enforcement (the action)
Based on the evaluation result, the module decides what happens to the request.
- Allow: the content satisfies policy. The filter chain continues, and the request is forwarded to the provider.
- Modify (redact): the module alters the payload before forwarding it. Redaction is the removal or masking of matched content, for example replacing a detected credit-card number with a placeholder such as
[REDACTED]. The request still proceeds, but the sensitive text never leaves the customer's environment. - Block: the content violates policy. The module stops the request, returns an error to the client indicating that the content was disallowed, and records the event. The provider never receives the request.
Stage 4: egress inspection (the response)
Guardrails are not limited to incoming prompts. Egress means traffic leaving toward its destination, and here it refers to the model's response on its way back to the application. When the model generates a response, that response flows back through the proxy, and the module inspects it using the same allow, redact, and block logic before it reaches the caller. Egress inspection is what catches a response that reproduces sensitive data or returns disallowed content, closing the gap that inspecting only the prompt would leave open.
The request path, end to end
Putting the four stages in order, a single guarded request moves like this:
- An application sends a prompt to the gateway. The Guardrails module intercepts the payload before it is forwarded (interception).
- The module evaluates the payload against policy, using local checks, a callout to an external service, or both (evaluation).
- The module acts on the verdict: it allows the request, redacts matched content and continues, or blocks the request outright (enforcement).
- If the request was allowed or redacted, the gateway forwards it to the provider, and the response returns through the proxy.
- The module inspects the response with the same logic before it reaches the application (egress inspection).
Every step happens inside the customer's data plane, the customer-managed environment that all AI traffic flows through. Local checks keep the data inside that environment entirely. A callout is the one point at which content is sent to an external service, and only when that level of analysis is switched on.
Why guardrails run as a module, not per application
Running guardrails inside a dynamic module rather than writing safety checks into each application has two consequences that motivate the design.
- Centralised governance: every application that uses the gateway is protected by the same rules automatically, with no per-application integration work. A rule changed once applies everywhere the moment it takes effect.
- Cost and safety at the edge: blocking a non-compliant or malicious prompt before it reaches the model means the provider is never invoked for a request that would fail policy. The organisation does not pay for a call it never wanted to make, and unsafe content never reaches the model in the first place.
Key terms
The vocabulary introduced above, collected for reference:
| Term | Meaning |
|---|---|
| Guardrail | A content rule inspected inline at the gateway that can allow, redact, or block a prompt or a response. |
| Prompt / response | The text sent to a model, and the text it returns. |
| Proxy | A server that receives a request, processes it, and forwards it on. The gateway is a proxy, built on Envoy. |
| Filter / filter chain | A single processing stage, and the ordered pipeline of such stages a request passes through in the proxy. |
| Dynamic module | High-performance add-on logic loaded into the proxy at runtime as a shared library, running inline with no extra network hop. |
| WebAssembly (Wasm) | A portable sandboxed code format; an older, slower way to extend a proxy than a dynamic module. |
| Shared library | A compiled file (for example a .so) that a program loads and runs as part of itself. |
| Inline processing | Work performed as part of handling the request, rather than in a separate service called over the network. |
| Payload | The content a request carries; here, the text of the prompt or response. |
| Personally Identifiable Information (PII) | Data that identifies a person, such as a name, email address, phone number, or payment-card number. |
| Redaction | Removing or masking matched content so the request can still proceed without the sensitive text. |
| Callout | A pause in processing to call an external service and resume on its reply; how remote evaluation is performed. |
| Azure Content Safety | An optional external machine-learning service the module can call out to for advanced content analysis. |
| Prompt injection / jailbreak | Text that tries to subvert the model; a jailbreak is a direct injection in the user's own prompt. |
| Egress | Traffic leaving toward its destination; here, the model's response inspected on its way back to the application. |
Where to go next
The operational configuration and developer-side use of guardrails are covered in the guides below:
- Configure vendor guardrails and Configure custom guardrails set up the policies the module enforces.
- Detect and redact sensitive data and Detect and block prompt injection cover the local and machine-learning-based checks.
- Protect requests with guardrails is the developer-side view.
Where to go next