Guardrails best practices and FAQ
Guardrails in Agent Router Enterprise run inline on model traffic that crosses the gateway: on both inputs and outputs to large language models (LLMs). Model Context Protocol (MCP) traffic is governed separately through identity, access control, and audit; content guardrails do not evaluate standalone MCP profile requests today. Rolling guardrails out safely means understanding how the guardrail provider kinds differ in latency and capability, promoting rules through monitor mode before enforcement, and choosing fail-open versus fail-closed behavior deliberately.
Persona: Platform operator or security stakeholder rolling out guardrails in production.
Estimated time: 15 to 20 minutes to read; ongoing for rollout and tuning.
Topics covered
- How the provider kinds (
regexp,builtin,tetrate) work and how they differ in latency and capability. - How to roll out guardrails without causing outages, using monitor mode and deliberate fail-open and fail-closed settings.
- How to investigate a blocked request and which permission to grant for read-only review.
- What guardrails protect at runtime, and what falls outside their scope.
Best practices
1. Roll out new guardrails in monitor mode first
Each rule has an action and a mode:
- Action: redact. Modifies the request or response inline, substituting anonymized placeholders or masking the matched content.
- Action: block. Stops the request and returns HTTP
403with an optional message (often including a correlation identifier). - Mode: monitor. The gateway records what it would have done without changing the request or response. Use this as a shadow mode to validate a rule against real traffic.
- Mode: enforce. Applies the configured action.
The most common cause of self-inflicted outages is promoting a rule straight to enforce with block before validating it. Start in monitor mode, confirm the trigger rate and false-positive rate are acceptable, then switch mode to enforce. The step-by-step configuration workflow is covered in Configure guardrails.
2. Order guardrails so cheap checks run first
Latency varies by provider kind:
regexp: near zero; in-process pattern matching against the Patterns catalog.builtin: near zero; in-process multi-layer keyword and regex pipeline.tetrate(and any other remote provider, including an external guardrail service): typically 100 ms or more, driven mostly by the network hop and inference.
Stack rules so cheap local providers evaluate first. A pattern match can short-circuit evaluation before a remote ML call ever runs, which keeps tail latency down on the common path. See Configure guardrail providers.
3. Set fail-open vs. fail-closed deliberately
Guardrails default to fail-closed. Both the failure behavior and the evaluation timeout are configurable per guardrail, not per rule. Choose based on the path: fail-closed when safety dominates, fail-open when availability dominates. Set an explicit timeout rather than relying on the default.
4. Scope code-execution guardrails narrowly
Guardrails that reject responses containing executable code will break any coding assistant if applied across the fleet. Apply them to contexts where generated code is never expected (a support agent, for example) rather than globally. See Protect coding assistant traffic for scoping guidance.
5. Treat tuning as a controlled change
Guardrails expose real configuration knobs. Tightening sensitivity too far produces false-positive blocks that affect end users. Validate threshold changes in monitor mode before applying them to live traffic, and restrict who can modify guardrail configuration (see the read-only permission below).
FAQ
Where do guardrails run? Does my data leave my network?
Local providers (regexp and builtin) run co-located with the AI Gateway, inside the deployment's own network. Data they evaluate does not leave that network. The tetrate provider is remote but still in-cluster, so its callout also stays inside the deployment's network. Content leaves the environment only where a rule names a third-party provider: azurecontentsafety, which sends content to Microsoft, or an external guardrail service, which sends it to whatever endpoint that service runs on. Neither exists in a project until an operator creates it. See Configure guardrail providers and Connect an external guardrail service.
Where does Agent Router Enterprise sit relative to an agent?
An agent is the combination of the client (the programmatic harness), the tools, and the LLM. Agent Router Enterprise sits between the client and the LLM and the MCP servers: in front of the model and the tools. Content guardrails inspect what the client sends to the model and what comes back on that model path. MCP profile traffic (/mcp/*) is subject to identity, access control, and audit, but not to content guardrails today. For the developer-side view of which traffic is protected, see Protect requests with guardrails.
What guardrail providers are supported?
Every project owns its guardrail providers, and a project is created already owning three of them, arranged as a cost-and-capability ladder:
regexp: fast pattern recognition for known formats such as email addresses and national identifiers. Uses the Patterns library; custom patterns are supported.builtin: a five-layer local keyword and regex pipeline (blocked keywords, allowlist gate, subject-plus-term co-occurrence, detection patterns, category keywords).tetrate: remote ML detection (embeddings, classifiers, NLI) for PII, toxicity, jailbreak, prompt injection, banned topics, and related check types.
Two more kinds are added deliberately, on the Guardrail Providers page. azurecontentsafety sends content to an Azure AI Content Safety resource and is the only kind that covers groundedness; it is offered as a catalog template rather than created with the project, because it needs an endpoint and a credential. An external guardrail service wraps a remote service, has no catalog template, declares the checks it supports per instance, and is connected by following Connect an external guardrail service. Configuration details are in Configure guardrails and Configure guardrail providers.
Can I plug in my own models?
The tetrate provider is backed by Tetrate's semantic router and ships with task-specific models for the supported check types. A customer's own model is connected as an external guardrail service: its checks are declared per instance and its wire format is the guardrail service contract. See Connect an external guardrail service.
Do guardrails run on outputs as well as inputs?
Yes. Guardrails evaluate both inputs and outputs on model traffic. PII can, for example, be redacted on the way in and disallowed content blocked on the way out. Standalone MCP profile traffic is not evaluated by content guardrails.
Note that not every check runs on both stages for every provider kind: some (provider, check) pairs are input-only or output-only, and a rule that leaves its Run on setting unset is asking for both stages. See the capability matrix.
Why was my rule or provider rejected with an invalid-argument error?
The API validates every guardrail rule and provider write against the Agent Router capability matrix: the check type must be supported by the provider's kind, and the requested stage (unset means both) must be within the stages that (kind, check) pair supports. Unsupported combinations are rejected at write time with an error naming the supported set, so a rule that could never fire cannot be created. Provider creation likewise rejects unknown kinds, and external-service providers must declare a valid, non-empty check list. See Write-time validation for the full contract.
An existing guardrail rule never fires. Why?
Rules created before write-time validation existed are grandfathered: they still run, but an unsupported (provider kind, check type, stage) combination scores zero and never fires. The management plane logs a warning (with the rule ID, provider kind, check type, and stage) whenever it ships such a rule, and the Admin Console shows an unsupported stored check as disabled in the rule form. The troubleshooting steps are in Rules that never fire.
A user was blocked and believes they shouldn't have been. How do I investigate?
Blocked responses carry a correlation identifier. The user provides that identifier, and an operator with the appropriate privileges can look up which rules triggered, the confidence score, and the actual input and output that caused the block. Use the correlation identifier to join against the organization's own logging system for the full request context. Request Logs and audit workflows are covered in Audit Agent Router activity.
How is read-only access to guardrail activity granted?
Through the guardrail_providers.read permission; there is no dedicated guardrail-reader role. A read-only grant of that permission lets security teams read which rules triggered and review guardrail activity without being able to modify guardrail settings. Reading unredacted inputs and outputs requires sufficient privilege.
Can guardrails enforce a budget or cost cap?
Guardrails do not, but budgets carry their own opt-in enforcement. A budget tracks spend against a limit and applies its configured action: the default Watch spend alerts without affecting traffic, Hard stop refuses requests once the limit is reached, and Degrade gracefully reroutes them to a cheaper model. Where the concern is that a mis-set limit could cause an outage, Degrade gracefully is the safer enforcing choice. To constrain abuse or runaway usage, rate limits are the separate control: token ceilings per hour, set on a project or an API key (organization and tag scopes are available through the management API). Request ceilings and per-minute windows arrive in 2026 Q4. See Working with budgets.
How are logs and metrics handled?
Request logs capture the full request and response payloads, subject to the deployment's retention and content-mode settings, which govern what is stored and for how long. Metrics are exposed on a Prometheus scrape endpoint for ingestion into an external monitoring system, and audit logs (configuration changes, administrative interface access, user management) can be written out for ingestion into a security information and event management (SIEM) system.
Can guardrails handle non-English or obfuscated attacks?
Shipped providers cover common cases. A language-allowlist rule (language check type) can block any request that is not in an approved language, which is a useful control for high-risk locales. Exotic evasion (rare languages, novel encodings, or techniques like poetry injection) may require a remote or external provider beyond local pattern matching. For jailbreak and injection detection specifically, see Detect and block prompt injection. This is an ongoing area: new evasion techniques appear regularly, and coverage is a moving target.
What do guardrails protect, and what is out of scope?
Agent Router Enterprise is a runtime layer. Content guardrails protect the model request and response path: traffic to and from an LLM. That path is, in effect, data loss prevention (DLP) that does not itself require inference. Standalone MCP profile traffic (/mcp/*) is out of content-guardrail scope today; it is governed by identity, access control, and audit instead.
Out of scope:
- Training-time and pre-deployment concerns: data poisoning, manipulation of model weights, and anything in the artificial intelligence software development lifecycle before the model is served.
- Client-side activity: lateral movement that manipulates the client itself, or data that never crosses the network (for example, a local file read by a coding agent).
- Direct client-to-retrieval access: the product protects data on its way to the model, but not the gap between a retrieval-augmented generation (RAG) or vector store and the client that queries it directly. If the same data is exposed over MCP instead, Agent Router can apply access-layer controls (identity, ACL, audit) on the MCP path; content guardrails apply only when that data later rides the model path.
- Standalone MCP profile traffic: client calls to
/mcp/*are not inspected by content guardrails. Govern which servers a profile may reach through Govern MCP server access. - Supply chain: Agent Router Enterprise ships as versioned Helm charts with pinned component versions. The rest of the supply chain (the agents, application programming interfaces (APIs), and MCP servers chosen for deployment) remains the deploying organization's responsibility. Version Agent Router configuration through automation as described in Manage configuration as code.
Where to go next
Configure guardrails
Define project policies and rules and pilot them in monitor mode before enforcement.
Configure guardrail providers
Choose regexp, builtin, or tetrate for each check type.
Protect requests with guardrails
The developer-side view of attaching a guardrail to a specific request path.
Detect and block prompt injection
Handle jailbreak and indirect injection separately from content guardrails.