# Tetrate Agent Router documentation > Enterprise AI gateway for routing, governance, and operations across AI models, agents, and MCP servers. This file contains all documentation content in a single document following the llmstxt.org standard. ## AI gateway for agents and MCP export const Pane = ({eyebrow, name, tagline, bestFor, ctaLabel, ctaHref, featured}) => ( {eyebrow} {name} {tagline} {bestFor} {ctaLabel} ); # An AI gateway for agents, models, and MCP servers Point your apps at one endpoint that speaks both the OpenAI and Anthropic APIs, and your agents at one MCP endpoint. Route to any model, fail over automatically, enforce rate limits and guardrails, track spend against budgets, and attribute spend by team and agent. Control which MCP tools each agent can reach, and manage policies centrally across a fleet of AI gateways. Provider-neutral, from a quick trial through enterprise rollout. ## Get started by product Not sure which product is right for you? Compare features → Agent Router Enterprise,Fully Managed} tagline="Team-scale governance on a dedicated instance, hosted end to end by Tetrate." bestFor="Best for getting admin controls without running infrastructure." ctaLabel="Fast Track Eval" ctaHref="/agent-router-enterprise/fully-managed/fast-track/" featured /> Agent Router Enterprise,Self-Hosted Data Plane} tagline="Same controls and dedicated management plane, but with data path on self-managed infra." bestFor="Best for teams that must keep data on premises." ctaLabel="Quick start" ctaHref="/agent-router-enterprise/self-hosted-data-plane/" /> ## Choosing the right Agent Router product {#which-is-right-for-you} This table compares Agent Router Service with Agent Router Enterprise. Within Enterprise, Fully Managed and Self-Hosted Data Plane share the same features and differ only in where the data plane runs: Fully Managed is hosted end to end by Tetrate; Self-Hosted Data Plane keeps AI traffic on your infrastructure while Tetrate still hosts the management plane. export const Yes = () => ( ✓ ); export const No = () => ( ); Developers Agent Router Service Operators & AI leaders Agent Router Enterprise AI Gateway with multi-model routing and auto-failover Model Context Protocol (MCP) Gateway: connect agents to tools securely Per-key token usage and cost logs OpenAI-compatible API: works with existing code Cross-team cost attribution, showback, and chargeback Admin controls: model and MCP access profiles by team Runtime AI Guardrails: PII redaction and policy enforcement Enterprise SSO: every request carries authenticated identity Distributed deployment: Fully managed Cloud, hybrid dataplane deploy at edge, per-region, or provider ## Connect to any provider and tool One endpoint fronts every model provider and slots into the SDKs, agents, and observability stack already in use. Supported providers The 200+ models and provider families the gateway routes to, how each provider is authenticated, and where the live model catalog is published. Browse providers → Integrations Everything that plugs into the gateway: SDKs and frameworks, coding agents, guardrail vendors, observability backends, and MCP clients. Explore integrations → ## Keep exploring Where to begin Role-based paths from sign-in to a first routed request, for developers and platform operators. Product overview What Agent Router is, how the architecture fits together, and the key concepts behind it. Reference Gateway APIs, supported providers, integrations, the SDK, and the CLI. --- ## Where to begin # Where to Begin This page is the way into the documentation. It covers the three things readers arrive wanting: the ordered path through a first setup, a task index for finding one specific guide, and a map of how the rest of the documentation is arranged so that the right section becomes predictable rather than a guess. New here Set it up Requirements, installation, and first request, in the order they are done. Already running Find a task Configuring SSO, provisioning models, setting a budget, and the rest, by goal. Checking a detail Look something up API surfaces, CLI flags, metric names, provider lists, and definitions. ## Which product {#which-product} Three products end at the same place, a working gateway serving a routed request. They differ in tier and in where the data plane runs. Developers Agent Router Service Multi-tenant and hosted by Tetrate. Nothing to install. 10 minute Quickstart → Teams Enterprise, Fully Managed A dedicated instance with operator controls, hosted end to end by Tetrate. Fully Managed → Teams Enterprise, Self-Hosted Data Plane The same controls, with AI traffic kept on customer-managed infrastructure. Self-Hosted Data Plane → Where the choice is still open, the [feature comparison](/#which-is-right-for-you) on the welcome page sets Service against Enterprise, and the flowchart under [Still choosing a product](#still-choosing-a-product) walks the deployment constraints. ## First time here {#first-time-here} Each path below runs from requirements through installation to a first routed request. The steps are ordered, so working down a column is the shortest route to a working deployment. Agent Router Service Nothing to install. Sign in and route a request. 10 minute Quickstart Dev Console quickstart Make an AI API call Guides for Developers Enterprise, Fully Managed Tetrate provisions and hosts everything. Onboarding configures it. Fast track evaluation Management Plane onboarding Admin Dashboard quickstart Enterprise guides Enterprise, Self-Hosted Data Plane Read the requirements first. The install runs in a customer-managed cluster. Prerequisites Self-Hosted onboarding Data plane installation Gateway Installation Admin Dashboard quickstart For a self-hosted install, [Prerequisites](/agent-router-enterprise/self-hosted-data-plane/prerequisites/) is the page to read before anything else. It covers the Kubernetes version, the connections a cluster needs, what a security review will ask for, and which version to install. ## Find a task {#find-a-task} Nine areas cover the operational surface. Each card opens that area's guides. Access and identity Single sign-on, role mapping, roles and permissions, developer onboarding. Models and providers Provider connections, enabling models, Bedrock, self-hosted models, BYOK. Routing and resilience Routing across providers, fallback chains, traffic splitting, integration. Cost and quotas Budgets, ceilings, rate limits, attribution, chargeback, spike alerts. Safety and guardrails PII redaction, prompt injection, vendor and custom guardrails. MCP and agents MCP profiles, server governance, OAuth clients, secrets and identity. Observability Usage analytics, request logs, OpenTelemetry export, custom backends. Projects and gateways Projects, members, project keys, gateway provisioning, data planes. Compliance and audit Data residency, log retention, audit trails, SIEM export. ### Most requested tasks | Task | Guide | | --- | --- | | Set up single sign-on | [Configure SSO](/agent-router-enterprise/guides/identity-and-access-management/single-sign-on/sso-configuration-guide/) | | Map identity provider claims to roles | [Configure SSO role mapping](/agent-router-enterprise/guides/identity-and-access-management/single-sign-on/sso-role-mapping-guide/) | | Drive access from existing Entra ID groups | [Map Entra groups to functions](/agent-router-enterprise/guides/identity-and-access-management/single-sign-on/map-entra-id-groups-to-business-functions/) | | Check what each role is allowed to do | [Manage roles and permissions](/agent-router-enterprise/guides/identity-and-access-management/manage-roles-and-permissions/) | | Add developers and issue them keys | [Onboard developers and issue keys](/agent-router-enterprise/guides/operate-and-govern/onboard-developers-and-issue-keys/) | | Add a provider and enable its models | [Provision models and providers](/agent-router-enterprise/guides/operate-and-govern/provision-models-and-providers/) | | Use AWS Bedrock models | [Provision AWS Bedrock models](/agent-router-enterprise/guides/operate-and-govern/provision-models-and-providers/provisioning-aws-bedrock-models/) | | Add a self-hosted or custom model | [Custom and self-hosted models](/agent-router-enterprise/guides/operate-and-govern/provision-models-and-providers/provision-custom-and-self-hosted-models/) | | Let callers supply their own provider key | [Use your own provider credentials](/agent-router-service/guides/use-your-own-provider-credentials/) | | Send requests to several providers through one endpoint | [Route requests across providers](/agent-router-service/guides/route-requests-across-providers/) | | Fail over automatically when a provider errors | [Improve resilience with fallbacks](/agent-router-service/guides/improve-resilience-with-fallbacks/) | | Split traffic between two models | [Reduce cost with traffic splitting](/agent-router-service/guides/reduce-cost-with-traffic-splitting/) | | Route on a logical model name | [Apply advanced routing rules](/agent-router-service/guides/apply-advanced-routing-rules/) | | Point an application or coding assistant at the gateway | [Integrate the gateway with an app](/agent-router-service/guides/integrate-the-gateway-with-an-app/) | | Decide between budgets, rate limits, and tags | [Choose the right cost control for each workload](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/choose-the-right-cost-control/) | | Cap what a team can spend | [Keep each team's spend inside an agreed ceiling](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/keep-team-spend-inside-a-ceiling/) | | Attribute cost to an app or project | [Know what every app and project actually costs](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/know-what-every-app-and-project-costs/) | | Bill spend back to the teams that incurred it | [Bill AI spend back to the teams that incur it](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/bill-ai-spend-back-to-teams/) | | Stop a runaway workload | [Stop runaway workloads before they burn the budget](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/stop-runaway-workloads/) | | Contain a leaked API key | [Contain a leaked key before it drains the budget](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/contain-a-leaked-key/) | | Get alerted when spend spikes | [Get alerted to cost spikes as they happen](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/get-alerted-to-cost-spikes/) | | Understand what guardrails do | [About guardrails](/agent-router-enterprise/guides/operate-and-govern/safety-and-data-protection/about-guardrails/) | | Redact personal data from prompts | [Detect and redact sensitive data](/agent-router-enterprise/guides/operate-and-govern/safety-and-data-protection/detect-and-redact-sensitive-data/) | | Block prompt injection | [Detect and block prompt injection](/agent-router-enterprise/guides/operate-and-govern/safety-and-data-protection/detect-and-block-prompt-injection/) | | Protect coding-assistant traffic | [Protect coding-assistant traffic](/agent-router-enterprise/guides/operate-and-govern/safety-and-data-protection/protect-coding-assistant-traffic/) | | Give agents a single MCP endpoint | [Aggregate MCP servers into a profile](/agent-router-service/guides/aggregate-mcp-servers-into-a-profile/) | | Control which MCP servers are reachable | [Govern MCP server access](/agent-router-enterprise/guides/operate-and-govern/govern-mcp-server-access/) | | Create a project | [Create and delete a project](/agent-router-enterprise/guides/operate-and-govern/project-and-data-plane-management/create-a-project/) | | Provision a gateway | [Provision a gateway for a project](/agent-router-enterprise/guides/operate-and-govern/project-and-data-plane-management/provision-a-gateway/) | | Issue a project-scoped API key | [Issue a project-scoped API key](/agent-router-enterprise/guides/operate-and-govern/project-and-data-plane-management/issue-a-project-api-key/) | | Add or remove project members | [Manage project members and access](/agent-router-enterprise/guides/operate-and-govern/project-and-data-plane-management/manage-project-members/) | | Install a data plane in a cluster | [Data plane installation](/agent-router-enterprise/self-hosted-data-plane/data-plane-installation/) | | Investigate one request | [Monitor traffic and usage](/agent-router-service/guides/monitor-traffic-and-usage/) | | Get the first readings out of analytics | [Usage Analytics Onboarding](/agent-router-enterprise/guides/observability-and-analytics/usage-analytics-onboarding/) | | Send traces and metrics to Grafana or Datadog | [Export telemetry to observability](/agent-router-enterprise/guides/observability-and-analytics/export-telemetry-to-an-observability-stack/) | | Review who changed what | [Audit Agent Router activity](/agent-router-enterprise/guides/operate-and-govern/audit-platform-activity/) | | Export audit records to a SIEM | [Export audit decisions to SIEM](/agent-router-enterprise/guides/operate-and-govern/compliance-and-governance/export-audit-and-policy-decisions-to-a-siem/) | | Keep data in a particular region | [Configure data residency](/agent-router-enterprise/guides/operate-and-govern/compliance-and-governance/configure-data-residency-and-no-retention/) | Anything not listed is usually fastest to reach through the search box in the top navigation, which indexes the full text of every page. ## How these docs are organised The sidebar has four sections below this page. Knowing what each one holds removes most of the guesswork. | Section | What it holds | Go here when | | --- | --- | --- | | [Agent Router Service](/agent-router-service/guides/) | Quickstarts and developer guides for the multi-tenant hosted product | An application or agent is being pointed at the gateway | | [Agent Router Enterprise](/agent-router-enterprise/guides/) | Operator guides, plus the Fully Managed and Self-Hosted install paths | Agent Router is being run and governed for a team | | [Product & Architecture](/product-architecture/product-overview/) | Planes and components, data flows, network and security, key concepts, evaluation material | The design has to be understood or explained to someone else | | [References](/reference/) | Gateway and management APIs, the CLI, the SDK, providers, glossary, best practices | An exact name, value, or schema is needed | Two rules of thumb hold across the set. Anything phrased as a goal ("keep team spend inside a ceiling") is a guide and lives under Service or Enterprise. Anything phrased as a thing ("audit log events", "OpenTelemetry traces and metrics") is reference material and lives under References. ## Look something up {#look-something-up} Glossary Definitions for data plane, fallback policy, MCP profile, guardrail, and the rest. Gateway APIs The inference endpoints the gateway speaks, with request and response shapes. Supported Providers The provider families the gateway routes to and how each is authenticated. Management API reference Generated reference for every control-plane service, from the OpenAPI specs. Agent Router CLI Reference Every `tare` command and flag, including install, upgrade, and diagnostics. OpenTelemetry traces and metrics Exact metric and span names for dashboards and alerting rules. ## Still choosing a product? {#still-choosing-a-product} The flowchart picks a path from the deployment constraints, chiefly where the data is allowed to go. Deployment models Where each part runs and who operates it, explained from first principles. Compliance Residency, retention, and the certifications an assessment tends to ask about. Evaluate Agent Router Run a proof of concept end to end, with scenarios and a scorecard. --- ## Aggregate MCP servers into a profile The Model Context Protocol ecosystem has expanded rapidly. A productive AI client today typically wants access to a file-system server, a code-search server, a ticketing-system server, a documentation server, sometimes a browser server, sometimes a shell server, and a handful of organisation-specific servers wrapping internal APIs. Wiring each of those servers into each AI client individually (one block of MCP configuration in Claude Code, the same block again in Cursor, and again in VS Code) produces unmanageable per-client configuration and offers no central control over what is actually reachable. It also forecloses the question of governance: each direct connection bypasses Agent Router entirely, so identity, audit, and access control do not apply. An MCP profile is Agent Router's answer to that fan-out. A profile is a named collection of MCP servers exposed through a single Agent Router URL. The AI client is configured once, against the profile URL; the profile decides which servers and which tools it exposes; and every request flowing through the profile is subject to Agent Router's identity, observability, and policy model. Adding a new MCP server to an organisation becomes a profile edit rather than a coordinated change across every developer's local setup. This guide covers how a profile is created, how security is layered on top of it, and how the major MCP clients are wired in. **Persona:** Developer working in the Developer Console. **Estimated time:** 15 to 20 minutes, including client-side configuration. ## When this guide applies A profile is the right level to engage with when any of the following is true: | Situation | Why a profile helps | | :---- | :---- | | Multiple AI clients need access to the same set of MCP servers | The clients all connect to one URL; server membership is edited centrally | | The set of available MCP servers changes frequently | Adding or removing a server is a profile edit, with no per-client configuration drift | | Access to MCP traffic needs to be governed | The profile applies Agent Router's identity, audit, and access-control model to every request | | Different developers need different subsets of the available servers | Multiple profiles can be defined, each exposing the subset appropriate to its consumer | For a single, ad-hoc server connection from a single client, profiles are still useful, but the operational benefit is smaller. The case grows quickly with the number of servers and clients in play. ## Outcomes By the end of this guide: - An MCP profile exists in the Console, exposing one or more MCP servers from the MCP catalogue. - The profile has an appropriate security type configured (API Key for programmatic use, Built-in OAuth for interactive clients, or another option as appropriate). - At least one MCP client (Claude Code, Cursor, or VS Code) is connected to the profile URL. - A tool invocation through the profile has completed successfully and is visible in Request Logs. ## Prerequisites - A working API key with a routing configuration attached, as set up in [Route Requests Across Providers](/agent-router-service/guides/route-requests-across-providers/). - One or more MCP servers already registered in the MCP catalogue by a platform operator. The operator-side step (adding servers to the catalogue and, where necessary, configuring OAuth clients for server-side authentication) is covered by the *Govern MCP Server Access* guide in *Guides for Platform Operators*. If the catalogue is empty, that work has to happen first; the Console exposes only what the catalogue holds. - An MCP-compatible client installed locally: Claude Code, Cursor, or VS Code with an MCP extension. ## Step 1: create the profile A profile is the unit of configuration on the developer side. Each profile has a name, a security type, a collection of server-tool selections, and a single URL that clients connect to. 1. Sign in to the Developer Console. 1. Open **Settings → MCP Profiles** from the sidebar. 1. Click **Create Profile**. 1. Enter a descriptive profile name. The name appears in Request Logs and is surfaced in the connection UI of MCP clients, so a phrase tied to the profile's purpose (`dev-tools`, `research-assistant`, or `support-team-kit`) is more useful than a generic value. 1. Proceed to security type selection. ![Create an MCP profile](/img/screenshots/console/mcp-profile-create.jpeg) The profile name and URL are visible only to the Console account that owns the profile; profiles are not shared across users automatically. Sharing a profile with another developer is a matter of sharing the profile URL plus the appropriate credentials. ## Step 2: choose the security type The security type controls how MCP clients authenticate to the profile URL. The right choice depends on whether the consumer is a person sitting at an interactive client, a piece of automation, or a third-party tool with its own SSO integration. | Security type | Authentication model | When to use it | | :---- | :---- | :---- | | **API Key** | The client presents an Agent Router API key as a bearer token | Programmatic access, CI pipelines, environments where an interactive OAuth flow would be intrusive | | **Built-in OAuth** | The client runs Agent Router's own OAuth flow on first use | Interactive clients (Claude Code, Cursor, VS Code) where a browser prompt at first use is acceptable | | **Custom OAuth** | The client authenticates through the organisation's own OAuth or OIDC provider | Environments with established SSO that should govern MCP access too | | **None** | No authentication is enforced on the profile URL | Local development on trusted networks only; not appropriate for any shared or production use | The choice can be revisited later, since security type is editable on the profile, but changing it does invalidate the previous connection method, so existing clients have to be reconfigured against the new model. :::caution The profile-level security type is independent of any **server-level** authentication the underlying MCP servers may require. A profile secured with Built-in OAuth still forwards requests to its servers with whatever credentials those servers expect on their own backends. Both layers operate, and both have to be correct for a tool call to reach the upstream MCP server. ::: ### Server authentication The server-level authentication each MCP server expects on its own backend is configured when the server is added to the profile. This layer is separate from the profile-level security type above. The MCP client only authenticates to the profile URL; Agent Router forwards each request to its server with the appropriate credentials. | Server auth | Description | | :---- | :---- | | **Open Access** | The MCP server requires no authentication | | **Bearer Token** | A static token passed to the MCP server, configured when the server is added to the profile | | **OAuth** | An OAuth flow between Agent Router and the MCP server backend | ## Step 3: add servers and choose tools Once the security type is set, the profile is populated with servers and the tools those servers expose. ### Browse the MCP Catalog The MCP Catalog, reached from **Catalog → MCP Catalog**, lists every server a platform operator has registered. The catalogue is presented as a grid of server cards; a search bar filters the grid by name. Each card shows the server name, a short description, and the auth type the server requires: **Open Access**, **Bearer Token**, or **OAuth**. Clicking a card opens a detail view listing the tools the server exposes and its authentication requirements. The catalogue is browse-only; servers are added to a profile from the profile editor, not from the catalogue page. The set of catalogued servers is managed by the platform operator and may grow over time. Examples of servers commonly available in the catalogue: | Server | Description | | :---- | :---- | | Google Drive | Access and search files in Google Drive | | Slack | Send messages and interact with Slack workspaces | | GitHub | Access repositories, issues, and pull requests | | Stripe | Query payment data and manage Stripe resources | | HuggingFace | Browse and interact with HuggingFace models and datasets | | Salesforce | Access Salesforce CRM data and records | | AWS Knowledge | Query AWS Knowledge Bases | | Context7 | Access up-to-date library documentation and code examples | ![MCP Catalog](/img/screenshots/console/mcp-catalog.jpeg) 1. In the profile editor, click **Add Server**. 1. The catalogue lists every MCP server the platform operator has registered, along with the tools each one exposes. 1. Select a server. 1. Review the tools it advertises and choose which ones to include in this profile. Selecting all is appropriate when the profile is meant to be a general-purpose toolkit; selecting a subset is appropriate when the profile is meant to be a focused, role-specific kit. 1. Repeat **Add Server** for each additional server the profile should expose. 1. Review the summary panel showing every selected server and tool. 1. Click **Save** or **Create** to finalise the profile. :::caution A single profile supports a maximum of **32 tools** across all of its servers. The limit reflects the practical attention budget of most AI clients; tool lists longer than this tend to produce worse model behaviour because the choice space becomes unmanageable. If more than 32 tools are needed in total, the right answer is usually two or more profiles, each focused on a distinct task or persona, rather than a single overloaded profile. ::: ![Select servers and tools for an MCP profile](/img/screenshots/console/mcp-profile-tools.jpeg) On save, the Console displays the **Profile URL**. This is the address the MCP client connects to. Copy it; subsequent steps depend on it. The path portion is server-assigned and takes the form `/mcp/-`. Paste the URL as the Console gives it rather than joining that path onto a host by hand. On a data plane that hosts Project Gateways, the shared workspace URL does not serve `/mcp/*` at all, and some hosts answer an unmatched `/mcp/*` request with `200` and an empty body, so a hand-built address can fail without looking like a failure. ### Retrieving the address through the API Profile reads return the same addresses in an `endpoints` field, so the join never has to be reproduced by a caller. Each entry carries an absolute `url` and the `projectGatewayId` serving it. See [Get MCP profile](/reference/api/mcp-catalog/agentrouter-mcp-v-1-mcp-profiles-service-get-mcp-profile/) in the API reference. Three properties of the field are worth knowing before it is consumed: - **More than one entry is normal.** A profile is served by every Project Gateway attached to its project. A project with no attached gateway resolves instead to its data plane's shared URL, a single entry whose `projectGatewayId` is empty. The empty value is the ordinary single-gateway case, not a missing one. - **An empty list is a valid answer.** It means no gateway URL is configured yet. That is a question for a platform operator rather than an error, and retrying does not resolve it. - **No polling is required.** The field is populated on the create response, including while `status` is still `pending`. The separate `url` field on the profile carries only the path, with no scheme or host. It exists for display and is not usable on its own. ## Step 4: connect from the MCP client The Console exposes two paths for getting the profile URL into the local client: a guided **Add to Client** flow that performs as much of the setup as possible automatically, and a **Get JSON Config** option that returns the raw configuration block for manual installation. The guided flow is the right starting point for most clients; the manual flow is useful when the client's MCP configuration is checked into version control. Both paths are reached from the **Connect** button on the profile row in the MCP Profiles page. - **Add to Client**: click **Connect**, select the target client (Claude Code, Cursor, VS Code, and so on), then follow the one-click installation instructions. - **Get JSON Config**: click **Connect**, select **Get JSON Config**, then copy the returned JSON block into the client's MCP configuration file. The Console returns the correct Profile URL; paste it as-is. ![Add an MCP profile to a client](/img/screenshots/console/mcp-connect-client.jpeg) ![Get the JSON configuration for an MCP profile](/img/screenshots/console/mcp-connect-json.jpeg) ### Claude Code A single command registers the profile with Claude Code. Paste the **Profile URL** from the Console in place of `PROFILE_URL`, which stands for the complete value, in the form `https:///mcp/-`: ```bash claude mcp add --transport streamable-http "my-profile" \ PROFILE_URL ``` `my-profile` is the local name the connection is given inside Claude Code. If the profile uses Built-in OAuth, Claude Code opens a browser window for authentication the first time a tool is invoked. If the profile uses API Key authentication, the key is supplied through an environment variable or through Claude Code's own settings. Claude Code also accepts a JSON configuration block, which is the form returned by **Get JSON Config**. For an API Key profile, the key is carried in an `Authorization` header: ```json { "mcpServers": { "agent-router": { "url": "PROFILE_URL", "headers": { "Authorization": "Bearer YOUR_API_KEY" } } } } ``` ### Cursor Cursor reads MCP configuration from `.cursor/mcp.json`, either in the project directory or in the user-level settings. The minimum configuration for an unauthenticated or OAuth profile is: ```json { "mcpServers": { "my-profile": { "url": "PROFILE_URL" } } } ``` For an API Key profile, an `Authorization` header is added: ```json { "mcpServers": { "my-profile": { "url": "PROFILE_URL", "headers": { "Authorization": "Bearer YOUR_API_KEY" } } } } ``` As an alternative to editing the file directly, the server can be added through the Cursor UI: open **Settings → MCP**, click **Add Server**, enter the profile URL and any authentication details, then save. Cursor picks up the change without a full restart in most cases; if the connection does not appear, restarting the client resolves it. ### VS Code VS Code's MCP configuration lives in `.vscode/mcp.json`, either at the workspace level or in user settings, and uses the same shape as Cursor's: ```json { "mcpServers": { "my-profile": { "url": "PROFILE_URL" } } } ``` For API Key profiles, the same `Authorization` header is added. For Built-in OAuth or Custom OAuth profiles, VS Code prompts for authentication through a browser flow when the connection is first established. ## Step 5: test the connection The simplest verification is to ask the AI client to use one of the profile's tools and confirm the call lands in Request Logs. 1. In the MCP client, confirm that the tools from the profile are listed and available. Most clients surface this through a tool inspector or a command palette. 1. Trigger one of the tools through a natural-language prompt, for example, asking the assistant to perform a task that requires the file-system tool or the search tool exposed by the profile. 1. Confirm the tool execution completes successfully. 1. In the Console, open **Monitoring → Request Logs**. 1. Locate the MCP request, where MCP traffic is distinguished from chat-completion traffic in the log surface, and expand it to confirm the routing was as expected. If a tool fails to invoke, the most common causes are profile-level authentication mismatches (the client's credentials do not match the profile's security type), server-level authentication failures (the upstream MCP server rejected the request), or a missing tool in the profile (the client requested a tool that was not included in the tool selection in Step 3). Each of these failure modes is distinguishable in Request Logs. ## Editing a profile after the fact Profiles are not write-once. Servers can be added or removed, tool selections can be tightened or expanded, the security type can be changed, and the profile can be renamed. Each edit takes effect on save without disturbing the profile URL itself. A profile **deletion**, in contrast, immediately invalidates the URL. Any client still connected to the URL will lose access to its tools on the next request, and the URL is not reissued if the profile is recreated under the same name. For one-off cleanup this is the right behaviour; for routine maintenance, editing the existing profile is the right approach. ## What to do next - **Integrate the gateway with an app**: expose the same gateway to non-MCP clients and SDKs. See [Integrate the gateway with an app](/agent-router-service/guides/integrate-the-gateway-with-an-app/). - **Test prompts in the Playground**: exercise an MCP profile interactively from inside the Console. See [Test prompts in the Playground](/agent-router-service/guides/test-prompts-in-the-playground/). - **Monitor traffic and usage**: track MCP request volume alongside chat-completion traffic. See [Monitor traffic and usage](/agent-router-service/guides/monitor-traffic-and-usage/). The profile and its client connections remain in place for subsequent guides. Where to go next Integrate the gateway with an app Expose the same gateway to non-MCP clients and SDKs. Test prompts in the Playground Exercise an MCP profile interactively from inside the Console. --- ## Apply advanced routing rules Most production AI traffic is routed with a combination of fallback policies and traffic splitting. Those two patterns cover the common needs: keep requests successful when a provider fails, and distribute traffic deliberately for cost or evaluation. There is, however, a category of needs that neither tool addresses cleanly: routing decisions that depend on the request itself rather than on the static configuration of the chain, and lifecycle concerns such as decoupling application code from the exact provider model identifier in use today. These needs are covered by Agent Router's advanced routing surface. Advanced routing is not a single feature so much as a collection of capabilities that activate on different signals. Some are configured explicitly: in particular, model-name virtualisation, which lets the application speak a stable logical name while the gateway resolves it to whichever provider model is currently appropriate. Others are present by default and apply transparently to every request without configuration: session affinity, dynamic backend selection, and request and response transformation across provider APIs. This guide covers both: how to configure the surface that needs configuration, and what to expect from the surface that does not. **Persona:** Developer working in the Developer Console. **Estimated time:** 15 to 30 minutes, depending on whether model-name overrides and A/B or canary patterns are configured during the same session. ## When this guide applies Advanced routing is the right level to engage with when one or more of the following are true: | Situation | Capability that addresses it | | :---- | :---- | | Application code keeps changing whenever the upstream provider releases a new model version | **Model-name override**: expose a stable logical model name and resolve it to the provider's identifier in the gateway | | Two model versions need to be compared head-to-head under real production traffic | **A/B with traffic splitting**: combine model-name overrides with weight-based distribution | | A new model version needs to be rolled out gradually to limit blast radius | **Canary deployment**: start with a small weight on the new model and shift over time | | Multi-turn conversations or agent workflows are involved | **Session affinity**: handled automatically; no configuration is required | | Backend load varies, and latency matters | **Dynamic backend selection**: handled automatically; no configuration is required | | The application speaks OpenAI but should be free to route to any provider | **Request and response transformation**: handled automatically; no configuration is required | For pure resilience or pure traffic distribution by weight, the simpler guides, *Improve Resilience with Fallbacks* and *Reduce Cost with Traffic Splitting*, are the right starting points. Advanced routing is layered on top of those patterns rather than replacing them. ## Outcomes By the end of this guide: - At least one logical model name has been defined that decouples the application from a specific provider model identifier. - The override has been combined with a traffic split to run an A/B between two model versions, or to operate a canary rollout for a new model version. - The behaviours that operate automatically (session affinity, dynamic backend selection, and cross-provider transformation) are understood, even though they require no configuration. ## Prerequisites This guide builds on the routing configuration patterns established in the earlier dev guides. Specifically: - A working API key with a routing configuration attached, as set up in [Route Requests Across Providers](/agent-router-service/guides/route-requests-across-providers/). - Familiarity with traffic splitting weights, as covered in [Reduce Cost with Traffic Splitting](/agent-router-service/guides/reduce-cost-with-traffic-splitting/). Model-version experiments are implemented as traffic splits over logical model names. - At least two enabled models in the Admin Dashboard, or one model with multiple versions available for promotion. ## Step 1: define a logical model name The most direct piece of advanced routing to configure is the **model-name override**, exposed in the routing configuration as the `modelNameOverride` field: a mapping from a logical name the application speaks to the specific provider model identifier the gateway dispatches against. Without this, every provider model version bump becomes an application change; with it, version changes become a routing-configuration change inside Agent Router. The override is attached to a route entry on the API key's routing configuration. The choice of logical name belongs to the application team and should be stable across versions. A few useful conventions: | Logical name | Resolves to | When to use this pattern | | :---- | :---- | :---- | | `my-gpt4` | `gpt-4o-2024-08-06` | A simple alias that hides the specific dated version from the application | | `my-claude` | `claude-sonnet-4-20250514` | An alias that hides a cross-provider model identifier behind a stable name | | `stable-chat` | `gpt-4o-mini-2024-07-18` | A long-lived alias used by parts of the system that prefer predictability over capability | | `next-gen-chat` | `gpt-4o-2024-11-20` | A name reserved for whichever version is currently being evaluated for promotion | Configure the override: 1. Open the detail page for the API key whose routing configuration should expose the logical name. 1. Open the routing configuration and add a route entry. 1. Set the logical name in the route entry (for example, `my-gpt4`). 1. Set the resolved provider model identifier (for example, `gpt-4o-2024-08-06`). 1. Save the configuration. The application now requests `my-gpt4` in the `model` field of its OpenAI-compatible payload, and the gateway forwards the request to the configured provider model. The application code is unaware of the underlying version. ## Step 2: combine overrides with traffic splitting Logical names are most useful when they are paired with the traffic-splitting mechanics covered in the previous guide. Two routes can resolve the same logical name to two different provider models, and a weighted split distributes requests between them. The application speaks one stable name; the gateway runs an A/B in the background. A typical A/B setup against the logical name `chat-model`: | Route | Logical name | Resolves to | Weight | | :---- | :---- | :---- | :---- | | A | `chat-model` | `gpt-4o-2024-08-06` | 50 | | B | `chat-model` | `gpt-4o-2024-11-20` | 50 | Configure: 1. In the routing configuration for the chosen API key, add two route entries with the same logical name and different resolved models. 1. Switch the routing strategy to **Traffic Splitting** if it is not already set. 1. Assign weights to each route entry (50/50 for an A/B; a heavier weight on the current version for an evaluation that should not perturb production much). 1. Save and confirm the **Active** toggle is on. Quality and performance comparisons are then made through Request Logs and Usage Analytics, both of which expose the resolved model per request even though the application only ever saw `chat-model`. A **canary deployment** is structurally identical to an A/B but with intentionally lopsided weights. A common progression: | Stage | Current version weight | New version weight | | :---- | :---- | :---- | | Initial canary | 95 | 5 | | Hold-and-observe | 80 | 20 | | Expand | 50 | 50 | | Cutover | 0 | 100 | The weights are adjusted in the Console at each stage; the application requires no change at any point. Once the cutover is complete, the route pointing at the old version can be removed entirely. ### Route by compliance, cost, or latency policy Weighted splits distribute one logical name across backends by chance. The same logical name can instead be resolved deterministically by keying the routing decision on attributes of the request, a policy condition rather than a weight. This is an application of the attribute-based dispatch already described: a routing rule inspects request attributes (for example, a tenant tag, a priority header, or a traffic classification) and selects the route entry whose condition matches, so that `chat-model` resolves to a different backend depending on the kind of request that arrived. Three policy conditions cover the common cases: | Condition on the request | Resolves `chat-model` to | Rationale | | :---- | :---- | :---- | | Tagged as regulated or residency-bound | An in-region, compliant provider | Keeps regulated traffic on a backend that satisfies data-residency and retention constraints | | Tagged as low-priority or bulk | A cheaper backend | Reserves premium capacity for traffic that needs it and lowers cost on the rest | | Tagged as interactive | A lower-latency backend | Protects the responsiveness of user-facing requests | The route entries are configured exactly as in the preceding split, the same logical name mapped to different resolved models, but each entry carries a match condition instead of a weight, and the gateway dispatches to the first entry whose condition the request satisfies. Where cost is the driver, this attribute-based approach complements the proportional split in [Reduce Cost with Traffic Splitting](/agent-router-service/guides/reduce-cost-with-traffic-splitting/): the split distributes by chance, whereas a policy condition routes by a known property of the request. Compliance-driven routing is most often enforced on the operator side, where the residency and no-retention guarantees are configured for the backend itself; see [Configure Data Residency and No-retention](/agent-router-enterprise/guides/operate-and-govern/compliance-and-governance/configure-data-residency-and-no-retention/). ## Step 3: understand the behaviour that requires no configuration A meaningful portion of Agent Router's advanced routing happens behind the scenes. The behaviours below apply to every request and do not need to be enabled or tuned. This step is short on configuration but useful to read once, because the behaviour shapes how the gateway responds under load and across providers. ### Dynamic backend selection When more than one backend is eligible to serve a request (for example, two replicas of a self-hosted model behind a load-balanced endpoint), the gateway evaluates live backend metrics and chooses the best target. The mechanism is an InferencePool paired with an Endpoint Picker Provider: rather than relying solely on static weights or ordered fallback lists, the Endpoint Picker evaluates live metrics for each candidate and routes to the backend with the best capacity and cache affinity, reducing latency and improving throughput. Three signals contribute to the decision: | Signal | What it captures | | :---- | :---- | | KV-cache usage | Memory pressure on each backend. Heavily loaded backends are deprioritised. | | Queue depth | Number of pending requests on each backend. Less-loaded backends are preferred. | | Prefix cache scoring | How well a backend's cache matches the request's prompt prefix. Better matches reduce latency. | Dynamic selection operates only within the eligible set defined by the routing configuration. A fallback policy that limits requests to a specific provider still constrains dynamic selection to that provider; the policy boundary always takes precedence. ### Session affinity Multi-turn conversations, agent loops, and MCP sessions benefit from being processed by the same gateway instance throughout their lifetime, because state and cache accumulate locally. The gateway architecture guarantees this affinity: - The proxy component and the external processor are deployed as a sidecar pair, so once a session is established, subsequent requests in that session route to the same processor. - MCP sessions use encoded multi-backend session identifiers that pin the session to whichever combination of backends it was established against. The affinity matters in three cases: multi-turn conversations where context accumulates on a specific backend, stateful agent interactions that maintain tool state across calls, and MCP sessions that manage connections to multiple tool servers. No configuration is required to obtain this behaviour; it is a property of how the data plane is deployed. ### Request and response transformation The gateway exposes an OpenAI-compatible request surface but routes to a wide range of provider APIs that do not natively speak OpenAI. The translation between the two happens transparently in the gateway's processing pipeline: | Transformation | Behaviour | | :---- | :---- | | Header mutations | Provider-specific authentication headers are set or replaced | | Body mutations | JSON fields are added or rewritten, for example, injecting a default `max_tokens` if the provider requires one | | Path rewriting | The OpenAI path is rewritten to the provider's native endpoint, for example, `/v1/chat/completions` becomes the Anthropic Messages endpoint at `/anthropic/v1/messages` | | Model field rewriting | The `model` field is rewritten to the provider-specific model identifier, picking up any logical-name override from Step 1 | | Response normalisation | The provider's response is translated back into the OpenAI-compatible shape the calling application expects | The result is that an application written against the OpenAI SDK can route through the gateway to OpenAI, Anthropic, Google, Azure OpenAI, Mistral, or any other supported provider with no per-provider integration code. #### Example transformation flow When an OpenAI Chat Completions request is routed to Anthropic Claude, the fields are transformed in sequence: 1. Path: `/v1/chat/completions` is rewritten to the Anthropic Messages endpoint. 1. Body: the `messages` array is converted to the Anthropic message format, and `max_tokens` is injected if absent. 1. Headers: `Authorization: Bearer` is replaced with the provider-specific auth header. 1. Model: the `model` field is verified against the provider model identifier. 1. Response: the Anthropic response is translated back into the OpenAI Chat Completions format. The calling application receives a response in the exact format it requested, regardless of which provider served it. ## What to do next - **Use your own provider credentials**: introduce BYOK credentials alongside the routes defined in this guide. The logical-name pattern composes naturally with BYOK, because the override is applied before credentials are selected. See [Use Your Own Provider Credentials](/agent-router-service/guides/use-your-own-provider-credentials/). - **Monitor traffic and usage**: evaluate the results of an A/B or canary by examining the per-resolved-model breakdown in usage analytics. See [Monitor Traffic and Usage](/agent-router-service/guides/monitor-traffic-and-usage/). - **Export telemetry to an observability stack**: push per-route metrics into an existing observability platform so quality comparisons can run alongside the rest of the application's data. See [Export Telemetry to an Observability Stack](/agent-router-enterprise/guides/observability-and-analytics/export-telemetry-to-an-observability-stack/). Where to go next Use your own provider credentials Introduce BYOK credentials alongside the routes defined in this guide. Reduce cost with traffic splitting Distribute traffic by weight, the mechanism behind A/B and canary routing. --- ## Connect Claude Code Get Claude Code running through Tetrate Agent Router in a few minutes. tare integrate is the preferred path, because it writes the config from the logged-in session. Choose managed mode where Agent Router should bill Anthropic traffic, or passthrough mode where a Claude Max (or Pro) subscription already exists and Anthropic should bill it directly while the gateway routes and observes. **Persona:** Developer running Claude Code against an Agent Router gateway. **Estimated time:** 10 minutes. ## Outcomes By the end of this guide: - Claude Code is installed and its `ANTHROPIC_BASE_URL` points at the Agent Router gateway. - Traffic is authenticated in either managed mode or passthrough mode, matching how it should be billed. - The gateway destination is confirmed with `/status`, and the requests appear in the request logs. ## Step 1 - Install Claude Code Install Claude Code with Anthropic's native installer, the recommended method, which auto-updates in the background and does not require Node.js. ### macOS, Linux, WSL ```bash curl -fsSL https://claude.ai/install.sh | bash ``` ### Windows PowerShell ```powershell irm https://claude.ai/install.ps1 | iex ``` Other supported methods (Homebrew, WinGet, apt/dnf/apk, npm) are documented in the [Claude Code setup guide](https://docs.claude.com/en/docs/claude-code/setup). ## Step 2 - Configure with tare integrate (recommended) The fastest path is [Configure coding tools with the tare CLI](/agent-router-service/guides/coding-agents/configure-tools-with-tare-cli/). After `tare api login`, preview then apply: ```bash tare integrate dataplanes tare integrate claude-code --dry-run --dataplane tare integrate claude-code --dataplane --yes ``` Add `--passthrough` for Claude Max (or Pro) billing with Agent Router attribution only: ```bash tare integrate claude-code --dataplane --passthrough --yes ``` The CLI writes `~/.claude/settings.json` (backing up any existing file), sets `0600` permissions, and validates the key against the gateway. `--dataplane` takes the **DATAPLANE** column id from `tare integrate dataplanes`, not the display name. **Connectors trade-off:** while managed mode sets `ANTHROPIC_API_KEY`, Claude Code disables claude.ai connectors, because an explicit API key takes precedence over the claude.ai login. Where connectors are needed, choose `--passthrough`, or remove the env block from `~/.claude/settings.json`. Where the CLI path does not fit the environment, use the manual configuration below. ## Step 3 - Manual configuration (fallback) Agent Router supports two auth modes for Claude Code. Pick the one that matches how traffic should be paid for. ### Option A: Managed mode (Agent Router-managed Anthropic key) Use this without a Claude Max subscription, or where all traffic should be billed through Agent Router using its managed Anthropic credentials. #### How it works - Agent Router authenticates the request with its managed Anthropic key. - Cost = base + Agent Router margin. - The Agent Router API key goes in `ANTHROPIC_API_KEY`. Add the following to the shell profile (for example `~/.zshrc` or `~/.bashrc`): ```bash # Use the proxy host (not a full /v1/messages URL): Anthropic clients append /v1/messages automatically export ANTHROPIC_BASE_URL="https://proxy.acme.example.com" export ANTHROPIC_API_KEY="your-tars-api-key" ``` Replace the base URL with the gateway proxy host from the Console Dashboard if it differs. While `ANTHROPIC_API_KEY` is set, Claude Code disables claude.ai connectors. Use passthrough mode (Option B) where connectors are required. ### Option B: Passthrough mode (an existing Claude Max subscription) Use this where a Claude Max (or Pro) subscription already exists and Claude Code should bill against it. Agent Router routes the request and observes it, but the Anthropic OAuth token passes through untouched, so Anthropic authenticates the user the same as on a direct connection. #### How it works - `ANTHROPIC_API_KEY` is left unset, because Claude Code's stored OAuth token is what authenticates with Anthropic. - `x-tars-api-key` authenticates the request with Agent Router for routing and observability. - Cost = the Claude Max plan, with no Agent Router margin on tokens. Passthrough requests are labeled `passthrough` in the Request Logs. First, log in to the Anthropic account once through Claude Code's normal flow, so it stores the OAuth token locally: ```bash claude /login ``` Then add the following to the shell profile: ```bash export ANTHROPIC_BASE_URL="https://proxy.acme.example.com" export ANTHROPIC_CUSTOM_HEADERS="x-tars-api-key: your-tars-api-key" ``` Replace the base URL with the gateway proxy host from the Console Dashboard if it differs. ## Step 4 - Start a session Change to the project directory and launch Claude Code: ```bash cd your-project claude ``` ## Step 5 - Verify Run the `/status` command inside the Claude Code session and confirm the base URL points to the gateway: ```text > /status Anthropic base URL: https://proxy.acme.example.com ``` The [Request Logs](/agent-router-service/guides/monitor-traffic-and-usage/) show the requests as they arrive. ## Troubleshooting | Issue | What to check | |---|---| | API key issues | In managed mode, `ANTHROPIC_API_KEY` must be set to the Agent Router API key. In passthrough mode, `ANTHROPIC_API_KEY` must be unset, and `ANTHROPIC_CUSTOM_HEADERS` (or the CLI-written settings) must carry `x-tars-api-key`. | | Connection errors | Verify that `ANTHROPIC_BASE_URL` points to the correct gateway URL, with no trailing `/v1/messages`, which Claude Code appends automatically. | | Passthrough returns 401 from Anthropic | The Claude Max OAuth token is missing or expired. Re-run `claude /login` and try again. | | claude.ai connectors unavailable | Managed mode's `ANTHROPIC_API_KEY` disables connectors. Use `--passthrough` with `tare integrate`, unset the API key, or remove the env block from `~/.claude/settings.json`. | | Permission denied | Claude Code needs read/write access to the project files. | | Context limits | For very large repositories, use `.claudeignore` to exclude irrelevant directories. | Where to go next Configure coding tools with the tare CLI Automate setup for Claude Code and other supported tools, including dry-run and key rotation. Project context (CLAUDE.md) Teach Claude Code Agent Router API patterns so generated app code targets the gateway. --- ## Claude Code Two separate jobs: route Claude Code's own traffic through the Tetrate Agent Router gateway, and optionally teach the agent how an application should call Agent Router. Start with Connect if Claude Code is not yet pointed at the gateway. Choose a path Connect Claude Code Point Claude Code at the gateway with managed billing, or with a Claude Max subscription in passthrough mode. Project context (CLAUDE.md) Add a CLAUDE.md to the repository so Claude Code uses Agent Router endpoints and routing patterns when it writes code. --- ## Project context (CLAUDE.md) Give Claude Code context about Tetrate Agent Router so it suggests the right API patterns, fallback routing, and cost tracking. Connecting only routes Claude Code's own calls; this file teaches the agent how an application should call Agent Router. If Claude Code is not yet pointed at the gateway, start with Connect Claude Code. **Persona:** Developer writing application code with Claude Code against an Agent Router gateway. **Estimated time:** 5 minutes. ## Outcomes By the end of this guide: - A `CLAUDE.md` file in the repository root describes the Agent Router base URL, authentication, and available models. - Code that Claude Code writes targets the gateway and reads its key from `AGENTROUTER_API_KEY`. - Fallback routing, traffic splitting, and per-feature keys are suggested as gateway configuration rather than application code. ## Setup Add the following to the project's `CLAUDE.md` file, creating one in the repository root if it does not exist: ```markdown title="CLAUDE.md" # Agent Router AI Gateway This project uses Agent Router (router.tetrate.ai) as an AI gateway. Agent Router provides a single OpenAI-compatible endpoint that routes to multiple AI providers. ## API Configuration - **Base URL:** `https://api.router.tetrate.ai/v1` (Service). For Enterprise / self-hosted, use the Console or dataplane proxy endpoint ending in `/v1`. - **Auth:** Bearer token with an Agent Router API key (`sk-...`) - **Protocol:** OpenAI Chat Completions API, so any OpenAI SDK works - **Environment variable:** Store the key as `AGENTROUTER_API_KEY` ## Making API Calls Use the standard OpenAI SDK. Only change `base_url` and `api_key`: Python: from openai import OpenAI import os client = OpenAI(api_key=os.environ["AGENTROUTER_API_KEY"], base_url="https://api.router.tetrate.ai/v1") response = client.chat.completions.create(model="gpt-5.6-terra", messages=[...]) TypeScript: import OpenAI from "openai"; const client = new OpenAI({ apiKey: process.env.AGENTROUTER_API_KEY, baseURL: "https://api.router.tetrate.ai/v1" }); const response = await client.chat.completions.create({ model: "gpt-5.6-terra", messages: [...] }); ## Available Models Agent Router routes to these providers transparently (sample, because the catalog changes often): - OpenAI: gpt-5.6-terra, gpt-5.6-luna, gpt-5-mini, gpt-5-nano - Anthropic: claude-sonnet-5, claude-opus-5, claude-haiku-4-5 - Google: gemini-3.1-pro-preview, gemini-2.5-flash - xAI / Groq / DeepInfra: use prefixed IDs from the live catalog Query the live catalog at https://router.tetrate.ai/api/public/models (or GET /v1/models with your key). ## Agent Router features When building features, consider suggesting these Agent Router capabilities: ### Fallback Routing If this feature needs high availability, suggest configuring fallback routing in the Agent Router Console. Set a primary model at priority 0 and backup at priority 1. Agent Router automatically retries on 5xx/timeout and falls back, with zero code changes needed. ### Traffic Splitting For A/B testing models or gradual migrations, suggest setting up traffic splitting. Configure weights (e.g., 70/30) at the same priority level in the Agent Router Console. ### Cost Tracking Agent Router tracks per-key usage and cost. Suggest using **separate API keys** for different features, services, or environments so costs can be broken down: - `AGENTROUTER_API_KEY_CHAT` for user-facing chat - `AGENTROUTER_API_KEY_SUMMARIZATION` for background summarization - `AGENTROUTER_API_KEY_EMBEDDINGS` for vector embeddings ### Streaming Agent Router supports streaming responses. Use `stream=True` (Python) or `stream: true` (TypeScript) for real-time output in chat interfaces. ### MCP (Model Context Protocol) Agent Router can serve as an MCP provider. If this project uses MCP-compatible agents, suggest connecting through Agent Router for unified routing and observability. ## Key Principles - Always use environment variables for API keys, never hardcode - Default to `gpt-5.6-terra` unless the user specifies a model - Use streaming for any user-facing chat interface - Suggest separate API keys when building multiple features that call AI - When a feature needs resilience, mention fallback routing as an Agent Router Console config - When comparing models, mention traffic splitting as a zero-code option ``` ## What the context provides With these instructions in `CLAUDE.md`, Claude Code: - Uses the correct Agent Router base URL and auth pattern automatically. - Defaults to `gpt-5.6-terra` through Agent Router instead of calling OpenAI directly. - Suggests separate API keys when multiple AI features are built. - Recommends fallback routing where high availability is needed. - Recommends traffic splitting when models are being compared. - Uses streaming for chat interfaces without being asked. Where to go next Connect Claude Code Point Claude Code at the gateway with managed or Max/passthrough auth. Cursor Give Cursor the same Agent Router context through a .cursorrules file. --- ## Coder Wire AI coding agents (such as Claude Code) running inside Coder workspaces to a Tetrate Agent Router gateway. Each developer stores their own Agent Router API key as a Coder user secret, and a platform operator adds one coder_env block to the workspace template, so every workspace routes through the gateway with per-developer spend attribution, budgets, and revocation. **Persona:** Developer with a Coder workspace, plus a platform operator who owns the workspace template. **Estimated time:** 10 to 15 minutes across both roles, plus a workspace restart. ## Outcomes By the end of this guide: - Each developer's Agent Router API key is stored as a Coder user secret and injected as `ANTHROPIC_API_KEY`. - The workspace template sets `ANTHROPIC_BASE_URL` to the Agent Router gateway URL. - Anthropic-protocol agents in every workspace built from that template route through the gateway. - Requests appear in the Agent Router request logs attributed to the individual developer's key. ## How it works Coder is a self-hosted cloud development environment: a platform operator owns the workspace *template*, and each developer owns their *workspaces* built from it. The integration splits along that line: - **Developer (once):** store the Agent Router API key as a Coder [user secret](https://coder.com/docs/user-guides/user-secrets). Coder injects it as `ANTHROPIC_API_KEY` into every workspace that developer owns, so the key never touches the template or version control. - **Platform operator (once per template):** add one `coder_env` resource that sets `ANTHROPIC_BASE_URL` to the Agent Router gateway URL. The URL is not a secret, so it is safe to commit with the template. Because each developer brings their own key, requests from workspaces appear in Agent Router request logs attributed per developer. **Requires Coder v2.34 or later** (user secrets, available in Coder OSS, with no Premium license needed). Works with any Agent Router gateway: Agent Router Service or an Enterprise (self-managed) data plane. ## Prefer tare integrate (recommended) The fastest path for the developer half is [Configure coding tools with the tare CLI](/agent-router-service/guides/coding-agents/configure-tools-with-tare-cli/). After `tare api login` and a `coder login` against the Coder deployment, preview then apply: ```bash tare integrate dataplanes tare integrate coder --dry-run --dataplane tare integrate coder --dataplane ``` The command resolves the gateway URL, reuses (or mints) an Agent Router API key, stores it as the Coder user secret, and prints the `coder_env` template block to hand to the platform operator. If the `coder` CLI is not installed or not logged in, it prints complete manual instructions instead of failing. The manual steps below achieve the same thing. ## Step 1 - Store the API key as a Coder user secret This step is performed by the developer. Log in to the Coder deployment, then pipe the Agent Router API key into a user secret (stdin keeps it out of shell history): ```bash coder login https://coder.example.com printf '%s' 'your-tars-api-key' | coder secret create tars-api-key \ --env ANTHROPIC_API_KEY \ --description 'Tetrate Agent Router inference key' ``` Use `coder secret update tars-api-key` instead if the secret already exists. The secret can also be created in the Coder dashboard under **Account > Secrets > Add secret**: set the name, the `ANTHROPIC_API_KEY` environment variable target, and paste the key as the value: ![Add secret dialog with name tars-api-key and environment variable ANTHROPIC_API_KEY](/img/screenshots/external/coder/01-add-secret.webp) Either way, the secret shows up with its environment-variable target, enabled for injection: ![Coder user secret tars-api-key with env target ANTHROPIC_API_KEY](/img/screenshots/external/coder/02-user-secret.webp) Coder injects the secret as `ANTHROPIC_API_KEY` into every workspace that developer owns, at workspace start. ## Step 2 - Point workspaces at the gateway This step is performed by the platform operator. Add the following to the workspace template (the URL is not a secret, so it is safe to commit): ```hcl # Routes Anthropic-protocol agents (Claude Code, ...) in every workspace # through Tetrate Agent Router. resource "coder_env" "anthropic_base_url" { agent_id = coder_agent.main.id name = "ANTHROPIC_BASE_URL" value = "https://your-gateway.example.com" } ``` Replace the value with the gateway proxy host: for Agent Router Service, the gateway URL from the Console Dashboard; for Enterprise, the data plane's proxy endpoint. Use the host only, not a full `/v1/messages` URL, because Anthropic clients append the path automatically. Push the template (`coder templates push`). The block is visible in the template's **Source Code** view: ![The coder_env block in the template Source Code view](/img/screenshots/external/coder/03-template-coder-env.webp) Every workspace built from it now routes Anthropic-protocol traffic through Agent Router. ## Step 3 - Verify from inside a workspace Start (or restart) a workspace so the secret and template env are injected: ![A running Coder workspace built from the updated template](/img/screenshots/external/coder/04-workspace-running.webp) Open a terminal in it and check that both halves landed, then run the agent as usual, for example `claude` (Claude Code picks up `ANTHROPIC_BASE_URL` and `ANTHROPIC_API_KEY` automatically): ```bash printenv ANTHROPIC_BASE_URL test -n "$ANTHROPIC_API_KEY" && echo "ANTHROPIC_API_KEY is set (injected by Coder user secret)" claude -p "Reply with exactly: routed through Agent Router" ``` ![Terminal in the workspace: env vars injected and Claude Code answering through Agent Router](/img/screenshots/external/coder/05-terminal-verify.webp) The gateway can also be probed directly: ```bash curl "$ANTHROPIC_BASE_URL/v1/models" -H "x-api-key: $ANTHROPIC_API_KEY" ``` Requests appear in the Agent Router request logs, attributed to the developer's API key. Per-developer spend, budgets, and revocation all work because each developer stores their own key: ![Agent Router request logs attributing the workspace traffic to the developer's API key](/img/screenshots/external/coder/06-request-logs.webp) ## Notes and limitations - **Restart to pick up changes:** user secrets are injected at workspace start. After creating or updating the secret, restart running workspaces. - **The secret follows the developer everywhere:** Coder injects user secrets into *every* workspace that developer owns on that deployment. Where some workspaces must talk to Anthropic directly, unset `ANTHROPIC_API_KEY` and `ANTHROPIC_BASE_URL` in those, or scope templates accordingly. - **Template parameters are not for keys:** Coder displays parameter values in cleartext across the product and [recommends against them for secrets](https://coder.com/docs/admin/security/secrets). User secrets exist for exactly this. - **OpenAI-protocol tools:** this guide covers the Anthropic protocol. OpenAI-shape tools inside workspaces can use the same pattern with a second secret (`--env OPENAI_API_KEY`) and a `coder_env` for the OpenAI-compatible base URL (the gateway URL with `/v1`). - **Coder AI Gateway is not the mechanism:** Coder's own AI Gateway (a Premium add-on) would demote Agent Router to an upstream provider. This integration deliberately uses plain environment wiring instead, so it needs only Coder OSS. Where to go next Connect Claude Code Point Claude Code at the gateway outside Coder, with managed or passthrough auth. Monitor traffic and usage Read the request logs that attribute workspace traffic to each developer's key. --- ## Codex CLI Point Codex CLI at a Tetrate Agent Router gateway with tare integrate, then give it project context so generated code uses the right API endpoint, auth patterns, and available models. **Persona:** Developer running Codex CLI against an Agent Router gateway. **Estimated time:** 10 minutes. ## Outcomes By the end of this guide: - Codex CLI sends its own model calls through the Agent Router gateway. - An `AGENTS.md` file in the repository root describes the gateway base URL, authentication, and available models. - Code that Codex writes targets the gateway and reads its key from `AGENTROUTER_API_KEY`. ## Prefer tare integrate (recommended) The fastest way to wire Codex itself to the gateway is [Configure coding tools with the tare CLI](/agent-router-service/guides/coding-agents/configure-tools-with-tare-cli/): ```bash tare integrate dataplanes tare integrate codex --dry-run --dataplane tare integrate codex --dataplane --yes ``` That writes the Codex provider block and API key as `AGENTROUTER_API_KEY` (with a backup of any existing config). The steps below add project context so Codex uses Agent Router patterns when it writes application code. ## Setup: project context Create an `AGENTS.md` file in the repository root, or add to an existing one: ```markdown title="AGENTS.md" # Agent Router AI Gateway This project uses Agent Router (router.tetrate.ai) as an AI gateway. Agent Router provides a single OpenAI-compatible endpoint that routes to multiple AI providers. ## API Configuration - Base URL: https://api.router.tetrate.ai/v1 (Service). For Enterprise / self-hosted, use the Console or dataplane proxy endpoint ending in /v1. - Auth: Bearer token with an Agent Router API key (`sk-...`) - Protocol: OpenAI Chat Completions API, so any OpenAI SDK works - Environment variable: Store the key as AGENTROUTER_API_KEY ## Code Patterns Python: from openai import OpenAI import os client = OpenAI(api_key=os.environ["AGENTROUTER_API_KEY"], base_url="https://api.router.tetrate.ai/v1") response = client.chat.completions.create(model="gpt-5.6-terra", messages=[...]) TypeScript: import OpenAI from "openai"; const client = new OpenAI({ apiKey: process.env.AGENTROUTER_API_KEY, baseURL: "https://api.router.tetrate.ai/v1" }); const response = await client.chat.completions.create({ model: "gpt-5.6-terra", messages: [...] }); ## Available Models - OpenAI: gpt-5.6-terra, gpt-5.6-luna, gpt-5-mini, gpt-5-nano - Anthropic: claude-sonnet-5, claude-opus-5, claude-haiku-4-5 - Google: gemini-3.1-pro-preview, gemini-2.5-flash - xAI / Groq / DeepInfra: use prefixed IDs from the live catalog Query https://router.tetrate.ai/api/public/models (or GET /v1/models with your key). ## Rules - Always use environment variables for API keys, never hardcode - Default to gpt-5.6-terra unless the user specifies a model - Use streaming for any user-facing chat interface - Suggest separate API keys when building multiple features that call AI - When a feature needs resilience, mention fallback routing as an Agent Router Console config ## Agent Router features - Fallback Routing: Configure in the Console for automatic provider failover - Traffic Splitting: A/B test models with weighted routing in the Console - Cost Tracking: Per-key usage and cost tracking. Use separate keys per feature - Streaming: Full support for streamed responses - MCP: Agent Router can serve as an MCP provider for agent workflows ``` ## What the context provides With this context in place, Codex CLI: - Uses the Agent Router endpoint (`https://api.router.tetrate.ai/v1`) instead of direct provider APIs. - Reads API keys from environment variables (`AGENTROUTER_API_KEY`). - Defaults to `gpt-5.6-terra` unless another model is specified. - Suggests streaming for chat interfaces. - Recommends separate API keys per feature for cost tracking. - Mentions Agent Router features such as fallback routing when building resilient applications. Where to go next Configure coding tools with the tare CLI Automate Codex and other tool gateway setup with tare integrate. Claude Code Connect Claude Code to the gateway, or add a CLAUDE.md for Agent Router patterns. --- ## Configure coding tools with the tare CLI tare integrate sets up popular AI coding tools to send their model calls through a Tetrate Agent Router gateway, using existing Agent Router credentials. Instead of copying a base URL and API key into each tool by hand, one command reads the logged-in session and writes the tool's config. **Persona:** Developer configuring a local coding tool against an Agent Router gateway. **Estimated time:** 5 to 10 minutes once the CLI is installed and authenticated. ## Outcomes By the end of this guide: - The `tare` CLI is installed and authenticated against the management plane. - The data plane (gateway) to route through is identified by its dataplane id. - At least one coding tool is configured through `tare integrate`, optionally previewed with a dry run first. - Keys and backed-up config files can be rotated or cleaned up safely. ## Prerequisites 1. Install the `tare` CLI: ```bash curl -fsSL https://tare.tetrate.ai/tools/install.sh | bash tare --version ``` 1. Log in so the CLI can reuse the session (data plane, model catalog, and API key): ```bash tare api login --base-url https://api.example.tetrate.ai ``` Login can be skipped by passing `--gateway-url`, `--api-key`, and `--model` explicitly, but then nothing is reused from the account. 1. Know which data plane (gateway) to route through. List them with: ```bash tare integrate dataplanes ``` Example output: ```text DATAPLANE NAME GATEWAY URL acme-prod Acme Production https://acme-prod.gateway.example.com acme-dev Acme Dev https://acme-dev.gateway.example.com ``` `--dataplane` takes the workspace id from the **DATAPLANE** column (for example `acme-prod`), not the display name. ## Step 1 - List the supported tools This command shows every supported tool, whether it is detected on the machine, and where its config file would go: ```bash tare integrate list ``` Supported tools include Aider, Claude Code, Claude Cowork, Cline, [Coder](/agent-router-service/guides/coding-agents/coder/), Codex, Continue, Hermes, and OpenClaw. ## Step 2 - Preview a change (dry run) ```bash tare integrate --dry-run --dataplane ``` Prints exactly what would be set, with the API key masked, and confirms no file is written. ### Claude Code ```bash tare integrate claude-code --dry-run --dataplane acme-prod ``` ```text Detected Claude Code: found `claude` at /Users/you/.local/bin/claude Claude Code (/Users/you/.claude/settings.json): set env.ANTHROPIC_BASE_URL = https://acme-prod.gateway.example.com set env.ANTHROPIC_API_KEY = sk-test1... set env.ANTHROPIC_MODEL = claude-sonnet-5 --dry-run: no changes written. ``` ### Codex Codex writes two files (`config.toml` for the provider block, `.env` for the key): ```bash tare integrate codex --dry-run --dataplane acme-prod ``` ```text Detected Codex: found `codex` at /Users/you/.bun/bin/codex Codex (/Users/you/.codex/config.toml): /Users/you/.codex/config.toml: set model = gpt-5 set model_provider = agentrouter set [model_providers.agentrouter] base_url/env_key/wire_api /Users/you/.codex/.env: set AGENTROUTER_API_KEY = sk-test1... --dry-run: no changes written. ``` ## Step 3 - Apply a change ```bash tare integrate --dataplane --yes ``` For example: ```bash tare integrate claude-code --dataplane acme-prod --yes ``` On apply: - The command reuses the logged-in session's data plane, model catalog, and API key by default. Override any of them with `--gateway-url`, `--api-key`, or `--model`. - It mints a new inference key only if one is not already cached for that profile and data plane; otherwise it reuses the cached key. - If a config file already exists, it is backed up first (`.bak-`) before being overwritten. - The written file gets `0600` permissions (owner read/write only). - After writing, it makes one call to the configured gateway to confirm the key works, and reports the result. ## Claude Code: managed mode vs. passthrough mode `tare integrate claude-code` supports two modes: - **Managed (default).** Sets `env.ANTHROPIC_BASE_URL`, `env.ANTHROPIC_API_KEY`, and `env.ANTHROPIC_MODEL` in `~/.claude/settings.json`. Every Claude Code call goes through the Agent Router gateway using the issued key. - **Passthrough** (`--passthrough`). Keeps the existing `claude /login` session for model calls, so usage is billed to the existing Claude plan. The Agent Router key is only sent as an `x-tars-api-key` header for attribution; `ANTHROPIC_API_KEY` is left unset. ```bash tare integrate claude-code --dataplane acme-prod --passthrough --yes ``` **Trade-off:** while managed mode's `ANTHROPIC_API_KEY` is set, Claude Code disables claude.ai connectors, because an explicit API key takes precedence over the claude.ai login. Choose passthrough mode where connectors are needed, or remove the env block from `~/.claude/settings.json`. For the full Claude Code walkthrough (install, verify with `/status`, troubleshooting), see [Connect Claude Code](/agent-router-service/guides/coding-agents/claude-code/connect/). ## Known issues **Some newer model names reject `temperature=0`.** Several OpenAI-compatible tools (Aider and similar) default to `temperature=0` for any model they do not recognize by name. If the selected model rejects a temperature of `0`, the first request fails immediately with something like `'temperature' is deprecated for this model`, even though the model works fine otherwise. Workaround: pin a non-zero temperature for that model in the tool's own per-model settings. For Aider, add an entry to `.aider.model.settings.yml` with `extra_params: {temperature: 1}`. ## Cleanup and rotation - An apply either mints a new inference key or reuses a cached one. Check which before revoking anything: revoking a key still referenced by a tool's config breaks that tool until the integration is re-applied. - To rotate a key, revoke the old one (`tare api keys revoke `), then re-run `tare integrate --dataplane --yes` to mint and write a fresh one. - Backed-up config files (`.bak-`) build up with every apply. Clean them out periodically where the history is not needed. Where to go next Connect Claude Code Verify the session, or fall back to manual env configuration when the CLI path does not fit. Codex CLI Add project context so Codex uses Agent Router patterns when it writes application code. --- ## Cursor Give Cursor context about Tetrate Agent Router so it generates the right API patterns, suggests Agent Router features, and follows recommended practices. **Persona:** Developer writing application code in Cursor against an Agent Router gateway. **Estimated time:** 5 minutes. ## Outcomes By the end of this guide: - A `.cursorrules` file in the repository root describes the Agent Router base URL, authentication, and available models. - Cursor's agent and inline chat generate code that targets the gateway instead of a single provider. - API keys are read from environment variables rather than hardcoded. ## Setup Create a `.cursorrules` file in the repository root: ```text title=".cursorrules" # Agent Router AI Gateway Context This project uses Agent Router (router.tetrate.ai) as an AI gateway that provides a single OpenAI-compatible endpoint routing to multiple AI providers. ## API Setup - Base URL: https://api.router.tetrate.ai/v1 (Service). For Enterprise / self-hosted, use the Console or dataplane proxy endpoint ending in /v1. - Auth: Bearer token using Agent Router API key (`sk-...`) - Protocol: OpenAI Chat Completions API - Store key as environment variable: AGENTROUTER_API_KEY ## Code Patterns Python: from openai import OpenAI import os client = OpenAI(api_key=os.environ["AGENTROUTER_API_KEY"], base_url="https://api.router.tetrate.ai/v1") response = client.chat.completions.create(model="gpt-5.6-terra", messages=[...]) TypeScript: import OpenAI from "openai"; const client = new OpenAI({ apiKey: process.env.AGENTROUTER_API_KEY, baseURL: "https://api.router.tetrate.ai/v1" }); const response = await client.chat.completions.create({ model: "gpt-5.6-terra", messages: [...] }); ## Available Models - OpenAI: gpt-5.6-terra, gpt-5.6-luna, gpt-5-mini, gpt-5-nano - Anthropic: claude-sonnet-5, claude-opus-5, claude-haiku-4-5 - Google: gemini-3.1-pro-preview, gemini-2.5-flash - xAI / Groq / DeepInfra: use prefixed IDs from the live catalog Query https://router.tetrate.ai/api/public/models (or GET /v1/models with your key). ## Rules - Always use environment variables for API keys, never hardcode them - Default to gpt-5.6-terra unless the user specifies a different model - Use stream=True / stream: true for any user-facing chat interface - Use separate Agent Router API keys for different features/services for cost breakdown: AGENTROUTER_API_KEY_CHAT for chat, AGENTROUTER_API_KEY_SUMMARIZATION for summarization, etc. ## Agent Router features (suggest when relevant) - Fallback Routing: Configure in the Agent Router Console for automatic provider failover. Set primary model at priority 0, backup at priority 1. Zero code changes. - Traffic Splitting: A/B test models with weighted routing. Configure weights at same priority level in the Console. - Cost Tracking: Per-key usage tracking. Use separate keys per feature. - Streaming: Full support for streamed responses through Agent Router. - MCP: Agent Router can serve as an MCP provider for agent-compatible workflows. ``` ## What the rules file provides With `.cursorrules` in the repository, Cursor: - Generates code with the correct Agent Router base URL and auth pattern. - Uses OpenAI SDK patterns rather than raw HTTP by default. - Suggests separate API keys when multiple AI features are built. - Adds streaming automatically for chat-style interfaces. - Treats fallback routing and traffic splitting as configuration options rather than code. Where to go next Claude Code Connect Claude Code to the gateway, or add a CLAUDE.md for Agent Router patterns. Codex CLI Configure OpenAI's Codex CLI to route through Agent Router. --- ## Set up Coding Agents & Tools Use tare integrate to point supported coding tools at a Tetrate Agent Router gateway in one command. For deeper tool-specific setup, and for tools the CLI does not cover yet, the per-tool guides below give project context and manual configuration. Preferred path Configure coding tools with the tare CLI Run tare integrate to detect a local tool, write its config from the logged-in Agent Router session, and validate the gateway key. Per-tool guides Visual Studio Code Install the Tetrate model provider extension so one key serves VS Code chat, agent mode, and every extension that uses the Language Model API. Claude Code Point Claude Code at the gateway (managed or Max), then optionally add a CLAUDE.md for Agent Router patterns. Cursor Add a .cursorrules file so Cursor's agent and inline chat default to Agent Router. Codex CLI Configure the gateway with tare integrate, then add project context so Codex uses Agent Router endpoints. Lovable Paste project context so apps generated in Lovable call the Agent Router API. Coder Route agents inside Coder workspaces: a user secret for the API key, one coder_env block for the gateway URL. ## How it works For tools that `tare integrate` supports, the CLI detects the install, reuses the logged-in Agent Router session, writes the tool's config (with a backup), and validates the key against the gateway. Preview with `--dry-run` before applying. For project context, and for tools configured by hand, the setup follows four steps: 1. Pick a tool above and copy its config. 1. Paste the config into the project, in the file the tool reads (a repo-root instructions file, a rules file, or a project context field). 1. Start prompting. The agent now understands the Agent Router base URL, authentication, and available endpoints. 1. Layer in routing, fallback, and cost-tracking patterns as the application grows. The config teaches the agent to use the Agent Router [Gateway APIs](/reference/supported-apis/) (chat completions, embeddings, image generation, and audio speech), along with routing, fallback, and cost-tracking patterns, so generated code points at the gateway from the first request. ## What every config provides With the config in place, the coding agent: - Uses the correct Agent Router base URL and Bearer-token auth pattern automatically. - Defaults to routing through the gateway instead of calling a single provider directly. - Suggests separate API keys when multiple AI features are built, so cost can be attributed per feature. - Recommends fallback routing when a feature needs high availability. - Recommends traffic splitting when models are being compared. - Uses streaming for user-facing chat interfaces. Where to go next Configure with the tare CLI Automate gateway setup for Claude Code, Codex, Aider, and other supported tools. Make an API call Send the first request through the gateway with the OpenAI SDK. Integrate the gateway with an app Point SDKs and frameworks at the gateway in application code. --- ## Lovable Give Lovable context about Tetrate Agent Router so the apps it generates call the gateway instead of a single provider, with the right base URL, authentication, and endpoints from the first prompt. **Persona:** Developer generating an application in Lovable that calls an Agent Router gateway. **Estimated time:** 5 minutes. ## Outcomes By the end of this guide: - Lovable's project context describes the Agent Router base URL, authentication, and available endpoints. - Generated applications call the gateway rather than a single provider directly. - The API key is read from an environment variable or a server-side secret, never embedded in client-side source. ## Setup Lovable reads project context from the knowledge or instructions field attached to a project, rather than from a file in a repository. Paste the following into that field before generating AI features: ```markdown title="Project context" # Agent Router AI Gateway This project uses Agent Router as an AI gateway. Agent Router exposes a single OpenAI-compatible endpoint that routes to multiple AI providers. ## API configuration - Base URL: the proxy endpoint shown on the Console Dashboard, ending in `/v1` - Auth: Bearer token with an Agent Router API key - Protocol: OpenAI Chat Completions API, so any OpenAI-compatible client works - Store the key as `AGENTROUTER_API_KEY`, never in generated source ## Making API calls Use a standard OpenAI-compatible client and change only the base URL and API key: - Point `baseURL` at the Agent Router proxy endpoint. - Read the API key from `AGENTROUTER_API_KEY`. - Default to a chat completions model unless a specific model is requested. ## Endpoints available through Agent Router - Chat completions - Embeddings - Image generation - Audio speech ## Patterns to prefer - Use streaming for any user-facing chat interface. - Use a separate API key per feature so cost can be attributed. - Rely on gateway-side fallback routing for high availability, configured in the Console rather than in application code. ``` :::caution Keep the key out of client-side source Generated apps run in the browser or in a deployed backend, so the Agent Router API key belongs in an environment variable or a server-side secret. Generated code must not embed the key in client-side source. ::: ## What the context provides With this context attached, Lovable: - Generates API calls that target the Agent Router base URL and Bearer-token auth pattern. - Routes through the gateway rather than calling a single provider directly. - Reads the API key from `AGENTROUTER_API_KEY` instead of hardcoding it. - Uses streaming for chat interfaces and separate keys per feature. Where to go next Claude Code Connect Claude Code to the gateway, or add a CLAUDE.md for Agent Router patterns. Integrate the gateway with an app Point SDKs and frameworks at the gateway in application code. --- ## Visual Studio Code The Tetrate Agent Router Model Provider extension registers Agent Router as a language model provider in Visual Studio Code. One API key brings every chat model the key can reach into VS Code chat, agent mode, and any extension that selects models through the vscode.lm API. **Persona:** Developer adding Agent Router models to a local Visual Studio Code install. **Estimated time:** 5 minutes, plus time to pick the models to enable. ## Outcomes By the end of this guide: - The Tetrate Agent Router Model Provider extension is installed and holds an Agent Router API key in secret storage. - The chosen models appear in the VS Code chat model picker and in agent mode. - Other extensions can select the same models through the `vscode.lm` API. - The base URL points at the hosted service, an Enterprise tenant, or a self-hosted deployment as required. ## Why a provider extension VS Code can already call an OpenAI-compatible endpoint through its built-in custom endpoint support (Bring Your Own Key, or BYOK), but that path serves only the chat view. Models configured that way are not offered to other extensions through the `vscode.lm` API, so each model-consuming extension falls back to asking for its own provider key. The extension registers as a [language model chat provider](https://code.visualstudio.com/api/extension-guides/ai/language-model-chat-provider) instead, under the vendor id `tetrate-agent-router`. A single Agent Router key then serves: - The chat view and agent mode - Any third-party extension that selects models through `vscode.lm` - Anthropic, OpenAI, Google, xAI, Groq, and DeepInfra models behind a single endpoint and a single bill Models are discovered from the endpoint at runtime, so newly released models appear in the picker without an extension update. `Tetrate Agent Router: Refresh Model List` forces a re-query on demand. ## Requirements | Requirement | Detail | |---|---| | VS Code | 1.106 or newer | | API key | An Agent Router API key from [the Console](https://router.tetrate.ai/) | | Network | Outbound HTTPS to the configured base URL, and to `router.tetrate.ai` for model metadata | ## Setup :::caution Do not skip step 3 Extension-contributed models start out hidden in the VS Code chat model picker. An installed extension with a valid key still shows nothing in the dropdown until the models are enabled. This is a VS Code default, not a bug, and it is the most common setup issue. ::: 1. Install the extension from the [Visual Studio Marketplace](https://marketplace.visualstudio.com/items?itemName=tetrate.tetrate-model-provider), or from a terminal: ```bash code --install-extension tetrate.tetrate-model-provider ``` 1. Run **Tetrate Agent Router: Set Agent Router API Key** from the Command Palette and paste the key. This step is optional: the extension prompts for the key the first time VS Code resolves models interactively. 1. Run **Chat: Manage Language Models**, select **Tetrate Agent Router**, and enable the required models. The enabled models appear in the chat model picker. Discovery is lazy, so an installed but unconfigured extension costs nothing at startup. :::tip For Enterprise tiers On an Enterprise tier, the base URL must point at the dedicated proxy instance. See [Point at an Enterprise or self-hosted endpoint](#point-at-an-enterprise-or-self-hosted-endpoint). ::: ### Where the key is stored The key is held in VS Code [secret storage](https://code.visualstudio.com/api/references/vscode-api#SecretStorage), backed by the operating system keychain. It is never written to a settings file, a log, or the output channel, and it is excluded from Settings Sync, so it stays on the machine where it was set. ## Configuration | Setting | Type | Default | Scope | Purpose | |---|---|---|---|---| | `tetrate-model-provider.baseUrl` | string | `https://api.router.tetrate.ai/v1` | machine | The OpenAI-compatible endpoint to call | | `tetrate-model-provider.modelFilter` | string[] | `[]` | window | Glob patterns limiting which models are offered; empty offers every chat model | | `tetrate-model-provider.requestHeaders` | object | `{}` | machine | Extra HTTP headers sent with every request | `baseUrl` and `requestHeaders` are machine-scoped, so they can be set in User settings but not in a workspace or folder `settings.json`. Both decide where the API key is sent, and a cloned repository must not be able to redirect it. `modelFilter` only narrows the picker, so it stays settable per workspace. Changing any of these reloads the model list. ### Point at an Enterprise or self-hosted endpoint The base URL is pre-configured for the hosted service. For a dedicated [Agent Router Enterprise](https://tetrate.io/agent-router-product) tenant or a self-hosted deployment, change it in Settings or with **Tetrate Agent Router: Set Base URL** from the Command Palette: ```json { "tetrate-model-provider.baseUrl": "https://router.tare-.tetrate.ai/v1" } ``` Input is normalized before use: surrounding whitespace and trailing slashes are stripped, and `/v1` is appended when no version segment is present. Both `http` and `https` are accepted, which allows a local proxy such as `http://localhost:8080`. ### Filter the model list The hosted catalog exposes more than 160 conversational models. To keep the picker manageable, restrict it by glob pattern: ```json title=".vscode/settings.json" { "tetrate-model-provider.modelFilter": ["claude-*", "gpt-5.6-*", "gemini-3.1-pro-preview"] } ``` Only `*` is special, and it matches within and across segments. Everything else, including `.` and `-`, compares literally; matching is case-insensitive. A model is offered when it matches at least one pattern. Because the filter is workspace-scoped, a team working under an approved model list can commit it to `.vscode/settings.json` and every developer on the project sees the same narrowed catalog. ### Add request headers Use `requestHeaders` for a routing hint or a tenant identifier required by a self-hosted deployment: ```json { "tetrate-model-provider.requestHeaders": { "X-Tenant-Id": "team-platform" } } ``` Do not put the API key here. It belongs in secret storage, and settings files are frequently committed to source control. An `Authorization` entry is discarded: that header is always derived from the stored key. ## Commands | Command | Effect | |---|---| | Tetrate Agent Router: Set Agent Router API Key | Store or replace the API key | | Tetrate Agent Router: Clear Agent Router API Key | Remove the stored key | | Tetrate Agent Router: Set Base URL | Change the endpoint, with validation | | Tetrate Agent Router: Refresh Model List | Discard the cached model list and re-query the endpoint | ## Use the models from another extension Extension authors need no dependency on this extension and no coordination with Tetrate. Select by vendor through the VS Code API: ```typescript const [model] = await vscode.lm.selectChatModels({ vendor: 'tetrate-agent-router', // family: 'anthropic', // id: 'claude-sonnet-5', }); if (!model) { return; } const response = await model.sendRequest( [vscode.LanguageModelChatMessage.User('Summarize this file.')], { justification: 'Generating a summary of the open file.' }, cancellationToken ); for await (const chunk of response.text) { process.stdout.write(chunk); } ``` The selector fields map onto the model list as follows: | Field | Meaning | Examples | |---|---|---| | `vendor` | Always `tetrate-agent-router` | `tetrate-agent-router` | | `family` | The upstream provider name | `anthropic`, `openai`, `gemini`, `xai`, `groq` | | `id` | The model id as the endpoint reports it | `claude-sonnet-5`, `gpt-5.6-terra`, `xai/grok-4.5` | VS Code asks the user for consent the first time an extension sends a request, so the `justification` string is worth writing carefully. Tool calling is supported: pass `tools` in the request options and read `LanguageModelToolCallPart` values from `response.stream`. Provider-specific options such as `temperature`, `max_tokens`, and `reasoning_effort` pass through unchanged via `modelOptions`. Full details, including request translation and streaming behaviour, are documented on the [extension's Marketplace page](https://marketplace.visualstudio.com/items?itemName=tetrate.tetrate-model-provider). ## Behaviour and limitations - **Output length.** No token cap is sent, so each model's server-side default applies. Set `max_tokens` or `max_completion_tokens` through `modelOptions`. - **Images** are sent only to models reporting vision support. Audio and PDF inputs are not forwarded. - **Reasoning traces** are not surfaced; the supported VS Code versions have no thinking part to render them into. - **Token counts** are estimated locally and deliberately pessimistically, since the endpoint exposes no token-counting route. - **Prompt caching** is not configured explicitly. Where the upstream provider applies it automatically, it still takes effect. - **Errors** propagate as-is: a 401 or 403 becomes `LanguageModelError.NoPermissions`, a 404 becomes `NotFound`, and everything else surfaces with the status and the server's message. ## Troubleshooting | Symptom | Resolution | |---|---| | No models in the picker | Models start hidden. Run **Chat: Manage Language Models** and enable them. | | No models after enabling | Check the key with **Set Agent Router API Key**, then **Refresh Model List**. | | Models missing after editing settings | `modelFilter` may exclude them. An empty array offers everything. | | A 401 on every request | The key is invalid or revoked. Set a fresh one from the Console. | | A 404 on every request | The base URL is wrong. It must end in `/v1` or another version segment. | | Fallback limits on every model | `router.tetrate.ai` is unreachable, so catalog metadata is unavailable. Discovery still works. | | Claude models appear twice | Another Claude provider extension is installed. Both contribute under separate vendors. | The **Tetrate Agent Router** output channel logs the discovered model count, configuration changes, and failures with status codes. Open it from **View → Output** and pick the channel from the dropdown. It never logs the API key or message content. Where to go next Make an API call Send the first request through the gateway with the OpenAI SDK, curl, or the tare CLI. Coding agents & tools Connect Claude Code, Cursor, Codex, and other tools to Agent Router. --- ## Generate embeddings Embeddings sit underneath semantic search, retrieval-augmented generation, clustering, deduplication, and recommendation, the workloads that turn a corpus of text into something a model can reason over. The mechanics are the same as any other gateway call: an OpenAI-compatible request is sent to one endpoint, an API key authenticates it, and Agent Router routes it to whichever embedding model has been enabled. Routing embedding traffic through Tetrate Agent Router means the same governance, cost tracking, and provider independence that apply to chat traffic apply to embeddings too: one endpoint, one credential, and one place to see what was spent and which model produced which vectors. This guide covers the embedding path end-to-end: why an embedding model is reached through Agent Router rather than directly, how an enabled embedding model is selected from the catalogue, how the OpenAI-compatible embeddings endpoint is called from curl and from Python, how a batch of inputs is embedded in a single call, how the resulting vectors feed a semantic-search or RAG pipeline, why an embedding-model version is pinned so that vectors stay comparable over time, and how the call is confirmed in Request Logs. It builds directly on the key and routing setup from Route requests across providers; the embeddings endpoint reuses that same key and proxy endpoint. **Persona:** Developer working in the Developer Console and in the application's own code. **Estimated time:** 15 to 20 minutes for the first run, including time to select a model in the Console and copy values into a terminal. ## When this guide applies This guide applies whenever an application needs vector representations of text and the goal is to obtain them through Agent Router rather than from a provider directly. Typical situations: - A retrieval-augmented generation pipeline embeds documents at ingestion time and embeds queries at request time. - A semantic-search feature ranks results by vector similarity instead of keyword match. - A clustering, deduplication, or classification job needs a stable numeric representation of a text corpus. - Embedding spend and usage need to be tracked and governed alongside chat traffic rather than billed and audited separately. The one precondition is that an embedding model has been enabled by an operator. Chat models and embedding models are provisioned the same way, but an embedding model has to be exposed in the catalogue before it can be selected. ## Outcomes By the end of this guide: - An enabled embedding model has been identified in the Console catalogue, with its exact identifier noted for use in requests. - A request to `POST /v1/embeddings` returns one or more embedding vectors through the gateway. - A batch of inputs has been embedded in a single call. - The returned vectors are understood well enough to feed a semantic-search or RAG pipeline. - An embedding-model version is pinned so that vectors generated now remain comparable to vectors generated later. - The embedding call is visible in **Request Logs**, with the resolved model, token counts, latency, and cost recorded. ## Prerequisites - A working API key with a routing configuration attached, as set up in [Route requests across providers](/agent-router-service/guides/route-requests-across-providers/). The embeddings endpoint uses the same key and proxy endpoint as chat traffic. - At least one embedding model enabled in the catalogue. Operators enable embedding models the same way they enable chat models; see [Provision custom and self-hosted models](/agent-router-enterprise/guides/operate-and-govern/provision-models-and-providers/provision-custom-and-self-hosted-models/). If no embedding model appears in the catalogue, that step has to happen first. - The gateway's proxy endpoint URL, displayed on the Console Dashboard. The examples below refer to it as `PROXY_URL`, and use `YOUR_API_KEY` for the key. - A terminal with `curl`, or a Python environment with the `openai` package, for the request steps. ## Step 1: select an embedding model from the catalogue Embedding models are listed in the same model catalogue as chat models, alongside their provider, pricing, and status. The identifier shown there is the value that goes in the `model` field of an embeddings request, so the first task is to find an enabled embedding model and note its exact name. 1. Sign in to the Developer Console. 1. Open the model catalogue (**Catalog → Model Catalog**). 1. Filter the list to embedding models. Searching for a known family name, for example `embedding`, narrows the list quickly, as does filtering by provider. 1. Confirm the model's **Status** is enabled. A disabled model cannot be reached even if its identifier is used in a request. 1. Note the exact model identifier and the output vector dimension. Both matter downstream: the identifier is sent on every request, and the dimension determines the width of the vector column in the vector store. Alternatively, discover embedding-capable models for your API key with authenticated `GET /v1/models` (for example, filter returned `id` values that contain `embedding`, or inspect the capability fields). See [Models API](/reference/supported-apis/#models-api-v1models). Embedding models differ from chat models in what they accept and return. An embedding model takes text and returns a fixed-length vector of floating-point numbers; it does not take a `messages` array and does not produce a chat completion. Two properties recorded in the catalogue are worth carrying forward: | Property | Why it matters | | :---- | :---- | | Model identifier | The value sent in the `model` field. Vectors are only comparable when they come from the same identifier. | | Output dimension | The length of every returned vector. The vector store's column width is sized to this value. | ## Step 2: call the embeddings endpoint The gateway exposes an OpenAI-compatible endpoint at `/v1/embeddings`. The request shape is the standard OpenAI embeddings payload, a `model` field and an `input` field, and the API key from the routing setup is presented as a bearer token. The same endpoint serves every enabled embedding model regardless of the upstream provider. In the examples below, `PROXY_URL` stands for the proxy endpoint from the Console Dashboard, scheme included and ending in `/v1`, `YOUR_API_KEY` for the API key, and `EMBEDDING_MODEL` for the identifier noted in Step 1. ### Using curl ```bash curl PROXY_URL/embeddings \ -H "Authorization: Bearer YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "EMBEDDING_MODEL", "input": "The quick brown fox jumps over the lazy dog." }' ``` ### Using Python ```python from openai import OpenAI client = OpenAI( base_url="PROXY_URL", api_key="YOUR_API_KEY", ) response = client.embeddings.create( model="EMBEDDING_MODEL", input="The quick brown fox jumps over the lazy dog.", ) vector = response.data[0].embedding print(len(vector)) ``` A successful call returns a response in the same shape the calling code would expect from OpenAI directly. The vector is found under `data[0].embedding`, and `usage.prompt_tokens` reports the tokens consumed: ```json { "object": "list", "data": [ { "object": "embedding", "index": 0, "embedding": [0.0023, -0.0091, 0.0145] } ], "model": "EMBEDDING_MODEL", "usage": { "prompt_tokens": 11, "total_tokens": 11 } } ``` The `embedding` array above is truncated for readability; a real response contains as many floating-point values as the model's output dimension. For the full embeddings request and response reference, see [Gateway APIs](/reference/supported-apis/). ## Step 3: embed a batch of inputs in one call Embedding workloads are rarely one string at a time. Ingesting a corpus means embedding thousands of chunks, and issuing one HTTP request per chunk is slow and wasteful. The `input` field accepts an array, so a batch of texts is embedded in a single call. Each returned object carries an `index` that maps it back to its position in the input array, so order is preserved without extra bookkeeping. ### Using curl ```bash curl PROXY_URL/embeddings \ -H "Authorization: Bearer YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "EMBEDDING_MODEL", "input": [ "The quick brown fox jumps over the lazy dog.", "A fast auburn fox leaps above a sleepy hound.", "Interest rates were left unchanged at the latest meeting." ] }' ``` ### Using Python ```python from openai import OpenAI client = OpenAI( base_url="PROXY_URL", api_key="YOUR_API_KEY", ) documents = [ "The quick brown fox jumps over the lazy dog.", "A fast auburn fox leaps above a sleepy hound.", "Interest rates were left unchanged at the latest meeting.", ] response = client.embeddings.create( model="EMBEDDING_MODEL", input=documents, ) vectors = [item.embedding for item in sorted(response.data, key=lambda d: d.index)] print(len(vectors), len(vectors[0])) ``` Batching reduces request overhead and is counted as a single entry in Request Logs, with token usage aggregated across the batch. Provider limits apply to the number of inputs and the total tokens accepted per call, so very large corpora are split into batches sized to stay within those limits. Work that runs long enough to need scheduling and resumption is better handled as a job; see [Run batch and long-running jobs](/agent-router-service/guides/run-batch-and-long-running-jobs/). ## Step 4: use the vectors for semantic search and RAG A vector on its own is not useful; value comes from comparing vectors. Texts with similar meaning produce vectors that sit close together, so similarity between two vectors approximates similarity in meaning. Cosine similarity is the usual measure. The end-to-end shape of a retrieval pipeline is consistent regardless of which embedding model produced the vectors: 1. Each document in the corpus is split into chunks sized to the model's input limit and the retrieval granularity required. 1. Every chunk is embedded (in batches, as in Step 3) and each vector is stored in a vector store alongside the source text and any metadata. 1. At query time, the incoming query is embedded with the same model and version used for the corpus. 1. The query vector is compared against the stored vectors, and the nearest matches are retrieved. 1. For RAG, the retrieved text is supplied as context to a chat completion sent through the same gateway; see [Route requests across providers](/agent-router-service/guides/route-requests-across-providers/). The constraint that governs the whole pipeline is consistency: query vectors and corpus vectors are only comparable when they come from the same model and the same version. Mixing vectors from different models, or from different versions of the same model, produces meaningless distances. That constraint is what makes versioning in the next step a requirement rather than a nicety. ## Step 5: pin an embedding-model version Embedding vectors are comparable only within a single model version. When a provider releases a new version of an embedding model, vectors produced by the new version do not align with vectors produced by the old one; the same text maps to a different point in a different space. A corpus embedded under one version and a query embedded under another will not retrieve correctly, even though both calls succeed and both return vectors of the expected dimension. Two practices keep a vector store internally consistent: - **Pin a specific version.** Where the catalogue exposes a dated or otherwise versioned identifier, that exact identifier is used for every embedding call against a given store, rather than a floating alias that may advance to a newer version. Pinning guarantees that today's query vectors and last month's corpus vectors share the same space. - **Re-embed on a deliberate change.** Moving to a different embedding model, or to a new version of the current one, is a corpus-wide operation: the entire corpus is re-embedded with the new model, and queries are switched to it only once that re-embedding is complete. Re-embedding is planned as a migration (run as a batch job, written to a separate index, and cut over atomically) rather than applied piecemeal. Recording the model identifier and version as metadata alongside each stored vector makes a later migration straightforward: the records embedded under the old version are identifiable, and the cutover can be verified. Because Agent Router records the resolved model on every request, Request Logs also provides an independent record of which model actually served each embedding call. ## Step 6: verify the call in request logs Issuing an embeddings request is not the same as confirming which model served it and what it cost. Request Logs is the developer-facing record of every request that flowed through the gateway under a given API key, embeddings included. 1. In the Console, open Request Logs (**Monitoring → Request Logs**). 1. Locate the embeddings request, the most recent entry under the API key used for the call. 1. Expand the row to view the detail panel. 1. Confirm the following fields are populated and consistent with the model selected in Step 1: | Field | What to check | | :---- | :---- | | Resolved model | Matches the embedding model identifier sent in the request, including its pinned version. | | Provider | Matches the upstream provider for that model. | | Token counts | Input tokens are present and non-zero; embeddings report prompt tokens only, with no completion tokens. | | Latency | The end-to-end time the gateway observed for the call. | | Cost | The computed cost based on the resolved model and token usage. | A batch call from Step 3 appears as a single entry, with token usage aggregated across every input in the batch. Confirming the resolved model and version here is the fastest way to catch a request that was unintentionally sent to the wrong embedding model, the failure mode that silently corrupts a vector store. The aggregated view of embedding spend and volume by model and key is covered in [Monitor traffic and usage](/agent-router-service/guides/monitor-traffic-and-usage/). ## What to do next - **[Route requests across providers](/agent-router-service/guides/route-requests-across-providers/)**: the key and routing setup the embeddings endpoint reuses, and the path for the chat completions that consume retrieved context in a RAG pipeline. - **[Run batch and long-running jobs](/agent-router-service/guides/run-batch-and-long-running-jobs/)**: embed a large corpus or re-embed after a version change as a scheduled, resumable job rather than a single request. - **[Provision custom and self-hosted models](/agent-router-enterprise/guides/operate-and-govern/provision-models-and-providers/provision-custom-and-self-hosted-models/)**: the operator-side work that enables an embedding model, including self-hosted and custom embedding endpoints. - **[Gateway APIs](/reference/supported-apis/)**: the full request and response reference for `/v1/embeddings` and the other supported formats. Where to go next Run batch and long-running jobs Embed a large corpus or re-embed after a version change as a scheduled, resumable job. Gateway APIs The full request and response reference for the embeddings endpoint. --- ## Improve resilience with fallbacks Production AI traffic is not nearly as reliable as it looks on a calm afternoon. The major commercial providers run high-availability infrastructure, but they still have bad hours: regional incidents, capacity exhaustion, rate-limit throttling, model deprecations rolled out faster than expected, and the occasional outright outage. An application that calls a single provider directly has no answer to any of these events; the user sees an error, and someone is paged. Agent Router's fallback model removes that single point of failure without requiring any retry or branching logic inside the application. A fallback policy is an ordered list of backends. The gateway tries the first one; if the call fails with a recoverable error, the gateway walks to the next backend in the list and tries again, continuing until either a backend succeeds or the chain is exhausted. The calling application sees a single request that either returns a response or returns a final error; the underlying failover happens inside the gateway and is invisible to the caller. This guide covers how to design that chain, configure it against an existing API key, exercise the failover path to confirm it behaves as expected, and verify in Request Logs which backend actually served each request. **Persona:** Developer working in the Developer Console. **Estimated time:** 10 to 20 minutes, plus a short period of intentional failure exercise. ## When this guide applies Fallback chains are the right tool when the goal is **resilience**: keeping requests successful in the face of provider issues, without distributing traffic deliberately across providers under normal conditions. The chain is walked only on failure; a healthy primary provider serves 100 % of traffic, and the secondary backends are effectively idle until something goes wrong. For other routing intents, different patterns apply: | Intent | Pattern | Where it is covered | | :---- | :---- | :---- | | Distribute traffic by weight (cost, evaluation, gradual rollout) | Traffic splitting | *Reduce Cost with Traffic Splitting* | | Route based on request attributes (tenant, header, task type) | Advanced routing rules | *Apply Advanced Routing Rules* | | Enforce data residency or compliance boundaries | Fallback policy with carefully chosen providers | This guide (see *Routing under compliance constraints* below) | Fallback and traffic splitting can be combined. A weighted split selects the primary backend for a given request, and an additional fallback chain takes over only when the chosen backend fails. The patterns are not mutually exclusive. ## Outcomes By the end of this guide: - A fallback policy with at least two backends from different providers is attached to a working API key. - The chain has been tested under at least one simulated failure, with the gateway falling through to the secondary backend. - Request Logs show, for each request, which backend actually handled it and whether a fallback event occurred. ## Prerequisites This guide builds directly on [Route Requests Across Providers](/agent-router-service/guides/route-requests-across-providers/). Specifically, the following should already be in place: - An API key created in the Console, with a working routing configuration attached. - At least two enabled models in the Admin Dashboard, ideally from different upstream providers. A chain with two backends from the same provider still works mechanically, but it provides no protection against a provider-wide outage, which is the most common failure the chain is designed to absorb. - A terminal with `curl` (or a Python environment with the `openai` package) for the exercise step. ## Step 1: choose the chain composition A good fallback chain is not "a primary model plus whatever else is enabled". The composition of the chain determines what kinds of failures it can absorb and how surprised the calling application will be when failover occurs. Three considerations matter when choosing backends: - **Provider independence.** The chain protects against provider failures only to the extent that its backends fail independently. A chain consisting of two OpenAI models still goes down when OpenAI does. A chain that alternates providers (for example, an OpenAI primary with an Anthropic or Google secondary) absorbs single-provider outages without disruption. - **Quality consistency.** When the chain falls through, the application gets a response from a different model. If the secondary model produces meaningfully worse output, the failover is visible to the end user even though the request technically succeeded. Where quality consistency matters, choose secondaries that are close to the primary in capability: frontier model behind frontier model, not frontier model behind a budget option. - **Cost profile.** Some chains are deliberately built so that the secondary is cheaper than the primary. This is acceptable when degraded-but-cheaper output is preferable to no output at all, and it has the side effect of reducing cost exposure during long-running incidents on the primary provider. A typical first chain pairs a primary frontier model with one secondary from a different provider: | Position | Example backend | Role | | :---- | :---- | :---- | | Priority 0 (Primary) | `gpt-4o` | Serves 100 % of traffic under normal conditions | | Priority 1 (Fallback) | `claude-sonnet-4-20250514` | Used when the primary returns a recoverable failure | Additional backends can be added at priority 2 and beyond. The marginal value drops quickly (two well-chosen backends absorb the vast majority of realistic incidents), but a third backstop is reasonable for workloads with strict SLA commitments. ## Step 2: configure the fallback policy The chain is attached to the API key created in the previous guide. The same key can hold only one routing configuration at a time, so editing this configuration replaces whatever was attached previously. 1. From the **API Keys** page in the Console, open the detail page of the key the configuration should apply to. 1. Scroll to the **Routing Configuration** section. 1. Click **Configure** or **Add Rule**. 1. Select **Fallback Policy**. 1. Add the primary model and select it at priority 0. 1. Click **Add Fallback** and choose the secondary backend. The Console assigns priority 1 automatically. 1. Optionally repeat to add further backups at priorities 2 and beyond. 1. Save the configuration and confirm the **Active** toggle is on. The Console exposes drag-and-drop reordering and a remove control on each row, which is useful when chains are being tuned over time, for example when a secondary provider is being promoted to primary after performing well during an incident. ![Fallback policy configuration](/img/screenshots/console/config-fallback.jpeg) ### Fallback policy field reference The Routing Configuration section exposes the following controls for a fallback policy: - **Priority.** Each backend is assigned a priority level that fixes its position in the chain. Priority 0 is the primary and receives all traffic under normal conditions; priority 1 is the first fallback, priority 2 the second, and so on. The error from the last attempted backend is returned to the caller only when every backend in the chain has failed. - **Weight.** Each backend in a fallback chain is assigned a weight of 1, so a single backend handles all traffic at its priority level. Weight is the mechanism used by traffic splitting rather than fallback; in a fallback policy it stays fixed at 1. - **Active/inactive toggle.** A policy can be switched inactive without being deleted. While inactive, the chain is not enforced and requests go directly to the model named in the API call. - **Reorder.** Drag a row, or use the priority controls, to change a backend's position in the chain, then save. - **Remove.** The remove control on a row drops that backend; the remaining backends keep their relative priority order. :::tip Keep one API key per logical chain composition rather than swapping the configuration on a shared key. Per-key routing is Agent Router's natural separation boundary, and Request Logs are filtered per key, so a separate key per chain keeps the analytics tidy and the rollback path obvious. ::: ## Step 3: test the failover path A fallback chain that has never failed over is effectively unverified. The simplest way to exercise the chain is to introduce a failure on the primary backend and observe whether the gateway successfully walks to the secondary. Two approaches are practical, depending on how much disruption is acceptable in the environment under test. - **Misconfigure the primary temporarily.** Change the primary backend in the routing configuration to a model that is enabled in the catalogue but lacks a working provider connection, then send a request. The gateway records the primary failure and falls through to the secondary, which should succeed. Restore the original primary after the test. - **Use a non-production API key.** Repeat the exercise against a key that is not serving production traffic, so that the experiment is fully isolated from real users. Send a test request after each change. `PROXY_URL` stands for the proxy endpoint from the Console Dashboard, scheme included and ending in `/v1`: ```bash curl PROXY_URL/chat/completions \ -H "Authorization: Bearer YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "gpt-4o", "messages": [{"role": "user", "content": "Confirm which backend served this request."}] }' ``` The request should still succeed. The `model` field in the response body reflects the backend that actually produced the answer, which, during a deliberate failover exercise, should be the secondary rather than the primary. :::note Not every provider failure is a recoverable one. The gateway walks the chain on transient errors (5xx responses, timeouts, connection failures, or rate-limit responses) but not on client-side errors such as malformed requests or authentication failures. Those are returned to the caller directly, because retrying them against a different backend would not change the outcome. ::: ## Step 4: verify the failover in request logs The exercise above is only useful if its result is observable after the fact. Request Logs records, per request, which backend actually served the response and whether a fallback event occurred. 1. In the Console, open **Monitoring → Request Logs**. 1. Filter by the API key used in the test. 1. Locate the request issued during the exercise. 1. Expand the row to see the detail panel. 1. Confirm the **Resolved model** field matches the secondary backend, not the primary. If a fallback event is recorded, it appears in the request timeline as a separate attempted-and-failed entry on the primary, followed by a successful entry on the secondary. The expected pattern in normal operation is the opposite: every request resolves to the primary, and no fallback events appear. A sudden cluster of fallback events in production is a strong leading indicator that something is happening on the primary provider's side, often visible in Request Logs before the provider's own status page acknowledges the incident. For richer filtering (by time range, by status, or by resolved model), see [Monitor traffic and usage](/agent-router-service/guides/monitor-traffic-and-usage/). ## Routing under compliance constraints Fallback chains are also the natural mechanism for policy-driven routing where compliance rather than resilience is the dominant concern. Because the chain contains only the backends explicitly added to it, the chain itself is the enforceable boundary: a chain that contains only EU-hosted providers cannot, by construction, route to a US-hosted provider, regardless of what the calling application requests. Three common patterns use this property directly: - **Data residency.** Build the chain from providers in the required region only (for example, Azure EU as the primary with GCP EU as the secondary) so failover never leaves the region. - **Approved providers.** Limit the chain to providers that have been reviewed and approved for the use case in question, and assign that chain to a key dedicated to that use case. - **Output consistency.** Use the same logical model class across positions in the chain to keep response quality steady during failover, even when the upstream provider changes. There is no "compliance mode" to enable. The boundary is enforced by the composition of the chain itself. ## What to do next - **[Reduce cost with traffic splitting](/agent-router-service/guides/reduce-cost-with-traffic-splitting/):** distribute traffic across backends by weight rather than priority, useful for cost management and gradual rollout. - **[Apply advanced routing rules](/agent-router-service/guides/apply-advanced-routing-rules/):** combine fallback semantics with attribute-based dispatch. - **[Use your own provider credentials](/agent-router-service/guides/use-your-own-provider-credentials/):** introduce BYOK credentials into one or more positions in the fallback chain. - **[Monitor traffic and usage](/agent-router-service/guides/monitor-traffic-and-usage/):** extend the verification step into ongoing dashboards, with cost and latency tracked per resolved model. The guide assumes the fallback chain configured above remains attached to the key. Subsequent guides build on this configuration rather than replacing it. Where to go next Reduce cost with traffic splitting Distribute traffic across backends by weight rather than priority. Apply advanced routing rules Combine fallback semantics with attribute-based dispatch. --- ## Guides for Developers Written for the developer persona: the engineer who consumes AI models through the gateway and works primarily in the Developer Console. Each guide is organized around a single goal rather than a single screen. It opens with why the goal matters and when it applies, names the persona, walks through the task end to end, and closes with a pointer to the next related goal. The recommended starting point is [Route requests across providers](/agent-router-service/guides/route-requests-across-providers/), which establishes the request path that every other developer guide builds on. The remaining guides can be read in any order as the need arises. Quickstarts Make an API call through Agent Router Use Python and the OpenAI SDK to make API calls through Agent Router. Build a chat app Use Python and the OpenAI SDK to build a chat app that uses some of Agent Router's core functionality. Common tasks Route requests across providers Create a key, attach a routing configuration, and send the first request through the gateway. Improve resilience with fallbacks Formalise the multi-provider chain into an ordered fallback policy and tune failover. Reduce cost with traffic splitting Distribute requests across backends by weight for cost management and gradual rollout. Apply advanced routing rules Use logical model names and attribute-based dispatch for A/B and canary patterns. Use your own provider credentials Register BYOK credentials and compose them with routing patterns. Aggregate MCP servers into a profile Combine multiple MCP servers into a single endpoint for AI clients. Integrate the gateway with an app Point SDKs, frameworks, and code assistants at the gateway. Test prompts in the Playground Experiment with prompts and compare models interactively. Monitor traffic and usage Investigate individual requests and aggregate usage across keys, models, and providers. Export telemetry to an observability stack Send traces and metrics to an external observability platform. --- ## Integrate the gateway with an app A working API key, a thoughtful routing configuration, and a tested fallback chain are all useful in isolation, but their value only materialises when an application actually talks to the gateway instead of talking to providers directly. In almost every case this is a trivial change: the gateway speaks the OpenAI HTTP API, so any tool, SDK, or framework that already supports a configurable OpenAI-compatible endpoint can be pointed at the gateway by changing two values: the base URL and the API key. The rest of the application code stays exactly as it was. This guide covers the integration patterns that come up most often: the OpenAI Python and JavaScript SDKs that the majority of applications use directly, the AI frameworks (LangChain, Vercel AI SDK, Pydantic AI) that wrap those SDKs, the code-assistant tools (Aider, Continue, Cline, Cursor, Roo Code) that accept a configurable endpoint, and the agent frameworks (OpenAI Agent SDK, CrewAI, Goose, Open WebUI) that drive multi-step workflows. The mechanic is the same in every case; only the location of the two settings changes. The Console surfaces the same recipes in the Build > Integrations panel, grouped by category, with a copy-paste snippet for each tool. ![Integrations](/img/screenshots/console/integrations.jpeg) **Persona:** Developer working in the Developer Console and in the application's own codebase or configuration. **Estimated time:** 5 to 15 minutes per integration, depending on whether the tool reads its configuration from environment variables, a config file, or an in-app settings panel. ## When this guide applies This guide applies whenever the goal is to send AI traffic from an application or tool through the gateway rather than directly to a provider. In practical terms, that is almost every situation: the gateway absorbs the variability of provider APIs and gives the routing, observability, and policy benefits covered in the earlier guides for free. There is rarely a reason to keep an application pointed directly at a provider once the gateway is in place. The one consistent precondition is OpenAI compatibility on the application side. Tools and SDKs that accept a configurable OpenAI-compatible base URL, which now includes virtually every general-purpose AI library, integrate without code changes. Tools that hardcode a specific provider's SDK without offering a base-URL override are the rare exception and may require a small wrapper. ## Outcomes By the end of this guide: - At least one application, SDK, or tool is pointed at the gateway through its base URL and API key. - A request issued from that integration completes successfully end-to-end. - The request appears in Request Logs, attributed to the API key used by the integration. - The mental model for adding further integrations ("point base URL at the gateway, present the API key, leave everything else alone") is established. ## Prerequisites - A working API key with a routing configuration attached, as set up in [Route Requests Across Providers](/agent-router-service/guides/route-requests-across-providers/). - The gateway's proxy endpoint URL, displayed on the Console Dashboard. In the examples below this is referred to as `PROXY_URL`. - The application, SDK, or tool that will be integrated. The integration steps in the rest of this guide assume it is already installed and working against a provider directly; the integration is a configuration change, not a fresh install. ## The general pattern Every integration in this guide follows the same shape: 1. The application's **base URL** (sometimes called the API base, the endpoint, or the proxy URL) is changed from the provider's default to the gateway's `PROXY_URL`. 1. The application's **API key** is changed from the provider's key to the Agent Router API key. 1. The application's **model identifier** is set to any model the routing configuration exposes: either a provider model name such as `gpt-4o` or a logical name defined under [Apply Advanced Routing Rules](/agent-router-service/guides/apply-advanced-routing-rules/). The rest of the application's code is unchanged. Where a snippet in this guide shows `PROXY_URL` and `YOUR_API_KEY`, those are placeholders for the values from the Console. ## SDK integrations The OpenAI SDKs are the most common starting point because so many applications are built directly against them. The integration is a two-line change. ### OpenAI Python SDK ```python from openai import OpenAI client = OpenAI( base_url="PROXY_URL", api_key="YOUR_API_KEY", ) response = client.chat.completions.create( model="gpt-4o", messages=[{"role": "user", "content": "Hello, world!"}], ) print(response.choices[0].message.content) ``` The same `client` object handles the Chat Completions API (above), the Responses API (`client.responses.create`), and streaming variants of both; the gateway accepts all three. For a full reference of the supported API shapes including streaming examples, see [Gateway APIs](/reference/supported-apis/). The Responses API is reached through the same client with no change to the base URL or key: ```python from openai import OpenAI client = OpenAI( base_url="PROXY_URL", api_key="YOUR_API_KEY", ) response = client.responses.create( model="gpt-4o", input="Hello, world!", ) print(response.output_text) ``` ### OpenAI JavaScript SDK ```javascript import OpenAI from "openai"; const client = new OpenAI({ baseURL: "PROXY_URL", apiKey: "YOUR_API_KEY", }); const response = await client.chat.completions.create({ model: "gpt-4o", messages: [{ role: "user", content: "Hello, world!" }], }); console.log(response.choices[0].message.content); ``` The same JavaScript client supports both the Chat Completions and Responses APIs. The Responses API is called on the same client: ```javascript import OpenAI from "openai"; const client = new OpenAI({ baseURL: "PROXY_URL", apiKey: "YOUR_API_KEY", }); const response = await client.responses.create({ model: "gpt-4o", input: "Hello, world!", }); console.log(response.output_text); ``` In environments where the API key cannot be hard-coded, which is most environments, standard secret-management practices apply: environment variables in development, secret managers or platform-supplied configuration in deployed environments. ## Framework integrations Higher-level frameworks wrap the OpenAI SDK and expose their own configuration surface. The mapping to a gateway integration is straightforward. ### LangChain ```python from langchain_openai import ChatOpenAI llm = ChatOpenAI( base_url="PROXY_URL", api_key="YOUR_API_KEY", model="gpt-4o", ) response = llm.invoke("Hello, world!") print(response.content) ``` LangChain's `ChatOpenAI` is implemented on top of the OpenAI SDK, so chains, agents, retrieval pipelines, and tool integrations all work without further change once the underlying client is pointed at the gateway. ### Vercel AI SDK ```typescript import { createOpenAI } from "@ai-sdk/openai"; import { generateText } from "ai"; const provider = createOpenAI({ baseURL: "PROXY_URL", apiKey: "YOUR_API_KEY", }); const { text } = await generateText({ model: provider("gpt-4o"), prompt: "Hello, world!", }); console.log(text); ``` The Vercel AI SDK's streaming, tool-use, and structured-output features all flow through the same `provider` definition. ### Pydantic AI ```python from pydantic_ai import Agent from pydantic_ai.models.openai import OpenAIModel model = OpenAIModel( "gpt-4o", base_url="PROXY_URL", api_key="YOUR_API_KEY", ) agent = Agent(model) result = agent.run_sync("Hello, world!") print(result.data) ``` ## Code-assistant integrations Code-assistant tools differ from SDKs in that they are configured outside the application's own codebase, through settings panels, configuration files, or environment variables. The mapping is still the same two-value change. For supported tools (including Aider, Claude Code, Cline, Codex, and Continue), prefer [Configure coding tools with the tare CLI](/agent-router-service/guides/coding-agents/configure-tools-with-tare-cli/): `tare integrate` detects the local install, writes the tool config from your logged-in session, and validates the key. Use the manual snippets below when the CLI does not cover your tool or you need to edit config by hand. ### Aider Aider reads its configuration from `OPENAI_API_BASE` and `OPENAI_API_KEY`: ```bash export OPENAI_API_BASE=PROXY_URL export OPENAI_API_KEY=YOUR_API_KEY aider --model gpt-4o ``` ### Continue Continue is configured through `~/.continue/config.json`: ```json { "models": [ { "title": "Agent Router", "provider": "openai", "model": "gpt-4o", "apiBase": "PROXY_URL", "apiKey": "YOUR_API_KEY" } ] } ``` ### Cline, Cursor, and Roo Code These tools accept an OpenAI-compatible endpoint through their settings UI. The steps are similar across all three: 1. Open the tool's settings or preferences. 1. Select **OpenAI Compatible** (or equivalent) as the API provider. 1. Set the base URL to `PROXY_URL`. 1. Enter the Agent Router API key. 1. Set the model to any model the routing configuration exposes. 1. Save and, if prompted, restart the tool. The exact wording of each setting varies between tools; the mapping is consistent. ## Agent-framework integrations Agent frameworks add their own abstractions over the model client but ultimately call out through an OpenAI-compatible interface. The integration pattern is unchanged. ### OpenAI Agent SDK ```python from openai import OpenAI from agents import Agent, Runner client = OpenAI( base_url="PROXY_URL", api_key="YOUR_API_KEY", ) agent = Agent( name="my-agent", instructions="You are a helpful assistant.", model="gpt-4o", ) result = Runner.run_sync(agent, "Hello, world!") print(result.final_output) ``` ### CrewAI ```python from crewai import Agent, Task, Crew, LLM llm = LLM( model="openai/gpt-4o", base_url="PROXY_URL", api_key="YOUR_API_KEY", ) agent = Agent( role="Researcher", goal="Find information", backstory="You are a research assistant.", llm=llm, ) task = Task( description="Summarize the latest AI trends.", agent=agent, expected_output="A summary of AI trends.", ) crew = Crew(agents=[agent], tasks=[task]) result = crew.kickoff() print(result) ``` ### Goose and other CLI agents CLI-based agents pick up the OpenAI base URL and key from environment variables in the same way as Aider: ```bash export OPENAI_API_BASE=PROXY_URL export OPENAI_API_KEY=YOUR_API_KEY goose session start --model gpt-4o ``` The same pattern works for any local agent or script that uses the OpenAI SDK without overriding its defaults. Where the agent reads its model from the environment as well, the model identifier can be exported alongside the base URL and key: ```bash export OPENAI_API_BASE=PROXY_URL export OPENAI_API_KEY=YOUR_API_KEY export MODEL_NAME=gpt-4o ``` ### Open WebUI Open WebUI is configured through its admin panel rather than environment variables: 1. Go to **Admin Settings → Connections**. 1. Under **OpenAI API**, set the base URL to `PROXY_URL`. 1. Enter the Agent Router API key. 1. Save and refresh the model list. Every model exposed by the routing configuration appears in the Open WebUI model picker. ## Verifying the integration A successful integration shows up in three places: in the application itself, in Request Logs, and in usage analytics. 1. Issue at least one request from the integrated application. The expected outcome is a successful response in the application's normal output: console output, web UI, or wherever the application surfaces model responses. 1. Open Request Logs and filter by the API key the integration is using. The request should appear within seconds. See [Monitor Traffic and Usage](/agent-router-service/guides/monitor-traffic-and-usage/). 1. Confirm the request's metadata matches expectations: the resolved model, the latency, and the token counts. For longer-term observation across many integrations, usage analytics aggregates traffic by API key and by model. A common operational pattern is to issue a separate API key per integrated application, which makes the per-application picture obvious without any further configuration. ## What to do next - **Test prompts in the Playground:** exercise model behaviour interactively before wiring an integration into application code. See [Test Prompts in the Playground](/agent-router-service/guides/test-prompts-in-the-playground/). - **Monitor traffic and usage:** track each integration's traffic separately by per-integration API key. See [Monitor Traffic and Usage](/agent-router-service/guides/monitor-traffic-and-usage/). - **Export telemetry to an observability stack:** push integration-level telemetry into the organisation's existing dashboards. Where to go next Test prompts in the Playground Exercise model behaviour interactively before wiring an integration into code. Monitor traffic and usage Track each integration's traffic separately by per-integration API key. --- ## Monitor traffic and usage The hardest part of running AI in production is rarely the first request. The hard part is what happens later: an unfamiliar bill at the end of the month, a sudden latency regression that no application change explains, a quiet drift in which model is actually serving a particular path, a key that has started sending traffic from somewhere it should not. Two complementary surfaces in the Console make monitoring easy: Request Logs is the per-request record of every gateway call, and Usage Analytics is the aggregate view of total requests and spend, broken down by model, provider, and API key. This guide covers when each surface is the right one, how the API-key-per-purpose pattern makes both of them dramatically more useful, and the most common monitoring workflows. **Persona:** Developer working in the Developer Console. **Estimated time:** 10 to 15 minutes for an initial tour of both surfaces; ongoing thereafter. ## When this guide applies The monitoring surfaces are useful whenever AI traffic is flowing. The guide is especially relevant in three recurring situations: | Situation | Surface that helps most | | :---- | :---- | | Debugging a specific bad request | Request Logs | | Verifying that a fresh integration is wired correctly | Request Logs | | Understanding cost trends and projecting future spend | Usage Analytics | | Comparing the volume served by two backends in a traffic split | Usage Analytics | | Detecting an unexpected usage spike on a specific API key | Usage Analytics, then Request Logs to drill in | For programmatic access to the same data (exporting to a long-term observability stack, building external dashboards, or alerting), the *Export Telemetry to an Observability Stack* guide is the right next step. ## Outcomes By the end of this guide: - The Request Logs and Usage pages have both been opened and the available filters exercised. - At least one request has been located in Request Logs and its detail panel reviewed. - Usage Analytics has been filtered by API key and inspected for the model and provider breakdown. - The API-key-per-purpose pattern is understood as the way to make both surfaces precise rather than blurry. ## Prerequisites - A working API key with at least a few requests of recent traffic against it, as set up in [Route Requests Across Providers](/agent-router-service/guides/route-requests-across-providers/) and exercised through any of the earlier guides. - Optional but recommended: a second API key created for a different purpose (Playground, an SDK integration, an experimental routing configuration), so the API-key filter is meaningful rather than degenerate. ## Step 1: find a specific request in request logs Request Logs is the closest Agent Router comes to a single source of truth for what actually happened on any given call. Each row corresponds to one request, in reverse chronological order. ![Request Logs](/img/screenshots/console/request-logs.jpeg) 1. Sign in to the Console. 1. Open **Monitoring → Request Logs** from the sidebar. 1. The table shows every recent request, with the following columns: | Column | What it captures | | :---- | :---- | | Timestamp | When the gateway received the request | | Model | The model that handled the request, the *resolved* model, after routing decisions are applied | | Tokens (Input) / Tokens (Output) | Token counts attributed to the request | | Cost | Computed from the resolved model's pricing and the token usage | | Latency | End-to-end time the gateway observed for the call | | Status | HTTP status code (`200` for success; `429` for rate limit; `5xx` for failure) | 1. Narrow the view using the available filters: - **Text search** finds requests by content keyword. - **API key filter** isolates traffic from one integration or team member. - **Time-range filter** preset windows of 1h, 24h, 7d, 30d, or 90d. 1. Click any row to open the request's detail panel, which contains: - The request headers (with the API key masked). - The full request body, including model, messages, and parameters. - The full response body. - A token-by-token breakdown of input and output. - The cost calculation, derived from the model's per-token pricing. - A latency breakdown: time-to-first-token and total response time. - Status and any error message, if the call failed. ![Request Detail](/img/screenshots/console/request-detail.jpeg) The detail panel is the level of granularity that settles most debugging questions: did the request reach the gateway, which backend handled it, what exactly did the model see, what exactly did the model return, and how long did it take. When the result set exceeds a single page, the pagination controls at the bottom of the table page through the remaining requests at a fixed number of rows per page. To debug a request that failed rather than verify one that succeeded, the table is best read from its status column inward: 1. Set the time-range filter to the window in which the failure occurred. 1. Scan for rows whose status is not `200`. 1. Open the row and read the error message in the response body, alongside the status code, to determine the cause. The Cost column is also the quickest way to attribute spend at the per-request level: with the API-key filter applied, the cost of each individual request made with that key is visible directly in the table. For aggregated cost across many requests, Usage Analytics is the better surface. ## Step 2: verify an integration in request logs A typical use of Request Logs is to confirm that a freshly-configured integration is doing what its author expected. 1. Issue at least one request from the integration: a Playground exchange, an SDK call, a tool invocation through an MCP profile, or an editor extension talking to the gateway. 1. Switch to **Monitoring → Request Logs** in the Console. 1. Filter by the API key the integration is using. 1. Locate the request in the table. 1. Confirm three things in the detail panel: - The **resolved model** matches what was expected. If a routing policy is in play, the resolved model might differ from the one the application asked for; both values are visible. - The **status** is `200`. A `429` typically indicates rate limiting on an upstream provider; `5xx` indicates an upstream failure. - The **request body** matches what the integration intended to send. Prompt templating bugs are the most common cause of "the model is misbehaving" reports, and they are immediately obvious here. If the integration is producing no log entries at all, the problem is almost always on the integration side: a wrong base URL, a missing API key, or a network configuration issue. The gateway logs everything it sees; absence of a log entry means the request never arrived. ## Step 3: look at aggregate behaviour in usage analytics Where Request Logs answers "what happened on this specific call", Usage Analytics answers "what is the overall picture across many calls". The page surfaces two summary metrics (**Total Requests** and **Total Spend**) and a set of breakdowns. ![Usage Analytics](/img/screenshots/console/usage-analytics.jpeg) 1. Open **Monitoring → Usage** from the sidebar. 1. Select a time range. Last 24 hours, Last 7 days, Last 30 days, and Last 90 days are typical presets. 1. Read the summary cards at the top of the page for an at-a-glance picture of activity. 1. Scroll through the available breakdowns: | Breakdown | What it shows | | :---- | :---- | | Usage trends over time | A time-series of request volume and cost across the selected window | | Requests by model | How many requests were routed to each model, surfacing which models the workload actually leans on | | Cost breakdown by provider | Total cost attributed to each provider, showing which provider is driving spend | 1. Apply the **API key filter** to restrict every panel on the page to a specific key. When no filter is applied, every panel reflects all API keys combined. The filter is what turns the page from a single number per organisation into a per-integration, per-team, or per-experiment view. The pattern that makes this work is covered in Step 4. The breakdowns also support three interpretive uses beyond reading the current numbers: - **Cost optimisation:** the cost-by-provider and requests-by-model breakdowns reveal where spend concentrates. Where a less expensive model can serve the same use case, the API key's routing configuration can be updated to prefer it, or traffic splitting can be used to shift load gradually. - **Capacity planning:** the usage-trends chart projects future request volume. Where usage is growing, the administrator can be engaged to confirm that provider rate limits and budgets remain sufficient. - **Key hygiene:** a key showing unexpected usage may indicate a leaked credential or a misconfigured integration. Such a key can be revoked and a replacement issued. ## Step 4: issue one API key per purpose Almost every interesting question that can be asked of the monitoring surfaces becomes trivially answerable once each distinct concern has its own API key. A few examples: - *"How much is the production checkout service spending on AI this month?"* trivial if `checkout-service-prod` is its own key. - *"Which integration produced this surprising spike?"* trivial if Playground, each SDK integration, and each CI job have separate keys. - *"How does the latency on the canary model compare to the baseline?"* trivial if the canary route is isolated on its own key. - *"Was this leaked credential used anywhere I should worry about?"* trivial if the leaked key is one of several, scoped to a narrow purpose, and easy to revoke. The pattern is cheap to set up (new keys are issued from the Console in seconds) and pays for itself the first time a question about traffic needs a precise answer rather than a guess. Conversely, a single organization-wide key that everything uses turns every monitoring question into a forensic exercise. Practical conventions that work well: - Name keys for their purpose, not their owner (`checkout-service-prod`, `rag-experiment-staging`, `ci-evaluation-runner`) because the name appears in every monitoring view. - Issue separate keys for production, staging, development, and Playground use within the same application. - Rotate keys when their purpose changes substantially; do not stretch a single key across two unrelated workloads. The detailed lifecycle for API keys (creation, naming, revocation, and rotation) is documented at [Route Requests Across Providers](/agent-router-service/guides/route-requests-across-providers/). ## Step 5: use the two surfaces together Request Logs and Usage Analytics are designed to be used together. A typical investigation flow: 1. Notice an anomaly on the Usage page: an unexpected spike, an unfamiliar model in the by-model breakdown, or an unusual provider cost. 1. Filter the Usage page by API key to identify which key is responsible. 1. Switch to Request Logs. 1. Apply the same API-key filter and a time range covering the anomaly. 1. Inspect specific requests to confirm what is actually happening at the per-request level. The reverse direction also works. A puzzling individual request seen in Request Logs is sometimes obvious in context once the surrounding traffic is viewed at the aggregate level: a single 429 in isolation looks like a fluke, while a 429 plus a cluster of latency spikes during the same minute looks like a provider incident. ## What to do next - **Export telemetry to an observability stack:** send the same data into the organisation's existing dashboards, alerting, and long-term storage. See [Export Telemetry to an Observability Stack](/agent-router-enterprise/guides/observability-and-analytics/export-telemetry-to-an-observability-stack/). - **Apply advanced routing rules:** use the by-model breakdown in Usage Analytics to evaluate A/B and canary deployments. See [Apply Advanced Routing Rules](/agent-router-service/guides/apply-advanced-routing-rules/). - **Reduce cost with traffic splitting:** watch the cost-by-provider breakdown evolve as splits are tuned. See [Reduce Cost with Traffic Splitting](/agent-router-service/guides/reduce-cost-with-traffic-splitting/). Where to go next Export telemetry to an observability stack Send the same data into existing dashboards, alerting, and long-term storage. Protect requests with guardrails Confirm guardrail actions on traffic using the Request Logs workflow. --- ## Reduce cost with traffic splitting Frontier models are remarkable but expensive, and most applications do not actually need their full capability on every request. A traffic split is a routing configuration in which all backends sit at the same priority and share traffic by weight: where a fallback chain reaches its secondary backends only on failure, a traffic split sends a deliberate fraction of requests to each backend on every call, so a 70/30 split produces approximately a 70/30 distribution across a reasonable sample. This guide covers how to configure that split, what proportions to start with, how to combine splits with fallback chains, and how to verify the distribution in Request Logs and usage analytics. **Persona:** Developer working in the Developer Console. **Estimated time:** 10 to 20 minutes for the initial configuration, plus an evaluation period that depends on traffic volume. ## When this guide applies Traffic splitting is the right tool when traffic should be distributed across backends **by design**, not only on failure. The three common use cases are: | Use case | What the split does | | :---- | :---- | | **Cost reduction** | Sends a portion of traffic to a cheaper backend while keeping the rest on a higher-capability model. The split can be widened over time as quality data is gathered. | | **A/B evaluation** | Distributes traffic between two backends so their behaviour can be compared under real production conditions, rather than synthetic benchmarks. | | **Gradual migration** | Rolls a new backend out behind an existing one, typically starting at a small weight (5 to 10 %) and increasing as confidence grows. | For pure resilience (keeping requests successful when the primary backend fails) the right tool is a fallback chain, covered in *Improve Resilience with Fallbacks*. The two patterns are not mutually exclusive; combining them is covered later in this guide. ## Outcomes By the end of this guide: - A traffic split with at least two weighted backends is attached to a working API key. - Test traffic has been generated and the resulting distribution observed in Request Logs. - The split has been adjusted at least once to confirm that weight changes take effect immediately. - Where appropriate, the split is layered with a fallback chain to combine deliberate distribution with on-failure resilience. ## Prerequisites This guide assumes the foundational developer setup from [Route Requests Across Providers](/agent-router-service/guides/route-requests-across-providers/) is in place: a working API key in the Console with a routing configuration attached. The configuration created in that guide, and refined in *Improve Resilience with Fallbacks*, is replaced or layered here, depending on the chosen approach. Other prerequisites: - Two or more enabled models in the Admin Dashboard. The models do not need to come from different providers; a within-provider split between a frontier and a budget model is the most common cost-reduction shape. - A terminal with `curl` (or a Python environment with the `openai` package) for the verification step. ## Step 1: choose the weight distribution A split is defined by which backends it includes and what weight each backend receives. The right starting weights depend on the goal of the rollout. - **Cost reduction.** Start conservative: 80 to 90 % on the primary backend, 10 to 20 % on the cheaper alternative. The conservative weight makes the rollout safe to leave running while the application team gathers quality data on the cheaper option's outputs. As confidence grows, the split is widened. - **A/B evaluation.** Use a balanced or near-balanced split (50/50, 60/40) so each backend serves a statistically usable sample within a reasonable evaluation window. Heavily skewed splits make the comparison slow. - **Gradual migration.** Start with a small canary weight on the new backend, 5 %, sometimes less. Increase in stages (5 % → 20 % → 50 % → 100 %), pausing between stages long enough to see error rates and latency stabilise. Weights are unitless and proportional. The Console accepts any non-negative integers; the gateway normalises them internally. A 70/30 split, a 7/3 split, and a 700/300 split all behave identically. Using values that sum to 100 is a common convention because the weights then read directly as percentages. A typical first cost-reduction split: | Position | Example backend | Weight | Share of traffic | | :---- | :---- | :---- | :---- | | 0 | `gpt-4o` | 80 | ~80 % | | 0 | `gpt-4o-mini` | 20 | ~20 % | Both entries sit at priority 0; the split is what determines which backend serves each request. ## Step 2: configure the traffic split The split replaces the existing routing configuration on the chosen API key, unless it is being layered with a fallback chain (see *Combining splits and fallbacks* below). 1. From the **API Keys** page in the Console, open the detail page of the key the split should apply to. 1. Scroll to the **Routing Configuration** section. 1. Click **Configure** or **Add Rule**. 1. Select **Traffic Splitting**. 1. Add the first model and set its weight (for example, `gpt-4o` at weight 80). 1. Click **Add Model**, add the second model, and set its weight (for example, `gpt-4o-mini` at weight 20). 1. Repeat to add further backends if a three-way or higher split is required. 1. Confirm the displayed traffic share matches the intended distribution. The Console renders each weight as a percentage of the total to make the resulting split obvious. 1. Save the configuration and confirm the **Active** toggle is on. Weight adjustments take effect immediately on save; no in-flight requests are affected, and the new distribution applies to every subsequent request. This makes incremental rollouts inexpensive to manage: a small weight change, observed for a few hours, then another small change, with no service restart anywhere. ![Traffic splitting configuration](/img/screenshots/console/config-traffic-split.jpeg) ### Traffic-split field reference The traffic-split configuration UI exposes the following controls: - **Model.** Selected from the model dropdown. Each entry sits at **priority 0**, so the weight, not the priority, determines the share of traffic the model receives. - **Weight.** A non-negative integer per model. The Console renders each weight as a percentage of the running total so the resulting split is visible as it is edited. - **Add Model.** Appends another backend to the split. After selection, a weight must be assigned to the new model. - **Remove.** A control next to each model removes it from the split. Remaining weights are then adjusted as needed before saving. ## Step 3: generate test traffic and observe the distribution A configured split is most useful when its real distribution has been confirmed against expectations. The distribution is probabilistic, so individual requests do not follow the weights exactly; the law of large numbers takes over only across a reasonable sample. A simple loop is sufficient for verification. `PROXY_URL` stands for the proxy endpoint from the Console Dashboard, scheme included and ending in `/v1`: ```bash for i in {1..50}; do curl -s PROXY_URL/chat/completions \ -H "Authorization: Bearer YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "gpt-4o", "messages": [{"role": "user", "content": "Reply with a single word."}] }' > /dev/null done ``` Then, in the Console: 1. Open **Monitoring → Request Logs**. 1. Filter by the API key the split is attached to. 1. Filter by the time window covering the loop above. 1. Group or sort by **Resolved model**, and confirm the proportions are in the neighbourhood of the configured weights. A 50-request sample of an 80/20 split typically lands somewhere between 75/25 and 85/15, which is the expected statistical spread; substantially wider deviations are worth investigating. For longer-term observation, **Usage Analytics** is the better surface, because it aggregates across larger windows and presents the distribution as cost and token counts in addition to request counts. ## Step 4: adjust and re-verify The point of the split is to be tuneable. After observing the initial distribution and any quality signals from downstream evaluation, the weights are adjusted: 1. Open the existing traffic-split configuration on the API key's detail page. 1. Edit the weight values. 1. Save. 1. Confirm in Request Logs that the new distribution takes hold from the moment of save. The same flow handles adding a third backend, removing a backend that has fallen out of favour, or rebalancing once a migration has progressed. The split can also be disabled without being deleted. Setting the **Active** toggle off causes subsequent requests to bypass the split entirely, which is a quick way to confirm that the split, rather than some other configuration, is responsible for the observed distribution. Toggling it back on restores the configured weights immediately. :::tip For migrations and canary rollouts, log the current weights in a tracking document along with the date and the rationale for each change. Distributions tend to drift over time as engineers iterate, and a short audit trail removes a lot of guesswork later about *why* a particular split looks the way it does. ::: ## Combining splits and fallbacks A traffic split decides which backend handles a given request under normal conditions. A fallback chain decides what happens when the chosen backend fails. The two compose naturally: a split sits at priority 0, and one or more fallback backends sit at priorities 1 and beyond. A common shape: | Priority | Backend | Weight | Role | | :---- | :---- | :---- | :---- | | 0 | `gpt-4o` | 80 | Primary, normal-traffic share | | 0 | `gpt-4o-mini` | 20 | Primary, cheaper share | | 1 | `claude-sonnet-4-20250514` | 1 | Fallback for both primaries | Under normal conditions, the split distributes traffic across the two priority-0 backends at the configured ratio. If the chosen priority-0 backend fails on a given request, the gateway walks to priority 1 and serves the request from the secondary. This composition gives the cost or evaluation benefits of the split *and* the resilience benefits of a fallback chain without forcing a choice between them. For the deeper fallback semantics, see [Improve Resilience with Fallbacks](/agent-router-service/guides/improve-resilience-with-fallbacks/). ## What to do next - **Apply advanced routing rules:** combine splits with attribute-based dispatch, so different request types are split differently. See [Apply Advanced Routing Rules](/agent-router-service/guides/apply-advanced-routing-rules/) for the underlying capabilities. - **Use your own provider credentials:** bring BYOK credentials into one or more positions in the split. See [Use Your Own Provider Credentials](/agent-router-service/guides/use-your-own-provider-credentials/). - **Monitor traffic and usage:** extend the verification step into ongoing dashboards, including cost-per-thousand-requests across the backends in the split. - **Export telemetry to an observability stack:** push the distribution and the per-backend cost into the organisation's existing observability platform. The split configured above remains attached to the key for subsequent guides. Where to go next Apply advanced routing rules Split different request types differently with attribute-based dispatch. Use your own provider credentials Bring BYOK credentials into one or more positions in the split. --- ## Route requests across providers Modern AI applications rarely depend on a single provider for long: a primary commercial model handles most of the traffic, a second is held in reserve in case the first one has a bad afternoon, and a cheaper or self-hosted option absorbs traffic where quality requirements allow. This is the foundational developer walkthrough, covering the request path end-to-end: how an API key is created in the Console, how a routing configuration is attached to that key so that more than one backend is available, how a request is sent to the gateway, and how the resulting traffic is verified in Request Logs. Once complete, the more specific routing patterns (fallback chains, traffic splitting, advanced rules, and BYOK) become incremental additions on top of the same setup. **Persona:** Developer working in the Developer Console. **Estimated time:** 15 to 20 minutes for the first run, including time to copy values between the Console and a terminal. ## Outcomes By the end of this guide: - A working API key exists in the Console, stored securely outside of Agent Router. - A routing configuration is attached to that key, referencing at least one, and ideally two, enabled models. - A request sent through the gateway returns a successful response from one of the configured backends. - The request is visible in **Request Logs**, with the resolved model, latency, token counts, and cost recorded. ## Prerequisites This guide assumes that a platform operator has already provisioned the models and providers that requests will be routed to. The operator-facing companion to this work (enabling models, configuring providers, and verifying provider connectivity) is covered by the *Provision Models and Providers* guide in *Guides for Platform Operators*. If models are not yet enabled in the Admin Dashboard, that step has to happen first; the Console can only build routing configurations on top of what Agent Router has been told to expose. In summary, before starting: - The Admin Dashboard shows at least one enabled model with a healthy provider connection. - Developer access to the Developer Console is in place. - A terminal with `curl` (or a Python environment with the `openai` package) is available for the request step. ## The Console and the proxy endpoint The Developer Console is the developer-facing application within Agent Router. It provides a single interface for routing requests across the supported models, managing API keys, and monitoring usage and costs. Access is through a deployment-specific URL; sign-in is by email and password or by corporate SSO. After sign-in, the landing page is the **Dashboard**. It surfaces the proxy endpoint, recent request activity, and shortcuts to common actions: - The **proxy endpoint** is displayed at the top of the Dashboard and on the API Keys page, and again after a key or routing configuration is created. It is the OpenAI- and Anthropic-compatible base URL that requests are sent to. The value includes the scheme and ends in `/v1`, so it serves as the `base_url` in a client configuration without modification. Any tool, SDK, or HTTP client that speaks the OpenAI Chat Completions or Anthropic Messages API can connect to it, authenticating with an API key passed in the `Authorization: Bearer` header. The copy icon next to the URL copies it to the clipboard. - **Summary metrics** cover the selected period: **Total Requests** across all keys, and **Total Spend** (shown only when billing is enabled). A time-range selector (last 24 hours, last 7 days, or last 30 days) adjusts the window. - An **API key summary card** per key shows recent activity at a glance. - **Quick actions** jump directly to API key creation or to the Playground. The concepts that recur throughout this guide are summarized below: | Concept | Description | | :---- | :---- | | Proxy endpoint | The OpenAI- and Anthropic-compatible URL that routes requests to the correct provider. | | API key | A credential prefixed with `sk-` that authenticates requests to the proxy endpoint. | | Routing configuration | Per-key settings for fallback policies and traffic splitting across models. | | MCP profile | A bundled set of Model Context Protocol (MCP) servers exposed as a single URL for tool-use clients. | :::note Endpoint terminology **Proxy endpoint** is the label the Console displays, and it is the term used here when referring to what appears on screen. **Gateway URL** refers to the same value where the product surface is not the Console, for example in installation and operator documentation. Both name one URL. **Data plane** is reserved for the deployment that hosts gateways, never for the URL itself. ::: ## Step 1: create an API key in the Console API keys are the credential applications present when they call the gateway. Each key is associated with a routing configuration, so creating the key comes first; the routing rules are attached to it in the next step. 1. Sign in to the Developer Console. 1. Open **Settings → API Keys** from the sidebar. 1. Click **Create API Key**. 1. Enter a descriptive name. Names are surfaced in Request Logs and usage analytics, so a phrase tied to the calling application or environment (`checkout-service-prod`, `rag-experiment-staging`) is more useful at audit time than a generic value. 1. Click **Create** and copy the generated key immediately. Keys are prefixed with `sk-` (for example, `sk-abc123def456...`). :::caution The full key string is displayed only once. It cannot be retrieved later from the Console; if it is lost, the key has to be revoked and a new one issued. Store it in a secrets manager, an environment variable, or whatever credential store the application stack already uses, not in source control. ::: ### What the API Keys page shows Once created, each key is listed on the **API Keys** page with its own usage statistics, so cost and volume can be tracked per integration, environment, or team member: | Column | Detail | | :---- | :---- | | Name | The descriptive label entered at creation; also surfaced in Request Logs and usage analytics. | | Total requests | The number of requests made with the key. | | Token counts | Input and output tokens consumed by the key. | | Total spend | The accumulated cost attributed to the key. | | Actions | Configure routing, set rate limits, or revoke the key. Revocation is immediate, and any further requests with the key return an authentication error. | Each key can also carry a per-hour rate limit. Rate limits are configured per key from the key's **Configure** action: the **Rate Limit** section is enabled with a toggle, after which three independent token-budget sliders become available. Each can be enabled separately and set with the slider or by typing a value: - **Total tokens**: maximum combined input and output tokens per hour. - **Input tokens**: maximum prompt tokens per hour. - **Output tokens**: maximum completion tokens per hour. Limits are evaluated on a rolling one-hour window that counts the current request together with all tokens consumed by the key in the preceding hour. Only enabled limits are enforced, and each is checked independently; a request is rejected if it would exceed any active limit. Requests that exceed a limit receive a `429 Too Many Requests` response. ### Revoke a key A key is revoked from its row on the **API Keys** page using the revoke action; the action requires confirmation. Revocation is immediate: in-flight and future requests presenting the revoked key return an authentication error. A revoked key cannot be restored; a replacement key has to be issued. ### Key hygiene - **Descriptive names**: name keys after their purpose (`ci-pipeline`, `staging-app`, `john-local`) so usage can be identified later. - **One key per integration**: a separate key per application, environment, or developer keeps usage tracking isolated. - **Rotate regularly**: keys are periodically revoked and replaced, especially for production systems. - **Never commit keys to source control**: environment variables or a secrets manager are used instead. ## Step 2: configure routing across providers Routing rules are attached to individual API keys: the same Console account can hold one key with a single-backend configuration and another key with a multi-backend chain, and the choice between them is made simply by which key the calling application presents. This per-key separation is useful for keeping experimental routing arrangements off the path of production traffic. 1. From the **API Keys** page, click the key that was just created to open its detail page. 1. Scroll to the **Routing Configuration** section. 1. Click **Configure** or **Add Rule**. 1. Select a routing strategy. For a first pass, **Fallback Policy** is the simplest choice, because it works equivalently with one backend or several: | Strategy | Behaviour | | :---- | :---- | | Fallback Policy | Backends are tried in priority order; the gateway walks to the next backend only when the previous one fails. | | Traffic Split | Backends share traffic by weight, on every request. Useful for evaluation and cost management. | | Advanced Routing Rules | Routing decisions consider request attributes such as the model field, custom headers, or other signals. | 1. Add the primary model as the first backend in the chain (priority 0 in a fallback policy, or the first weighted entry in a traffic split). The model picker draws from the **Model Catalog** (**Catalog → Model Catalog**), which lists every enabled model with the detail needed to choose one: | Column | Detail | | :---- | :---- | | Model name | The identifier used in API requests (for example, `gpt-4o`, `claude-sonnet-4-20250514`). | | Provider | The upstream provider hosting the model. | | Context window | The maximum number of tokens the model accepts in a single request. | | Pricing | Cost per input token and per output token. | | Status | Whether the model is enabled or disabled for the organization. | The catalog spans 200+ models from providers including OpenAI, Anthropic, Google, Azure, Mistral, Cohere, Together, and Perplexity, sorted newest first by default. A search bar filters by name in real time, and the list can be narrowed by one or more providers and by status (enabled or disabled). A **Group Snapshots** toggle collapses dated model versions under their parent model for a cleaner view, or expands them to show every individual version. Selecting a model opens a detail page with its full name and provider, context window (input and output limits), pricing per million input and output tokens, supported capabilities (chat completions, function calling, vision, and others), and current status. 1. Optionally add a second backend from a different provider, for example, a Claude model behind a GPT model, or vice versa. This is what turns the configuration from "send everything to one provider" into "route across providers". Even with only one backend configured, the surrounding mechanics are identical; adding a second one later does not require any application change. 1. Save the configuration. 1. Confirm that the **Active** toggle on the routing configuration is on. An inactive configuration is saved but not enforced; requests bypass it and go directly to whatever model the calling code specifies. ![Routing Configuration overview](/img/screenshots/console/config-overview.jpeg) ### Routing configuration reference A few constraints apply to every routing configuration, regardless of strategy: - **Per-key scope**: Rules are attached to one API key. A test key can run an experimental configuration while a production key holds a stable one. - **Active/inactive toggle**: An inactive configuration is retained but not enforced; toggling it back to active restores the rules without reconfiguring them. - **Model availability**: Every model referenced in a rule must be enabled in the model catalog. If a model is later disabled by an administrator, rules that reference it may stop behaving as expected. The defaults applied by the Console (priority 0 for the first model, weight 1 per priority level) are the right starting point for the cases covered in this guide. The deeper mechanics of each strategy (chain ordering, weight tuning, and attribute-based dispatch) are covered in the dedicated *Improve Resilience with Fallbacks*, *Reduce Cost with Traffic Splitting*, and *Apply Advanced Routing Rules* guides. ## Step 3: send a request through the gateway The gateway exposes an OpenAI-compatible endpoint at `/v1/chat/completions`. The request shape is the standard OpenAI Chat Completions payload, and the API key from Step 1 is presented as a bearer token. The same endpoint serves every backend Agent Router supports, regardless of which underlying provider ultimately handles the request. The proxy endpoint is specific to the deployment and is copied from the Console Dashboard or the API Keys page, as described above. In the examples below, `PROXY_URL` stands for that value, scheme included and ending in `/v1`, and `YOUR_API_KEY` for the key copied in Step 1. ### Using curl ```bash curl PROXY_URL/chat/completions \ -H "Authorization: Bearer YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "gpt-4o", "messages": [{"role": "user", "content": "Explain the concept of AI routing in one sentence."}] }' ``` ### Using Python ```python from openai import OpenAI client = OpenAI( base_url="PROXY_URL", api_key="YOUR_API_KEY", ) response = client.chat.completions.create( model="gpt-4o", messages=[{"role": "user", "content": "Explain the concept of AI routing in one sentence."}], ) print(response.choices[0].message.content) ``` A successful call returns a response in the same shape the calling code would expect from OpenAI directly; which model actually produced the response is determined by the routing configuration, not by the value of the `model` field. :::note The `model` field in the request is treated as a hint rather than a strict instruction once routing rules are in place. If the configured routing strategy resolves to a different backend (for example, a fallback chain has fallen through to its secondary model), the gateway uses that backend regardless of what the application sent. The exact resolution rules are documented in *Gateway behavior* under *Reference*. ::: ## Step 4: verify the route in request logs Sending a request is not the same as confirming the routing behaved as expected. Request Logs is the developer-facing record of every request that flowed through the gateway under a given API key, with enough detail to settle questions about which backend served the request and how it performed. 1. In the Console, open **Monitoring → Request Logs**. 1. Locate the test request, the most recent entry under the API key created in Step 1. 1. Expand the row to view the detail panel. 1. Confirm the following fields are populated and consistent with the routing configuration: | Field | What to check | | :---- | :---- | | Resolved model | Matches the backend the gateway actually used, not necessarily the value sent in the `model` field. | | Provider | Matches the upstream provider for the resolved model. | | Token counts | Input and output tokens are present and non-zero. | | Latency | The end-to-end time the gateway observed for the call. | | Cost | The computed cost based on the resolved model and token usage. | ![Console Request Logs](/img/screenshots/console/request-logs.jpeg) If the resolved model differs from the primary backend in the routing configuration, the chain has fallen through, either because the primary backend returned an error or because the configured strategy distributes traffic by weight. Both are expected behaviours; the deeper failure-mode handling is covered in *Improve Resilience with Fallbacks*. Request Logs can be filtered by API key, time range, model, or response status, and each row expands to a detail panel showing the full request and response content. The aggregated view (spend and volume broken down by model, provider, and key) is covered in *Monitor Traffic and Usage*. ## What to do next The foundational route is in place. The remaining developer guides each add a specific behaviour on top of it without reworking what has already been built: - **[Improve resilience with fallbacks](/agent-router-service/guides/improve-resilience-with-fallbacks/)**: formalise the multi-provider chain into an explicit ordered fallback policy and tune the failover behaviour. - **[Reduce cost with traffic splitting](/agent-router-service/guides/reduce-cost-with-traffic-splitting/)**: distribute requests by weight rather than priority. - **[Apply advanced routing rules](/agent-router-service/guides/apply-advanced-routing-rules/)**: dispatch requests based on attributes of the request itself. - **[Use your own provider credentials](/agent-router-service/guides/use-your-own-provider-credentials/)**: introduce BYOK credentials into the routing chain. - **[Monitor traffic and usage](/agent-router-service/guides/monitor-traffic-and-usage/)**: expand on Step 4 with usage analytics and cost reporting across API keys. Each of those guides assumes the setup completed here is already in place. Where to go next Improve resilience with fallbacks Formalise the multi-provider chain into an explicit ordered fallback policy. Monitor traffic and usage Expand on Step 4 with usage analytics and cost reporting across API keys. --- ## Run batch and long-running jobs Not all AI work is interactive. A large share of it is offline: scoring an overnight queue of support tickets, generating embeddings for a million-document corpus, running an evaluation suite across every model under consideration. Batch processing fits this shape of work: instead of one request per item held open until it returns, a single file of requests is submitted, processed asynchronously by the provider, and collected later as a single file of results. Tetrate Agent Router exposes this through an OpenAI-compatible batch interface, so the same gateway, API key, and observability surfaces that handle synchronous traffic also handle batch traffic. This guide covers when batching is the right choice, how a batch job is submitted, polled, and retrieved, how large workloads are structured into manageable batches, how batch traffic interacts with rate limits and budgets, and how it appears in the Console's monitoring surfaces. **Persona:** Developer working in the Developer Console and in application code. **Estimated time:** 20 to 30 minutes to submit a first batch and retrieve its results; batch completion itself runs asynchronously and may take from minutes to hours depending on size and provider. ## When this guide applies Batching fits high-volume, latency-tolerant work. Synchronous and streaming calls remain the right tool when a human or a downstream system is waiting on the answer. The distinction usually comes down to who, or what, is blocked on the result. | Situation | Recommended approach | | :---- | :---- | | An offline corpus of thousands of items is processed on a schedule | Batch | | Embeddings are generated in bulk for a vector index | Batch | | An evaluation suite is run across many prompts or many models | Batch | | A nightly or weekly summarisation, classification, or enrichment job | Batch | | A user is waiting on a response in a UI | Synchronous, often streaming | | Tokens should appear progressively as they are produced | Streaming | | A single ad-hoc request is being tested or debugged | Synchronous | | Per-item latency matters more than throughput or unit cost | Synchronous | The decision is rarely permanent. A pipeline that starts as a synchronous loop during prototyping is a natural candidate to convert to batch once its volume grows and its latency tolerance becomes clear. ## Outcomes By the end of this guide: - The conditions under which batch processing is preferable to synchronous or streaming calls are understood. - A batch input file has been constructed in the OpenAI-compatible JSONL format and submitted through the gateway. - A batch job's status has been polled to completion and its output file retrieved and parsed. - A long-running workload has been structured into appropriately sized batches. - The interaction between batch traffic, per-key rate limits, and budgets is understood. - Batch traffic has been located in Request Logs and Usage Analytics. ## Prerequisites - A working API key with a routing configuration attached, as set up in [Route requests across providers](/agent-router-service/guides/route-requests-across-providers/). - Confirmation that the target model is enabled in Agent Router and that its upstream provider supports batch processing. Batch availability is provider-specific; a model that serves synchronous traffic does not necessarily expose a batch endpoint. - A terminal with `curl`, or a Python environment with the `openai` package, for the submission and retrieval steps. - For bulk-embedding workloads, familiarity with the embeddings endpoint described in [Gateway APIs](/reference/supported-apis/). The proxy endpoint is specific to the deployment and is displayed on the Console Dashboard. In the examples below, `PROXY_URL` stands for that value, scheme included and ending in `/v1`, and `YOUR_API_KEY` for a key from the Console. ## Step 1: decide whether to batch The first decision is whether the workload belongs in a batch at all. Three properties together make a workload a good fit: - **Volume**: the work consists of many independent items, typically hundreds to millions, rather than a handful. - **Latency tolerance**: nothing is blocked waiting on an individual result. The job can complete minutes or hours after submission without affecting a user or a time-sensitive process. - **Independence**: each item is self-contained. Batch processing does not preserve any conversational state between items, and items are not guaranteed to be processed in submission order. Where all three hold, batching converts a fragile, rate-limit-prone loop into a single asynchronous submission. Where any one of them fails (a user is waiting, the items depend on each other, or there are only a few of them), a synchronous call through `/v1/chat/completions`, or a streaming call for progressive output, remains the correct choice. The synchronous request path is the subject of [Route requests across providers](/agent-router-service/guides/route-requests-across-providers/); streaming is documented in [Gateway APIs](/reference/supported-apis/). ## Step 2: build the batch input file The gateway accepts batch input in the OpenAI-compatible JSONL format: one JSON object per line, each describing a single request. Every line carries a `custom_id` used to correlate the eventual result back to its input, the HTTP `method` and `url` of the endpoint being called, and a `body` containing the request payload that would otherwise be sent synchronously. A chat-completions batch with two requests: ```json {"custom_id": "ticket-001", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gpt-4o", "messages": [{"role": "user", "content": "Classify this ticket: cannot reset password"}]}} {"custom_id": "ticket-002", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gpt-4o", "messages": [{"role": "user", "content": "Classify this ticket: invoice shows wrong amount"}]}} ``` The `custom_id` must be unique within the file. Because results are not guaranteed to return in input order, the `custom_id` is the only reliable way to match an output line back to the request that produced it; deriving it from a stable key in the source data (a record ID, a document path) avoids a brittle reliance on line position. The `body` of each line is the same payload accepted by the corresponding synchronous endpoint, so any request shape valid for `/v1/chat/completions` is valid here. The `model` field within each `body` is subject to the same routing configuration as a synchronous request; Agent Router resolves it according to the rules described in [Route requests across providers](/agent-router-service/guides/route-requests-across-providers/). Generating the file programmatically keeps it consistent at scale: ```python import json records = load_source_records() # application-specific with open("batch_input.jsonl", "w") as f: for record in records: line = { "custom_id": record["id"], "method": "POST", "url": "/v1/chat/completions", "body": { "model": "gpt-4o", "messages": [ {"role": "user", "content": record["prompt"]}, ], }, } f.write(json.dumps(line) + "\n") ``` ## Step 3: submit the batch job Submitting a batch is a two-stage operation: the input file is uploaded first, then a batch job is created that references the uploaded file by its identifier. ### Upload the input file ```bash curl PROXY_URL/files \ -H "Authorization: Bearer YOUR_API_KEY" \ -F purpose="batch" \ -F file="@batch_input.jsonl" ``` The response contains a file identifier, conventionally prefixed `file-`. That identifier is the handle for the next call. ### Create the batch ```bash curl PROXY_URL/batches \ -H "Authorization: Bearer YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "input_file_id": "file-abc123", "endpoint": "/v1/chat/completions", "completion_window": "24h" }' ``` The same two stages in Python: ```python from openai import OpenAI client = OpenAI( base_url="PROXY_URL", api_key="YOUR_API_KEY", ) input_file = client.files.create( file=open("batch_input.jsonl", "rb"), purpose="batch", ) batch = client.batches.create( input_file_id=input_file.id, endpoint="/v1/chat/completions", completion_window="24h", ) print(batch.id, batch.status) ``` The create call returns immediately with a batch identifier and an initial status; the work itself proceeds asynchronously on the provider side. The `endpoint` field declares the endpoint every line in the file targets, and `completion_window` expresses the window within which completion is requested. The application does not hold a connection open for the duration; the batch identifier is the only state that needs to be retained. :::note Batch support and the specific values accepted for `endpoint` and `completion_window` are provider-dependent. If a batch creation call is rejected, the most common cause is that the resolved model's upstream provider does not offer a batch endpoint for the requested endpoint type. Confirm batch availability for the target model before building a pipeline around it. ::: ## Step 4: poll status and retrieve results A batch job moves through a sequence of states (typically validating, in progress, finalising, and completed), and the application discovers its progress by polling rather than by holding a connection open. Failed, expired, and cancelled are also terminal states that any poller must handle. ```python import time while True: batch = client.batches.retrieve(batch.id) print(batch.status, batch.request_counts) if batch.status in ("completed", "failed", "expired", "cancelled"): break time.sleep(30) ``` A polling interval of roughly 30 seconds to a few minutes is appropriate; tighter intervals add load without materially improving the time to discover completion. The retrieve response also reports request counts (total, completed, and failed), which gives a coarse sense of progress while the job is in flight. Once the status reaches `completed`, the batch object carries an `output_file_id` for successful results and, where any requests failed, an `error_file_id` for the failures. Both are retrieved through the same files interface used for the upload: ```python output = client.files.content(batch.output_file_id) with open("batch_output.jsonl", "wb") as f: f.write(output.read()) ``` The output file is JSONL, one line per processed request. Each line echoes the `custom_id` from the input alongside the response body, which is why a stable `custom_id` matters: results are correlated by that field, not by position. A line whose request failed carries an error rather than a response, so each line is checked individually rather than assuming the whole batch succeeded or failed as a unit. ```python import json with open("batch_output.jsonl") as f: for line in f: result = json.loads(line) custom_id = result["custom_id"] if result.get("error"): handle_failure(custom_id, result["error"]) else: handle_success(custom_id, result["response"]["body"]) ``` A batch that has not yet completed can be cancelled if the work is no longer needed: ```bash curl -X POST PROXY_URL/batches/batch_abc123/cancel \ -H "Authorization: Bearer YOUR_API_KEY" ``` ## Step 5: structure long-running and large workloads A workload of a few thousand items maps cleanly onto a single batch. Larger workloads (bulk embeddings for a large corpus, an evaluation sweep across many models and prompts, or an enrichment pass over a full dataset) benefit from being split into several smaller batches rather than submitted as one enormous file. Smaller batches fail in smaller, more recoverable units; they make progress observable as each batch completes; and they sidestep any per-file size or request-count ceilings the upstream provider enforces. A few patterns make large workloads manageable: - **Chunk the input**: the source dataset is divided into batches of a consistent size (for example, a few thousand to tens of thousands of requests each), and each chunk is submitted as its own job. The chunk index is folded into the `custom_id` so that results from different batches remain unambiguously correlated. - **Track batch identifiers durably**: each batch identifier is persisted alongside the chunk it represents, so a long-running job survives a restart of the submitting process. Recovery becomes a matter of re-polling known identifiers rather than resubmitting work. - **Process results as each batch completes**: rather than waiting for every batch, each output file is consumed as its batch reaches `completed`, which spreads the downstream work and surfaces problems early. - **Bulk embeddings**: an embeddings workload follows the same shape, with each line targeting `/v1/embeddings` and its `body` carrying the input text. The endpoint is described in [Gateway APIs](/reference/supported-apis/). - **Evaluations**: an evaluation sweep is expressed as a batch whose lines vary the `model` field across the candidates under test, with a `custom_id` encoding both the prompt and the model so that results can be pivoted by either dimension afterwards. Splitting also interacts favourably with rate limits and budgets, which is the subject of the next step: several moderate batches submitted in sequence are easier to keep within a key's quota than one batch large enough to exhaust it. ## Step 6: understand rate limits and budgets Batch traffic is metered the same way as synchronous traffic. The tokens consumed by every request in a batch count against the same per-key rate limits and contribute to the same usage totals; submitting work as a batch does not exempt it from a key's quota. A per-key rate limit set on a rolling hourly window can therefore reject batch requests with a `429` response in the same way it rejects synchronous ones, particularly when a large batch lands within a single window. Several practices keep batch work inside its budget: - **Size batches against the key's limit**: where a key carries an hourly token ceiling, batches are sized so that a single submission does not blow through the window. Splitting a large workload across several batches, as in Step 5, is the primary lever. - **Isolate batch work on its own key**: a dedicated key for batch pipelines keeps their consumption separate from interactive traffic, so a heavy overnight job cannot starve a user-facing path of its quota, and the cost of the batch work is attributable on its own. - **Treat the provider's batch discount as part of the budget**: many providers price batch work below synchronous work. Where that discount applies, the saving shows up in the usage totals for the batch key and can be planned for rather than discovered after the fact. The mechanics of per-key rate limits (the rolling window, the independent token sliders, and the `429` behaviour) and the broader discipline of bounding spend are covered in [Working with budgets](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/choose-the-right-cost-control/). Coordinate with the platform operator who owns those limits before pointing a high-volume batch pipeline at a production key. ## Step 7: observe batch traffic in the Console Batch requests flow through the same gateway as synchronous requests, so they appear in the same monitoring surfaces with the same detail. Once a batch has completed, its constituent requests are visible in the Console. - **Request Logs**: each request within a batch is recorded as an individual row, with its resolved model, token counts, latency, cost, and status, exactly as a synchronous request would be. Filtering by the batch key isolates the batch traffic from everything else; scanning the status column surfaces any requests that failed within an otherwise successful batch. - **Usage Analytics**: the aggregate view attributes the batch's volume and spend to the submitting key, broken down by model and provider. With batch work isolated on its own key, the cost of an entire batch run is read directly from the per-key totals. Because batch and synchronous traffic share these surfaces, the API-key-per-purpose convention is what keeps them distinguishable: a key reserved for batch pipelines turns "how much did last night's job cost" into a single filtered reading rather than a forensic exercise. The full treatment of both surfaces, including the per-purpose key pattern, is in [Monitor traffic and usage](/agent-router-service/guides/monitor-traffic-and-usage/). ## What to do next - **[Monitor traffic and usage](/agent-router-service/guides/monitor-traffic-and-usage/)**: inspect the individual requests a batch produced and read its aggregate cost in Usage Analytics. - **[Working with budgets](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/choose-the-right-cost-control/)**: set and tune the per-key rate limits that batch traffic is metered against. - **[Gateway APIs](/reference/supported-apis/)**: the full endpoint reference, including the embeddings endpoint used for bulk-embedding batches and the streaming surfaces that batch deliberately trades away. - **[Integrate the gateway with an app](/agent-router-service/guides/integrate-the-gateway-with-an-app/)**: wire the batch submission and retrieval flow into an application's SDK configuration. Where to go next Monitor traffic and usage Inspect the individual requests a batch produced and read its aggregate cost. Working with budgets Set and tune the per-key rate limits that batch traffic is metered against. --- ## Test prompts in the playground Iterating on a prompt usually involves the same boring loop: edit the prompt in source, redeploy or restart the application, send a request, read the output, repeat. The Playground breaks that loop. It is an interactive chat surface in the Console that talks to the same gateway, with the same routing rules and the same observability as production traffic, but without any of the round-trip cost of editing source code. This guide covers what the Playground is for, how it differs from production traffic, and the workflows it supports most cleanly: single-shot prompt evaluation, multi-turn conversation testing, head-to-head model comparison, and reproduction of issues seen in Request Logs. **Persona:** Developer working in the Developer Console. **Estimated time:** 5 to 10 minutes for the basic flow; longer for iterative prompt development. ## When this guide applies The Playground is the right surface in any of these situations: | Situation | Why the Playground helps | | :---- | :---- | | Iterating on a system prompt or user prompt | Edits are free; the next attempt is one message away | | Choosing between two models for the same task | The model selector switches between any enabled models without leaving the conversation | | Verifying that the account can reach a specific model | A successful Playground exchange confirms end-to-end connectivity, credentials, and routing | | Reproducing an issue seen in Request Logs | The same prompt, the same model, and the same account context can be replayed instantly | | Showing a colleague how a model behaves | Screen-sharing the Playground avoids the need to share or paste application code | For programmatic access from an application, the Playground is the wrong tool; the [Integrate the Gateway with an App](/agent-router-service/guides/integrate-the-gateway-with-an-app/) guide covers SDK and framework integrations. ## Outcomes By the end of this guide: - At least one message has been sent through the Playground and the response has been read. - A multi-turn conversation has been exercised, so the conversation-history behaviour is understood. - The same prompt has been compared across at least two models using the model selector. - The Playground request has been located in Request Logs, confirming that Playground traffic flows through the same gateway as everything else. ## Prerequisites - A Console account with access to at least one enabled model in the Admin Dashboard. - A working API key associated with that account, as set up in [Route Requests Across Providers](/agent-router-service/guides/route-requests-across-providers/). The Playground uses the account's own credentials; no separate setup is required. ## Step 1: open the playground and send a message The Playground is reached from the Console's primary navigation. ![Playground](/img/screenshots/console/playground.jpeg) 1. Sign in to the Developer Console. 1. Open **Build → Playground** from the sidebar. 1. From the model selector at the top of the chat panel, choose a model. The selector lists every model enabled in the organisation's Model Catalog; an empty list indicates that no models are enabled yet, which is a platform-operator concern rather than a developer one. 1. Type a message into the input area at the bottom of the chat panel. 1. Press **Send**, or use the Enter key. The model's response appears in the chat panel within a few seconds. Two pieces of metadata are surfaced underneath each response and are worth getting in the habit of glancing at: - **Token count.** Input and output token counts for the exchange. Useful for understanding how a particular prompt sits relative to the model's context window, and for back-of-envelope cost estimation. - **Latency.** End-to-end time the gateway observed for the call. The first request to a freshly-warmed model tends to be slower than subsequent ones; sustained latency is a more useful signal than a single measurement. ### Playground controls The chat panel exposes a small set of controls: | Control | Location | Purpose | | :---- | :---- | :---- | | Model selector | Top of the chat panel | Lists every model enabled in the organisation's Model Catalog; selects the backend the next message is routed to | | Input area | Bottom of the chat panel | Accepts the message text; submitted with **Send** or the Enter key | | New Chat | Chat panel | Clears the current session and starts a fresh conversation with no prior context | | Token count | Beneath each response | Input and output token counts for the exchange | | Latency | Beneath each response | End-to-end time the gateway observed for the call | ## Step 2: have a multi-turn conversation The Playground maintains conversation context within a session. Follow-up messages are sent along with the full prior history, just as a real application would have to do explicitly when calling the gateway. The cumulative context is visible in the chat panel. Two implications of this are worth keeping in mind: - **Cost grows turn over turn.** Each subsequent message in a conversation includes every previous message as input. A long conversation can use significantly more tokens per request than a short one, even if the visible message at the bottom of the screen is short. The token counter under each response reflects the cumulative input cost rather than the cost of the latest message alone. - **The Playground does not summarise or truncate.** When a conversation approaches the model's context window, the model itself will start to misbehave: losing track of earlier turns, returning errors, or both. This is a feature for testing how a model degrades, but it is also a reason to start a fresh session (**New Chat** in the chat panel, or a page refresh) when the current one has run long. ## Step 3: compare models on the same prompt Switching models mid-conversation is the Playground's most useful comparison tool. The model selector at the top of the panel can be changed at any time; the next message sent will be routed to the new model. Previous messages in the conversation remain visible and are included in the next request as context, which means a conversation can carry the same setup across models without retyping it. A typical comparison workflow: 1. Send a representative prompt to the first model. 1. Read the response. 1. Switch to the second model using the selector. 1. Send a follow-up that asks the second model to perform the same task, for example, "answer the question I just asked, in the same format". 1. Compare the two responses side by side in the chat panel. This approach is faster and more honest than running two separate Playground sessions, because both models see the same prior context. Where the goal is to compare cold responses to the same prompt, **New Chat** before switching models keeps each side clean. For larger comparison campaigns (statistically significant evaluations rather than informal side-by-sides), a traffic split is the right tool. See [Reduce Cost with Traffic Splitting](/agent-router-service/guides/reduce-cost-with-traffic-splitting/) and [Apply Advanced Routing Rules](/agent-router-service/guides/apply-advanced-routing-rules/) for the patterns that handle that case at scale. ## Step 4: find the request in request logs The Playground is not a separate surface from the gateway. Every Playground request flows through the same data plane, the same routing configuration, and the same observability path as a request from an application or an SDK integration. This is occasionally surprising, but it is the foundation of the Playground's usefulness for debugging. 1. After sending a Playground message, open **Monitoring → Request Logs** in a new tab. 1. Filter by the API key associated with the Playground session, or by the most recent time window. 1. Locate the Playground request; it appears alongside any other recent gateway traffic. 1. Expand the row to view the full request and response, the resolved model, the token counts, and the latency. Two consequences of this design: - A Playground session **counts toward usage and cost reporting** like any other traffic. For experimental sessions that should not contaminate production cost dashboards, consider issuing a dedicated API key for exploratory work and using a different API key for production integrations. - A Playground session **exercises the routing configuration** on the account's API key. If a fallback chain is attached, a Playground request might be served by a secondary backend just as a production request would. The Resolved model field in Request Logs is the authoritative record of which backend actually answered. ## Reproducing issues from request logs The Playground is the most ergonomic surface for reproducing issues seen in Request Logs. The workflow: 1. In Request Logs, locate the problematic request. 1. Note the model that served it (or the logical name, if model-name overrides are in use) and the prompt content. 1. Open the Playground in a new tab. 1. Select the same model in the model selector. 1. Paste the prompt into the input area. 1. Send the message and observe. If the issue reproduces, the prompt itself is at fault, which is a great outcome, because the Playground is then the right place to fix it. If the issue does not reproduce, the difference is somewhere in the application layer: a different system prompt, different sampling parameters, a different conversation history, or a different account context. Each of these is worth ruling out one at a time before assuming the gateway or the model is at fault. ## What to do next - **Integrate the gateway with an app**: once a prompt or a model choice has stabilised in the Playground, move the same configuration into application code. See [Integrate the Gateway with an App](/agent-router-service/guides/integrate-the-gateway-with-an-app/). - **Monitor traffic and usage**: track Playground and production traffic separately by per-purpose API key. See [Monitor Traffic and Usage](/agent-router-service/guides/monitor-traffic-and-usage/). - **Apply advanced routing rules**: once a logical model name has been introduced, the Playground can drive it directly. See [Apply Advanced Routing Rules](/agent-router-service/guides/apply-advanced-routing-rules/). Where to go next Integrate the gateway with an app Move a stabilised prompt or model choice into application code. Monitor traffic and usage Track Playground and production traffic separately by per-purpose API key. --- ## Use your own provider credentials Real organisations rarely use only one provider account per provider: a business unit insists on its own OpenAI account because the spending sits in their cost centre, a regulated subsidiary holds an Anthropic enterprise contract, a research team has a fixed quota for evaluating a new provider. Bring Your Own Key (BYOK) is the mechanism for these cases. When a BYOK credential is configured for a provider, requests to models hosted by that provider are authenticated with the consumer-supplied key instead of the centrally managed one, so rate limits, access tiers, billing, and provider-side usage records all attach to the BYOK account directly. This guide covers how to register a BYOK credential, how requests are authenticated once it is in place, and how the credential composes with the routing patterns covered in the earlier dev guides. **Persona:** Developer working in the Developer Console. **Estimated time:** 10 to 15 minutes per provider credential, including a verification step against the provider's own usage dashboard. ## When this guide applies BYOK is the right tool when any of the following are true: | Situation | Why BYOK helps | | :---- | :---- | | Billing and cost centre attribution require that a specific team's traffic shows up on a specific provider account | The provider sees the BYOK account directly, so its usage records and invoices match the responsibility boundary | | Rate limits or access tiers on a private account exceed those available on the centrally managed account | Traffic that needs the higher ceiling can be routed through the BYOK account specifically | | Compliance or contractual obligations require that traffic land on a particular provider relationship | Routing the relevant API key through BYOK ensures the contractual identity is the one that actually serves the request | | A new provider or a new model from an existing provider is being evaluated against a fixed quota | Isolating the traffic on its own credential keeps the evaluation clean | For traffic that is purely centrally managed (general developer access on centrally managed credentials), BYOK is unnecessary. The two models coexist on the same Console without conflict. ## Outcomes By the end of this guide: - A BYOK credential is registered against at least one supported provider. - A test request has been routed through the BYOK credential, and the request appears in the provider's own usage dashboard. - The credential's behaviour against rotation and removal has been exercised, so the operational lifecycle is understood. - Where appropriate, the BYOK credential is used in combination with the routing patterns from earlier in the dev guides. ## Prerequisites - A working API key with a routing configuration attached, as set up in [Route Requests Across Providers](/agent-router-service/guides/route-requests-across-providers/). - A valid API key for one of the supported upstream providers: OpenAI, Anthropic, Google (Gemini), Azure OpenAI, Mistral, Together, Perplexity, or Cohere. Additional providers become available over time. - Access to that provider's usage dashboard for the verification step. For OpenAI, this is `platform.openai.com/usage`; the other providers have equivalent surfaces. ## Step 1: register the BYOK credential BYOK credentials are configured at the Console account level, not per API key. Once a credential is registered for a provider, every request the account makes to models hosted by that provider uses the BYOK credential, regardless of which Agent Router API key was presented. 1. Sign in to the Developer Console. 1. Open **Settings → Bring Your Own Key** from the sidebar. 1. The page lists the supported providers, each with the current configuration status. Locate the provider whose credential should be registered. 1. Click the provider card to open the configuration form. 1. Paste the provider API key into the **API Key** field. Each provider has a distinctive key format (OpenAI keys begin with `sk-`, and most other providers follow comparable conventions), and verifying the prefix before saving avoids the most common mistake. 1. Click **Save**. 1. Confirm that the provider card updates to show the credential as **Active** or **Configured**. ![Bring Your Own Key settings](/img/screenshots/console/byok.jpeg) The credential is encrypted before being stored and is never returned to the Console after the initial save. Only a masked version of the key is shown on subsequent views. This is deliberate: even the user who saved the credential cannot retrieve the original value back through the UI. ![BYOK provider configured](/img/screenshots/console/byok-configured.jpeg) ### BYOK settings field reference The settings page exposes one configuration card per supported provider: - **Supported providers.** BYOK can be configured for OpenAI, Anthropic, Google (Gemini), Azure OpenAI, Mistral, Together, Perplexity, and Cohere, with additional providers added over time. Keys for several providers can be configured at once; each provider operates independently, so configuring an OpenAI key has no effect on how Anthropic requests are handled. OpenAI, Anthropic, Google (Gemini), Azure OpenAI, Mistral, Together, Perplexity, and Cohere, with additional providers added over time. Keys for several providers can be configured at once; each provider operates independently, so configuring an OpenAI key has no effect on how Anthropic requests are handled. - **Rotation behaviour.** Saving a new key over an existing one replaces the old credential immediately; the next request to that provider uses the new key. No in-flight requests are interrupted. - **Removal behaviour.** Removing a credential, or clearing the field and saving, reverts that provider's models to the centrally managed credential on the next request. - **Security model.** All BYOK keys are encrypted at rest before storage, are never logged or exposed in API responses, and can only be updated or removed by the account that owns them. :::caution A BYOK credential applies account-wide for the provider, not per API key. Subsequent requests from this Console account to any model hosted by the configured provider will use the BYOK credential rather than the centrally managed one, on every Agent Router API key. If the credential should apply to only a subset of traffic, the cleanest approach is to use a separate Console account for that subset, or to manage the credential's lifecycle in step with the traffic that needs it. ::: ## Step 2: send a request through the BYOK credential The BYOK credential takes effect immediately on save. Sending a request to any model from the configured provider exercises it. `PROXY_URL` stands for the proxy endpoint from the Console Dashboard, scheme included and ending in `/v1`. ```bash curl PROXY_URL/chat/completions \ -H "Authorization: Bearer YOUR_AGENT_ROUTER_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "gpt-4o", "messages": [{"role": "user", "content": "Confirm this request is routed through the BYOK credential."}] }' ``` The request looks identical to any other gateway request; nothing in the request shape indicates whether BYOK or centrally managed credentials are being used. The decision happens inside the gateway, based on the BYOK configuration of the account. For each request, the provider that hosts the target model is resolved, and that provider's BYOK configuration is checked: - When a BYOK credential exists for the provider, the request is forwarded upstream with the consumer-supplied key, and usage is tracked by the provider directly against the BYOK account. - When no BYOK credential exists for the provider, the request is served with the centrally managed credential instead. The Playground is an alternative way to exercise the credential without leaving the Console: 1. Open **Build → Playground** in the Console sidebar. 1. Select a model from the BYOK-configured provider. 1. Send a short test message. 1. Confirm a successful response is returned. ## Step 3: verify the credential was used Three signals confirm that the BYOK credential rather than the centrally managed credential served the request. Two of them are visible inside the Console; the third is the most authoritative, because it confirms the request actually reached the provider account. 1. In the Console, open **Monitoring → Request Logs**. 1. Filter by the API key used in the test. 1. Locate the request and expand its detail row. 1. Confirm the request metadata indicates the BYOK credential was used. The exact field name depends on the Console build, but a BYOK-served request is clearly distinguished from a centrally served one. 1. Cross-reference with the provider's own usage dashboard. For OpenAI, the request should appear at `platform.openai.com/usage`, attributed to the account that owns the BYOK credential, within a few minutes of the call. For the other providers, the equivalent surface offers the same confirmation. The provider-side check is the strongest evidence because it cannot be faked by Agent Router: if the request shows up under the BYOK account, the credential was used. ## Step 4: rotate or remove the credential Provider credentials are not forever. Keys are rotated on a schedule, revoked when an employee changes role, or replaced when a quota tier changes. Agent Router supports the full lifecycle in a single screen. To rotate a credential: 1. Open **Settings → Bring Your Own Key**. 1. Select the provider whose credential should be rotated. 1. Paste the new key into the **API Key** field. 1. Click **Save**. The old credential is replaced on save; the next request to that provider uses the new key. No in-flight requests are affected, but in-flight requests that were authorised with the old key still complete with it. To remove a credential entirely: 1. Open **Settings → Bring Your Own Key**. 1. Select the provider. 1. Click **Remove**, or clear the key field and save. Subsequent requests to that provider's models revert to the centrally managed credential. The transition is immediate; the next request will be served by Agent Router account. :::tip Rotate BYOK credentials in step with the provider's own key-rotation schedule. A credential that has expired on the provider side but is still configured in the Console produces authentication failures that look, at first glance, like provider outages. A scheduled rotation cadence on the Console side (written into the team's runbook rather than Agent Router) eliminates this category of incident. ::: ## How BYOK composes with routing configurations BYOK credentials and routing configurations are independent: the routing configuration decides *which backend* serves a request, and the BYOK configuration decides *which credential* is used to reach that backend. The two compose without conflict. A few useful patterns: - **Fallback with BYOK primary and centrally managed secondary.** The fallback chain has a BYOK-hosted model at priority 0 and a centrally managed model at priority 1. Under normal conditions, the BYOK account serves all traffic; if the BYOK account experiences a quota or availability problem, traffic falls through to Agent Router account without disruption. - **Traffic split between BYOK and centrally managed accounts.** Two routes resolve the same logical name; one is hosted by the BYOK provider and one by Agent Router. A weight distribution shifts traffic between them, useful when a BYOK quota needs to be smoothed out across the month. - **BYOK plus model-name override.** Logical names defined under [Apply Advanced Routing Rules](/agent-router-service/guides/apply-advanced-routing-rules/) compose naturally with BYOK, because the override is applied before credential selection. ## What to do next - **Aggregate MCP servers into a profile**: the next agent-infrastructure concern. See [Aggregate MCP Servers into a Profile](/agent-router-service/guides/aggregate-mcp-servers-into-a-profile/). - **Monitor traffic and usage**: distinguish BYOK and centrally managed traffic in usage analytics over time. See [Monitor Traffic and Usage](/agent-router-service/guides/monitor-traffic-and-usage/). - **Export telemetry to an observability stack**: push the BYOK/centrally managed distinction into existing dashboards so cost reporting picks it up. The BYOK credentials configured above remain attached to the Console account for subsequent guides. Where to go next Aggregate MCP servers into a profile The next agent-infrastructure concern after credentials. Monitor traffic and usage Distinguish BYOK and centrally managed traffic in usage analytics over time. --- ## Build a chat app Build a streaming chat application that uses Agent Router Enterprise for model routing, with multi-turn conversation support, streaming responses, and automatic failover. :::info All code snippets in this guide are Python. Any OpenAI SDK is supported; Python is not required. ::: ## Architecture Requests flow from the frontend through the application backend to Agent Router, which routes to a provider and streams the response back over server-sent events (SSE). ```mermaid flowchart LR U[User] --> F[Frontend] F -->|POST /api/chat| B[Your Backend] B -->|stream: true| T[Agent Router] T --> P1[OpenAI] T -.->|fallback| P2[Anthropic] P1 -->|SSE stream| B B -->|SSE stream| F style T fill:#4f46e5,color:#fff ``` | Built by the application | Handled by Agent Router | | --- | --- | | Frontend UI, conversation state, API route | Provider routing, streaming, failover, cost tracking | ## Step 1 - Set up the client This step stands up the backend that the chat application calls. A FastAPI application exposes a single `POST /api/chat` route, and an OpenAI client is configured with Agent Router API key and base URL. On each request, the route reads the message array from the request body, opens a streaming completion against Agent Router, and relays each token to the caller as a server-sent event. The `stream_options={"include_usage": True}` flag requests token-usage data in the final chunk, and the result is returned as a `StreamingResponse` with the `text/event-stream` media type, which is what allows the frontend to render the reply as it arrives rather than after the full response completes. ```python title="Python" from fastapi import FastAPI, Request from fastapi.responses import StreamingResponse from openai import OpenAI import os import json app = FastAPI() client = OpenAI( api_key=os.environ["AGENTROUTER_API_KEY"], base_url="https://api.router.tetrate.ai/v1", ) @app.post("/api/chat") async def chat(request: Request): body = await request.json() messages = body.get("messages", []) stream = client.chat.completions.create( model="gpt-5.6-terra", messages=messages, stream=True, stream_options={"include_usage": True}, ) async def generate(): for chunk in stream: delta = chunk.choices[0].delta if chunk.choices else None if delta and delta.content: yield f"data: {json.dumps({'content': delta.content})}\n\n" # Usage comes in the final chunk if hasattr(chunk, 'usage') and chunk.usage: yield f"data: {json.dumps({'usage': {'prompt_tokens': chunk.usage.prompt_tokens, 'completion_tokens': chunk.usage.completion_tokens}})}\n\n" yield "data: [DONE]\n\n" return StreamingResponse(generate(), media_type="text/event-stream") ``` ## Step 2 - Multi-turn conversations This step gives the assistant memory of the conversation so far. The chat completions API is stateless and retains nothing between calls, so the full conversation must accompany every request. A running `conversation` list holds that history: the system prompt seeds it, each user message is appended before the request, and each assistant reply is appended after it. Every turn therefore sends the complete dialogue, which is what lets the model resolve follow-up questions such as "Can you give me an example?" against everything said earlier. The same array is what the streaming route in Step 1 receives from the frontend. Conversation history is maintained by passing the full message array on each request: ```python title="Python" conversation = [ {"role": "system", "content": "You are a helpful assistant."}, ] def chat_turn(user_message: str) -> str: conversation.append({"role": "user", "content": user_message}) response = client.chat.completions.create( model="gpt-5.6-terra", messages=conversation, ) assistant_message = response.choices[0].message.content conversation.append({"role": "assistant", "content": assistant_message}) return assistant_message # Each turn includes full history print(chat_turn("What's Agent Router?")) print(chat_turn("How does fallback routing work?")) print(chat_turn("Can you give me an example?")) ``` ## Step 3 - Add fallback for production This step protects the chat application against a single provider failing. A fallback policy is an ordered list of providers attached to the API key. When the primary provider returns a recoverable error, a rate-limit response, or a timeout, Agent Router automatically retries the next provider in the list, so one outage does not interrupt the conversation. The policy is configured in Agent Router's dashboard rather than in code, which is why the backend from Step 1 stays unchanged: the same `chat.completions.create` call gains failover the moment the policy is in place. No code changes are required; Agent Router handles failover automatically. ## Step 4 - Track usage This step surfaces what each conversation costs. Because every request passes through Agent Router, token usage and cost are recorded centrally, with no per-application instrumentation. When streaming with `stream_options: { include_usage: true }`, the prompt and completion token counts set in Step 1 are included in the final SSE event, even for providers that do not include them natively, so the frontend can show a running token or cost figure as the reply streams in. Per-model cost breakdowns are available in Agent Router's dashboard for review after the fact, or through the Usage API for programmatic reporting. Where to go next Make an AI API call The single-request basics: API key, base URL, streaming, embeddings, and images. Improve resilience with fallbacks Formalise the failover policy behind the chat app into an explicit fallback chain. --- ## Agent Router Service Agent Router Service For individual developers on the Service tier. Tetrate hosts the platform; no installation is required. Service quickstart Sign up at router.tetrate.ai, add an API key, and make a first routed AI request against the OpenAI-compatible endpoint. No installation. Console quickstart Sign in, issue an API key, and make a first routed AI request from the Developer Console. Not sure which path fits? See Deployment models. --- ## Make an AI API call Agent Router Enterprise routes any model request through a single endpoint. Agent Router exposes one API that works with every model (OpenAI, Anthropic, Google, and others), with automatic failover, usage tracking, and cost controls. :::info All code snippets in this guide are Python. Any OpenAI SDK is supported; Python is not required. ::: ## 1. Get an API key This step obtains the credential that authenticates every call. Each key identifies the calling consumer and carries the routing policy, budgets, and usage tracking attached to it, so a request is both authorised and governed by the key it presents. Sign up at [router.tetrate.ai](https://router.tetrate.ai) and create an API key from the dashboard. The key appears under **Settings → API Keys**. An API key has the form `sk-...` and is required for every request. Keep it secure. ## 2. Make the first request This step confirms that the credential and endpoint work by sending a single chat completion. Because Agent Router implements the OpenAI API, an existing OpenAI SDK needs only two changes: the base URL and the API key. The request names a model, Agent Router resolves that model to a provider, forwards the call with the right credentials, and returns a standard chat completion response. No provider-specific SDK or code path is involved. Agent Router is OpenAI-compatible. Point any OpenAI SDK at `https://api.router.tetrate.ai/v1`. ```python title="Python" from openai import OpenAI client = OpenAI( api_key="your-api-key", base_url="https://api.router.tetrate.ai/v1", ) response = client.chat.completions.create( model="gpt-5.6-terra", messages=[{"role": "user", "content": "What is Agent Router?"}], ) print(response.choices[0].message.content) ``` :::tip Replace `gpt-5.6-terra` with any model in Agent Router's catalog, such as `claude-sonnet-5` or `gemini-2.5-flash`. Agent Router routes to the right provider automatically. To see which models are routable for your API key, call `GET /v1/models` or `client.models.list()` — see [Models API](/reference/supported-apis/#models-api-v1models). ::: ## 3. Stream responses This step returns the response incrementally instead of in a single block. Setting `stream=True` makes Agent Router forward tokens as the provider generates them, which lowers the time to first token and suits chat-style interfaces. The streamed chunks use the OpenAI event format, so client code written for OpenAI streaming works without modification. Add `stream: true` for real-time token streaming. This works identically to the OpenAI streaming API. ```python title="Python" stream = client.chat.completions.create( model="gpt-5.6-terra", messages=[{"role": "user", "content": "Explain Agent Router in one paragraph."}], stream=True, ) for chunk in stream: if chunk.choices[0].delta.content: print(chunk.choices[0].delta.content, end="") ``` ## 4. Generate embeddings This step produces vector embeddings for search, retrieval, or clustering workloads. Embeddings convert text into numeric vectors that capture meaning, and they are requested through the same client and endpoint as chat completions: only the method and model change. The same routing, usage tracking, and cost controls apply. Agent Router routes embedding requests the same way: one endpoint, any provider. ```python title="Python" response = client.embeddings.create( model="text-embedding-3-small", input="What is an AI gateway?", ) print(f"Dimensions: {len(response.data[0].embedding)}") ``` ## 5. Generate images This step generates images through the same gateway used for text. An image request reaches the provider through the identical endpoint and key, and the response returns a URL to the generated image. Cost is recorded per image by size, quality, and count, and appears in the same usage reports as chat and embedding spend. Image generation is routed through Agent Router with cost tracking by size, quality, and count. ```python title="Python" response = client.images.generate( model="dall-e-3", prompt="A futuristic city powered by AI, digital art", size="1024x1024", ) print(response.data[0].url) ``` ## Use the tare CLI and typed SDKs The steps above use a generic OpenAI SDK pointed at the gateway. Agent Router also ships the `tare` CLI for minting keys and typed SDKs for Go, Python, and TypeScript, each with a runnable `examples/quickstart`. This path suits teams that prefer a typed client and bundled examples over raw HTTP. All downloads are public, no sign-in required. Substitute the placeholders as follows: replace `` with the version shown on the [SDKs](https://sdk.tetrate.ai/sdks.html) page, `.` with a key minted below, and `api.` with the gateway host for the deployment. ### Install the CLI and mint a key `tare` is the official Agent Router CLI and the fastest way to verify reach. ```bash curl -fsSL https://tare.tetrate.ai/tools/install.sh | bash tare --version ``` Authenticate, then create a long-lived key the SDKs use as `AGENTROUTER_API_KEY` (the `secret` field is shown exactly once): ```bash tare api login --issuer https://idp.example.com tare api whoami tare api keys create --name quickstart ``` See the [TARE CLI reference](/reference/tare-cli/) for the available commands. ### Download and run a typed SDK Each SDK tarball ships a runnable `examples/quickstart` so a first call needs no copied snippets. Download from the public stable channel, install from the local file (no public registry), then run the example. #### Go ```bash curl -fLO https://tare.tetrate.ai/tools/sdks/stable/agentrouter-go-.tar.gz mkdir -p third_party && tar -xzf agentrouter-go-.tar.gz -C third_party/ # go.mod: require github.com/tetrateio/agentrouter-go v0.1.0 replace github.com/tetrateio/agentrouter-go => ./third_party/agentrouter-go- go mod tidy ``` From the extracted SDK root: `go run ./examples/quickstart`. #### Python ```bash curl -fLO https://tare.tetrate.ai/tools/sdks/stable/agentrouter-python-.tar.gz pip install ./agentrouter-python-.tar.gz ``` The sdist unpacks `examples/quickstart` next to the package: `python examples/quickstart/quickstart.py`. #### TypeScript ```bash curl -fLO https://tare.tetrate.ai/tools/sdks/stable/agentrouter-typescript-.tgz npm install ./agentrouter-typescript-.tgz ``` From the extracted SDK root run `bun install` once, then `cd examples/quickstart && bun install && bun run quickstart.ts`. #### curl No SDK needed, call the API directly with the key: ```bash export AGENTROUTER_API_KEY=. curl -sS https://api./v1/me \ -H "Authorization: Bearer ${AGENTROUTER_API_KEY}" ``` Where to go next Build a chat app Turn the single call into a streaming, multi-turn chat application with failover. Route requests across providers Attach a routing configuration so requests span more than one backend. --- ## Quickstarts The shortest path from an API key to working code. Each quickstart takes minutes, uses the OpenAI SDK unchanged, and ends with real traffic visible in the Console. Make an API call Authenticate, create an API key, and send a first routed request through the gateway. Build a chat app Build a small chat application that exercises the gateway's core functionality. --- ## Dev Console quickstart This 10-minute walkthrough takes you from sign-in to your first routed AI request through the gateway. By the end you will have an API key, a proxy endpoint URL, a working request in at least one language, and a verified entry in Request Logs. The Console is the developer-facing application. It gives you everything you need to call AI models, configure routing policies, and monitor usage, without managing provider credentials directly. All AI requests pass through the gateway, which handles authentication, routing, fallback, guardrail enforcement, and observability transparently. :::info The screenshots in this guide show a redacted URL. Your URL in a production environment will be something like `router..tetrate.ai`. Your console URL will differ based on deployments in production and non-production. ::: ## Step 1: sign in 1. Navigate to your Console URL (e.g., `https://router.poc.tetrate.ai`) ![Sign in page](/img/screenshots/console/quickstart-sign-in.jpeg) 1. Click **Sign In with Corporate SSO** to authenticate through your organization's identity provider. If your deployment uses email/password authentication, enter your credentials directly. ![Click Sign In with Corporate SSO](/img/screenshots/console/quickstart-sign-in-sso.jpeg) 1. You land on the **Dashboard**, which displays your proxy endpoint URL and a summary of recent usage activity. Note the proxy endpoint URL; you will need it in Step 3. ## Step 2: create an API key API keys authenticate your application's requests to the gateway. Each key can be assigned its own routing policy (fallback chain, traffic splitting), rate limits, and model scope. For this quickstart, you will create a basic key with default settings. 1. In the sidebar, click **API Keys** ![Click API Keys in sidebar](/img/screenshots/console/quickstart-api-keys-nav.jpeg) 1. Click **Add API Key** ![Click Add API Key](/img/screenshots/console/quickstart-api-keys-add.jpeg) 1. Enter a descriptive name for your key (e.g., `my-agent-router-key`). Choose a name that identifies the application or workload this key will serve; the key value is shown only once, but you can always identify keys by name in logs and usage reports. ![Enter key name](/img/screenshots/console/quickstart-api-key-name.jpeg) 1. Click **Create key** ![Click Create key](/img/screenshots/console/quickstart-api-key-create.jpeg) 1. Copy the generated key immediately and store it in a secure location (a secrets manager or environment variable). This is the only time the key value is displayed. ![Copy the API key](/img/screenshots/console/quickstart-api-key-copy.jpeg) ![API key created](/img/screenshots/console/quickstart-api-key-done.jpeg) :::caution API keys grant the bearer the ability to route requests through the gateway and consume AI quota. Store your key securely; do not commit it to source control or include it in logs or error messages. ::: ## Step 3: find your proxy endpoint Your proxy endpoint URL is the address your application calls instead of calling AI providers directly. When a request arrives at the proxy, the gateway: 1. Authenticates the request using your API key 2. Applies the routing policy configured for that key (fallback chain, traffic splitting rules) 3. Translates the request format if routing to a different provider (e.g., OpenAI format to Anthropic backend) 4. Returns the normalized response, including error normalization if the upstream provider returns an error The proxy endpoint URL is displayed on your Dashboard and typically looks like: ```text https://proxy.poc.tetrate.ai/v1 ``` Append the appropriate path for the API format you intend to use: | Format | Path | Description | | --- | --- | --- | | OpenAI Chat Completions | `/v1/chat/completions` | Most widely supported format; compatible with all OpenAI-compatible SDKs and agent frameworks | | OpenAI Responses | `/v1/responses` | Newer OpenAI Responses API with a simplified interface | | Anthropic Messages | `/v1/messages` | Anthropic native format for Claude models | For detailed examples of each format including streaming, see **[Gateway APIs](/reference/supported-apis/)**. ## Step 4: make your first request Replace `YOUR_API_KEY` with the key you created in Step 2. All examples below call `gpt-4o`, but you can substitute any model name from your [Model Catalog](/agent-router-service/guides/route-requests-across-providers/). To discover models programmatically for the same key, call `GET /v1/models` — see [Models API](/reference/supported-apis/#models-api-v1models). The only difference between calling the gateway and calling a provider directly is the `base_url` (or the curl URL). All other parameters, response formats, and SDK behaviours are identical. ### Using curl ```bash curl https://proxy.poc.tetrate.ai/v1/chat/completions \ -H "Authorization: Bearer YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "gpt-4o", "messages": [{"role": "user", "content": "Hello, world!"}] }' ``` ### Using Python ```python from openai import OpenAI client = OpenAI( base_url="https://proxy.poc.tetrate.ai/v1", api_key="YOUR_API_KEY" ) response = client.chat.completions.create( model="gpt-4o", messages=[{"role": "user", "content": "Hello, world!"}] ) print(response.choices[0].message.content) ``` ### Streaming with curl ```bash curl https://proxy.poc.tetrate.ai/v1/chat/completions \ -H "Authorization: Bearer YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "gpt-4o", "messages": [{"role": "user", "content": "Hello, world!"}], "stream": true }' ``` ### Streaming with Python ```python from openai import OpenAI client = OpenAI( base_url="https://proxy.poc.tetrate.ai/v1", api_key="YOUR_API_KEY" ) stream = client.chat.completions.create( model="gpt-4o", messages=[{"role": "user", "content": "Hello, world!"}], stream=True ) for chunk in stream: content = chunk.choices[0].delta.content if content: print(content, end="", flush=True) ``` ### Using the playground The Playground lets you test model routing interactively without writing any code, useful for evaluating models or verifying routing behaviour before integrating into an application. 1. In the sidebar, go to **Build > Playground** 2. Select a model from the dropdown. The list shows all models currently enabled by your administrator. 3. Type a message and press **Send** 4. View the response alongside token usage, latency, and the upstream provider that served the request ## Step 5: check your request logs Every request the gateway processes is recorded in Request Logs. This is your primary tool for debugging, auditing, and analysing cost. 1. Go to **Monitoring > Request Logs** 2. You should see your request with the model name, upstream provider, token counts, estimated cost, and total latency 3. Click any row to view the full request and response payloads, including which provider was selected and whether any fallback attempts were made Request Logs also display the `x-request-id` correlation header that the gateway attaches to every response. Use this ID to locate the corresponding span in your OpenTelemetry tracing backend. See [Gateway Behavior](/reference/gateway-behavior/) for details on correlation IDs and the full debugging workflow. ## Step 6: view usage analytics Usage Analytics provides aggregated metrics across all your requests. Use it to track consumption trends, compare model costs over time, and understand traffic distribution across API keys. 1. Go to **Monitoring > Usage** 2. Select a time range (e.g., **Last 24 hours**) 3. View breakdowns by model and API key, including total tokens consumed, estimated cost, and request volume ## Evaluation checkpoint - [ ] Successfully signed in to the Console - [ ] Created an API key and stored it securely - [ ] Located the proxy endpoint URL on the Dashboard - [ ] Made a successful request via curl, Python, or the Playground - [ ] Confirmed the request appears in Request Logs with provider, latency, and token details - [ ] Reviewed usage analytics for the time period Where to go next Route requests across providers Browse the model catalog and configure routing policies. Improve resilience with fallbacks Configure fallback chains so the gateway reroutes to a backup provider on errors. --- ## 10 minute Quickstart This 10-minute walkthrough takes you from sign-up to your first routed AI request on the hosted Agent Router Service, with no installation and nothing to deploy. By the end you will have an account, an API key, the OpenAI-compatible base URL, a working request in at least one language, and a verified entry in Request Logs. The Service is Tetrate-hosted: sign up, add a key, and route requests through the gateway, which handles authentication, routing, fallback, and observability transparently. It exposes a single OpenAI-compatible endpoint, so pointing an existing agent or SDK at it is a base-URL and key change rather than a rewrite. :::info The Agent Router Service is the self-serve, Tetrate-hosted tier for individual developers. Everything in this guide happens at [router.tetrate.ai](https://router.tetrate.ai); there is no cluster to install. For the enterprise, dedicated-tenant experience, see the [Dev Console quickstart](/agent-router-service/quickstarts/quickstart-console/). ::: ## Step 1: create an account 1. Go to [router.tetrate.ai/sign-in](https://router.tetrate.ai/sign-in). 1. Sign up with a GitHub or Google account. Signing up with a business email grants a small starting credit balance for evaluation. 1. You land on the dashboard, which shows your credit balance, recent usage, and the entry points for API keys and the Playground. ## Step 2: understand pricing and credits The Service is pay-as-you-go. Each request is charged at the underlying model's cost plus a small service fee, drawn down from a prepaid credit balance that can be set to replenish automatically. No commitment or subscription is required to evaluate. 1. Open the billing area from the dashboard to view your current balance. 1. Optionally, add credits or enable auto-replenishment so an evaluation is not interrupted when the starting balance runs low. :::info New accounts created with a business email start with free evaluation credit. Costs are itemized per request in Request Logs (Step 5), so the exact spend for each call is visible as it happens. ::: ## Step 3: create an API key API keys authenticate your application's requests to the gateway. Each key routes through the Service with its own usage tracking and can be pointed at different routing strategies. For this quickstart, create a basic key with default settings. 1. In the dashboard, open the [API Keys](https://router.tetrate.ai/api-keys) page. 1. Click **Add API Key** and enter a descriptive name (for example, `my-explore-key`). Choose a name that identifies the application or workload the key will serve; keys are identified by name in logs and usage reports. 1. Create the key, then copy the generated value immediately and store it in a secure location (a secrets manager or environment variable). This is the only time the key value is displayed. :::caution API keys grant the bearer the ability to route requests through the gateway and consume credit. Store your key securely; do not commit it to source control or include it in logs or error messages. ::: ## Step 4: find your base URL The Service exposes a single OpenAI-compatible endpoint. Your application calls this base URL instead of calling AI providers directly, and the gateway authenticates the request, applies the routing strategy for the key, translates formats when routing to a different provider, and returns a normalized response. The base URL for the Service is: ```text https://api.router.tetrate.ai/v1 ``` Append the appropriate path for the API format you intend to use: | Format | Path | Description | | --- | --- | --- | | OpenAI Chat Completions | `/v1/chat/completions` | Most widely supported format; compatible with all OpenAI-compatible SDKs and agent frameworks | | OpenAI Responses | `/v1/responses` | Newer OpenAI Responses API with a simplified interface | | Anthropic Messages | `/v1/messages` | Anthropic native format for Claude models | For detailed examples of each format including streaming, see **[Gateway APIs](/reference/supported-apis/)**. ## Step 5: make your first request Replace `YOUR_API_KEY` with the key you created in Step 3. The examples below call `gpt-4o`, but you can substitute any model available to your account. To discover models programmatically for the same key, call `GET /v1/models`; see [Models API](/reference/supported-apis/#models-api-v1models). The only difference between calling the gateway and calling a provider directly is the `base_url` (or the curl URL). All other parameters, response formats, and SDK behaviours are identical. ### Using curl ```bash curl https://api.router.tetrate.ai/v1/chat/completions \ -H "Authorization: Bearer YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "gpt-4o", "messages": [{"role": "user", "content": "Hello, world!"}] }' ``` ### Using Python ```python from openai import OpenAI client = OpenAI( base_url="https://api.router.tetrate.ai/v1", api_key="YOUR_API_KEY" ) response = client.chat.completions.create( model="gpt-4o", messages=[{"role": "user", "content": "Hello, world!"}] ) print(response.choices[0].message.content) ``` ### Streaming with curl ```bash curl https://api.router.tetrate.ai/v1/chat/completions \ -H "Authorization: Bearer YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "gpt-4o", "messages": [{"role": "user", "content": "Hello, world!"}], "stream": true }' ``` ### Streaming with Python ```python from openai import OpenAI client = OpenAI( base_url="https://api.router.tetrate.ai/v1", api_key="YOUR_API_KEY" ) stream = client.chat.completions.create( model="gpt-4o", messages=[{"role": "user", "content": "Hello, world!"}], stream=True ) for chunk in stream: content = chunk.choices[0].delta.content if content: print(content, end="", flush=True) ``` ### Using the Playground The Playground lets you test model routing interactively without writing any code, useful for comparing models or verifying routing behaviour before integrating into an application. 1. Open the **Playground** from the dashboard. 1. Select a model from the dropdown. The list shows all models available to your account. 1. Type a message and send it. 1. View the response alongside token usage, latency, and the upstream provider that served the request. ## Step 6: check your request logs Every request the gateway processes is recorded in Request Logs. This is your primary tool for debugging, auditing, and analysing cost. 1. Open **Request Logs** from the dashboard. 1. Confirm your request appears with the model name, upstream provider, token counts, estimated cost, and total latency. 1. Click any row to view the full request and response payloads, including which provider was selected and whether any fallback attempts were made. ## Evaluation checkpoint - [ ] Created an account on the Agent Router Service - [ ] Reviewed the credit balance and pricing - [ ] Created an API key and stored it securely - [ ] Made a successful request via curl, Python, or the Playground - [ ] Confirmed the request appears in Request Logs with provider, latency, and token details Where to go next Route requests across providers Browse the model catalog and configure routing policies. Use your own provider credentials Bring your own OpenAI, Anthropic, or other provider keys and route them through the gateway. --- ## Fast track evaluation import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; # Fast track evaluation Evaluate the Fully Managed tier in an afternoon: first routed request in about 15 minutes after onboarding; a small team and cost readout by the end of the day. Nothing to install. ## What this evaluation covers Tetrate Agent Router is an [AI gateway](/product-architecture/product-overview/#product-overview) between applications or coding agents and providers (OpenAI, Anthropic, and others). One OpenAI-compatible endpoint handles routing, credentials, cost, and policy; every request is recorded and attributed to the key that made it. On the [Fully Managed tier](/#which-is-right-for-you), Tetrate runs both the [management plane](/product-architecture/planes-and-components/#operating-model) and the [data plane](/product-architecture/planes-and-components/#core-components-of-the-data-plane), so the evaluation starts at first sign-in. Results are only convincing on real traffic, so the fast track routes work that is already being paid for: an existing provider account via Bring Your Own Key (BYOK), or a Claude Pro/Max plan via [passthrough mode](/reference/glossary/#passthrough-mode) through the gateway. Upstream billing stays as before, and Agent Router observes and attributes the traffic. Nothing is locked in, and the evaluation can be stopped at any point. By the end, the environment produces: - Real traffic flowing through the gateway, still billed upstream as before (BYOK or Claude Code [passthrough mode](/reference/glossary/#passthrough-mode)). - A cost and token breakdown per request, model, and key, with a monthly projection. - A budget with a live meter, and multiple teammates routing through the same gateway. The result is the raw material for a rollout decision: what current AI usage actually costs, where it goes, and how it compares across a team. ## Before beginning :::info Want to know more about Agent Router? - Check the [product overview](/product-architecture/product-overview/) page for more information. - Or, consult the [where to begin](/where-to-begin/) page as a starting point. ::: The fast track assumes the following are on hand. Note: URLs below use `` for the tenant hostname segment Tetrate assigned; Fully Managed evaluation tenants look like `tare-pn-` (for example `tare-pn-a1b2c3d4`). - **The Admin Dashboard URL** Tetrate provided for the evaluation tenant, for example `https://dashboard.tare-pn-.tetrate.ai`. The Admin Dashboard is the operator view: onboarding, identity, providers, models, budgets, and Agent Router settings. - **The Dev Console URL** Tetrate provided for the evaluation tenant, for example `https://router.tare-pn-.tetrate.ai`. The Dev Console is the developer view: API keys, the Base URL, request logs, and usage. - **The Gateway URL** Tetrate assigned for the evaluation tenant, listed in the provisioning email, for example `https://proxy.tare-pn-.tetrate.ai`. Nothing is entered with it during onboarding; the same value appears automatically as the **Base URL** in the Dev Console once setup is complete. - **An LLM provider API key**, for example an OpenAI or Anthropic key. It is added once during onboarding (Step 1), and that key routes real traffic on its own. Adding a **BYOK** key (Step 3) is optional, and is only needed to bill each person's spend to their own provider account. Alternatively, a Claude Max or Pro subscription, for Claude Code **[passthrough mode](/reference/glossary/#passthrough-mode)** instead. - **An identity provider**. An [Auth0](https://auth0.com) account works well for an evaluation and takes minutes to set up; corporate IdP admin access can be used instead where it is available and approved. *Learn how to set up Auth0 [here](/product-architecture/product-evaluation/auth0-for-evaluation/)*. - **A terminal or HTTP client** (curl, Python, or Postman) for the first request. ## Step 1: Initial sign in and onboard (Admin Dashboard) {#sign-in-and-onboard} Access is the first thing the evaluation establishes, because every later action happens under a real signed-in identity. The first sign-in launches a one-time [onboarding](/agent-router-enterprise/fully-managed/onboarding/) wizard that configures the organization: single sign-on, at least one provider, and the models to expose. Single sign-on is configured up front for two reasons. It puts the evaluation on the same footing as a real deployment from the start, and it turns teammate onboarding later in the fast track into a single shared link with no separate account administration. Auth0 is recommended for an evaluation because it stands up an identity provider in minutes without waiting on corporate IT. The configuration is identical to a corporate IdP, so nothing about the evaluation changes if the tenant later moves to permanent corporate SSO. 1. Open the **Admin Dashboard** URL (`https://dashboard.tare-pn-.tetrate.ai`) in a web browser of choice. 1. The first sign-in launches a welcome screen and the onboarding wizard. 1. Click button **Start setup** to begin the onboarding. Row numbers in the table below match the wizard screens.
No corporate IdP access? Use [auth0.com](https://auth0.com) for evaluation SSO Only change the settings listed below. Leave everything else at its default. 1. At [auth0.com](https://auth0.com), sign up and create a tenant. Go to **Applications → Create Application → Regular Web Application**. Leave **"This application is owned by a third party"** toggled **off**: this cannot be changed later, and enabling it breaks sign-in. 1. Enter a name, such as `agent-router-` and click **Create** . - *Note: Replace `` with the actual ID, which is visible in the dashboard URL*. 1. From tab **Settings**, copy the **Domain**, **Client ID**, and **Client Secret** to a text editor or note taking app. 1. Scroll down to Application URIs. Set **Allowed Callback URLs** to `https://auth.tare-pn-.tetrate.ai/api/auth/sso/callback/corporate`, and **Allowed Web Origins** to these three hostnames (`https://auth.tare-pn-.tetrate.ai`, `https://dashboard.tare-pn-.tetrate.ai`, `https://router.tare-pn-.tetrate.ai`), followed by clicking button **Save** . - *Note: Replace `` with the actual ID*. 1. Create an admin user: **User Management → Users → Create User** with a controlled email address. Use this email as an **Admin email** in the wizard below. If it is skipped, the first sign-in lands as a non-admin user with no in-product way to fix it. Full walkthrough: [Set up Auth0 for evaluation use](/product-architecture/product-evaluation/auth0-for-evaluation/).
| Wizard step | Section | What to do | | :---------- | :---------------- | :--------- | | 1 | Configure Organization Name | Set **Organization Name**. On Fully Managed there is no data-plane hostname to enter here, because Tetrate already runs the data plane and its gateway. The resulting **Base URL** appears later, in the Dev Console. | | 2 | Configure Corporate Login | **No corporate IdP access?** Expand the Auth0 tip above and use those Domain / Client ID / Client Secret values here.Fill in the OIDC values from **the IdP** (not Tetrate):- **Issuer**: `https:///` (trailing slash)- **Client ID**: `` - **Client Secret**: `` - **Scopes**: `openid profile email`- **Admin emails**: at least one administrator email address, added as a default project member. Optional; add members later from the Admin Dashboard (default project → **Members**) if skipped.- **Attribute mapping**: Set to `email`.- **Role Mapping (optional)**: skip for now.Click button **Next**. | | 3 | Configure LLM Provider Keys | Enable at least one (1) provider (for example OpenAI or Anthropic).- Click button **Enable** on an accessible provider.- Enter the API key into the **Provider secret** field and click button **Rotate** to persist it.- Toggle **Enable provider** to **on**.- Click button **Save**. | | 4 | Configure LLM Model | Review the model list.- Leave the defaults as is, or (de)select as needed.- This can be changed later in the Admin Dashboard. | | 5 | Review AI Gateway | Nothing to configure. Tetrate already provisioned the gateway, so this step only reports its status.Click button **Next**, which is enabled regardless of the status shown (see [what each status means](/agent-router-enterprise/fully-managed/onboarding/#step-5-review-ai-gateway)). | | 6 | Complete | Click button **Complete setup** to close the onboarding wizard. | :::important Fully Managed credential handling Tetrate manages the installed data-plane credential for Fully Managed deployments, and there is no credential control in the onboarding wizard to regenerate it. Later, from the Admin Dashboard's project **Gateway** tab, regenerating a data-plane credential is a rotation operation that can break the managed data plane. If it is regenerated by mistake, contact Tetrate to restore service. ::: After onboarding, sign out, then sign back in with SSO as the admin user. Onboarding sets up the default project automatically: the gateway URL, the providers connected in wizard step 3, and the models selected in wizard step 4 are already in place. Project membership comes from Step 2's Admin email field. A quick look confirms all of it: ### Confirm the default project (optional) 1. Log in to the **Admin Dashboard** (`https://dashboard.tare-pn-.tetrate.ai`). 1. In the left-hand sidebar, navigate to **Directory → Projects**. 1. Select the **default** project from the list of projects. - Note: at this point, only the default project will be visible. 1. Select tab **Gateway** and confirm the default gateway's **endpoint** matches the Gateway URL from the provisioning email. The value is already set, with nothing to configure. 1. Select tab **Members** and confirm the admin user is listed. 1. Select tab **Providers** and confirm the providers connected during onboarding are listed and enabled. 1. In the top-left corner, switch to project **default**, navigate to **Catalog → Models**, and confirm the models selected during onboarding are linked and enabled. If something looks wrong, for example an expected provider is not listed, each tab above allows a direct fix with **Add to project** or **Grant access** as needed. **Outcome** The Admin Dashboard loads under SSO, the default project points at the managed gateway endpoint, and the configured providers and models are available to the project. **Relevant resources** - [Management Plane onboarding](/agent-router-enterprise/fully-managed/onboarding/) - [Set up Auth0 for evaluation use](/product-architecture/product-evaluation/auth0-for-evaluation/) ## Step 2: route the first request (Dev Console) {#first-request} Before any real traffic runs, a single request confirms the path end to end. Two things are created here. The **Base URL** is the OpenAI-compatible URL that fronts every configured provider; applications point at it instead of at a provider directly. The **API key** is the credential a client presents to the gateway, and it is also the unit that request logs, usage, and budgets attribute to, so it is worth naming keys meaningfully. Because the endpoint is OpenAI-compatible, most existing code and SDKs work by changing only the base URL and the key. The `curl` and Python examples below are the same request expressed two ways. ### Create an API key 1. Open the Dev Console (`https://router.tare-pn-.tetrate.ai`) in a web browser. If needed, log in using the account set up in step 1. 1. In the left-hand sidebar, navigate to **Settings → API Keys**. 1. Click button **Add API Key**. 1. Set field **Key type** to **Inference key** from the drop-down list. 1. Enter a Key name (e.g. `eval-dev-key1`) 1. Click button **Create key**. 1. From the popup, copy the values of **Your API key** and **base URL** to a text editor or note taking app. - The API key will not be visible again, so verify it has been copied before continuing. 1. Click button **Done**. 1. Note the **Base URL** is displayed at the top of the page. - This is the project's gateway endpoint, provisioned automatically by Tetrate rather than entered during onboarding, usually with `/v1` appended, for example `https://proxy.tare-pn-.tetrate.ai/v1`. ### Test the API key 1. Send a request using curl or Python. - In the code examples, replace `YOUR_BASE_URL` and `YOUR_API_KEY` with the values copied earlier. - In the code examples, replace `gpt-4o` with any model in the catalog. ```bash export AGENTROUTER_API_KEY="" export BASE_URL="" curl ${BASE_URL}/chat/completions \ -H "Authorization: Bearer $AGENTROUTER_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "gpt-4o", "messages": [{"role": "user", "content": "Hello, world!"}] }' ``` ```python from openai import OpenAI client = OpenAI(base_url="", api_key="YOUR_API_KEY") resp = client.chat.completions.create( model="gpt-4o", messages=[{"role": "user", "content": "Hello, world!"}], ) print(resp.choices[0].message.content) ``` **Expected result:** a completion, and the request in the Dev Console under **Monitoring → Request Logs** with the model, token counts, cost, and latency. Example `curl` output: ```bash {"id":"msg_011CdLiqXzSDUMMYvVKFSGnW","choices":[{"finish_reason":"stop","index":0,"message":{"role":"assistant","content":"Hello! 👋 It's nice to meet you. How can I help you today?"}}],"created":1234567890,"model":"claude-haiku-4-5-20251001","object":"chat.completion","usage":{"completion_tokens":23,"prompt_tokens":11,"total_tokens":34}} ``` **Relevant resources** - [Make an API call](/agent-router-service/quickstarts/make-an-api-call/) - [Quickstart: the Dev Console](/agent-router-service/quickstarts/quickstart-console/) ## Step 3: put real traffic through the gateway (Dev Console) {#your-real-traffic} The synthetic request in Step 2 proves the plumbing; this step makes the evaluation convincing by running real work, a coding task or an agent run, through the gateway. The provider key added during onboarding already routes that traffic, so a client can be pointed at the proxy directly. The two paths below only change *whose* provider bill the tokens land on; pick one if it applies. ### Optional: Bring Your Own Key (per-person billing) **Bring Your Own Key (BYOK)** registers a provider API key already in use (OpenAI, Anthropic, and others), so the provider bills that account instead of the org key from onboarding. Use it to put each person's spend on their own provider invoice, or to reach a provider the onboarding key does not cover. The gateway measures and attributes the traffic; it does not rebill those tokens. This is not Claude Code passthrough mode; that path is below. 1. In the Dev Console, go to **Settings → Bring Your Own Key** and add credentials for a provider already in use. Credentials are encrypted at rest and never returned. 1. Point a coding agent or OpenAI-compatible client at the gateway. Use the **Base URL** from Step 2 and the **Agent Router API key** (not the provider key added above). ```bash # Example: OpenAI-compatible client (Base URL includes /v1) export OPENAI_BASE_URL=https://proxy.tare-pn-.tetrate.ai/v1 export OPENAI_API_KEY=YOUR_AGENT_ROUTER_API_KEY ``` Cursor and similar tools take the same two values in their settings. For Claude Code **without** a Max/Pro subscription (managed Anthropic credentials on the gateway, not passthrough mode), follow [Connect Claude Code → Option A (managed mode)](/agent-router-service/guides/coding-agents/claude-code/connect/#option-a-managed-mode-agent-router-managed-anthropic-key): proxy host without `/v1`, and the Agent Router API key in `ANTHROPIC_API_KEY`. 1. Run a real task with the agent or client. **Expected result:** the requests in Dev Console **Monitoring → Request Logs**, attributable to the Agent Router key. ### Alternative: Claude Code [passthrough mode](/reference/glossary/#passthrough-mode) (Pro or Max subscription) **Passthrough mode** keeps the flat Claude Max or Pro subscription fee in place while Agent Router Enterprise monitors the traffic through Request Logs, usage, and the Admin Dashboard Spend overview. Anthropic still authenticates the session with Claude Code's stored OAuth token, and the gateway routes and observes without putting those tokens on Agent Router billing. In the Spend overview, **passthrough usage** is the theoretical API-rate cost that traffic would carry without the subscription, and it does not count toward the budget. This is separate from BYOK: a Claude subscription is **not** pasted into Bring Your Own Key. 1. Follow [Connect Claude Code → Option B (passthrough mode)](/agent-router-service/guides/coding-agents/claude-code/connect/#option-b-passthrough-mode-an-existing-claude-max-subscription): - From a command prompt, type `claude /login` - Leave `ANTHROPIC_API_KEY` unset - Set `ANTHROPIC_BASE_URL` to the proxy host and `ANTHROPIC_CUSTOM_HEADERS` with `x-tars-api-key` (the Agent Router API key from Step 2). 1. Start a session (type `claude` from a command prompt) and run a real task. - Confirm with `/status` that the Anthropic base URL is the proxy host. - In Request Logs, these requests are labeled **passthrough**. **Expected result:** passthrough-labeled requests in Dev Console **Monitoring → Request Logs**, attributable to the Agent Router key. **Relevant resources** - [Use your own provider credentials](/agent-router-service/guides/use-your-own-provider-credentials/) - [Coding agents and tools](/agent-router-service/guides/coding-agents/) - [Connect Claude Code](/agent-router-service/guides/coding-agents/claude-code/connect/) (Option A managed, Option B passthrough) ## Step 4: see what the traffic costs (Admin Dashboard) {#see-the-cost} With real traffic flowing, cost questions move to the **Admin Dashboard**. Under the sidebar **Usage** section, start with **Analytics** to baseline spend. Step 5 attaches a personal budget under **Budgets** and reads the **Spend overview** (billed against the ceiling; an optional **passthrough** line if Claude Code passthrough mode was used). That matches [Keep team spend inside a ceiling](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/keep-team-spend-inside-a-ceiling/) and [See budget exhaustion coming](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/see-budget-exhaustion-coming/). ### Baseline cost in Analytics Before a ceiling is useful, read the same dimension the budget will use. 1. Open the Admin Dashboard (`https://dashboard.tare-pn-.tetrate.ai`) in a web browser. If needed, log in using the account set up in step 1. 1. Ensure **Organization** level is selected in the top left corner. 1. In the left-hand sidebar, navigate to **Usage → Analytics**. 1. Select tab **Cost**. 1. Use the drop-down box (top right corner) to set a time range that covers the traffic from Step 3 (for example last 24 hours or last 7 days). 1. Use the drop-down box directly next to the search box and select **Group by Mode** to visualize model usage across managed, BYOK, and passthrough traffic by request count, tokens, and estimated cost, per key, model, and provider. - Once additional users are added, **Grouping by User or Team** provides critical insight into model usage across developers. Size any monthly ceiling from this baseline: above normal burn, below a number that would be a problem. A first evaluation budget can start tight, and can be raised later. Write down a monthly projection (a representative day × working days, or a week × about 4.3). That list-price-style figure is the evaluation's cost signal, comparable to a subscription price or a direct-provider invoice. **Expected result:** a user row in Cost with tokens and estimated spend, enough to size a personal ceiling in Step 5. **Relevant resources** - [Keep team spend inside a ceiling](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/keep-team-spend-inside-a-ceiling/) - [Read the shadow bill](/agent-router-enterprise/guides/observability-and-analytics/usage-analytics-onboarding/read-the-shadow-bill/) (Dev Console view of list-price cost on passthrough traffic) ## Step 5: set a budget for one user (Admin Dashboard) {#set-a-budget} With a Cost baseline from Step 4, attach a ceiling to a single user's billed spend. Use the **One person** scope: a monthly USD limit on that user alone, independent of any team. ### Create a budget 1. Open the Admin Dashboard (`https://dashboard.tare-pn-.tetrate.ai`) in a web browser. If needed, log in using the account set up in step 1. 1. Ensure **Organization** level is selected in the top left corner. 1. In the left-hand sidebar, navigate to **Usage → Budgets**. 1. Click button **Add budget**. 1. Set **Scope** to **One person**. 1. Select the user (use search if the list is long). 1. Set **Spend limit** to a given amount in USD (for example, sized from the Analytics Cost baseline in Step 4), or pick a preset ($100, $500, $1,000, …). - The budget resets at the start of each calendar month (UTC); a mid-month create still applies to spend already in that month. 1. Name it (for example `Eval personal cap`) and review the summary. 1. Click button **Create budget**. 1. Verify the new budget is listed in the overview. ### Read the Spend overview 1. In the left-hand sidebar, navigate to **Usage → Budgets**. 1. Open the budget created in the previous steps. - The detail view has configuration on the left (scope, name, spend limit, **Active**) and **Spend overview** on the right. 1. Read the headline meter. - **billed** spend against the ceiling (for example `$36.01 of $100.00 · monthly`), plus **Used**, **Remaining**, and **Days left**. - Only **billed** amounts count toward the limit and drive alerts. 1. If **Claude Code passthrough mode** was configured in Step 3 and that traffic was sent, look for **passthrough usage (not counted toward budget)** under the meter. **Claude Code passthrough mode** keeps Max/Pro paying the flat subscription fee. The **passthrough** line is the theoretical cost of those same requests **without the subscription**, priced at standard API rates (`≈`). It is for scale and comparison only; it never draws down the budget. On a team **per user** budget, each member row can show **billed** next to **passthrough** the same way, with the member's billed amount against their cap. **If Claude Code passthrough mode was not configured**, the Spend overview stays simpler: billed against the ceiling, progress bar, Used / Remaining / Days left, with no passthrough line. **Expected result:** the budget under **User Budgets**, **Active**, with billed on the meter. A passthrough `≈` total appears only after Claude Code passthrough traffic. **Relevant resources** - [Keep team spend inside a ceiling](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/keep-team-spend-inside-a-ceiling/) - [See budget exhaustion coming](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/see-budget-exhaustion-coming/) - [Raise a spend ceiling](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/raise-a-spend-ceiling/) ## Step 6: bring in teammates (Admin Dashboard and Dev Console) {#bring-teammates} A single user proves the mechanism; a small group turns it into decision material. Cost and usage patterns only become comparable when there is more than one person to compare, so this step repeats the personal setup for a handful of the heaviest AI users on the team. Those users produce the most representative spread, and the SSO configured in Step 1 makes onboarding them a matter of a shared link. The payoff is team-level attribution: spend broken down by team, model, and application, which is the evidence a rollout decision rests on. ### Create two or more users Additional users must be created on the IdP (identity provider) so they can sign in with single sign-on. On a corporate IdP, follow the administrator's standard user-provisioning process instead of the steps below. For Auth0 (from Step 1): 1. Navigate to https://auth0.com and sign in. 1. In the left-hand navbar, select **User Management → Users**. 1. Click button **Create User** and select **Create via UI**. 1. Set **Connection** to **Username-Password-Authentication**. 1. Enter an email for the new user (e.g. `elmer.fudd@acme.com`) and set a strong password. 1. Click button **Create Another** and repeat the process one or more times. 1. End by clicking button **Create**. ### Set up for users and teams 1. Share the Dev Console URL (`https://router.tare-pn-.tetrate.ai`) with the users just created 1. Have them sign in with SSO using the email address and password entered in the previous steps. 1. Each user appears in the Admin Dashboard under **Directory → Users**, after their first login. 1. Each teammate repeats **Step 3: BYOK** with their provider API key, or Claude Code **passthrough mode** if they use Claude Max/Pro. 1. Assign each team a budget under **Usage → Budgets** (Whole team or Each teammate), sized from the Analytics baseline in Step 4 and the personal ceiling from Step 5. **Expected result:** spend comparable across teammates by model and app, with each team's meter drawing down. **Relevant resources** - [Onboard developers and issue keys](/agent-router-enterprise/guides/operate-and-govern/onboard-developers-and-issue-keys/) - [Use your own provider credentials](/agent-router-service/guides/use-your-own-provider-credentials/) ## Go deeper (optional) The fast track leaves behind a working environment and a first set of numbers. The following pages build on it: - [Graded evaluation scenarios](/product-architecture/product-evaluation/evaluation-scenarios/): a scored scorecard for a structured, comparable decision. - [Configure SSO](/agent-router-enterprise/guides/identity-and-access-management/single-sign-on/sso-configuration-guide/): move off the temporary Auth0 tenant to permanent corporate SSO. - [Guides for Developers](/agent-router-service/guides/) and [Guides for Platform Operators](/agent-router-enterprise/guides/operate-and-govern/): the full task-by-task reference. --- ## Agent Router Enterprise Fully Managed Agent Router Enterprise · Fully Managed Tetrate runs both planes, so there is nothing to install. Onboard the management plane, confirm the default project is ready for traffic, then validate the Admin Dashboard and Dev Console. :::important Fully Managed data-plane credentials Do not create, download, or regenerate a data-plane service-account credential during Fully Managed onboarding. Tetrate operates the data plane and manages its installed credential. Regenerating that credential is a rotation operation that can break the managed data plane. If it is regenerated by mistake, contact Tetrate to restore service. ::: Fully Managed sequence Step 1 Management plane onboarding Organization details, SSO, providers, and models. Skip any data-plane credential screen. Management plane onboarding → Step 2 Admin Dashboard quickstart Confirm the default project endpoint, membership, providers, models, users, and usage before developer handoff. Admin Dashboard quickstart → Step 3 Console quickstart Create an inference key and send a first routed request through the Base URL. Console quickstart → --- ## Management Plane onboarding Onboard a Fully Managed deployment through a one-time Admin Dashboard wizard: organization details, SSO, providers, and models. Complete Steps 1 to 4, then review Step 5 (AI Gateway), which requires no configuration because Tetrate has already provisioned the gateway, and click Complete setup. Data-plane credentials are never created, downloaded, or regenerated during Fully Managed onboarding. ## The onboarding wizard The wizard launches automatically on the first sign-in to the Admin Dashboard; there is no separate onboarding URL. Each step has **Next** and **Previous** controls. **Next** is enabled once the current step's required fields are filled; **Previous** returns to any earlier step to review or change a value. Step 1 Organization Name Identify the tenant. Step 2 Corporate Login Configure SSO over OpenID Connect (OIDC). For evaluations without corporate IdP admin access, use Auth0 as a temporary IdP. Step 3 LLM Providers Connect providers such as OpenAI and Anthropic. Step 4 LLM Models Choose which models are exposed. Step 5 Review AI gateway No action needed. This step summarizes the onboarding status. Step 6 Setup Complete The Admin Dashboard can now be accessed by signing in with SSO. :::important Step 5 is a status review, not a form On Fully Managed, Step 5 is labeled **Review AI Gateway**, not **Configure AI Gateway**. It reports whether Tetrate's gateway is ready and asks for nothing: there is no credential to create, regenerate, or download. That credential workflow only appears on [Self-Hosted Data Plane](/agent-router-enterprise/self-hosted-data-plane/onboarding/) installs. **Next** is enabled regardless of what the status shows, so **Complete setup** is always reachable. ::: Plan for 15 to 30 minutes, plus any back-and-forth with the identity provider (IdP) team during Step 2. ## Prerequisites Have the following on hand before launching the wizard. Step 2 is hard to complete without the IdP details: - Access to the Admin Dashboard. Tetrate provides the URL during provisioning. - Initial admin credentials for the first sign-in. SSO is configured in the wizard; the first login uses these credentials. - Your API credentials for at least one LLM provider, for example an OpenAI API key. **Important: make sure you have at least one (1) API key available during onboarding.** - OIDC IdP details: issuer URL, client ID, client secret, scopes. Agent Router supports **OIDC only**; SAML and other protocols are not supported. For an evaluation without IdP admin access, use the [Set up Auth0 for evaluation](/product-architecture/product-evaluation/auth0-for-evaluation/) path instead. - An IdP administrator reachable during Step 2. The wizard shows a redirect URI that must be registered on the IdP side before SSO works. - The **Gateway URL** Tetrate assigned for the deployment, listed in the provisioning email, for example `https://proxy.example.com`. Nothing is entered with it in the wizard; the same value appears automatically as the default project's gateway **endpoint** once setup completes. ## Get started The wizard appears automatically when the Admin Dashboard URL is first visited. Click **Start setup** to begin. ### Step 1: configure organization name Fill in: | Field | Required | Description | Sample value | | :---- | :---- | :---- | :---- | | Organization Name | Yes | The organization's display name | `Acme Corp.` | | Data plane URL | No | The public URL where the gateway (proxy) is reachable. It is provided by Tetrate. | `https://proxy.tare-pn-12345678.tetrate.ai` | :::info About the data plane URL Fully Managed does not ask for a data-plane hostname here. Tetrate already operates the data plane and its gateway, so there is nothing to register. Confirm the gateway's endpoint after the first SSO sign-in (see [Before the first request](#before-the-first-request) below). ::: ### Step 2: configure corporate login (single sign-on) SSO lets users sign in with the organization's existing identity provider. :::tip Evaluating without corporate IdP admin access? Use Auth0 as a temporary IdP rather than the corporate IdP. The [Set up Auth0 for evaluation](/product-architecture/product-evaluation/auth0-for-evaluation/) guide is the full spine: create the tenant, copy the Redirect URI from this step, register the Auth0 app, and paste the credentials back into this form. ::: :::warning Agent Router supports **OIDC only**. SAML, LDAP, and other protocols are not supported. ::: At Step 2, the wizard displays the **OIDC redirect URI** for the deployment, following the pattern: ```text https://dashboard./login ``` Register Agent Router as an OIDC application in the IdP using that exact URI as an authorized redirect URI. Once the IdP-side application is registered and the issuer URL, client ID, and client secret are in hand, fill in: | Field | Required | Description | Sample value | | :---- | :---- | :---- | :---- | | Issuer (IdP URL) | Yes | OIDC issuer URL exposed by the IdP | `https://login.example.com/` | | Client ID | Yes | OAuth client ID created for Agent Router | `agent-router` | | Client Secret | Yes | OAuth client secret paired with the Client ID | `••••••••` | | Discovery URL | No | OIDC discovery document URL. If omitted, Agent Router constructs it as `/.well-known/openid-configuration`. Provide it only if the IdP serves discovery at a non-standard path. | `https://login.example.com/.well-known/openid-configuration` | | Scopes | Yes | OAuth scopes to request | `openid profile email` | | Admin email | Optional | Set one or more emails as admin. Emails set here are added as default project members. If skipped, add members later from the Admin Dashboard (default project → **Members**). | `admin@example.com` | | Attribute mapping | Yes | Maps an IdP claim to the Agent Router user's email. The typical value is `email`. | `email` | See the [SSO configuration guide](/agent-router-enterprise/guides/identity-and-access-management/single-sign-on/sso-configuration-guide/) for details. :::info Recovery If SSO is misconfigured and blocks sign-in, sign back in with the initial admin credentials and reset the SSO configuration from the Admin Dashboard settings panel. ::: ### Step 3: configure LLM provider keys Connect Agent Router to one or more customer-managed LLM providers. At least one **enabled** provider is required to advance to Step 4. The wizard shows a list of supported providers, and the set varies by deployment; see [Provisioning models and providers](/agent-router-enterprise/guides/operate-and-govern/provision-models-and-providers/) for the authoritative list. Click **Enable** next to a provider to open its form, then fill in: | Field | Required | Description | Sample value | | :---- | :---- | :---- | :---- | | Display Name | No | Optional label shown in tables and filters | `OpenAI` | | API Base URL | Yes | Base URL for the provider API. Pre-filled for supported providers. | `https://api.openai.com/v1` | | Auth method | Yes | Authentication method. Pre-filled per provider. | `API Key` | | Secret Type | Yes | `Password` for single-string API keys; `Service Account JSON` for providers that authenticate with a JSON credential file. | `Password` | | Provider Secret | Yes | API key, token, or service-account JSON. Use **Rotate** in the UI to replace a stored secret without clearing the rest of the form. | `sk-proj-...` | | Enable provider | Yes | **Off by default.** Switch it on before clicking **Save**. | toggle | :::warning Enable provider is off by default Clicking **Enable** in the provider list only opens the form; it does not itself enable the provider. Inside the form, the **Enable provider** toggle starts off. Filling in the secret and clicking **Save** without switching it on saves the provider but leaves it hidden and disabled, and Step 3's "at least one provider" requirement is still not met. ::: Repeat for each provider to expose. The wizard returns to the provider selection screen after each one is saved. ### Step 4: configure LLM models The wizard shows a single list of all models Agent Router supports, maintained by Tetrate. By default, a selective set of common models is enabled. Deselect models to control cost, meet data-handling requirements, or scope to the intended use case. Models remain configurable later from the Admin Dashboard. If a required model is missing, contact Tetrate support. :::note Custom model entries are not configurable in onboarding. ::: ### Step 5: review AI Gateway On Self-Hosted Data Plane, Step 5 is **Configure AI Gateway** and downloads a `service-account.json` credential used to install a customer-operated data plane. **Fully Managed shows a different step: Review AI Gateway.** There is no credential form, because Tetrate already provisioned the gateway, so the step only reports its status. The first column quotes the wizard's own wording: | Status shown | Meaning | What to do | | :---- | :---- | :---- | | Checking your gateway status… | The wizard is querying the gateway for the first time. | Wait a moment. | | Your AI gateway is provisioned and ready. | Managed by Tetrate, no action needed. | Click **Next**. | | Your AI gateway is still being provisioned. | Normal shortly after the tenant is created. The page checks again automatically. | Click **Next** and continue; check back later if needed. | | Your AI gateway isn't ready. | Something is wrong on Tetrate's side. | Click **Next** to continue onboarding, and contact a Tetrate representative. | | We couldn't check your AI gateway status. | The status check itself failed, likely transient. | Click **Next** to continue onboarding; contact a Tetrate representative if it persists. | **Next** is enabled for every status above, so Step 5 never blocks onboarding on Fully Managed. If the gateway is still provisioning at this step, finish onboarding and check its status again later from the Admin Dashboard's project **Gateway** tab (see [Before the first request](#before-the-first-request) below). ### Complete the process After Steps 1 to 4, and reviewing Step 5, click **Complete setup** to finish onboarding. The Admin Dashboard can now be accessed by signing in with SSO. ## Before the first request Onboarding sets up the default project automatically: by the time **Complete setup** finishes, the gateway endpoint, the providers connected in Step 3, and the models selected in Step 4 are already in place. Project membership comes from Step 2's Admin email field; add members from the Admin Dashboard (default project → **Members**) if that was left blank. Once project access is in place, continue in the Dev Console: - In the Dev Console, create an **Inference** key for gateway requests. Do not use an Admin Dashboard organization/API key for inference traffic. - The **Base URL** shown alongside the key is the project's gateway endpoint, already set, with nothing to configure. If something looks wrong, for example an expected provider is not listed, the default project's **Gateway**, **Members**, **Providers**, and project-scoped **Models** pages in the Admin Dashboard show its current state. ## Where to go next Admin Dashboard quickstart Review models, providers, users, and usage before developer handoff. Console quickstart Issue an API key and make a first routed request. --- ## Create and manage teams Enterprise Tier A team is a named group of users in the Tetrate Agent Router directory. Teams exist so that access can be granted once to a group instead of person by person, and so that cost and usage can be attributed to a group rather than only to individuals. A team is created with the New Team wizard on Directory → Teams, which collects a name, an optional description, and the initial members. Membership is exclusive: a user belongs to one team at a time, and selecting a user who already belongs to another team moves them. **Persona:** Platform operator working in the Admin Dashboard, holding a role that can manage users and teams. **Estimated time:** 5 minutes to create a team; recurring thereafter as people join, move, and leave. ## What a team is used for A team carries no permissions of its own. Its value comes from the places that accept a team where they would otherwise accept a single user. | Purpose | Where it applies | | :---- | :---- | | Grant a role to a group once rather than per person | **Directory → Access**, described in [Manage roles and permissions](/agent-router-enterprise/guides/identity-and-access-management/manage-roles-and-permissions/) | | Add a group of people to a project in one step | The Members step of the Create Project wizard, described in [Create and delete a project](/agent-router-enterprise/guides/operate-and-govern/project-and-data-plane-management/create-a-project/) | | Attribute cost and usage to a group | The Team filter in Analytics, described in [Compare user and team spend in Analytics](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/compare-user-and-team-spend-in-analytics/) | | Identify which group a person belongs to | The team is shown in user listings | ## Outcomes By the end of this guide: - A team exists with a name and, optionally, a description. - The initial members have been selected, and any move out of a previous team has been seen before it was applied. - The effect of exclusive membership on inherited access is understood. - The team is ready to be used as the target of a role grant, a project membership, or an Analytics filter. ## Prerequisites - Access to the Admin Dashboard (`https://dashboard..tetrate.ai`) with a role that can manage users and teams, such as **Super Admin** or **User Admin**. - The intended members already exist as users. A user record is created on first sign-in, as described in [Onboard developers and issue keys](/agent-router-enterprise/guides/operate-and-govern/onboard-developers-and-issue-keys/). ## In this guide - **Tasks:** create a team and populate it - [Step 1: open the New Team wizard](#step-1-open-the-new-team-wizard) - [Step 2: name the team](#step-2-name-the-team) - [Step 3: select the members](#step-3-select-the-members) - [Step 4: review and create](#step-4-review-and-create) - **Concepts** - [A user belongs to one team at a time](#a-user-belongs-to-one-team-at-a-time) - [Putting the team to use](#putting-the-team-to-use) - **Reference** - [Operational notes](#operational-notes) ## Step 1: open the New Team wizard 1. In the Admin Dashboard, go to **Directory → Teams**. 1. Select **Add team**. ![Teams page before any team exists, showing the Add team button](/img/screenshots/admin/teams-empty.jpeg) Where no team exists yet, the page shows a **Create your first team** placeholder carrying the same **Add team** control. The wizard opens in a panel beside the Teams page and runs in three steps: **Identity**, **Members**, and **Review**. The step counter and the **Continue** control sit in the panel footer, and each completed step is marked along the top. **Cancel** on the first step and **Back** on later steps leave or reverse the flow. Once anything has been entered the footer warns that closing the panel discards the progress, so the wizard is completed in one pass. ## Step 2: name the team 1. Enter a **Name**. The name identifies the team in user listings and in Analytics, so a name matching the organizational unit that owns the spend, such as `Platform Engineering`, carries more meaning at reporting time than an internal abbreviation. 1. Optionally enter a **Description** recording what the team is for. It is shown to administrators reviewing the team later and has no functional effect. 1. Select **Continue**. ![Identity step of the New Team wizard with the team name entered](/img/screenshots/admin/create-team-wizard-identity.jpeg) The name is the only required value in the wizard: **Continue** stays disabled until it is filled. The panel header previews the name as it is typed, which is the quickest check that the value is the one intended. ## Step 3: select the members 1. Select the users who should belong to the team. Each row carries a checkbox, the person's name, and their email address, and the counter above the list reports how many are selected. 1. For a long directory, narrow the list using **Search users by name or email**, or jump to a surname using the alphabetical index down the right edge. 1. Select **Continue**. ![Members step of the New Team wizard with one user selected](/img/screenshots/admin/create-team-wizard-members.jpeg) This step is optional. A team can be created empty and populated afterwards, which is the sensible order where the team is being created ahead of the people who will join it. Nothing is applied to the selected users at this point. Membership is written only when the wizard is submitted in the next step, so a selection made here can still be reversed without consequence. ## Step 4: review and create 1. Check the summary. It restates the name and description, the number of **members** the team will hold, and the number of **moves**, meaning how many of those users are being taken out of another team. 1. Remove anyone selected in error using the **×** beside their name under **Initial members**. 1. Select **Create team**. ![Review step of the New Team wizard showing the members and moves counts](/img/screenshots/admin/create-team-wizard-review.jpeg) The moves count is the figure worth reading closely. A move is not additive: it removes the person from the team they were in, along with any access that team conferred. Where the count is higher than expected, the cause is usually a name selected from the directory without noticing an existing membership, and the **×** control resolves it before submission rather than after. ## A user belongs to one team at a time Membership is recorded on the user as a tag carrying the team's identifier, and a user carries one such tag. Three consequences follow: - Selecting a user who already belongs to another team **moves** them into the new team rather than giving them a second membership. The Review step counts these separately so that the reassignment is visible before it is applied. - Reassignment is therefore performed by selecting the person into the destination team. Removing them from their current team first is neither necessary nor sufficient. - Access inherited from the previous team ends at the same moment. Where a role had been granted to that team, the person loses it on the move, and where a role has been granted to the destination team, they gain it. The rules governing inherited access are in [Manage roles and permissions](/agent-router-enterprise/guides/identity-and-access-management/manage-roles-and-permissions/). Because membership is exclusive, a team is best modelled on the unit a person genuinely belongs to, typically the one that owns their budget, rather than on a working group they participate in temporarily. Cross-cutting access is expressed with a role granted directly to the person, which coexists with their team membership. ## Putting the team to use A team on its own changes nothing. It takes effect when it is named somewhere: - **To give the whole team the same Admin Dashboard access**, grant a role to the team on **Directory → Access**. Adding someone to the team then grants that access, and moving them out withdraws it. See [Manage roles and permissions](/agent-router-enterprise/guides/identity-and-access-management/manage-roles-and-permissions/). - **To give the team access to a project**, select the team in the Members step of the Create Project wizard. The team's members at that moment are added individually, so people who join the team later are not added to the project retrospectively. See [Create and delete a project](/agent-router-enterprise/guides/operate-and-govern/project-and-data-plane-management/create-a-project/). - **To report on the team's spend**, apply the Team filter in Analytics. See [Compare user and team spend in Analytics](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/compare-user-and-team-spend-in-analytics/) and, for chargeback, [Bill AI spend back to teams](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/bill-ai-spend-back-to-teams/). ## Operational notes - The name is shown in user listings and in Analytics, so renaming a team changes how historical usage is labelled in reports. Agreeing the naming scheme with whoever consumes the cost reports before the first team is created avoids relabelling later. - A team confers nothing until it has members, and a role granted to an empty team has no effect at all. - Administrative changes are recorded and can be reviewed under **Audit Logs**, as described in [Audit Agent Router activity](/agent-router-enterprise/guides/operate-and-govern/audit-platform-activity/). - Teams group users for access and reporting. They are not an isolation boundary: the boundary for models, keys, MCP servers, and policy is the [project](/product-architecture/key-concepts/#projects). Where to go next Manage roles and permissions Grant a role to the team once, at organization or project scope, instead of granting it to each member. Compare user and team spend in Analytics Use the Team filter to compare groups side by side and attribute cost to the unit that owns it. --- ## Manage roles and permissions Enterprise Tier Access to the Tetrate Agent Router Admin Dashboard is governed by roles. A role is a named set of permissions, and the role granted to a user or a team determines which menus, pages, and actions that person is shown. A role applies either across the whole organization or inside a single project, so a delegated operator can be given exactly one project to administer and nothing beyond it. **Persona:** Platform operator working in the Admin Dashboard, holding a role that can manage access. **Estimated time:** 10 to 20 minutes to define a role and grant it; recurring thereafter as people join, move, and leave. ## When this guide applies | Situation | What it covers | | :---- | :---- | | A new administrator needs a subset of Admin Dashboard capabilities | Selecting a built-in role, or defining a custom one | | An operator should administer one project and nothing else | Granting a project-scoped role | | Several people all need the same access | Granting a role to a team once instead of per person | | Access has become broader or narrower than intended | Reviewing, editing, and revoking grants on the Access page | | A granted role must be confirmed to restrict as intended | Verifying the result from the recipient's own session | | Someone has changed job, or access was granted too broadly | Editing a grant in place, or revoking it | A single worked example runs through the tasks below: a custom **Guardrails Admin** role is created, granted to a user across the whole organization, and then revoked. ## Outcomes By the end of this guide: - The difference between read and write permissions is clear, along with how each is reflected in the Admin Dashboard. - A built-in role has been selected, or a custom role has been created. - That role has been granted to a user and to a team, at organization or project scope. - The combined effect of overlapping grants is understood. - The restriction has been verified from the recipient's session. - A grant has been edited in place, and one has been revoked. ## Prerequisites - Access to the Admin Dashboard (`https://dashboard..tetrate.ai`) with a role that can manage access, such as **Super Admin**. - The users involved already exist in Agent Router. A user record is created on first sign-in, as described in [Onboard developers and issue keys](/agent-router-enterprise/guides/operate-and-govern/onboard-developers-and-issue-keys/). - For team-based grants, a team under **Directory → Teams** with the intended members. The two pages used throughout this guide sit in the sidebar under **Directory**: **Roles** and **Access**. ## In this guide - **Concepts:** how a role decides what a person is shown - [How access is decided](#how-access-is-decided) - [Scope: organization or project](#scope-organization-or-project) - [Built-in roles](#built-in-roles) - **Tasks:** define access, grant it, and take it away - [Step 1: create a custom role](#step-1-create-a-custom-role) - [Step 2: grant a role to a user](#step-2-grant-a-role-to-a-user) - [Step 3: grant a role to a team](#step-3-grant-a-role-to-a-team) - [Step 4: edit or revoke a grant](#step-4-edit-or-revoke-a-grant) - [Step 5: verify the effect](#step-5-verify-the-effect) - **Reference** - [Operational notes](#operational-notes) ## How access is decided Every menu and page in the Admin Dashboard is tied to a permission. A menu is shown only when the role includes its **read** permission, and the page itself rejects anyone who arrives without that permission. Enforcement happens in two places at once. Hiding a menu is therefore a genuine restriction, not a cosmetic one. Permissions come in two kinds: - **Read:** view a resource, for example view models, view users, or view audit logs. Read permissions determine which menus are visible. - **Write:** create, edit, or delete a resource. Write permissions determine whether the **New**, **Edit**, and **Delete** buttons appear and whether the action is accepted. ```text role permissions: { models.read, providers.read } ↓ sidebar shows: Models, Providers (their read permissions) buttons shown: none (no write permissions) other pages: redirect to "unauthorized" ``` A role carrying read permissions only is view-only: every change button is hidden, and the server refuses the change even when it is submitted directly. ## Scope: organization or project Scope appears in two distinct places, and they answer different questions. | Where | What it decides | | :---- | :---- | | **Scope levels**, on the role | Where the role is *allowed* to be granted. A role can permit one level or both, so a role marked **Organization & Project** can be used for either kind of grant. | | **Apply to**, on the grant | Where this particular grant *actually* applies: the whole organization, or named projects. | The two levels cover the following. | Scope | What it covers | | :---- | :---- | | **Organization** | The whole organization, including organization-wide resources such as users, teams, API keys, billing, and the shared model, provider, and Model Context Protocol (MCP) catalog. | | **Project** | A single project only: the models, providers, MCP servers, and members of that project, and nothing outside it. | A user can hold an organization role, one or more project roles, or a combination of the two. Someone holding a project role alone is shown only that project's pages; the organization-wide administrative pages (Users, Roles, Settings, and Audit Logs) remain closed. ## Built-in roles The following roles are predefined. Each can be granted as-is, or copied as the starting point for a custom role. The permission counts are those shipped with the current release and are shown in the **Permissions** column on the Roles page. ### Organization roles | Role | Slug | Permissions | What it is for | | :---- | :---- | :---- | :---- | | Super Admin | `super_admin` | 163 | Full access to every resource and setting. | | Viewer | `read_only` | 37 | Read-only access to every resource: all menus are visible, nothing can be changed. | | Billing Admin | `billing_admin` | 14 | Manage billing, service accounts, and bonus claims. | | User Admin | `user_admin` | 12 | Manage users, teams, and API keys. | | Model Admin | `model_admin` | 8 | Manage models and providers. | | Provider Admin | `provider_admin` | 4 | Manage providers. | | MCP Admin | `mcp_admin` | 4 | Manage MCP servers. | ### Project roles | Role | Slug | Permissions | What it is for | | :---- | :---- | :---- | :---- | | Project Owner | `project_owner` | 18 | Full administration within the project, covering models, providers, MCP, members, settings, and the gateway. | | Project Member | `project_member` | 2 | Work within the project without managing its members or settings. | Note that the Viewer role's slug is `read_only`, which is the value to use where a role is referenced by slug rather than by display name, as in identity-provider claim mapping. ## Step 1: create a custom role :::info Optional step The built-in roles cover most deployments. This step applies only where no built-in role matches the intended access, for example a role confined to guardrails, or one shown the Models and Providers menus and nothing else. ::: The Roles page lists every role with its slug, scope, permission count, and type, where **Built-in** marks a shipped role and **Custom** marks one defined locally. The list can be narrowed with the name and slug filter, restricted by scope, and built-in roles can be hidden altogether to leave only the custom ones in view. 1. In the Admin Dashboard, go to **Directory → Roles**. 1. Select **New role**, or open an existing role and select **Copy** to start from its permission set. 1. Enter the **Slug**. It must be lower case with words joined by underscores, for example `guardrails_admin`. The slug is the role's stable identifier. 1. Enter the **Display name**, which is the label shown everywhere the role is offered, for example `Guardrails Admin`. 1. Optionally enter a **Description**. 1. Under **Scope levels**, select **Organization**, **Project**, or both. This sets where the role is permitted to be granted, not where it is granted. Levels required by the permissions selected below are added automatically. 1. Select the permissions the role carries. Include a resource's **read** permission to make its menu visible; add its **create**, **edit**, and **delete** permissions to allow changes. 1. Select **Create role**. ![New role panel with slug, display name, scope levels, and the guardrails permission group selected](/img/screenshots/admin/roles-new-role-panel.jpeg) Permissions are grouped by resource, and each group carries a `selected/total` count, so `guardrails 4/4` means all four of that resource's permissions are held. Selecting a group's checkbox takes every permission in it, expanding the group with the chevron selects them individually, **Filter permissions** narrows a long list, and **All** takes everything at once. In the example, three groups are taken in full (`guardrails`, `guardrail_patterns`, and `guardrail_providers`), giving a role of 10 permissions that can administer guardrails and nothing else. On submission the role appears in the list marked **Custom**, with the scope shown as **Organization & Project** where both levels were selected. ![Roles page listing the new Guardrails Admin role as a custom role](/img/screenshots/admin/roles-list-custom-role.jpeg) ### Editing, copying, and deleting a role Selecting a role opens it for editing. The slug and scope are shown for reference, the permission tree is editable, and a running total is reported beneath it (`10 permission(s) selected`). Three actions sit in the panel footer: **Copy** creates a new role starting from this permission set, **Save** applies the changes, and **Delete** removes the role. ![Role panel showing the permission summary with Delete, Copy, and Save actions](/img/screenshots/admin/roles-edit-panel.jpeg) Built-in roles are editable in the same way. Copying one and editing the copy is the safer pattern, because the original stays intact for comparison and reuse. ### Example: a role limited to the catalog menus An organization role named **Gateway Viewer** that carries only the read permissions for Models and Providers restricts its holder to the catalog. The sidebar shows Home, the Catalog group, Models, and Providers, and nothing else; every other address redirects to the unauthorized page. Adding the Models and Providers edit permissions extends the same role to changing those two resources, while the rest of the Admin Dashboard stays closed. ## Step 2: grant a role to a user The Access page is the register of who holds what. Each row is a **principal**, a user or a group, and each grant that principal holds appears as a chip reading the role followed by the scope it applies at, such as `Guardrails Admin · organization` or `Project Owner · project: default`. Grants can be filtered by principal, role, or project. ![Access page listing each principal with their grant chips](/img/screenshots/admin/access-list.jpeg) 1. Go to **Directory → Access**. 1. Select **Grant access**. 1. Leave **Principal type** set to **User**. 1. Under **Principal**, type a name or email and select the person from the results. 1. Leave **Grant** set to **A role**. 1. Under **Roles**, select one or more roles. Each is listed with its permitted scope on the right, and built-in roles are marked as such. **Filter roles** narrows the list. 1. Under **Apply to**, choose **Whole organization** or **Specific projects**. 1. Select **Grant access**. ![Grant access panel with a role selected and the Apply to options open](/img/screenshots/admin/access-grant-panel.jpeg) Three things about this panel are worth knowing before using it. - **Any role can be granted at either scope.** The role's scope levels say where a grant is permitted; **Apply to** decides where this grant lands. A role marked **Organization & Project** can therefore be granted org-wide to one person and scoped to a single project for another. - **A principal can hold several roles at once**, and several can be selected in a single pass. The result is additive, as described under [When a person has access from more than one place](#when-a-person-has-access-from-more-than-one-place). - **Only access the operator already holds can be granted.** A role carrying permissions beyond the operator's own cannot be handed out, which prevents privilege escalation through the grant form. Setting **Grant** to **Direct permissions** instead of **A role** assigns individual permissions to the principal without going through a role. This is worth avoiding as a habit: a direct grant is invisible to anyone reviewing roles and has to be found on the Access page, whereas a role is reusable and self-documenting. A confirmation step restates who is about to receive the access and at what scope. ![Confirmation dialog naming the user and the organization-wide scope](/img/screenshots/admin/access-grant-confirm.jpeg) Once confirmed, the new chip appears against that principal on the Access page. ![Access page showing the new Guardrails Admin grant chip](/img/screenshots/admin/access-granted.jpeg) The grant takes effect on the recipient's next page load; no sign-out is required. ## Step 3: grant a role to a team Granting a role to a **team** once, rather than to each member separately, is the lower-maintenance option for a group of people who need the same access: adding someone to the team grants it, and removing them withdraws it. 1. Confirm the team exists and holds the intended members. Teams are managed under **Directory → Teams**, as described in [Create and manage teams](/agent-router-enterprise/guides/identity-and-access-management/create-and-manage-teams/). 1. Go to **Directory → Access** and select **Grant access**. 1. Change **Principal type** from **User**, then select the team under **Principal**. 1. Select the role or roles, and set **Apply to**. 1. Select **Grant access**, then confirm. The panel is the same one used in Step 2, and everything in it behaves identically. Only the principal differs, which is why the Access page describes its rows as assignments "to users and groups" and labels each row with a **Type**. Team grants behave as follows: - A role granted to a team confers nothing until someone is a member of that team. - A member inherits exactly the team's role and no more. A team holding a view-only role gives its members view-only access. - The grant is live: a person added to the team gains the access on their next page load, and a person removed loses it immediately. ### When a person has access from more than one place A person can hold roles granted to them directly and also inherit roles granted to a team they belong to. The resulting access is the sum of all of them: every permission from every role held directly, plus every permission inherited through team membership. Grants only ever add access; there is no deny grant that cancels another. ```mermaid flowchart LR A["Direct grant:Model Admin"] --> U["Effective permissions:union of all grants"] B["Team grant:Viewer"] --> U U --> V["Visible menus andenabled actions"] ``` For example, a person holding a direct role that permits editing models, who also belongs to a team holding a view-only role, can edit models (from the direct role) and view everything else (from the team). Removing them from the team leaves only the model editing the direct role provides; removing the direct role leaves only the view-only access inherited from the team. ## Step 4: edit or revoke a grant Access is not set once. People change job, a grant turns out to be broader than intended, or a role was scoped org-wide when a single project would have done. Both corrections start from the grant chip on the Access page. 1. Go to **Directory → Access**. 1. Select the grant chip to be changed. It opens for editing. 1. To adjust it, change the **Role**, the type of **Grant**, or the **Apply to** scope, then select **Save changes**. 1. To remove it, select **Revoke**, then confirm. ![Grant open for editing, with the Revoke action in the footer](/img/screenshots/admin/access-edit-revoke.jpeg) The principal cannot be changed here. Moving access from one person to another is done by revoking the grant and creating a new one, which keeps the audit trail honest about who held what and when. Revoking asks for confirmation and names what will be lost. ![Confirmation dialog stating that the user will lose this grant](/img/screenshots/admin/access-revoke-confirm.jpeg) The chip then disappears from that principal's row, which is the confirmation that the access is gone. ![Access page after revocation, with the grant chip removed](/img/screenshots/admin/access-revoked.jpeg) Two points to note before revoking: - **Revocation is immediate**, unlike granting, which lands on the recipient's next page load. Anyone relying on that access loses it at once, so a revocation made during someone's working day is felt during it. - **Removing one grant does not remove the rest.** Where a person holds several grants, or inherits access through a team, revoking a single chip only withdraws what that chip represents. The remaining chips on the row are what still applies, which is why the row, rather than the individual grant, is the thing to read when the question is what someone can still do. A padlock appears on the operator's own **Super Admin** grant in the screenshots above. Grants marked this way are protected from removal on this page. ## Step 5: verify the effect Verification is carried out from the recipient's session rather than the administrator's, because a broader administrative role masks the restriction under test. Once the recipient has reloaded the Admin Dashboard, confirm that: - The **sidebar** lists only the menus the read permissions allow. - Action buttons (**New**, **Edit**, **Delete**, and **Grant access**) appear only where the matching write permission is held. - Entering the address of a page the role does not cover redirects to the unauthorized page. If a page that should be closed is still reachable, the likely cause is a second grant. The Access page lists every grant held by that person, including access inherited through team membership. ## Operational notes - Built-in roles can be edited, but copying a built-in role and editing the copy is the safer pattern: the originals stay intact for comparison and reuse. - A role's **slug** is its stable identifier and is the value used where a role is named rather than picked from a list. The display name can be changed freely; treat the slug as fixed once anything references it. - Deleting a role is offered from the role's own panel. Where the role is still granted to someone, confirm on the Access page what would be withdrawn before deleting it, since the grant chips are the only place that relationship is visible. - Prefer a role over **Direct permissions**. Direct grants work, but they are discoverable only by inspecting each principal on the Access page, so a permission audit that reviews roles alone will miss them. - Role and grant changes are recorded. Every change writes an entry that can be reviewed under **Audit Logs**, as described in [Audit Agent Router activity](/agent-router-enterprise/guides/operate-and-govern/audit-platform-activity/). - Developers working in the Console rather than the Admin Dashboard are not governed by these dashboard roles. Their access to a gateway follows [project](/product-architecture/key-concepts/#projects) membership. - Where roles are assigned from identity-provider claims, the mapping is evaluated on every login and overwrites the stored role. The precedence rules are documented in [Configure SSO role mapping](/agent-router-enterprise/guides/identity-and-access-management/single-sign-on/sso-role-mapping-guide/). Where to go next Configure SSO role mapping Drive role assignment from identity-provider claims so directory changes propagate on each login. Onboard developers and issue keys Confirm user registration, review roles, and manage API keys across the organization. --- ## Map Entra groups to functions Enterprise Tier An organisation that has already invested in an identity provider has, almost by definition, already decided who belongs to which team, department, and cost centre. That structure lives in directory groups (eng-platform, data-science-emea, finance-analytics, and the rest), maintained continuously by the identity team as people join, move, and leave. This guide describes how Microsoft Entra ID (formerly Azure Active Directory) groups are surfaced to Agent Router Enterprise (Agent Router) and mapped to Agent Router roles and business functions, so that a developer's membership in an Entra ID group is what decides their role, the rate limits they are subject to, and the budget their consumption draws down. When the directory changes, Agent Router changes with it on the next sign-in, without an operator touching the Admin Dashboard. This guide builds directly on the single sign-on (SSO) role-mapping material and assumes that material has been read first. **Persona:** Platform operator working in the Admin Dashboard, in close coordination with the identity team that owns the Microsoft Entra ID tenant. **Estimated time:** 30 to 60 minutes, most of which is spent coordinating the group-claim configuration with the identity team rather than working in the Admin Dashboard. ## When this guide applies This guide is the right starting point in any of these situations: | Situation | What it covers | | :---- | :---- | | Access decisions should follow existing Entra ID groups rather than manual role edits | Emitting group claims and mapping them to roles | | Rate limits and budgets should be scoped to teams or departments | Driving group-scoped limits from directory membership | | Manual role assignment has become unmanageable as the user base has grown | Replacing per-user edits with claim-driven mapping | | A reorganisation has changed team membership in the directory | How membership changes propagate on the next sign-in | This guide is a continuation of the SSO setup. SSO must already be configured against the identity provider, and a sign-in verified, before group claims can be read; see [Configure SSO](/agent-router-enterprise/guides/identity-and-access-management/single-sign-on/sso-configuration-guide/). The mechanics of translating a claim into an Agent Router role are documented in the [SSO role mapping guide](/agent-router-enterprise/guides/identity-and-access-management/single-sign-on/sso-role-mapping-guide/); the two should be read together, and this guide assumes familiarity with both. ## Outcomes By the end of this guide: - Microsoft Entra ID has been configured to emit a group claim in the SSO assertion or token. - The group claim is read by Agent Router and its values are confirmed. - Each relevant Entra ID group is mapped to a Agent Router role, a business function, or both. - Group membership is used to scope rate limits and budgets, so that consumption is governed at the team or department level rather than per user. - The behaviour on each sign-in, and on a change to a user's group membership, is understood. ## Prerequisites - Administrator access to the Admin Dashboard, typically the `super_admin` role. - SSO already configured against the Microsoft Entra ID tenant, with a sign-in verified. See [Configure SSO](/agent-router-enterprise/guides/identity-and-access-management/single-sign-on/sso-configuration-guide/). - Familiarity with claim-driven role assignment as described in the [SSO role mapping guide](/agent-router-enterprise/guides/identity-and-access-management/single-sign-on/sso-role-mapping-guide/). - Cooperation from the identity team that owns the Entra ID tenant. The group-claim configuration is performed on the Entra ID side, and the set of groups and their identifiers comes from that team. - A decision, made jointly with the identity team, about which directory groups correspond to the business functions Agent Router needs to recognise. Mapping every group in a large tenant is rarely useful; mapping the handful that correspond to real access and budgeting boundaries is. ## Step 1: decide which groups represent business functions Before any configuration is touched, the set of groups that matter has to be agreed. A large Entra ID tenant contains hundreds of groups, most of which have nothing to do with Agent Router access. The groups worth mapping are the ones that correspond to a real boundary: a team that should hold a particular role, a department whose spend is tracked separately, or a cost centre that owns a budget. A working approach: 1. List the business functions Agent Router needs to distinguish: for example, platform engineering, data science, and a finance analytics team. 1. For each, identify the existing Entra ID group that already represents it. The identity team owns this mapping and can confirm which group is authoritative when more than one looks plausible. 1. Record the group identifier that will appear in the claim. Depending on how the claim is configured, this may be the group's display name or its object ID (a GUID). The object ID is stable across renames and is the safer choice where Agent Router can match on it. 1. Note, for each group, the intended Agent Router role and the business function it represents. This table becomes the mapping configured in later steps. Keeping the mapped set small and deliberate matters here. Every mapped group is a rule that has to be maintained; unmapped groups have no effect in Agent Router. ## Step 2: configure Microsoft Entra ID to emit a group claim Entra ID does not include group information in the SSO assertion or token by default. The identity team configures the application registration, the one created when SSO was set up, to add a group claim. This is performed entirely on the Entra ID side; Agent Router only consumes the result. The configuration on the Entra ID application registration determines: 1. That a groups claim is emitted at all. 1. Which groups are included: typically all groups assigned to the application, or all security groups, depending on tenant policy. Restricting the claim to groups assigned to the application keeps the claim small and avoids the token-size limits that large tenants run into when every group is emitted. 1. What value identifies each group in the claim: the group object ID (a GUID) or, where the directory is configured to emit them, the group display names. The object ID is recommended because it does not change when a group is renamed. For a token that emits group object IDs, the relevant portion of the decoded claim set looks roughly like this: ```json { "groups": [ "8f4e1c20-1d3a-4b9e-9b2a-3c7d5e6f1a90", "b1c2d3e4-5f60-4a1b-8c9d-0e1f2a3b4c5d" ] } ``` For a deployment configured to emit display names instead, the same claim carries readable values: ```json { "groups": [ "eng-platform", "finance-analytics" ] } ``` The exact claim name and shape depend on how the identity team configures the application registration. The name of the claim is one of the values that has to be confirmed jointly; Agent Router needs to know which claim to read, and that is settled in the next step. Large tenants should also confirm the identity team's handling of the group overage case: when a user belongs to more groups than the token can carry, Entra ID emits a reference to be resolved against the directory rather than the group list itself, and the deployment has to account for that behaviour. ## Step 3: confirm Agent Router is reading the group claim With Entra ID emitting the claim, Agent Router has to be pointed at it. The SSO role-mapping configuration in the Admin Dashboard is where the claim is named and its values are inspected; the full mechanics are in the [SSO role mapping guide](/agent-router-enterprise/guides/identity-and-access-management/single-sign-on/sso-role-mapping-guide/), and only the group-specific concerns are repeated here. 1. Sign in to the Admin Dashboard. 1. Open the SSO role-mapping configuration. 1. Identify the claim that carries the group values: the same claim name agreed with the identity team in the previous step. 1. Sign in once as a test user who belongs to one of the mapped groups, and confirm that the expected group values appear in the token Agent Router receives. The verification steps in [Configure SSO](/agent-router-enterprise/guides/identity-and-access-management/single-sign-on/sso-configuration-guide/#step-5-verify) exercise this path. Confirming the claim values before mapping them avoids the most common failure mode: a mapping rule written against a display name when the token actually carries object IDs, or against a claim name that the token does not contain. The values seen here are the values the mapping rules must match exactly. ## Step 4: map each group to a role and a business function With the group values confirmed, each mapped group is associated with a Agent Router role, a business function, or both. The role governs what members of the group can do; the business function is the label that subsequent rate-limit and budget scoping is built on. For each group identified in Step 1: 1. Add a mapping rule that matches the group's claim value: its object ID or display name, matching exactly what Step 3 confirmed. 1. Assign Agent Router role that members of the group should hold. The role model is the same one used throughout the Admin Dashboard: for example, `user` for developers who consume Agent Router, or one of the administrative roles for an operations team. The full role list is in [Onboard developers and issue keys](/agent-router-enterprise/guides/operate-and-govern/onboard-developers-and-issue-keys/). 1. Record the business function the group represents, so that rate limits and budgets can later be scoped to it. Where a user belongs to more than one mapped group, the precedence rules in the [SSO role mapping guide](/agent-router-enterprise/guides/identity-and-access-management/single-sign-on/sso-role-mapping-guide/) decide which role wins. This case is worth designing for deliberately: a developer who is also a member of an administrative group should resolve to the role the deployment actually intends, not to whichever rule happens to be evaluated first. The result of this step is that a developer's role is no longer set by hand. It is computed from their directory membership on every sign-in, and the directory becomes the source of truth. ## Step 5: scope rate limits and budgets to the mapped groups Once group membership drives role assignment, the same membership can govern consumption. The point of mapping groups to business functions is that limits and budgets are then maintained against the organisation's structure rather than against an ever-changing list of individual users. - **Rate limits.** A rate limit scoped to a business function applies to every member of the corresponding group collectively, rather than to each developer in isolation. A data-science team with a shared throughput ceiling is governed as one unit; adding a developer to the directory group brings them under the existing ceiling automatically, with no separate limit to create. - **Budgets.** A budget scoped to a group draws down against the combined spend of its members, which is what makes department- or cost-centre-level budgeting tractable. The budget is attached to the business function once; membership changes in the directory move spend in and out of it without further configuration. The mechanics of defining and attaching budgets, including group-scoped budgets, are in [Working with budgets](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/choose-the-right-cost-control/). The advantage of scoping at the group level is the same advantage that motivates the whole guide: there is one rule per business function, not one per person. A reorganisation that moves a team between departments is handled by the identity team in the directory, and Agent Router's limits and budgets follow. ## Step 6: understand how the mapping is applied on each sign-in The mapping is not a one-time import. It is re-evaluated every time a user authenticates, which is what keeps Agent Router aligned with the directory. On each sign-in: 1. The user authenticates at Microsoft Entra ID. 1. Entra ID returns the assertion or token, including the group claim configured in Step 2. 1. Agent Router reads the group claim and evaluates it against the mapping rules from Step 4. 1. The user's role is set to the result, overwriting whatever role was previously recorded for them. 1. The business-function association is refreshed, so the rate limits and budgets the user is subject to reflect their current group membership. Because the role is recomputed on every login, a manual role edit in the Admin Dashboard does not survive the user's next sign-in for any user covered by the mapping. This is the same precedence rule documented in [Onboard developers and issue keys](/agent-router-enterprise/guides/operate-and-govern/onboard-developers-and-issue-keys/) and the [SSO role mapping guide](/agent-router-enterprise/guides/identity-and-access-management/single-sign-on/sso-role-mapping-guide/): for mapped users, the directory wins. Manual edits remain appropriate only for users no mapping rule covers. When a user's group membership changes (they join a team, move departments, or leave), the change takes effect on their next sign-in, not instantly. Adding a user to a mapped group grants the associated role and brings them under the group's rate limits and budgets the next time they authenticate. Removing them has the mirror effect: the role falls back to whatever the remaining rules resolve to, or to the default role if no mapped group applies. A membership change made in the directory while a user holds an active session does not interrupt that session; it is applied the next time the session is re-established. Deployments that need a removal to take effect immediately, an offboarding, say, should treat the directory change as necessary but not sufficient, and follow the credential-revocation steps in [Onboard developers and issue keys](/agent-router-enterprise/guides/operate-and-govern/onboard-developers-and-issue-keys/). ## What to do next - **Working with budgets.** Attach group-scoped spending limits to the business functions mapped in this guide. See [Working with budgets](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/choose-the-right-cost-control/). - **Onboard developers and issue keys.** The role model, the precedence rules, and the credential-revocation steps that complement directory-driven role assignment. See [Onboard developers and issue keys](/agent-router-enterprise/guides/operate-and-govern/onboard-developers-and-issue-keys/). - **SSO role mapping.** The underlying claim-to-role mechanism that this guide builds on, including the multi-group precedence rules. See the [SSO role mapping guide](/agent-router-enterprise/guides/identity-and-access-management/single-sign-on/sso-role-mapping-guide/). Where to go next Onboard developers and issue keys The role model, precedence rules, and credential revocation that complement directory-driven roles. Configure SSO The SSO setup that must be in place before group claims can be read. --- ## Configure SSO Enterprise Tier Access to the Developer Console runs exclusively through single sign-on (SSO) using an existing OpenID Connect (OIDC) identity provider. Email and password sign-in is not supported. **Persona:** Platform operator completing onboarding, working with the identity team that owns the corporate identity provider. **Estimated time:** 30 to 60 minutes, most of it spent registering the OIDC client and exchanging values with the identity team. ## Overview Delegating authentication to the corporate identity provider avoids the problems a local credential database accumulates over time: passwords spread across forgotten accounts, no central offboarding signal when someone leaves, no way to enforce multi-factor authentication, and no audit trail tied to the corporate directory. Agent Router has no local password database at all, so the identity provider is the only authority on who may sign in. Agent Router exposes two web interfaces; both authenticate through the same SSO configuration: - The **Admin Dashboard** (`dashboard..tetrate.ai`), for platform operators. - The **Developer Console** (`router..tetrate.ai`), for developers. This guide covers SSO configuration against an existing identity provider. The work falls into two parts: - **Part 1: [Agent Router configuration](#part-1-agent-router-configuration)**: form expectations, verification, and troubleshooting. - **Part 2: [Identity provider configuration](#part-2-identity-provider-configuration)**: how to register an OIDC client in the identity provider and collect the values needed by the form. Walkthroughs are available for: - [Microsoft Entra ID](#microsoft-entra-id) - [Google Workspace](#google-workspace) - [Auth0](#auth0) - [Okta](#okta) - [Generic/other OIDC](#other-oidc-providers-keycloak-ping-and-custom) ## When this guide applies This guide is the right starting point in any of these situations: | Situation | What it covers | | :- | :- | | First-time SSO configuration on a new tenant | Collecting the Redirect URI, registering an OIDC client, and completing the Configure SSO form | | Registering Agent Router in a particular identity provider | The per-provider walkthroughs in Part 2 | | Rotating a client secret | Replacing the credential without interrupting sign-in | | Switching identity providers | Editing the saved configuration and re-verifying before users are affected | | Investigating a failed sign-in | The troubleshooting table and the error parameters the identity provider returns | For the related work of mapping identity-provider claims to Agent Router roles, the most common follow-up to a verified sign-in, see [Configure SSO role mapping](/agent-router-enterprise/guides/identity-and-access-management/single-sign-on/sso-role-mapping-guide/). ## Outcomes By the end of this guide: - An OIDC client representing Agent Router is registered in the corporate identity provider, with the tenant's Redirect URI registered against it. - The Issuer, Client ID, Client Secret, and Discovery URL are recorded and entered in the Configure SSO form. - At least one administrator email is set, so that the first SSO sign-in produces an account with administrative permissions. - A sign-in has been completed end to end through the Admin Dashboard or the Developer Console. - The relationship between this configuration and claim-driven role mapping is clear. ## Prerequisites - Access to the Tetrate onboarding wizard or, once onboarding is complete, the **Settings** page in the Admin Dashboard. - Cooperation from whoever owns the corporate identity provider. Most of the values required by the form come from that provider, and the tenant's Redirect URI has to be registered there. - Permission in the identity provider to create an application registration and read its client secret. - The email addresses of the people who should hold administrative permissions from the first sign-in onwards. Agent Router has no password fallback, so this decision cannot be deferred. ## Process at a glance 1. **Tetrate onboarding wizard**: open the Configure SSO step and click **IdP setup guide** to reveal the Redirect URI. Copy it. 1. **Identity provider**: register an OIDC client using that Redirect URI. The output is an **Issuer**, **Client ID**, and **Client Secret**. 1. **Configure SSO form**: paste the three values and submit. 1. **Verify**: sign in on the Admin Dashboard or Developer Console sign-in page. Open the Configure SSO step in the onboarding wizard first. The Redirect URI shown there is tenant-specific, and the IdP side cannot proceed without it. ## Part 1: Agent Router configuration ### Step 1: obtain the Redirect URI In the Configure SSO step, click the **IdP setup guide** button at the top of the form. A panel opens, titled "Configure your SSO provider", with step-by-step instructions and the **Redirect URI** for the tenant. The format is: ```text https://auth..tetrate.ai/api/auth/sso/callback/corporate ``` Use the copy icon to copy the URL. - The hostname is tenant-specific. - The trailing path is the same for every customer. - Most IdPs require the URL to match character-for-character. Any mismatch (scheme, trailing slash, missing path segment) produces a `redirect_uri_mismatch` error at sign-in. :::tip Always copy the URL from the IdP setup guide panel. Don't type it from memory or copy it from this document. ::: ### Step 2: register an OIDC client in the identity provider Use the Redirect URI from Step 1 to register an OIDC client in the identity provider. Walkthroughs for each supported IdP are in [Part 2: Identity provider configuration](#part-2-identity-provider-configuration). The output is an **Issuer**, **Client ID**, and **Client Secret**. ### Step 3: fill in the Configure SSO form The Configure SSO step collects the values below. #### Issuer (identity provider URL) The base URL of the OIDC provider. Agent Router uses it to discover other endpoints (authorization, token, userinfo, JWKS) and to validate the `iss` claim in incoming ID tokens. Common values: | IdP | Issuer format | | :- | :- | | Entra ID | `https://login.microsoftonline.com/` | | Google | `https://accounts.google.com` | | Auth0 | `https:///` | | Okta | `https://.okta.com` or `https://.okta.com/oauth2/default`, depending on the [authorization server](#choose-an-authorization-server) | | Keycloak | `https:///realms/` | The Issuer must match the `iss` claim the IdP puts in ID tokens exactly. Sign-in failures with no obvious cause usually trace back to an Issuer typo: trailing slash, missing tenant ID, wrong region. #### Client ID The OIDC client ID issued by the IdP. #### Client secret The confidential credential paired with the Client ID. This value is shown only once at creation. Copy it into the form immediately, or store it in a secrets manager. #### Discovery URL The full URL to the IdP's OpenID Connect metadata document: ```text /.well-known/openid-configuration ``` :::note Agent Router validates the URL on submit by fetching it and confirming the JSON includes `issuer`, `authorization_endpoint`, and `token_endpoint`. If those are missing or the URL is unreachable, the form returns *Invalid OIDC discovery URL*. ::: #### Scopes Use `openid, profile, email`. Some providers gate group memberships behind an additional scope, which matters for deployments that drive roles from directory groups. Okta is the common case and needs `groups` added here; see [Emit group claims for role mapping](#emit-group-claims-for-role-mapping). Microsoft Entra ID emits groups and app roles through token configuration instead, with no extra scope. #### Admin emails The email addresses granted Agent Router administrator permissions on first SSO login. This is the bootstrap mechanism for administrative access, and at least one address is always required. Without it, the initial SSO sign-in lands in a standard user account with no administrative privileges, and because Agent Router has no email and password fallback, the result is a permanent lockout from administrative functions. :::warning Add at least one address here before submitting the form. If lockout does occur, the only remedy is to contact Tetrate Support to reset the SSO configuration. ::: #### Attribute mapping Maps the Agent Router email user attribute to the matching claim name in the IdP's ID token. The default is `email → email`, which works for Google, Auth0, Okta, and Keycloak as registered in this guide. Entra requires an optional claim to be added before it emits `email`; see the [Microsoft Entra ID](#microsoft-entra-id) section. ### Step 4: submit the form Click **Configure**. Agent Router validates the discovery URL synchronously. A successful response means: - The discovery document was reachable from Agent Router's auth pod. - It contained the required OIDC metadata fields. It does **not** mean the Client ID or Client Secret are correct. Those are exercised on the first sign-in attempt. ### Step 5: verify After the form accepts the configuration, sign in via the **Admin Dashboard** or **Developer Console** sign-in page. Both delegate to the IdP. A successful flow lands back in Agent Router signed in as the IdP user. If the bounce-back lands at an error page instead of Agent Router, check the URL. Most IdPs include an error query parameter (`redirect_uri_mismatch`, `invalid_client`, `consent_required`) that pinpoints the cause. If sign-in succeeds but the dashboard shows an account with an empty email or no admin permissions, see the troubleshooting table below. ### Confirm the claims in the ID token Three of the most common SSO problems are invisible from the sign-in screen and obvious in the ID token: an `iss` value that does not match the configured Issuer, a missing `email` claim, and a missing groups or roles claim. Reading the token settles which side of the integration is at fault before any configuration is changed. This applies to every provider in Part 2, since all of them are being asked for the same claims. Agent Router exchanges the authorization code server-side, so the token never passes through the browser. To obtain one, run the same flow by hand against the identity provider: 1. Register a second, throwaway redirect URI on the OIDC client, such as `http://localhost:8080/callback`. Every provider in Part 2 accepts more than one. Leave the Tetrate Redirect URI in place. 1. Read the endpoints from the discovery document: ```bash curl -s "/.well-known/openid-configuration" \ | jq -r '.authorization_endpoint, .token_endpoint' ``` 1. Open the authorize URL in a browser and authenticate as a test user. Add `groups` to the scope list when the point of the exercise is to confirm a group claim: ```text ?client_id=&redirect_uri=http://localhost:8080/callback&response_type=code&scope=openid%20profile%20email&state=check ``` 1. The browser lands on a dead `localhost` address carrying `?code=...`. Copy the code value out of the address bar. 1. Exchange the code for tokens: ```bash curl -s -X POST "" \ -u ":" \ -d grant_type=authorization_code \ -d code="" \ -d redirect_uri=http://localhost:8080/callback \ | jq -r .id_token ``` 1. Remove the throwaway redirect URI once the check is finished. Authorization codes are single-use and expire quickly, so the exchange has to follow the browser step promptly. Where a provider rejects the credentials sent by `-u`, send them as form fields instead by adding `-d client_id= -d client_secret=`. Where the client is configured to require PKCE, the manual flow additionally needs a `code_challenge` and `code_verifier` pair; none of the walkthroughs in Part 2 enable PKCE. Decode the payload locally rather than pasting the token into an online decoder. An ID token identifies a real user and is best treated as a credential: ```bash jq -R 'split(".")[1] | gsub("-";"+") | gsub("_";"/") | @base64d | fromjson' <<<"$ID_TOKEN" ``` What to look for in the output: | Claim | Expectation | | :- | :- | | `iss` | Matches the **Issuer** in the Configure SSO form character-for-character, including the presence or absence of a trailing slash | | `aud` | Contains the **Client ID** configured in Agent Router | | `email` | Present and populated. An absent or differently named claim is the cause behind the *empty email* row in the troubleshooting table, and is corrected through **Attribute Mapping** | | `groups` or `roles` | Present only where the provider has been configured to emit it. Claim-driven role mapping depends on it, and the name seen here is the one the mapping configuration has to reference | | `exp` | A Unix timestamp in the future. A token that already looks expired points at clock skew rather than at configuration | Two providers offer a shortcut that avoids the manual flow. Okta exposes a **Token Preview** tab on a custom authorization server, which mints a token for a chosen user and client with no browser round-trip, although it is unavailable on the org authorization server this guide recommends. Google decodes and validates a token in a single call to `https://oauth2.googleapis.com/tokeninfo?id_token=`, which Google supports for debugging only. ### Troubleshooting and common errors | Symptom | Cause | Fix | | :- | :- | :- | | *Invalid OIDC discovery URL* on form submit | Issuer typo, IdP unreachable from the auth pod, or IdP returns 404 / non-JSON at `/.well-known/openid-configuration` | Verify the discovery URL is reachable: `curl /.well-known/openid-configuration \| jq -r '.issuer, .authorization_endpoint, .token_endpoint'` | | Signed in but Agent Router shows an empty email | ID token has no `email` claim, or claim name is non-standard | [Decode the ID token](#confirm-the-claims-in-the-id-token) to find the actual claim name, then update **Attribute Mapping** to point at it, or add `email` as an emitted claim in the IdP | | Signed in but no admin permission | Email wasn't in **Admin Emails** at the time of first SSO sign-in | Contact Tetrate Support to reset SSO and re-run with **Admin Emails** populated | For IdP-side errors (`redirect_uri_mismatch`, `invalid_client`, callback mismatch), see the IdP's documentation. ### Edit the SSO configuration after onboarding The same fields are editable from the **Settings** page in the dashboard sidebar (**SSO** tab) after onboarding completes. The form is identical; only the route differs. SSO cannot be turned off, but the values can be swapped to point at a different IdP. To rotate the Client Secret without downtime, create a new secret in the IdP, update **Client Secret** in Agent Router, verify that sign-in works, then delete the old secret in the IdP. ### Switch identity providers Agent Router wires one identity provider at a time. Switching providers replaces the saved configuration; there is no parallel-run mode in which both the outgoing and the incoming provider are accepted, which makes this the most disruptive operation on this surface. The sequence that causes the least disruption: 1. Register the OIDC client in the incoming identity provider and collect its Issuer, Client ID, Client Secret, and Discovery URL before anything is changed in Agent Router. 1. Where a separate test tenant exists, point that tenant at the new provider first and verify a sign-in there. 1. In production, edit **Settings → SSO** and replace the Issuer, Client ID, Client Secret, and Discovery URL in a single change. 1. Verify a sign-in immediately, using the checks in [Step 5](#step-5-verify). Because the change is a replacement rather than an addition, a mistake affects every user at once. 1. Coordinate the cutover with the identity team so that the Agent Router-side change and any provider-side change happen close together. 1. Announce the change at least one business day in advance for any deployment with a non-trivial number of users. See [Communicate with Agent Router users](/agent-router-enterprise/guides/operate-and-govern/communicate-with-platform-users/). ### Multi-factor authentication Multi-factor authentication is whatever the identity provider enforces. Agent Router delegates the entire login screen to the provider, so MFA prompts, conditional access policies, and device checks are all applied there and require no configuration in Agent Router. ### How SSO interacts with role mapping SSO authentication settles who is signing in. Role mapping settles a separate question: which Agent Router role that user holds. The two are configured on the same surface but are otherwise independent. Without role mapping, every user provisioned through SSO lands with the default user role, apart from the addresses listed in **Admin Emails**, and elevated roles are assigned by hand in the Admin Dashboard. That is workable for a small deployment, but it leaves a synchronisation gap: a change in the directory has no effect on Agent Router until an operator notices and edits the user record. With role mapping, the claims the identity provider returns (assigned application roles, or group memberships) are translated into Agent Router roles on every login, which makes the directory the source of truth and Agent Router-side role assignment redundant. The complete walkthrough is in [Configure SSO role mapping](/agent-router-enterprise/guides/identity-and-access-management/single-sign-on/sso-role-mapping-guide/), and it is the right next read once a sign-in has been verified here. ### SSO changes in the audit log A change to the SSO configuration is recorded in the audit log as an `sso.configured` event, alongside every other administrative change. See [Audit Agent Router activity](/agent-router-enterprise/guides/operate-and-govern/audit-platform-activity/). That trail is the first place to look when SSO behaviour starts to surprise people. A sudden run of failing sign-ins is usually preceded either by a recent `sso.configured` event, meaning the configuration itself was changed, or by a change on the identity-provider side that no Agent Router audit log can capture. Ruling out the first narrows the search considerably. --- ## Part 2: identity provider configuration This section covers how to register an OIDC client in each supported identity provider. Each walkthrough opens with the values it produces, so it can be read as a checklist and handed to the identity team. Only the section for the provider in use needs to be followed. ### Microsoft Entra ID #### What it produces | Value | Where it comes from | | :- | :- | | **Issuer** | App registration → Overview → Endpoints → Authority URL | | **Discovery URL** | App registration → Overview → Endpoints → OpenID Connect metadata document | | **Client ID** | App registration → Overview → Application (client) ID | | **Client Secret** | App registration → Certificates & secrets → New client secret | One value is registered **in Entra**: the **Redirect URI** for the Tetrate tenant. #### Prerequisites - Access to the Microsoft Entra ID instance. - The Tetrate **Redirect URI**. Open the **IdP setup guide** panel in the Configure SSO step to find it. #### Step 1: create an app registration In the Entra admin center, go to **App registrations** and click **+ New registration**. Fill in the name and complete the form. #### Step 2: add the Redirect URI In the app registration, click **Authentication** in the left sidebar. - Click **+ Add a platform**, then choose **Web**. - Paste the Redirect URI from the Tetrate onboarding form. - Click **Configure**. #### Step 3: add optional claims In the app registration, click **Token configuration** in the left sidebar, then **+ Add optional claim**. | Field | Value | | :- | :- | | Token type | **ID** | | Claims | Tick `email` | Click **Add**. #### Step 4: generate a client secret In the app registration, click **Certificates & secrets** in the left sidebar, then **+ New client secret**. - Add a description, choose an expiry, and click **Add**. - Copy the Value column immediately and store it in a secrets manager. Once the page closes, Entra does not reveal it again. #### Step 5: collect the Issuer URL and Client ID In the app registration **Overview**: - Click **Endpoints** in the top toolbar. A drawer opens with several URLs. - Copy the **Authority URL** (use as the Tetrate **Issuer**). - Copy the **OpenID Connect metadata document** URL (use as the Tetrate **Discovery URL**). - Copy **Application (client) ID** (use as the Tetrate **Client ID**). #### Done in Entra The collected values are: - Issuer, the Authority URL. - Discovery URL, the OpenID Connect metadata document URL. - Client ID. - Client Secret. Return to [Part 1: Agent Router configuration](#part-1-agent-router-configuration) and paste them into the Configure SSO form. ### Google Workspace #### What it produces | Value | Where it comes from | | :- | :- | | **Issuer** | `https://accounts.google.com` (fixed) | | **Discovery URL** | `https://accounts.google.com/.well-known/openid-configuration` (fixed) | | **Client ID** | APIs & Services → Credentials → OAuth client ID | | **Client Secret** | APIs & Services → Credentials → OAuth client ID | Two values are registered **in Google**: the **Authorized redirect URI** and the **Authorized JavaScript Origins** for the Tetrate tenant. #### Prerequisites - A Google Cloud project. - IAM permission on the project to manage OAuth credentials. - The Tetrate **Redirect URI**. Open the **IdP setup guide** panel in the Configure SSO step to find it. #### Step 1: create OAuth credentials Go to **Google Cloud Console → APIs & Services → Credentials** and click **Create Credentials → OAuth client ID**. #### Step 2: configure the consent screen Add: - App name. - Support email. - Scopes: `openid`, `profile`, `email`. #### Step 3: set the Redirect URI Set the **Authorized redirect URI** to the Tetrate Redirect URI: ```text https://auth..tetrate.ai/api/auth/sso/callback/corporate ``` #### Step 4: set authorized JavaScript origins Set **Authorized JavaScript Origins** to the three Tetrate hostnames for the tenant: ```text https://auth..tetrate.ai https://dashboard..tetrate.ai https://router..tetrate.ai ``` #### Step 5: save credentials Copy: - **Client ID**. - **Client Secret**. #### Done in Google The collected values are: - Issuer: `https://accounts.google.com`. - Discovery URL: `https://accounts.google.com/.well-known/openid-configuration`. - Client ID. - Client Secret. Return to [Part 1: Agent Router configuration](#part-1-agent-router-configuration) and paste them into the Configure SSO form. ### Auth0 :::tip Evaluation walkthrough For a ground-up Auth0 path (new tenant through credentials ready for onboarding), see [Set up Auth0 for evaluation](/product-architecture/product-evaluation/auth0-for-evaluation/). ::: #### What it produces | Value | Where it comes from | | :- | :- | | **Issuer** | Build from the Auth0 Domain: `https:///` *(trailing slash)* | | **Discovery URL** | Build from the Auth0 Domain: `https:///.well-known/openid-configuration` | | **Client ID** | Applications → app → Settings | | **Client Secret** | Applications → app → Settings | Three values are registered **in Auth0**: the **Callback URL**, **Allowed Web Origins**, and (optionally) the **Logout URL** for the Tetrate tenant. #### Prerequisites - An Auth0 tenant. - A role that allows creating applications and viewing secrets. - The Tetrate **Redirect URI**. Open the **IdP setup guide** panel in the Configure SSO step to find it. #### Step 1: create the application - Go to the **Auth0 Dashboard**. - Select **Applications**. - Click **+ Create Application**. - Select **Create Manually**. - Enter a recognisable name, such as `Tetrate Agent Router Enterprise`. - Select **Regular Web Application**. - Click **Create**. #### Step 2: configure application details On the **Settings** tab of the new application: - Under **Basic Information**, copy the `Domain`, `Client ID`, and `Client Secret` values into a secrets manager or scratch file. The Issuer is built from the Domain in Step 4. - Under **Application URIs**, set **Allowed Callback URLs** to the Tetrate Redirect URI: `https://auth..tetrate.ai/api/auth/sso/callback/corporate` - Under **Application URIs**, set **Allowed Web Origins** to the Tetrate hostnames for the tenant: ```text https://auth..tetrate.ai https://dashboard..tetrate.ai https://router..tetrate.ai ``` #### Step 3: set the logout URL (optional) Still under **Application URIs**, set **Allowed Logout URLs** to: ```text https://auth..tetrate.ai ``` Click **Save**. #### Step 4: save credentials, if not already done From the application **Settings** tab, copy: - **Domain**: build the Issuer as `https:///` (with trailing slash). For example, `acme.us.auth0.com` becomes `https://acme.us.auth0.com/`. - **Client ID**. - **Client Secret**. #### Done in Auth0 The collected values are: - Issuer: `https:///` *(trailing slash)*. - Discovery URL: `https:///.well-known/openid-configuration`. - Client ID. - Client Secret. Return to [Part 1: Agent Router configuration](#part-1-agent-router-configuration) and paste them into the Configure SSO form. ### Okta One decision shapes everything else in an Okta integration and is best settled before the wizard is opened: which authorization server issues the tokens. It determines the Issuer value, the discovery URL, and where group claims are configured later. #### Choose an authorization server Okta can issue tokens from either of two authorization servers, and they are not interchangeable. | | Org authorization server | Custom authorization server | | :- | :- | :- | | Issuer | `https://.okta.com` | `https://.okta.com/oauth2/`, commonly `.../oauth2/default` | | Discovery URL | `https://.okta.com/.well-known/openid-configuration` | `https://.okta.com/oauth2//.well-known/openid-configuration` | | Availability | Present in every Okta org | Requires the API Access Management add-on, which is licensed separately in production orgs | | Group claims configured on | The application's **Sign On** tab | The authorization server's **Claims** tab | | Custom scopes and claims | Not supported | Supported | The org authorization server is the right default. It exists in every org, it issues the ID token Agent Router consumes, and it supports the groups claim that role mapping depends on. A custom authorization server is only necessary where the same server also protects the organisation's own APIs, and where API Access Management is already licensed. Deployments that pick `.../oauth2/default` without that entitlement will find the endpoint absent, which surfaces as an unreachable discovery URL when the Configure SSO form is submitted. Agent Router validates the ID token only, so the [caveats Okta documents around access-token validation](https://developer.okta.com/docs/concepts/auth-servers/) on the org authorization server do not apply here. #### What it produces | Value | Where it comes from | | :- | :- | | **Issuer** | Security → API → **Authorization Servers** → **Issuer URI** of the server chosen above | | **Discovery URL** | Build from the Issuer: `/.well-known/openid-configuration` | | **Client ID** | Applications → app → **General** tab → **Client Credentials** | | **Client Secret** | Applications → app → **General** tab → **Client Credentials** | Two values are registered **in Okta**: the **Sign-in redirect URI** and, optionally, the **Sign-out redirect URI** for the Tetrate tenant. #### Prerequisites - An Okta org, and an administrator role that allows creating app integrations and reading client secrets. - The Tetrate **Redirect URI**. Open the **IdP setup guide** panel in the Configure SSO step to find it. - A decision on the authorization server, as above. #### Step 1: create the app integration In the Okta Admin Console, go to **Applications → Applications** and click **Create App Integration**. Where the org offers a choice of wizard, select **Classic experience**. - Sign-in method: **OIDC - OpenID Connect**. - Application type: **Web Application**. - Click **Next**. Application type matters more than it appears. A Web Application is a confidential client, which is what allows Okta to issue the client secret Agent Router authenticates with. Selecting a single-page or native application produces a public client with no secret, and the sign-in flow then fails at the token exchange. #### Step 2: configure the redirect URIs Under **General Settings**: - **App integration name**: a recognisable name, such as `Tetrate Agent Router Enterprise`. - **Grant type**: leave **Authorization code** selected. It is required for this flow and cannot be cleared. - **Sign-in redirect URIs**: the Tetrate Redirect URI, exactly as copied from the IdP setup guide panel. ```text https://auth..tetrate.ai/api/auth/sso/callback/corporate ``` - **Sign-out redirect URIs** (optional): `https://auth..tetrate.ai`. Absolute URIs only; wildcard subdomains are not accepted in this field. Leave **Allow wildcard in sign-in redirect URI** unselected. A single tenant needs exactly one redirect URI, and Okta cautions that subdomain wildcards can allow authorization codes to be sent to an attacker-controlled page. The **Trusted Origins** section of the wizard, with its optional **Base URIs** field, governs cross-origin browser calls to the Okta API. It is not required for the server-side authorization code exchange Agent Router performs. #### Step 3: set client authentication In **Client Credentials**, confirm **Client authentication** is set to **Client secret** rather than **Public key / Private key**. Agent Router authenticates to the token endpoint with a client secret, and there is no field in the Configure SSO form for a private key. **Proof Key for Code Exchange (PKCE)** may be left at its default. #### Step 4: set controlled access The **Assignments** section governs who may sign in. Under **Controlled access**, either allow everyone in the organisation, or select **Limit access to selected groups** and name the groups that should reach Agent Router. Click **Save**. Okta enforces this before Agent Router ever sees the user, so an unassigned user is refused at the identity provider no matter how Agent Router is configured. Assignment is also the mechanism for restricting Agent Router to a pilot group during a phased rollout. Choosing **Skip group assignment for now** leaves the application with no one assigned, and every sign-in attempt then fails. #### Step 5: collect the Client ID and Client Secret On the application's **General** tab, under **Client Credentials**, copy: - **Client ID**. - **Client secret**. Copy it into the Configure SSO form immediately or store it in a secrets manager. Okta reveals the value once. #### Step 6: confirm the Issuer URL Go to **Security → API → Authorization Servers** and copy the **Issuer URI** of the server chosen in the first step. Rather than transcribing it, confirm the canonical value by reading the discovery document, and use whatever the `issuer` field returns: ```bash curl -s "/.well-known/openid-configuration" | jq '.issuer, .authorization_endpoint, .token_endpoint' ``` An Issuer mismatch is the most common cause of an Okta sign-in that fails with no informative error, and Okta issuers have several ways of looking almost right: - No trailing slash. Neither form of Okta issuer ends in `/`, unlike Auth0. - The domain suffix follows the cell the org lives in: `okta.com`, `okta-emea.com` for EMEA cells, or `oktapreview.com` for preview and sandbox orgs. A value copied from a sandbox runbook will not work against production. - Where the org uses a [custom domain](https://developer.okta.com/docs/guides/custom-url-domain/main/), an `issuer_mode` setting decides whether tokens carry the Okta org URL or the custom domain. The configured Issuer has to match whichever one the tokens actually assert. Disabling a custom domain resets `issuer_mode` back to the org URL, which changes the `iss` value and breaks sign-in with no change on the Agent Router side. The `openid`, `profile`, and `email` scopes are standard, and Okta emits a standard `email` claim in the ID token, so the default `email → email` attribute mapping needs no change. #### Done in Okta The collected values are: - Issuer, the Issuer URI of the authorization server. - Discovery URL, the Issuer with `/.well-known/openid-configuration` appended. - Client ID. - Client Secret. Return to [Part 1: Agent Router configuration](#part-1-agent-router-configuration) and paste them into the Configure SSO form. #### Emit group claims for role mapping Group claims are only needed where roles, rate limits, or budgets are driven from directory membership. Base sign-in works without them. Okta does not emit group memberships by default. On the org authorization server, the claim is configured on the application rather than centrally: 1. In the Okta Admin Console, go to **Applications → Applications** and select the application created above. 1. Open the **Sign On** tab and click **Edit** in the **OpenID Connect ID Token** section. 1. Set **Group claim type** to **Filter**. 1. Under **Group claims filter**, leave the claim name as `groups`, or enter it if the field is empty. 1. Set the filter to **Matches regex** with a value of `.*` to return every group the user belongs to. A narrower regex is preferable in a large org, so that the token carries only the groups Agent Router maps. 1. Click **Save**, then return to the applications list, open the **More** menu, and choose **Refresh Application Data**. On a custom authorization server the equivalent is configured under **Security → API**, on the chosen server's **Claims** tab, which also allows the claim in access tokens. Okta's [groups claim guide](https://developer.okta.com/docs/guides/customize-tokens-groups-claim/main/) covers both paths, including the expression syntax needed for groups sourced outside Okta, such as those synchronised from Active Directory. Two details decide whether the claim actually arrives: - **The `groups` scope has to be requested.** Okta returns the claim only when `groups` is among the requested scopes, so the Scopes field in the Configure SSO form needs `openid, profile, email, groups` rather than the default three. - **The flow has to return a full ID token.** Okta issues what it calls a thin ID token, carrying base claims only and omitting groups, for implicit and several other flows. The authorization code flow Agent Router uses returns the full token, so the claim is present once the two points above are satisfied. At the mapping layer Agent Router draws no distinction between providers: it reads the group values it receives and matches them against the configured rules, so Okta groups drive role assignment, rate limits, and budgets exactly as Entra ID groups do. The claim-to-role mechanics are in [Configure SSO role mapping](/agent-router-enterprise/guides/identity-and-access-management/single-sign-on/sso-role-mapping-guide/), and the end-to-end treatment of group-to-business-function mapping, written against Entra ID but applicable to Okta groups unchanged, is in [Map Entra ID groups to business functions](/agent-router-enterprise/guides/identity-and-access-management/single-sign-on/map-entra-id-groups-to-business-functions/). #### Rotate an Okta client secret Okta supports two concurrent client secrets on an application, both active at once, which is what makes the rotation sequence in [Edit the SSO configuration after onboarding](#edit-the-sso-configuration-after-onboarding) possible without a sign-in outage: 1. On the **General** tab, under **Client Credentials**, click **Generate new secret**. Both secrets are now valid. 1. Update **Client Secret** in Agent Router with the new value and confirm a sign-in succeeds. 1. Set the old secret's status to **Inactive** in Okta. 1. Delete the inactive secret. The limit is two, so the old one has to be removed before the next rotation. #### Okta-specific troubleshooting | Symptom | Cause | Fix | | :- | :- | :- | | Okta reports the user is not assigned to the application | The signing-in user is not in the application's **Assignments**, directly or through a group | Assign the user or their group under **Assignments**, or widen **Controlled access** | | `redirect_uri_mismatch` at sign-in | **Sign-in redirect URIs** does not match the Tetrate Redirect URI character-for-character | Re-copy the value from the **IdP setup guide** panel; check scheme, trailing slash, and the full `/api/auth/sso/callback/corporate` path | | `invalid_client` at the token exchange | Wrong Client ID or Client Secret, or the app was created as a public client | Re-copy both values from **Client Credentials**; confirm the app is a **Web Application** with **Client authentication** set to **Client secret** | | Sign-in succeeds but no groups reach Agent Router | The `groups` scope is not requested, or no group claim is configured on the app | Add `groups` to the Scopes field and configure the claim as described above, then [decode the ID token](#confirm-the-claims-in-the-id-token) to confirm the claim now arrives | | Sign-in worked previously and now fails with no configuration change | A custom domain was enabled or disabled in Okta, changing the `iss` value the tokens assert | Re-read the discovery document and update the **Issuer** to match | ### Other OIDC providers (Keycloak, Ping, and custom) The flow matches the supported IdPs above; only the IdP UI changes. From the IdP, register a **Web application**, set the Redirect URI to the value Agent Router's form supplies, enable the `openid`, `profile`, `email` scopes, generate a client secret, and collect Issuer, Client ID, and Client Secret. For the Issuer, find the IdP's OIDC discovery URL and check that it returns valid JSON with `issuer`, `authorization_endpoint`, and `token_endpoint`: ```bash curl -s "/.well-known/openid-configuration" \ | jq '.issuer, .authorization_endpoint, .token_endpoint' ``` The value `.issuer` returns is what goes into the Agent Router **Issuer** field. Even small differences (trailing slash, version path) break sign-in. ## What to do next - **SSO role mapping.** Map identity-provider claims to Agent Router roles so that directory changes propagate without manual intervention. See [Configure SSO role mapping](/agent-router-enterprise/guides/identity-and-access-management/single-sign-on/sso-role-mapping-guide/). - **Map Entra ID groups to business functions.** Drive roles, rate limits, and budgets from directory group membership. See [Map Entra ID groups to business functions](/agent-router-enterprise/guides/identity-and-access-management/single-sign-on/map-entra-id-groups-to-business-functions/). - **Audit Agent Router activity.** The configuration change made here is one of the administrative events the audit log records. See [Audit Agent Router activity](/agent-router-enterprise/guides/operate-and-govern/audit-platform-activity/). - **Run multiple Agent Router instances.** SSO is configured per instance, and the multi-instance considerations are non-trivial. See [Run multiple Agent Router instances](/agent-router-enterprise/guides/operate-and-govern/run-multiple-platform-instances/). Where to go next Console quickstart Issue an API key and confirm a first routed request once SSO is working and the gateway is reachable. Admin Dashboard quickstart Review models, providers, users, and usage analytics from the management plane. SSO role mapping (optional) After base SSO is verified, map OIDC roles and groups to Agent Router roles. Self-hosted path Where SSO was deferred during onboarding, this is the next step before developer handoff. The full self-hosted sequence. --- ## Configure SSO role mapping Agent Router assigns its own roles automatically from the claims returned by any OIDC identity provider. Once the mapping is in place, a directory change propagates to Agent Router on the next login, with no manual work in the Admin Dashboard. :::info Optional configuration Base SSO must already be configured. Skip this guide if you assign roles manually in the Admin Dashboard or your PoC does not use claim-driven role mapping. ::: ## Mapping OIDC roles and groups to Agent Router roles Most organisations already model who-can-do-what in their identity provider, and duplicating that model inside Agent Router quickly becomes a maintenance burden. This guide describes how Agent Router is configured to assign its own roles automatically, driven by the claims returned by any OpenID Connect compliant (OIDC) identity provider. Once the mapping is in place, an administrator added to a group in the directory shows up as an administrator in Agent Router on their next login, with no manual intervention in the Admin Dashboard. The same configuration shape works for **Microsoft Entra ID**, **Okta**, **Auth0**, **Keycloak**, and any other OIDC provider. The pattern is consistent: a claim that carries role or group values is selected, and the values inside that claim are mapped to Agent Router roles. :::note This guide uses Microsoft Entra in its examples, but the overall process is the same for every identity provider. Only the location of settings and the exact claim names tend to differ between vendors. The Entra examples below use app roles because they produce stable, readable values in the `roles` claim: values such as `tare-billing-admin` are easy to recognise during audits and easy to map. If group claims are used instead, the same Agent Router mapping fields apply to the `groups` claim with no change to Agent Router side. For Entra, the recommended setup is **app roles assigned to users or groups**. Direct group-claim mapping also works when Entra reliably emits the `groups` claim, but it usually exposes raw group object IDs and can hit group-overage limits, both of which tend to make day-two operations harder than they need to be. ::: ## Outcomes By the end of this guide: - App roles (or groups) are defined and assigned in the identity provider. - Agent Router is configured with a role-claim path and a value-to-role mapping. - Verified users receive the correct Agent Router role on every login, with no per-user action required inside Agent Router itself. Plan for 15 to 30 minutes for the configuration itself, plus a few minutes for Entra to propagate the new app role assignments to test users. In larger tenants, propagation may take noticeably longer. ## In this guide - **Identity Provider (IdP) configuration** - define app roles or groups and confirm they appear in OIDC claims - [Step 1: Create app roles in your IdP](#step-1-create-app-roles-in-idp) - [Step 2: Assign roles to users or groups](#step-2-assign-roles-to-users-or-groups) - **Agent Router configuration** - map IdP claim values to Agent Router roles - [Step 3: Configure the SSO provider with role mapping](#step-3-configure-the-sso-provider-with-role-mapping) - **Verify** - confirm the mapping works end-to-end - [Step 4: Verify the mapping](#step-4-verify-the-mapping) Optional: - [Appendix A: Control who can log in with Assignment required](#appendix-a-control-who-can-log-in-with-assignment-required) - [Appendix B: Role mapping precedence and defaults](#appendix-b-role-mapping-precedence-and-defaults) - [Appendix C: App roles vs group claims](#appendix-c-app-roles-vs-group-claims) ## How it works When a user signs in through the OIDC identity provider, the provider returns a set of claims describing that user (name, email, group memberships, assigned application roles, and so on). Agent Router inspects the configured claim path, looks up each value found there in the configured mapping table, and applies the matching Agent Router role to the user record. This evaluation runs on every login, not just the first, so the directory always remains the source of truth. ```text [OIDC] token/userinfo: { "roles": ["tare-billing-admin"] } ↓ roleClaimPath: "roles" roleMapping: { "tare-billing-admin": "billing_admin" } ↓ [Agent Router] user.role = "billing_admin" ``` From Agent Router's perspective, app roles and group claims are interchangeable: both arrive as arrays of string values inside a named claim, and both are consumed through the same two configuration fields. | Field | What it does | Example | | :---- | :---- | :---- | | `roleClaimPath` | Dot-notation path to the claim to read | `roles`, `groups`, `resource_access.tars.roles` | | `roleMapping` | Maps each IdP claim value to an Agent Router role | `{ "tare-admin": "super_admin" }` | A typical app role configuration looks like this: ```json { "roleClaimPath": "roles", "roleMapping": { "tare-admin": "super_admin", "tare-user": "user" } } ``` A group claim configuration uses the same structure, with group identifiers in place of role strings: ```json { "roleClaimPath": "groups", "roleMapping": { "00000000-0000-0000-0000-000000000000": "super_admin", "11111111-1111-1111-1111-111111111111": "user" } } ``` :::note Entra's `/userinfo` endpoint sometimes serializes claims such as `roles` as a JSON-stringified array (`'["tare-billing-admin"]'`) instead of a native array. This is a known quirk of the Microsoft Graph response shape. Agent Router handles both formats transparently: no special configuration is needed, and the same `roleClaimPath` value works in either case. ::: ## Prerequisites - An OIDC application already configured for Agent Router corporate SSO login. For Entra, this is an App registration created during initial SSO setup. - Permission to configure claims and assignments in the identity provider. For Entra, the Owner or Application Administrator role in the tenant is required; lesser roles cannot edit app roles or manage user assignments. - Access to the Agent Router admin dashboard with the `super_admin` role, which is required to view and edit SSO provider configuration. ## Step 1: create app roles in IdP App roles are the building blocks of the mapping. Each role defined here corresponds to an entry that will eventually appear in the `roles` claim and, through the mapping, to a specific Agent Router role. :::info App roles are defined on the **App registration**, not the Enterprise application. The two blades are linked but distinct: App registration owns the application's identity and capabilities, while Enterprise application owns the user-facing assignment surface. ::: This example is based on Microsoft Entra. Comparable steps exist for every OIDC provider; the navigation differs, but the underlying objects are the same. 1. Go to **Azure Portal → App registrations** and open the Agent Router app. 1. In the left sidebar, click **App roles**. 1. Click **+ Create app role**. 1. Fill in the fields: | Field | Value | | :---- | :---- | | Display name | A human-readable label (for example, `Agent Router Billing Admin`) | | Allowed member types | **Users/Groups** | | Value | The string Agent Router will see in the token (for example, `tare-billing-admin`) | | Description | Brief description for admins | | Do you want to enable this app role? | **Checked** | 1. Click **Apply**. 1. Repeat for each required role. The **Value** field is the one that matters, because it is the exact string that Entra emits in the `roles` claim and that Agent Router will look up in its mapping. Display names are for humans browsing the Entra UI; values are for machines. :::tip Use a consistent naming prefix such as `tare-` (for example `tare-model-admin`, `tare-billing-admin`) to make role assignments easy to filter and audit in Entra. A prefix also helps avoid collisions when the same tenant hosts multiple applications with their own role schemes. ::: ### Available Agent Router roles The mapping target on the Agent Router side is always one of the roles listed below. The role determines which areas of the Console and Admin Dashboard the user can access, and which actions the API will accept on their behalf. | Agent Router role | Typical use | | :---- | :---- | | `super_admin` | Full platform administration | | `model_admin` | Manage model routes and backends | | `provider_admin` | Manage upstream AI providers | | `mcp_admin` | Manage MCP profiles and routing | | `user_admin` | Manage users and org membership | | `billing_admin` | View and manage billing and usage | | `user` | Default: standard access, no admin permissions | Most deployments only need two or three of these roles. A common starting point is `super_admin` for Agent Router owners, one or two functional admin roles (such as `model_admin` or `billing_admin`) for delegated operators, and `user` for everyone else. ## Step 2: assign roles to users or groups Defining app roles is only half the picture; the roles must also be assigned to the principals (users or groups) who should receive them. Role assignments live on the **Enterprise application**, a separate blade from App registrations that controls who can use the app and what they appear as inside it. ### Step 2.1: find the enterprise application The App registration and the Enterprise application are two views of the same underlying object in Entra. The fastest way to jump from one to the other is through the App registration's overview page. From inside the App registration: 1. Open the Agent Router app in **App registrations → Overview**. 1. In the **Essentials** section, click the **Managed application in local directory** link. This opens the matching Enterprise application directly, with the correct context preserved. ### Step 2.2: assign a role to a user A single assignment grants exactly one app role to one principal. Multiple assignments can be stacked for the same user when more than one role is required. 1. In the Enterprise application, click **Users and groups** in the left sidebar. 1. Click **+ Add user/group**. 1. Under **Users**, click **None Selected**, search for the user, select them, and click **Select**. 1. Under **Select a role**, click **None Selected**, choose the app role created in Step 1 (for example, `Agent Router Billing Admin`), and click **Select**. 1. Click **Assign**. The assignment becomes visible to Agent Router on the user's next sign-in. Existing sessions are not affected until the next token refresh or login. :::tip To assign the same role to multiple users at once, use a security group: add the group under **Users and groups**, assign the role to the group, and manage membership in the group itself. The `roles` claim is populated with roles assigned to the user directly **and** roles inherited through group membership, so the two approaches can be mixed without conflict. ::: :::note A user can hold only **one app role per assignment row**. To grant multiple Agent Router roles to a user, add them more than once with a different role each time, or consider whether a single higher-privilege role (such as `super_admin`) covers the use case more cleanly. In practice, Agent Router applies the highest-privilege matching role, so stacking lower roles on top of a higher one offers little additional value. ::: ## Step 3: configure the SSO provider with role mapping With the identity provider side in place, the remaining task is to tell Agent Router which claim to read and how to translate the values it finds there. This is done once per SSO provider configuration and applies to every user who signs in through it. ### Step 3.1: required fields Two fields drive the entire mapping. Both are edited together in the Admin Dashboard. | Field | Description | | :---- | :---- | | `roleClaimPath` | The claim path to read from the token or userinfo response. Use `roles` for Entra app roles, `groups` for OIDC group claims, or a nested path such as `resource_access.tars.roles` for providers like Keycloak. | | `roleMapping` | A JSON object mapping IdP claim values to Agent Router role names. | ### Step 3.2: apply the configuration in Agent Router 1. Open the Agent Router admin dashboard. 1. Go to **Settings → SSO**. 1. Open the OIDC SSO provider configuration. 1. In the role mapping section, set **Role claim path** to the claim Agent Router should read. The correct value depends on the identity provider and the claim shape configured for the app: | Source | Role claim path | | :---- | :---- | | Entra app roles | `roles` | | Entra group claims | `groups` | | Okta groups claim | `groups` | | Keycloak client roles | `resource_access.tars.roles` | 1. Add one mapping row for each IdP value that should grant an Agent Router role. Any value not listed here simply produces no admin role and falls through to the default `user` role: | IdP claim value | Agent Router role | | :---- | :---- | | `tare-billing-admin` | `billing_admin` | | `tare-model-admin` | `model_admin` | | `tare-super-admin` | `super_admin` | 1. Save the SSO provider configuration. The new mapping takes effect on the user's next login. There is no need to invalidate sessions or restart Agent Router; the configuration is read fresh on each authentication callback. :::note `roleMapping` keys must match the exact claim values emitted by the IdP, character for character and case for case. For Entra app roles, the app role **Value** strings set in Step 1 are what flows through the claim, not the display names shown in the portal. For Entra group claims, the group object IDs are emitted unless a provider-specific claim configuration that emits names has been added separately. Values on the right-hand side must be valid Agent Router role names from the table above; any mapping that points to an unrecognised role name is silently ignored, so misspellings can be a frustrating source of "why is my admin still a regular user?" tickets. ::: ### Step 3.3: provider examples The same two-field pattern covers most realistic deployments. The differences between providers are largely about which claim is the natural fit and what shape its values take. | Provider pattern | Typical `roleClaimPath` | `roleMapping` keys | | :---- | :---- | :---- | | Entra app roles | `roles` | App role **Value** strings, such as `tare-billing-admin` | | Entra group claims | `groups` | Group object IDs, only when Entra emits the `groups` claim for the user | | Okta groups claim | `groups` | Group names or IDs, depending on the Okta claim configuration | | Auth0 namespaced custom claim | `https://example.com/roles` | Role or group values emitted in the custom claim | | Keycloak client roles | `resource_access.tars.roles` | Client role names | ## Step 4: verify the mapping Before announcing the new SSO configuration to the broader user base, the mapping should be exercised end-to-end with at least one test user per Agent Router role. This catches both directory-side misconfiguration (missing assignments, wrong claim names) and mapping-side typos in a low-stakes setting. ### Step 4.1: log in as a test user Sign in via corporate SSO as a user assigned the expected IdP app role or group. After login completes, open **Users** in the Agent Router admin dashboard and confirm that the user's role matches the mapped Agent Router role. For example, a user with the `tare-billing-admin` Entra app role should appear as `billing_admin` in Agent Router immediately after their first login under the new configuration. ### Step 4.2: what to check if the role is wrong The table below covers the issues that account for most "role not applied" support cases. Each one is straightforward to diagnose once the right place to look is known. | Symptom | Likely cause | Fix | | :---- | :---- | :---- | | User gets `user` role despite assignment | Role or group value mismatch between the IdP claim and Agent Router role mapping | Confirm the exact app role value or group object ID in the IdP and make the Agent Router mapping key match it exactly, including case. | | `roles` or `groups` claim is absent from the token | Claim not configured, app role or group not assigned, or assignment not yet propagated | Confirm the IdP emits the claim for this application and that the user has the expected assignment. For Entra app roles, check **Enterprise application → Users and groups**. Allow ~1 minute for propagation. | | Existing user's role did not change after adding mapping | User has a role from a previous login; the mapping applies on the next login | Have the user log out and log back in; the mapping is evaluated on every login. | | Role reverts to `user` after next login | Mapping is applied on every login and overwrites the stored role | This is expected. Manage role changes via Entra assignments, not manual Agent Router edits. | ## Appendix a: control who can log in with assignment required By default, any user in the Entra tenant can sign in to Agent Router via SSO. Users without an app role assignment land in the `user` role and have access to standard Console features but no administrative surface. For deployments where Agent Router should be visible only to an explicit allow-list, Entra's **Assignment required** setting locks the application down at the directory level, before any token ever reaches Agent Router. To restrict access so that **only assigned users can log in**: 1. Open the **Enterprise application** for the Agent Router app. 1. Click **Properties** in the left sidebar. 1. Set **Assignment required** to **Yes**. 1. Click **Save**. The combinations of Assignment required and role assignment produce the following outcomes: | Assignment required | User has a role assigned | Result | | :---- | :---- | :---- | | No | No | User can log in; `roles` claim is empty, falls through to default `user` role in Agent Router | | No | Yes | User can log in; `roles` claim contains their role, mapped in Agent Router | | Yes | No | **Login blocked by Entra**, user never reaches the Agent Router callback | | Yes | Yes | User can log in; `roles` claim contains their role, mapped in Agent Router | When **Assignment required** is enabled, at least one app role must be assigned to every user or group that needs access. A user with no role assignment will be blocked at the Entra consent screen even if they have a corporate email in the allowed domain list, because Entra refuses to issue a token in the first place. :::warning Enable **Assignment required** only after confirming that all current users have an assignment. Flipping this on without checking will lock out existing users immediately, including Agent Router administrators if their accounts are not explicitly assigned a role. ::: ## Appendix b: role mapping precedence and defaults Agent Router does not consider the IdP claim mapping in isolation. Several signals can determine a user's role, and Agent Router evaluates them in a fixed order on every login. The first rule that produces a match wins, and later rules are not consulted. | Priority | Rule | When it applies | | :---- | :---- | :---- | | 1 | **Admin email list**: user's email is in `adminEmails` (SSO provider) or `CORPORATE_ADMIN_EMAIL` (env var) | Always checked first. Use this for emergency super-admin grants. | | 2 | **IdP claim mapping**: `roleClaimPath` resolves to a value that matches a key in `roleMapping` | Applied when the email list does not match. Overwrites the stored role on every login. | | 3 | **Retain existing role**: user already has a role in Agent Router from a previous login | Used when neither rule above matches. | | 4 | **First-user bootstrap**: new user, no `super_admin` exists in Agent Router yet | One-time only; promotes the first user to `super_admin` for fresh deployments. | | 5 | **Default** | New user with no matching rule gets `user`. | The most important consequence of this ordering is that rule 2 runs on every login and **overwrites** the stored role. Manual edits to a user's role in the Admin Dashboard are therefore not durable for any user whose role is also managed via claim mapping: the next login will revert the change. Once a user is brought under claim-driven management, their role should be changed exclusively through Entra assignments. For environments where some users are claim-managed and others are not, the cleanest approach is to omit the unmanaged users from `roleMapping` entirely so that rule 3 keeps their manually assigned role in place. ## Appendix c: app roles vs group claims Agent Router treats app roles and group claims identically once they are present in OIDC claims: `roleClaimPath` selects the claim, and `roleMapping` translates the values inside it to Agent Router roles. The difference is entirely on the identity provider side: in what the IdP emits, how stable those values are over time, and how easy they are to govern. ### Entra app roles assigned to groups For organisations that already use **security groups** to manage application access, Entra app roles still work without needing to flatten everything onto individual users. The group itself can be assigned the app role under **Enterprise application → Users and groups**, and every user who is a member of that group inherits the assignment. Entra populates the `roles` claim with every app role the user inherits through group memberships, plus any direct assignments, in one combined array. The Agent Router-side configuration is unchanged: ```json { "roleClaimPath": "roles", "roleMapping": { "tare-billing-admin": "billing_admin", "tare-super-admin": "super_admin" } } ``` This is usually the cleanest Entra setup for customer environments because app role values are readable, application-specific, and decoupled from the directory's broader group taxonomy. A rename of a security group does not break the mapping, and the value strings that appear in audit logs are immediately recognisable. ### Entra group claims In tenants that already manage application access purely through groups, and prefer not to introduce app roles at all, group claims provide a direct path. Agent Router maps group IDs to roles in exactly the same way, but two Entra-specific behaviours are worth being aware of before committing to this approach. :::important Entra group claims usually emit group object IDs, not display names. Map those IDs unless a provider-specific claim configuration that emits another stable value has been set up. Object IDs are opaque UUIDs, which makes the mapping table harder to read at a glance than the app role equivalent. ::: :::caution If a user is a member of too many groups, Entra may omit the full groups list and emit a group-overage indicator instead of the actual values. Agent Router does not resolve group overage through Microsoft Graph, so users in this state will be treated as if they had no groups assigned for the application. Entra app roles assigned to groups avoid this entirely, because only the roles relevant to the application are ever emitted, and they are the recommended option for the most predictable customer setup. ::: When Entra is configured to emit a `groups` claim for the app, the mapping looks like this: ```json { "roleClaimPath": "groups", "roleMapping": { "00000000-0000-0000-0000-000000000000": "billing_admin", "11111111-1111-1111-1111-111111111111": "super_admin" } } ``` The UUIDs above are placeholders. For Entra group mapping, the group's real **Object ID** is required, not its display name. The Object ID is found in **Microsoft Entra ID → Groups → the group → Overview → Object ID**, or with the Azure CLI: ```bash az ad group show --group "" --query id -o tsv ``` Group claims are a good fit when the identity provider already governs application access entirely through groups and reliably emits the relevant group values in OIDC claims. For Entra specifically, app roles assigned to groups are usually easier to operate over the long run: they avoid exposing broad directory group IDs directly to the application, sidestep Entra's group-overage behaviour, and produce mapping tables that an auditor can read without cross-referencing a group registry. Where to go next Configure SSO The base SSO setup that role mapping builds on: connecting an OIDC identity provider. Data plane installation Bring up the data plane on AWS, Azure, or GCP. --- ## Agent Router Enterprise Guides Enterprise Tier Goal-oriented guides for running Tetrate Agent Router as a shared service: admitting people and mapping what they may do, seeing what the traffic costs and where it goes, and governing the projects, models, spend, and safety controls behind it. Each guide is organised around a single operational goal rather than a single screen, names the persona it is written for, and closes with a pointer to the next related goal. Unless a guide says otherwise, the work is done in the Admin Dashboard by a platform operator. A new deployment is usually taken in roughly this order: establish the **project** that owns models, keys, and gateways; provision the **providers and models** the project routes to; then onboard the **developers** who consume them and map their access from the identity provider. Cost controls, guardrails, telemetry export, and compliance settings are layered on afterwards and can be read in any order as the need arises. Guides for the developers consuming the gateway are collected separately under [Guides for Developers](/agent-router-service/guides/). Identity and access management Connect the identity provider, then decide what the identities it supplies are allowed to do. Configure SSO Connect an OIDC identity provider so people sign in with corporate credentials. Configure SSO role mapping Translate the claims an identity provider sends into Agent Router roles. Map Entra groups to functions Drive access from existing Entra ID groups rather than a parallel membership list. Manage roles and permissions Review the available roles and the permissions each one carries. Observability and analytics Read what the traffic is doing inside Agent Router, then send the same signals to the observability stack the organisation already runs. Usage Analytics Onboarding Get the first useful readings out of the analytics, from initial insights to profiling agent efficiency. Export telemetry to observability Send traces and metrics to Grafana, Datadog, New Relic, or another OpenTelemetry consumer. Custom observability backends Point the exporters at a backend that is not one of the presets. Operate and govern The largest group, covering the day-to-day operation of the service and the controls placed around it. Each area below collects several goals; the individual goals are listed together under [Guides for Platform Operators](/agent-router-enterprise/guides/operate-and-govern/), which also carries the standalone tasks such as onboarding developers, governing MCP server access, and auditing activity. Project and data plane management Create the isolation boundary that owns models, keys, and gateways, then provision and operate the data planes that host them. Provision models and providers Configure provider connections, enable the models developers route to, and grant them to a project. Cost and quota management Pick between budgets, rate limits, and attribution tags, then hold spend inside a ceiling and bill it back. Safety and Data Protection Enforce guardrails inline: redact sensitive data, block prompt injection, and protect coding assistant traffic. Compliance and Governance Set data residency and retention, and export audit and policy decisions to a SIEM. Before the guides apply These guides assume a running deployment. Where one does not exist yet, start with the installation path that matches the deployment model. Agent Router Enterprise Fully Managed Onboarding for the Tetrate-hosted management plane, with no data plane to run. Agent Router Enterprise Self-Hosted Data Plane Install and operate the data plane and gateways in a customer-managed Kubernetes cluster. --- ## Custom observability backends Enterprise Tier The data plane runs its own OpenTelemetry collector, and everything the management plane learns about traffic passes through infrastructure Tetrate operates. Many organisations need a second copy — the same telemetry, delivered to systems they run themselves: a SIEM that must see every guardrail firing, a logging stack that must hold full request content for compliance, a metrics platform that already drives the on-call rotation. The global.observability.backends.custom[] Helm values list is the single mechanism for all of it. Each entry attaches one OTLP destination to the data-plane collector and subscribes it to a chosen subset of signals; the collector then fans each signal out to every subscribed backend, independently of what does or does not travel to the management plane. Configuration is deliberately GitOps-only — adding a backend is a reviewed infrastructure change by the cluster owner, not a dashboard click — because two of the signals carry full prompt, response, and matched-content data. **Persona:** Platform operator who owns the data-plane Helm values (usually through a GitOps flow), typically in coordination with security or compliance stakeholders for the content-bearing signals. **Estimated time:** 15 to 30 minutes to add and verify a backend; longer if the receiving side (network reachability, credentials) still has to be arranged. ## When this guide applies | Situation | What a custom backend adds | | :---- | :---- | | A SIEM or logging stack must receive every guardrail trigger with full matched content | Subscribe a backend to `guardrailtrigger`; the copy is always full content, regardless of the dashboard content-mode setting | | Full request/response bodies must land in a self-managed store, possibly while the management plane receives less | Subscribe a backend to `requestlog` | | Data-plane metrics, traces, or logs should also feed an existing observability platform | Subscribe a backend to any of the four legacy signals | | One signal must reach several destinations | Add several entries; every subscribed backend receives its own copy | Two adjacent surfaces are **not** this mechanism: trace export configured from the Developer Console (see [Export telemetry to observability](/agent-router-enterprise/guides/observability-and-analytics/export-telemetry-to-an-observability-stack/)) and the management-plane request-log modes set in the Admin Dashboard (see [Control request log data](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/configuring-request-logs/)). Custom backends are configured in Helm values only; there is no dashboard or API surface for them. ## Outcomes By the end of this guide: - Every `backends.custom[]` field is understood, including the explicit-subscription rule for the content-bearing signals. - A backend is subscribed to `requestlog` and/or `guardrailtrigger` and verified receiving records. - The delivery guarantees — sized in-memory queues, drop-on-overflow, bounded retry — are understood well enough to size the queue and pick a protocol deliberately. - The relationship between the always-full collector copy of guardrail triggers and the dashboard's management-plane content-mode setting is clear. ## How the fan-out works The data-plane cluster runs a local OpenTelemetry collector (`otel-gateway`, a stateless two-replica Deployment in the system namespace). Data-plane components push their telemetry to this collector, and the collector forwards it: management-plane-bound copies travel on their own pipelines, and each `backends.custom[]` entry adds one more OTLP exporter that joins the pipelines of the signals it subscribes to. The management-plane copies are unchanged by any backend you add. Six signals exist, in two groups with different subscription semantics: | Signal | Content | Subscription | | :---- | :---- | :---- | | `metrics` | Data-plane metrics | Default set | | `traces` | Request traces | Default set | | `logs` | Application logs of the data-plane components | Default set | | `accesslog` | Envoy access logs (requires the access-log lane to be enabled with an endpoint: `global.observability.accessLog`) | Default set | | `requestlog` | **Full request-log records, including request and response bodies** | **Explicit only** | | `guardrailtrigger` | **Full guardrail-trigger records, including matched and sanitized message content** | **Explicit only** | **The explicit-subscription rule:** a backend whose `signals` list is omitted or empty receives the four legacy signals — `metrics`, `traces`, `logs`, `accesslog` — and **never** `requestlog` or `guardrailtrigger`. The content-bearing signals only flow to a backend that names them, for example `signals: [requestlog]`. This is a deliberate privacy guardrail: upgrading a chart, or adding a catch-all backend, can never silently start shipping full prompt and response content somewhere new. For the two content-bearing signals, the chart wires the entire path from one predicate — `global.observability.enabled` AND `signals..enabled` AND at least one subscribed backend. When it holds, the collector renders a dedicated receiver and pipeline for the signal **and** the emitting component is configured (via chart-injected environment variables on the egress pod) to push to it; when it does not hold, both disappear together and nothing is emitted. There is nothing to enable by hand on the emitting side — subscribing a backend is the switch. Each signal that a backend does not subscribe to, and each signal whose `signals..enabled` toggle is off, simply does not flow to it; a backend subscribed to a disabled signal keeps its exporter but receives nothing until the signal is enabled. ## The `backends.custom[]` fields A complete entry, with every field: ```yaml global: observability: backends: custom: - name: siem # required endpoint: otel.security.example.com:4317 # required protocol: otlp # otlp (default) | otlphttp signals: [guardrailtrigger] # omitted/empty = metrics, traces, logs, accesslog compression: none # omitted = gzip tls: insecure: false insecureSkipVerify: false auth: headers: Authorization: { secretKey: SIEM_AUTH_HEADER } # from the credentials Secret X-Scope-OrgID: { value: acme } # non-secret literal sendingQueue: { queueSize: 64, numConsumers: 4 } ``` ### `name` (required) Lowercase letters, digits, and hyphens (`[a-z0-9-]`). The name becomes part of the collector exporter's identity (`otlp_grpc/custom-` or `otlphttp/custom-`), which is how the backend appears in the collector's own logs and metrics — pick something that will make sense in a 3 a.m. dashboard. ### `endpoint` (required) Where the backend listens. The expected shape depends on `protocol`: - `protocol: otlp` (OTLP/gRPC): host and port, for example `otel-collector.observability.svc.cluster.local:4317`. - `protocol: otlphttp` (OTLP/HTTP): a full `http(s)://` URL. ### `protocol` `otlp` (OTLP/gRPC, the default) or `otlphttp` (OTLP/HTTP). Choose `otlphttp` in two situations: 1. **Oversized single records.** gRPC receivers commonly cap messages at 4 MiB by default, and a single record larger than the receiving side's cap can *never* be delivered over gRPC — it is retried for the full retry budget and then dropped. Request-log records for large prompts and responses can reach low-single-digit MiB, so a `requestlog` subscriber whose records may approach its receiver's cap should either raise the cap on its receiver or subscribe over `otlphttp`, which has no 4 MiB framing default. 1. **Intermediate infrastructure.** Load balancers and proxies between the data plane and the backend that do not pass gRPC through cleanly. OTLP/HTTP traverses ordinary HTTP infrastructure. ### `signals` The subset of signals to ship, from the six in the table above. Omitted or empty means the four legacy signals and never the content-bearing two — see the explicit-subscription rule. ### `compression` Omit for the collector default (gzip). Set `compression: none` for backends that do not accept gzip-compressed OTLP. ### `tls` `tls.insecure: true` disables TLS entirely (plaintext — in-cluster hops only). `tls.insecureSkipVerify: true` keeps TLS but skips certificate verification. Leave both unset for full TLS verification against system CAs. ### `auth.headers` Headers attached to every export request. Each header takes one of two value forms: - `{ value: }` — a non-secret literal, rendered into the collector config as-is. - `{ secretKey: }` — the value is read from the key `` of the well-known Kubernetes Secret **`otel-gateway-backend-credentials`** in the collector's (release) namespace. Keys must be valid environment-variable names (`[A-Za-z_][A-Za-z0-9_]*`), because the Secret is loaded onto the collector pod as environment variables. Create the Secret manually, with External Secrets Operator, sealed-secrets, or any equivalent — the Secret is optional, so the collector starts fine when no backend needs credentials. ```yaml apiVersion: v1 kind: Secret metadata: name: otel-gateway-backend-credentials namespace: stringData: SIEM_AUTH_HEADER: "Bearer " ``` ### `sendingQueue` Per-backend sizing of the exporter's in-memory queue: `{ queueSize, numConsumers }`. **`queueSize` counts batches, not bytes or records**, so its memory bound is `queueSize × the pipeline's batch cap`. The collector default (1000) is fine for the low-volume signals, but on the `requestlog` pipeline — where a batch can reach ~4 MiB — the default would bound roughly 4 GiB of queue memory, past the collector pod's 2 Gi limit. **Size the queue down on every `requestlog` subscriber**; `queueSize: 64` bounds about 256 MiB. The collector's memory limiter (75 % of the pod limit) is the backstop, but a deliberately sized queue fails cleanly (dropping the oldest data for one backend) instead of pressuring the whole collector. ## Worked example: request logs to a logging stack The scenario from [Control request log data](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/configuring-request-logs/): full request-log records, bodies included, must land in a self-managed logging stack. ```yaml global: observability: enabled: true signals: requestlog: enabled: true # chart default backends: custom: - name: request-log-sink endpoint: otel-collector.logging.svc.cluster.local:4317 signals: [requestlog] # explicit — never implied tls: { insecure: true } # in-cluster plaintext hop sendingQueue: { queueSize: 64 } ``` Applying this makes the chart render the collector's dedicated request-log lane (an OTLP/gRPC receiver on port 3501, raised message-size cap, plus an OTLP/HTTP receiver on 3502) and configure the gateway's data-plane module to push request logs to it over gRPC. Verify end to end: 1. Confirm the emission wiring and the lane exist: ```shell kubectl get envoyproxy -n -o yaml | grep -A1 OTEL_ kubectl get cm otel-gateway-config -n -o yaml | grep -A5 requestlog ``` 1. Send a prompt through the gateway (the MANGO smoke test from the request-logs guide works well). 1. At the backend, expect one record per request: the body is the readable request-log JSON (`request_id`, model, token and cost fields, request and response bodies inline), record attributes include `event.type: request_log` plus `gen_ai.*` and `tars.*` dimensions, and the resource carries `log.type: request_log`. Remove the `signals: [requestlog]` subscription and the whole lane — receiver, pipeline, and module emission — disappears together on the next upgrade. ## Worked example: guardrail triggers to a SIEM Security wants every guardrail firing, full matched content included, in the corporate SIEM — while the management plane keeps only metadata (see the content-mode section below for why those two settings are independent). ```yaml global: observability: enabled: true signals: guardrailtrigger: enabled: true # chart default backends: custom: - name: siem protocol: otlphttp # SIEM sits behind an HTTP load balancer endpoint: https://otel.security.example.com/otlp signals: [guardrailtrigger] auth: headers: Authorization: { secretKey: SIEM_AUTH_HEADER } ``` Create the `otel-gateway-backend-credentials` Secret with the `SIEM_AUTH_HEADER` key as shown above, apply the values, then fire a deterministic rule — for example a `regexp` guardrail matching `\d{3}-\d{2}-\d{4}` (SSN-shaped) in enforce mode — and send a request through the gateway containing a matching string. One record per rule evaluation arrives at the SIEM; the record shape is specified in the next section. ## Guardrail-trigger destinations ### The collector copy is always full content Guardrail triggers exist in two copies with different owners: - **The management-plane copy** — what the dashboard's trigger views show — is scoped by the dashboard content-mode setting described below. - **The collector fan-out copy** — what subscribed custom backends receive — is **always full content**. The content-mode setting does not touch it. `Off` stops the management-plane copy entirely; a subscribed backend still receives every trigger, matched messages included. This is deliberate. The fan-out exists precisely so an organisation can hold full trigger content in its own systems while independently throttling what leaves for the management plane; coupling the two would make the strictest privacy setting silently break the customer's own security pipeline. The consequence for auditing is equally deliberate: **`backends.custom[]` subscriptions, not the dashboard content modes, are the authority on where full content goes.** The only ways to stop full triggers reaching a backend are to remove its `guardrailtrigger` subscription or disable the signal. ### The dashboard setting scopes only the management-plane copy In the Admin Dashboard, **Settings → Guardrails**, the **Guardrail triggers** card sets "What guardrail trigger data gets sent to the management plane": | Mode | Management-plane copy | Collector fan-out copy | | :---- | :---- | :---- | | **Full** | Matched prompt and output text, including sanitized messages, is stored on the management plane. | Full content | | **Metadata only** | Scores, check types, actions, durations and ids are stored; matched message content is not sent to the management plane. | Full content | | **Off** | No trigger records reach the management plane; dashboard trigger views are empty. | Full content | The card itself says so: "Additional observability destinations are configured by your platform team via helm values" — that sentence points here. ### The record consumers see Each rule evaluation produces **one OTLP log record**. A trigger in which three rules evaluated emits three records that share the trigger's `id` in the body, so a backend can treat each record as a self-contained per-rule event (the natural unit for filtering and alerting) or re-correlate the whole trigger on `id` without unpacking arrays. **Body** — a camelCase JSON object: - The trigger's common fields at the top level: `id`, `triggeredAt`, `guardrailId`, `consumerId`, `resourceId`, `executionDurationMs`, `metadata`, `customerId`, `projectId`. - `requestId` — the originating request's `x-request-id` (omitted when the request carried none). - A single `ruleEvaluation` object for this record's evaluation: `guardrailRuleId`, `score`, `action`, `mode`, `executedOn`, `checkType`, `providerId`, `executeOnContentType`, `executionDurationMs`, and the `inputResults` / `outputResults` arrays with `message` / `sanitized_message` content. **Record attributes** — for filtering and aggregation without parsing the body: | Attribute | Value | | :---- | :---- | | `event.type` | `guardrail_trigger` | | `tars.request.id` | The request's `x-request-id` — the **same key stamped on request-log records**, so triggers and request logs join on one attribute across backends | | `tars.customer.id`, `tars.project.id`, `tars.id`, `tars.user.id`, `tars.gateway.id`, `tars.dataplane.id` | The TARS identity context, using the same keys as the platform's guardrail metrics, so logs and metrics slice on the same dimensions (empty values omitted) | | `tars.guardrail.id`, `tars.guardrail.rule.id`, `tars.consumer.id` | Guardrail, rule, and consumer identity | | `gen_ai.provider.name`, `gen_ai.request.model` | Provider and model, per GenAI semantic conventions | | `tars.guardrail.rule.action`, `tars.guardrail.rule.mode` | The rule's configured action and mode, as enum names (e.g. `RULE_ACTION_REDACT`) | | `tars.guardrail.input.redacted`, `tars.guardrail.output.redacted` | Booleans, **derived rather than parroted from config**: true only when the action is REDACT, the mode actually enforced it, and the sanitized message is a real rewrite of the original | **Severity** is `ERROR` exactly when the evaluation actually blocked the request or response (a BLOCK action in an enforcing mode); everything else — including a BLOCK rule in monitor mode — is `INFO`. Alert on severity, not on the action attribute, to page only on requests that were actually stopped. **Resource attributes**: `log.type: guardrail_trigger`, `service.name: aidiscovery`. The record timestamp is the trigger's `triggeredAt`. On the emitting side, trigger export rides a bounded in-memory queue (2 048 records, oldest dropped on overflow, batches of up to 512 flushed every second, 30 s export timeout): a slow or unreachable collector drops trigger copies after bounded retry and can never stall guardrail evaluation or management-plane reporting. ## Delivery semantics: what is and is not guaranteed Fan-out delivery is engineered to protect the traffic path and the sibling backends, not to guarantee delivery to any single backend. Concretely, per hop: | Hop | Guarantee | Where data can be lost | | :---- | :---- | :---- | | Emitting component → collector | OTLP request/response; the sender sees failures and retries within its own bounded buffer | Records emitted during a full collector outage are lost to the fan-out path (management-plane copies travel separately and are unaffected) | | Collector receiver → per-backend queue | Non-blocking enqueue into a **sized, in-memory** queue | **Overflow drops** (counted in the collector's own metrics) rather than blocking the pipeline — a slow backend never backpressures ingest or the sibling backends; a collector crash loses whatever was queued | | Per-backend exporter → backend | Retry with backoff: 5 s initial, 30 s max interval, up to 300 s total per batch | After the 300 s budget the batch is **dropped**. A single record larger than the receiver's message cap can never succeed and is dropped after the same budget — the fix is `protocol: otlphttp` (or raising the receiver's cap), not more retries | The queues are in-memory by design: the collector is a stateless Deployment, and pretending to offer restart durability would be a promise the architecture cannot keep. If a backend must not miss records across collector restarts, put a collector or queue you operate (with its own persistence) close to the data plane as the subscribed backend, and let it forward onward. Batching on the request-log pipeline is byte-aware: small batches (up to 8 records, ~4 MiB typical, flushed every 2 s) so that assembled batches stay under common gRPC frame caps on the receiving side. ## Frequently asked questions ### Can the dashboard's request-log or trigger content modes reduce what a custom backend receives? No. Both content modes scope only the management-plane copy. The collector fan-out copies of `requestlog` and `guardrailtrigger` are always full content for every subscribed backend — that is the contract, and subscriptions are therefore the thing to review when auditing data flows. ### Why does my backend receive metrics and traces but no request logs? Almost certainly the explicit-subscription rule: an omitted `signals` list means the four legacy signals only. Add `signals: [requestlog]` (plus whatever else the backend should receive — an explicit list replaces the default set, so name the legacy signals too if they should keep flowing). ### Does adding a backend change what the management plane receives? No. Management-plane copies travel on their own pipelines and transports; `backends.custom[]` entries only add destinations. ### The collector logs show queue-overflow drops for one backend. What do I do? That backend is slower than the signal's ingest rate. Raise its `sendingQueue.queueSize` (respecting the batch-size arithmetic above), raise `numConsumers`, or fix the backend's throughput. The drops are confined to that backend; siblings and the management-plane copies are unaffected. Where to go next Control request log data The management-plane content modes, and the collector-routed path for shipping full request logs to your own backend. Protect requests with guardrails The guardrail model behind the trigger records this guide ships to your backends. --- ## Export telemetry to observability The Console's Request Logs and Usage Analytics surfaces are sufficient for almost any single-question investigation. They are not, however, sufficient for the questions an observability team is paid to answer at three o'clock in the morning. Those questions live in dashboards that already exist, alerting rules that already page the right people, long-term storage that already complies with the right retention policies, and correlation models that already join AI traffic to the rest of the application stack. Recreating any of that inside a vendor-specific console would be both wasteful and politically unviable. Agent Router's answer is to emit telemetry in open formats and let the organisation's existing observability stack absorb it. Two integration paths are supported, and they are complementary rather than competing. OpenTelemetry trace export sends per-request traces with rich attributes to any OTLP-compatible backend: Jaeger, Datadog, Grafana Tempo, Honeycomb, New Relic, or a self-hosted collector. Prometheus metrics are exposed on a scrape endpoint by each data plane component and can be pulled by any metrics agent, or transformed into OTLP metrics by a Grafana Alloy (or similar) collector. Most production deployments use both: traces for per-request investigation, metrics for dashboards and alert rules. This guide covers configuring trace export, what to expect from the scrape endpoint, and how the result composes with the alerting and storage that the observability team already runs. **Persona:** Developer working in the Developer Console, in collaboration with whoever owns the organisation's observability stack. **Estimated time:** 20 to 30 minutes for the trace-export configuration, plus whatever time is required to confirm the traces land in the destination platform. ## When this guide applies Telemetry export is the right next step in any of these situations: | Situation | What export adds over the in-Console surfaces | | :---- | :---- | | Dashboards and alerts already exist in Datadog, Grafana, Honeycomb, New Relic, or similar | The same dashboards and alerts can include AI traffic alongside the rest of the application stack | | AI traffic needs to be correlated with downstream service behaviour | Trace context flows through to the observability backend, where the join across services is trivial | | Long-term retention beyond the in-Console window is required | Storage is whatever the observability platform already provides | | Alerting on elevated error rates, latency spikes, or rate-limit saturation is required | Alerts are configured in the alerting platform the team already operates | | Compliance or SIEM workflows depend on a specific log format | The OTLP and Prometheus formats integrate with the standard tooling these workflows already use | For ad-hoc investigation by a single developer, the Console's Request Logs and Usage Analytics are still the right surfaces; export is the layer that scales beyond that. The two surfaces remain available regardless of whether export is configured. ## Outcomes By the end of this guide: - Trace export to an OTLP-compatible backend is configured, enabled, and reporting a **Connected** status. - At least one request issued through the gateway has produced a trace in the destination observability platform. - The Prometheus scrape endpoint exposed by each data plane component is understood, even where it is not yet wired into the metrics collection stack. - The boundary between platform-emitted telemetry and observability-platform-owned concerns (storage, alerting, retention) is clear. ## Prerequisites - A working API key with traffic flowing through it, as set up in [Route Requests Across Providers](/agent-router-service/guides/route-requests-across-providers/). - An OTLP-compatible observability backend. The endpoint URL, transport protocol (gRPC or HTTP), and authentication credentials must be obtained from whoever owns that backend. - Confirmation that the backend's OTLP endpoint is reachable from the Agent Router data plane. This is a network reachability question, not a Console question; if there is doubt, the operator who installed Agent Router is the right person to ask. ## Step 1: configure trace export Trace export is configured once per Agent Router instance, from inside the Console. The configuration applies to all traffic flowing through the data plane; individual API keys do not need any further setup. 1. Sign in to the Developer Console. 1. Open **Settings → OpenTelemetry Export** from the sidebar. The same page is reachable at `/system-integrations`. 1. Click **Configure** or **Enable Export**. ![OpenTelemetry Export settings](/img/screenshots/console/otel-export.jpeg) 1. Enter the **Endpoint URL** of the OTLP collector. Examples by backend: | Backend | Typical endpoint | Default protocol | | :---- | :---- | :---- | | Jaeger | `http://jaeger:4317` | gRPC | | Grafana Tempo | `http://tempo:4317` | gRPC | | Datadog | `https://trace.agent.datadoghq.com` | HTTP | | Honeycomb | `https://api.honeycomb.io` | gRPC with API key | | New Relic | `https://otlp.nr-data.net:4317` | gRPC with API key | | Self-hosted OTLP collector | Collector address and port | Backend-specific | 1. Select the transport **Protocol**: - **gRPC** is the most common choice and is the default on port 4317. - **HTTP** uses OTLP over HTTP on port 4318 and is the better choice when gRPC is blocked by intermediate infrastructure. 1. Configure **Authentication**. The right option depends on the backend: | Auth mode | When to use it | | :---- | :---- | | None | Internal endpoints on a trusted network | | Bearer Token | OTLP collectors that accept a static bearer token | | Basic Auth | Backends that require username and password (rare for OTLP) | | API Key | Hosted backends such as Honeycomb, New Relic, and Datadog. The header name is often vendor-specific and configurable here | | Custom Headers | Backends that require one or more arbitrary headers, for example, `DD-API-KEY` for Datadog | 1. Optionally set the **Service name** under which traces appear in the destination backend. Left unset, spans arrive under the data plane's own service name, `ai-gateway-extproc`. A more specific value is useful in tenants that run multiple Agent Router instances (`agent-router-prod`, `agent-router-staging`) so the destination dashboards can distinguish them. 1. Optionally set the **Sampling rate**. 100 % is appropriate for evaluation and low-volume environments; high-volume production typically samples lower. 1. Click **Save**. 1. Enable the export toggle. 1. Confirm the status indicator transitions to **Connected**. ![OpenTelemetry Export active](/img/screenshots/console/otel-export-active.jpeg) The status indicator is the most useful signal during initial setup. It reports one of three states: | Status | Meaning | | :---- | :---- | | Connected | Traces are being exported successfully | | Disconnected | Export is disabled or not yet configured | | Error | Export is enabled but the endpoint is unreachable or returning errors | A persistent **Error** state usually points to one of three issues: the endpoint is unreachable from the data plane (network or DNS), the protocol does not match the collector's expectation (gRPC vs HTTP), or the authentication credentials are wrong. Each of these is checkable independently. The configuration can be revised at any time from the same page. Changing the endpoint URL, protocol, or authentication settings and clicking **Save** applies the new values immediately; no restart of the data plane is required. Toggling the export switch off stops trace export immediately while preserving the saved configuration, so export can be re-enabled later without re-entering the endpoint and credentials. ## Step 2: generate traffic and inspect the destination Once export is active, every gateway request produces a trace. A small burst of test traffic is enough to confirm the pipeline end-to-end. `PROXY_URL` stands for the gateway URL for the deployment with `/v1` appended, the same value the Console shows as the proxy endpoint. ```bash for i in {1..5}; do curl -s PROXY_URL/chat/completions \ -H "Authorization: Bearer YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "gpt-4o", "messages": [{"role": "user", "content": "Generate trace data."}] }' > /dev/null done ``` Then, in the destination observability platform: 1. Search for traces under the service name that the data plane reports. Spans are emitted by the ai-gateway extproc, which reports `ai-gateway-extproc` unless the deployment overrides it. 1. Open a recent trace. 1. Confirm one span per request is present, named for the endpoint that served it. The five requests above are chat completions, so five spans named `ChatCompletion` are expected. Each request produces a single flat span. There are no gateway, routing, or inference child spans, and the Envoy hop is not instrumented, so no proxy span accompanies it. Where the calling client is itself instrumented, the span attaches to the client's trace through W3C trace context rather than appearing as a trace root. 1. Confirm the expected attributes are populated on the span: - `llm.model_name`, the model that served the request. - `llm.token_count.prompt`, `llm.token_count.completion`, and `llm.token_count.total`. - `openinference.span.kind`, set to `LLM`. - `request.id`, the per-request correlation ID. - `input.value` and `output.value`, subject to the active logging mode. Request latency is read from the span's own duration; it is not stamped as an attribute. Failed calls carry an `exception` event with `exception.type` and `exception.message` rather than an HTTP status attribute. The full span model, including every attribute and event and the fields that are deliberately absent, is enumerated in [OpenTelemetry metrics and attributes](/reference/otel-metrics/). If traces appear in the destination but with missing attributes, the integration is working but the receiver may be filtering or sampling on its own side; the destination platform's documentation is the right place to investigate. If traces do not appear at all and the Console status indicator is **Connected**, the most common cause is a service-name filter or a search range that excludes the test traffic. ### OpenInference semantic conventions On top of the OpenTelemetry (OTLP) export described above, the gateway annotates LLM spans with OpenInference semantic conventions, the open standard for large-language-model telemetry attributes. Base OTLP defines how a span is structured and transported, but it says nothing about what an LLM call means. OpenInference fills that gap with a shared vocabulary for model, token, and prompt/response attributes, so that LLM-aware observability tools can interpret the spans without custom parsing. What this convention adds beyond base OTLP is a set of LLM-specific span attributes, including: - The span kind (`openinference.span.kind`), which marks the span as an LLM, embedding, or reranker operation. - The model name and the invocation parameters supplied with the request. - Prompt and completion token counts captured per span, including cache-read, cache-write, and reasoning subtotals where the provider reports them. - The input and output of the call, captured per span subject to the active logging mode; prompt and response content appears only when the logging mode permits it. Token counts are treated as metadata and are recorded regardless. See [Configure request logs](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/configuring-request-logs/) for how the logging mode governs whether request and response bodies are recorded. Because these attributes follow a published standard rather than a Agent Router–specific schema, traces become meaningful in any tool that understands the convention; model, token, and prompt/response fields are recognised and rendered without bespoke mapping. The exact attribute set is enumerated in the reference; see [OpenTelemetry metrics and attributes](/reference/otel-metrics/) for the full list. ## Step 3: add Prometheus metrics to the pipeline Each data plane component exposes a Prometheus-compatible metrics endpoint, which is the second integration path. This is independent of trace export; metrics can be enabled regardless of whether traces are being exported, and the two together give a substantially fuller picture than either alone. Agent Router does not include a metrics collector itself. Instead, an existing metrics agent (Prometheus, a Grafana Alloy collector, the Datadog agent in OpenMetrics mode, or any other scraper) pulls metrics from Agent Router's scrape endpoints and forwards them to whichever destination the observability stack uses. A reference Grafana Alloy configuration is supplied that scrapes the data plane components and forwards the metrics as OTLP to a downstream collector; it is a useful starting point even for teams that ultimately route metrics through a different agent. Two consequences of this design are worth noting: - **The scrape endpoint is internal to the data plane** and is reachable from inside the same Kubernetes cluster. Exposing it outside the cluster is a deployment decision rather than a Console one. - **Metrics and traces share enough dimensions to be joined** in the destination platform. The model is the most reliable join key, appearing as `llm.model_name` on spans and as a model label on metrics. Where per-request correlation is needed, the `request.id` span attribute is the right handle, since it is the same correlation ID carried by Request Logs. ## Alerting and storage: where each one belongs Two operational concerns are deliberately delegated to the observability platform rather than handled inside Agent Router. ### Alerting Agent Router emits metrics with the dimensions an alerting rule needs (error rate, latency, rate-limit indicators, model, provider, and key) but it does not drive alerts itself. Alerting rules live in the observability platform, where they can be combined with rules covering the rest of the application stack, escalated through the same paging chains, and tuned with the same noise-reduction tooling. Suppressing alerts for invalid payloads, expected policy denials, or scheduled maintenance windows happens in the alerting platform's rule engine rather than in the gateway. The benefit of this split is that an organisation that already has a mature alerting setup does not have to maintain a second, vendor-specific one inside Agent Router. The cost is that a fresh deployment that does not yet have such a setup will need one before alerting on AI traffic is feasible. ### Storage For trace data, the destination observability platform owns retention, indexing, and query performance. For in-Console request logs, Tetrate manages the underlying storage infrastructure on the management plane (capacity, index health, and query performance) so there is no storage management burden on the customer side. The implication is that the in-Console surfaces are the right place for "last 90 days" investigations, while the exported telemetry is the right place for longer-horizon retention, compliance archiving, and SIEM workflows. ## What to do next The dev guides have now covered the end-to-end developer surface: foundational routing, resilience, cost management, advanced patterns, BYOK, MCP, integrations, the Playground, in-Console monitoring, and external telemetry. From here: - **Govern MCP server access** (operator side): the natural companion to *Aggregate MCP Servers into a Profile*. See [Govern MCP server access](/agent-router-enterprise/guides/operate-and-govern/govern-mcp-server-access/) in the Admin Dashboard. - **Audit Agent Router activity** (operator side): the operator equivalent of the monitoring surfaces covered in this and the previous guide. See [Audit Agent Router activity](/agent-router-enterprise/guides/operate-and-govern/audit-platform-activity/). Where to go next Monitor traffic and usage The in-Console request log and usage surfaces that complement exported telemetry. Govern MCP server access The operator-side companion for controlling what MCP traffic Agent Router exposes. --- ## Level 1: First insights in ten minutes Ten minutes on a timer: traffic in, insight out. The first level of the usage analytics onboarding path proves that requests sent through the gateway become visible, costed, and explainable within minutes. It is played against the clock, and the clock is part of the result: time-to-first-insight is one of the numbers an evaluation exists to produce. **Persona:** Developer or engineering leader working in the Developer Console. **Estimated time:** 10 minutes, played against a timer. **Progression:** Level 1 of 3 in the usage analytics onboarding path. Completing it unlocks [Level 2: Read the shadow bill](/agent-router-enterprise/guides/observability-and-analytics/usage-analytics-onboarding/read-the-shadow-bill/). ## The rules - A timer starts at Step 1 and runs until the checklist at the bottom is fully ticked. - Each step has a minute budget and ends with a **time check**. A failed time check is not a failure of the run; note where the time went, because that observation is itself an evaluation finding. - The elapsed time of the first full run is the **personal best**. Every later run tries to beat it. ## Outcomes By the end of this level: - A burst of requests has been sent and located in Request Logs within minutes of sending. - The cost, token counts, and latency of a single request have been read from its detail panel and explained. - The most expensive request of the run has been identified, with the reason articulated in one sentence. ## Prerequisites - A working API key, as issued in [Make an API Call](/agent-router-service/quickstarts/make-an-api-call/). - A way to send a handful of requests: a terminal with curl, an SDK snippet, or a coding agent pointed at the gateway. - A timer. ## Step 1 (minutes 0 to 2): open the scoreboard 1. Start the timer. 1. Sign in to the Console. 1. Open **Monitoring → Request Logs** in one browser tab and **Monitoring → Usage** in another. These two pages are the scoreboard for the rest of the onboarding path; everything the next two levels measure is read from them. **Time check, minute 2:** both pages are open, showing either existing traffic or an empty state. ## Step 2 (minutes 2 to 5): send a burst 1. Send 5 to 10 requests through the gateway: a short script loop, a few Playground exchanges, or one small task given to a coding agent. 1. Vary the prompts; identical requests make the later steps less interesting. 1. Refresh Request Logs. **Time check, minute 5:** the burst appears at the top of Request Logs, newest first. ## Step 3 (minutes 5 to 8): read one request end to end 1. Click any row from the burst to open its detail panel. 1. Read, in order: the resolved model, input and output token counts, the cost calculation, and the latency breakdown (time-to-first-token and total). **Time check, minute 8:** the cost of a single request can be stated in dollars and explained as tokens multiplied by the model's per-token pricing. ## Step 4 (minutes 8 to 10): find the big ticket 1. Scan the burst for the request with the highest value in the Cost column. 1. Open it and explain why it cost the most; the longest output is the usual culprit. **Time check, minute 10:** the priciest request of the run is identified, and the reason fits in one sentence. ## Level complete - [ ] The burst was located in Request Logs within minutes of sending. - [ ] One request's cost, tokens, and latency were read and explained. - [ ] The most expensive request was identified and the reason articulated. - [ ] The elapsed time was recorded as the personal best. Bonus challenge: speedrun Run the level again from a cold start (new browser session, fresh burst). A second run under five minutes demonstrates that the workflow, not the learning curve, is what takes the time. Log both times; the gap between them is the onboarding cost for the next person. Where to go next Level 2: Read the shadow bill Put a dollar figure on what subscription traffic would cost at API list prices. Monitor traffic and usage The full reference for Request Logs and Usage Analytics, beyond the timed tour. --- ## Usage Analytics Onboarding A three-level path through the monitoring surfaces, played against a timer. Each level is short, ends with a number worth repeating in a meeting, and unlocks the next: first insights inside ten minutes, then the shadow bill, then an agent efficiency scorecard built to be beaten. Level 1: First insights in ten minutes Send a burst of traffic and explain one request's cost, tokens, and latency, on the clock. Level 2: Read the shadow bill Put a dollar figure on what subscription traffic would cost at API list prices. Level 3: Profile agent efficiency Token splits, cached-request share, and cost per task: a baseline scorecard to beat. The path is also part of a wider evaluation: the [Fast-Track Evaluation](/agent-router-enterprise/fully-managed/fast-track/) includes all three levels in its Day 1 milestone. --- ## Level 3: Profile agent efficiency An agent's bill is a behavior, not a number: how much context it re-reads, how often its requests hit the cache, how many model calls one task really takes. The final level of the usage analytics onboarding path turns one agent's day of traffic into an efficiency scorecard in twenty timed minutes. The scorecard is a baseline, and baselines exist to be beaten; that is what makes this level the one that keeps getting replayed. **Persona:** Developer or engineering leader working in the Developer Console. **Estimated time:** 20 minutes, played against a timer. **Progression:** Level 3 of 3 in the usage analytics onboarding path. Requires [Level 2](/agent-router-enterprise/guides/observability-and-analytics/usage-analytics-onboarding/read-the-shadow-bill/). ## Outcomes By the end of this level: - One agent's traffic has been isolated and profiled: input/output token split, cached-request share, and cost per task. - A baseline efficiency scorecard has been filled in. - One number on the scorecard has been chosen as the target to beat on the next run. ## Prerequisites - Levels 1 and 2 of this path completed. - A coding agent or application running on its own API key, as set up in [Coding Agents and Tools](/agent-router-service/guides/coding-agents/), with at least one working session of traffic behind it. ## Step 1 (minutes 0 to 4): isolate the agent 1. Start the timer. 1. Open **Monitoring → Usage** and filter by the agent's key over the window of its last working session. 1. Note the totals: requests and spend for the session. **Time check, minute 4:** the agent's session is isolated, with its request count and total spend visible. ## Step 2 (minutes 4 to 9): read the token split 1. From the filtered view, compare total input tokens against total output tokens. 1. Open two or three individual requests in **Request Logs** and confirm the pattern in their token breakdowns. Coding agents typically send far more input than they receive as output, because context is resent on every call. The input-to-output ratio is the first efficiency number: a rising ratio means the agent is re-reading more than it is producing. **Time check, minute 9:** the session's input-to-output token ratio is written down. ## Step 3 (minutes 9 to 13): find the cached share 1. In the request detail panels, look at the token breakdown for input tokens served from cache, where the provider reports them. 1. Estimate the share of the session's requests that benefited from caching. Cached input tokens are billed at a fraction of the list price, so the cached share is leverage: the same agent behavior at a higher cache-hit rate is the same work for less money. **Time check, minute 13:** the cached-request share is estimated, even roughly. ## Step 4 (minutes 13 to 17): compute cost per task 1. Count the tasks the agent actually completed in the session (pull requests, tickets, answered questions; whatever the unit of work is). 1. Divide the session's total spend by that count. Cost per task is the number that travels: unlike tokens or requests, it can be compared across agents, models, and weeks, and it is the number a budget discussion eventually lands on. **Time check, minute 17:** cost per task is computed for the session. ## Step 5 (minutes 17 to 20): fill the scorecard Record the baseline: | Measure | Baseline (this session) | Next run | Best | | :---- | :---- | :---- | :---- | | Requests | | | | | Input / output token ratio | | | | | Cached-request share | | | | | Cost per task | | | | | Total spend | | | | Pick one number to beat on the next run and write down the intended lever: a tighter system prompt, better cache reuse, a cheaper model on a routing rule for the easy calls. **Time check, minute 20:** the scorecard's baseline column is full and a target is chosen. ## Level complete - [ ] The agent's session isolated and totalled. - [ ] Input-to-output token ratio recorded. - [ ] Cached-request share estimated. - [ ] Cost per task computed. - [ ] Scorecard baseline filled and a number chosen to beat. Bonus challenges Head-to-head: run the same profile for a second agent or teammate and put the two scorecards side by side; the differences are where the interesting conversations start. Cache detective: find the most-repeated prompt prefix behind the cached share and name which part of the agent's context makes it cacheable. Beat the baseline: improve one scorecard number on the very next session and record what changed. Where to go next Fast-Track Evaluation: The First Week The first-week evaluation this onboarding path belongs to. Catch spend drift early Turn the one-off profile into a recurring usage review across keys and teams. --- ## Level 2: Read the shadow bill Subscription traffic feels free at the margin, which is exactly why nobody can say what it is worth. The gateway prices every request from the resolved model's per-token pricing even when billing goes to a subscription, so traffic routed in passthrough mode accumulates a shadow bill: the amount the same usage would have cost at API list prices. This level surfaces that number in fifteen timed minutes, and it tends to be the moment an evaluation stops being abstract. **Persona:** Developer or engineering leader working in the Developer Console. **Estimated time:** 15 minutes, played against a timer. **Progression:** Level 2 of 3 in the usage analytics onboarding path. Requires [Level 1](/agent-router-enterprise/guides/observability-and-analytics/usage-analytics-onboarding/first-insights-in-ten-minutes/); completing it unlocks [Level 3: Profile agent efficiency](/agent-router-enterprise/guides/observability-and-analytics/usage-analytics-onboarding/profile-agent-efficiency/). ## Outcomes By the end of this level: - Subscription (passthrough) traffic has been isolated on its own key in the monitoring surfaces. - The list-price cost of individual subscription-billed requests has been read from Request Logs. - The shadow bill for a chosen window has been totalled and set against the subscription's flat price. - A month-scale projection has been written down. ## Prerequisites - Level 1 of this path completed. - A personal provider subscription connected in passthrough mode, on its own API key, as set up in [Use Your Own Provider Credentials](/agent-router-service/guides/use-your-own-provider-credentials/). - At least a few hours of real traffic through that key; a working day is better. ## Step 1 (minutes 0 to 3): isolate the subscription lane 1. Start the timer. 1. Open **Monitoring → Request Logs** and filter by the passthrough key. 1. Confirm the traffic shown is the subscription-billed workload and nothing else. The one-key-per-purpose pattern from [Monitor Traffic and Usage](/agent-router-service/guides/monitor-traffic-and-usage/) is what makes this filter meaningful; if subscription and API-billed traffic share a key, split them before continuing. **Time check, minute 3:** the filtered view shows only subscription traffic. ## Step 2 (minutes 3 to 8): price a single free-feeling request 1. Open the detail panel of a recent request. 1. Read its cost calculation: tokens multiplied by the resolved model's per-token pricing. That figure was not billed to anyone; the subscription absorbed it. It is one line of the shadow bill. **Time check, minute 8:** the list-price cost of one subscription-billed request can be stated in dollars. ## Step 3 (minutes 8 to 12): total the window 1. Switch to **Monitoring → Usage** and apply the same key filter. 1. Select the window with real traffic (last 24 hours or last 7 days). 1. Read **Total Spend** for the filtered view: that is the shadow bill for the window. **Time check, minute 12:** the shadow bill for the chosen window is written down, next to the number of requests that produced it. ## Step 4 (minutes 12 to 15): the reveal 1. Scale the window to a month (a 24-hour figure times working days, or a 7-day figure times 4.3). 1. Write the projection next to the subscription's flat monthly price. 1. Record which is larger, and by how much. Either outcome is a finding. A shadow bill above the subscription price quantifies the subscription's value; one below it flags a subscription that may not be earning its fee. **Time check, minute 15:** the sentence "this subscription is worth $X per month against list prices" can be completed with a real number. ## Level complete - [ ] Subscription traffic isolated on its own key. - [ ] One request's list-price cost read and understood. - [ ] The window's shadow bill totalled. - [ ] The monthly projection written next to the subscription price. - [ ] Elapsed time recorded. Bonus challenges Break-even: compute the daily traffic level at which the shadow bill exactly pays for the subscription, and check which side of it the current usage sits on. Streak: log the shadow bill at the end of each workday for five consecutive days; a five-day streak turns a one-off reveal into a trend line, and the trend is what convinces the rest of the team. Where to go next Level 3: Profile agent efficiency Token splits, cached-request share, and cost per task: a baseline scorecard to beat. Use your own provider credentials The passthrough (BYOK) setup this level measures. --- ## Audit Agent Router activity Enterprise Tier Two questions land on a platform operator's desk more reliably than any other: who did this?, usually after a configuration changes unexpectedly, a key disappears, or a model behaves differently than it did yesterday; and what is Agent Router actually doing?, usually before a quarterly review, a cost forecast, or a compliance attestation. Both are answered from the same place: Agent Router's record of activity, surfaced through two complementary screens in the Admin Dashboard. Audit Logs is the immutable per-event record of every administrative action, and Usage Analytics is the aggregated traffic view. This guide covers both, the investigation patterns that combine them, and the compliance posture the audit surface supports. **Audit Logs** is the per-event record. Every administrative action (secret rotation, guardrail change, project membership edit, API key lifecycle event, data-plane change, and similar mutations) is captured with a timestamp, an actor, an `actionType`, the affected resource, and structured context. The log is immutable; nothing in Agent Router can edit or delete an entry once it has been written. **Usage Analytics** is the aggregated traffic view: organization-wide request counts, token totals, costs, and breakdowns by user, model, provider, and API key, with time-range filtering and export. Together they answer the two questions: Audit Logs for "who did what", Usage Analytics for "what is happening". For the full wire schema, see [Audit log events](/reference/audit-log-events/). **Persona:** Platform operator working in the Admin Dashboard, often under time pressure during an incident or a compliance review. **Estimated time:** 10 to 15 minutes for an initial tour; ongoing thereafter. ## When this guide applies This guide is the right surface in any of these situations: | Situation | Surface that helps most | | :---- | :---- | | Investigating a configuration that has changed unexpectedly | Audit Logs, filtered by resource and time range | | Identifying who revoked an API key, or when | Audit Logs, filtered by action type | | Producing a SOC 2, ISO 27001, or other compliance attestation | Audit Logs export, filtered by the relevant time window | | Forecasting cost across the rest of the quarter | Usage Analytics with a long time range | | Identifying the heaviest-consuming user or team | Usage Analytics with the By User breakdown | | Detecting an unexpected spike in spend | Usage Analytics with anomaly inspection, then Audit Logs to correlate to a recent change | For the developer-side equivalents of the same data (per-request investigation and per-key usage), the *Monitor Traffic and Usage* developer guide is the corresponding surface. ## Outcomes By the end of this guide: - The Audit Logs surface has been opened, exercised, and at least one audit entry has been located with the filters. - The Usage Analytics surface has been opened and the user, model, provider, and API key breakdowns reviewed. - A combined investigation pattern ("use Usage Analytics to spot the anomaly, then Audit Logs to find the cause") has been walked through at least once. - The compliance and retention model is understood, including the immutability guarantee on the log. ## Prerequisites - Administrator access to the Admin Dashboard, typically the `super_admin` or `billing_admin` role for Usage Analytics, and `super_admin` for the full Audit Logs surface. - Some recent activity in Agent Router. A freshly-installed Agent Router with no traffic and no administrative changes will surface empty screens that are technically working but uninformative. ## Step 1: open audit logs and orient The Audit Logs surface is the closest Agent Router comes to a single source of truth for what has been changed and by whom. Every administrative action surfaces here, with no editing or deletion possible after the fact. 1. Sign in to the Admin Dashboard. 1. Open **Audit Logs** from the sidebar. ![Audit Logs](/img/screenshots/admin/audit-logs.jpeg) 1. Review the columns surfaced for each entry: | Field | What it carries | | :---- | :---- | | Timestamp | Exact date and time in UTC | | User / Actor | The acting user id (and email when available) | | Action | Wire `actionType` such as `CREATE_SECRET`, `SET_GUARDRAIL_STATUS`, or `DELETE` | | Resource | `resourceType` plus `resourceId`, often with a human-readable label in the UI | | Source | Origin of the request when known | | Result | Success or failure, with HTTP status when available | 1. Note that entries appear in reverse chronological order. The most recent action is at the top of the list. Action names are stable uppercase strings on the wire (`CREATE_SECRET`, `UPDATE_GUARDRAIL`, `ADD_PROJECT_MODEL`), not dotted `.` labels. Common examples: | Action type | Generated when | | :---- | :---- | | `CREATE_SECRET` / `DELETE_SECRET` / `ROTATE_SECRET` | Secret lifecycle | | `SET_GUARDRAIL_STATUS` / `CREATE_GUARDRAIL` / `UPDATE_GUARDRAIL` | Guardrail enablement and edits | | `ADD_PROJECT_MODEL` / `REMOVE_PROJECT_MODEL` | Project model catalogue changes | | `ADD_PROJECT_PROVIDER` / `REMOVE_PROJECT_PROVIDER` | Project provider attachment | | `CREATE_CLIENT_KEY` / `DISABLE_CLIENT_KEY` / `DESTROY_CLIENT_KEY` | API key lifecycle | | `ADD_PROJECT_MEMBER` / `UPDATE_PROJECT_MEMBER` / `REMOVE_PROJECT_MEMBER` | Project membership | | `CREATE_DATAPLANE` / `UPDATE_DATAPLANE` / `DELETE_DATAPLANE` | Data plane lifecycle | | `CREATE_TELEMETRY_CONFIG` / `UPDATE_TELEMETRY_CONFIG` | Telemetry / SIEM forward configuration | See [Audit log events](/reference/audit-log-events/) for the full field list and a broader vocabulary sample. ## Step 2: investigate a specific change Most investigations into the Audit Logs surface come down to a sequence of filter applications. The available filters can be combined; the more specific the combination, the smaller and more useful the result set. 1. Set the **time range** to the window around the moment of interest. A narrower window is almost always more useful than a wider one. 1. Apply the **Actor** filter to a specific user if there is reason to suspect a particular operator's actions. 1. Apply the **Action** filter if the type of change is known, for example `REMOVE_PROJECT_PROVIDER` if a provider detachment is suspected. 1. Apply the **Resource** filter to a specific resource if the affected object is known: a model id, a provider id, a secret name, or an API key id. 1. Open **View** on the resulting entries. `metadata`, request context, and result fields usually identify what changed. A worked example. A developer reports that requests targeting `gpt-4o` started failing yesterday afternoon. To investigate: 1. Set the time range to yesterday afternoon, plus a couple of hours of buffer on either side. 1. Filter on the model or provider resource associated with `gpt-4o`. 1. Review the resulting entries. An `REMOVE_PROJECT_MODEL` or `REMOVE_PROJECT_PROVIDER` action in that window would explain the symptom immediately. 1. If no model-level change shows up, broaden the search by removing the resource filter and looking for related `actionType` values on the same provider in the same window. The same pattern applies to almost every "what changed" investigation: narrow on time first, then on resource or action, then read the entry detail. ## Step 3: open usage analytics and orient Where Audit Logs answers "what changed", Usage Analytics answers "what happened". The two are designed to be used together, and they share the same time-range model. 1. Open **Usage → Analytics** from the sidebar. ![Usage Analytics](/img/screenshots/admin/usage-analytics.jpeg) 1. Review the top-level metrics: | Metric | What it captures | | :---- | :---- | | Total Requests | All API requests in the selected time range | | Total Tokens | Combined input and output token totals | | Total Cost | Estimated cost from model pricing tiers and token totals | | Unique Users | Number of distinct users who issued requests | | Unique Models | Number of distinct models that received requests | 1. Apply a **time range** at the top of the page. Presets cover last 24 hours, last 7 days, and last 30 days; a custom range covers anything longer or more specific. 1. Switch between the available breakdowns: | Breakdown | What it reveals | | :---- | :---- | | By User | Ranked list of users by requests, tokens, or cost; useful for identifying heavy consumers or unusual patterns | | By Model | Which models the workload actually leans on; useful for retiring rarely-used models and for negotiating with providers | | By Provider | Cost and volume across providers; useful for contract negotiation and capacity planning | | By API Key | Per-key request and cost picture; useful for spotting integrations that might need rate limiting | The By User breakdown supports drill-down: clicking a user's bar or table row focuses on that person. To select several users or teams at once and compare them side by side, see [Compare user and team spend in Analytics](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/compare-user-and-team-spend-in-analytics/). The distinction between this surface and the developer-side Usage page in the Console is worth keeping clear: | Feature | Admin Usage Analytics | Console Usage | | :---- | :---- | :---- | | Scope | All users and API keys | The signed-in user's own data | | Breakdowns | User, model, provider, API key | Model, API key | | Cost analysis | Organization-wide cost totals and per-user costs | Individual user costs | | Export | Full Agent Router data export | Personal data export | | Audience | Agent Router administrators | Individual developers | ## Step 4: combine the two surfaces in an investigation The interesting investigations almost always cross both surfaces. The pattern that works most reliably: 1. **Spot the anomaly on Usage Analytics.** An unexpected cost spike on the cost-by-provider chart, an unfamiliar model in the by-model breakdown, an outlier in the by-user list. 1. **Identify the resource.** Filter Usage Analytics down to the specific user, model, provider, or key the anomaly points at. 1. **Switch to Audit Logs.** Apply the same time window plus the resource filter. 1. **Look for an administrative event.** Most surprising traffic patterns are explained by a recent configuration change: a new model enabled, a routing change, a user role modified, or an API key reissued. 1. **Reconstruct the timeline.** The combination of the audit entry and the usage chart shows what changed, who changed it, and the resulting traffic shift. The reverse direction (starting from an audit event and looking for its traffic consequences) is equally common. A recent `ADD_PROJECT_PROVIDER` event makes a new provider show up in the by-provider breakdown; a recent `ADD_PROJECT_MODEL` makes a model appear in the by-model breakdown; these correlations are how the "did the change have the intended effect" question gets answered. ## Step 5: export for compliance and reporting Usage Analytics supports export of the filtered data: 1. Apply the time range and breakdown filters that match the report being produced. 1. Click **Export**. 1. Select the format (typically CSV). 1. The downloaded file contains the metrics for the selected filters and time range. Audit Logs export, where supported by the deployment, follows the same shape. The exported records are the same immutable rows that the screen displays, with all of the audit fields included. ## Compliance posture The audit surface is deliberately built to satisfy the audit and integrity requirements of SOC 2, ISO 27001, and similar frameworks. Three properties are worth being able to assert: - **Immutability.** Audit entries cannot be edited or deleted by any user, regardless of role. There is no "delete from the log" operation anywhere in the Admin Dashboard. - **Coverage.** Every state-modifying administrative action generates an entry. There is no path through the Admin Dashboard that changes Agent Router state without leaving a record. - **Retention.** The retention period for audit data is set by the deployment configuration. Long-horizon retention, including compliance-archival use cases, is supported through export to an external system; the in-Console window is intended for active investigation rather than indefinite storage. ## What to do next - **Configure SSO**: the SSO and role-mapping surface generates audit events that this guide makes visible. See [Configure SSO](/agent-router-enterprise/guides/identity-and-access-management/single-sign-on/sso-configuration-guide/). - **Working with budgets**: budgets track spend and raise alerts, and are visible in Usage Analytics. See [Choose the right cost control for each workload](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/choose-the-right-cost-control/). - **Run multiple Agent Router instances**: multi-instance deployments need to be audited per instance; the audit surface helps confirm that boundary. See [Run Multiple Agent Router Instances](/agent-router-enterprise/guides/operate-and-govern/run-multiple-platform-instances/). The audit and usage surfaces remain available across every subsequent operational activity. Where to go next Configure SSO The SSO surface generates audit events this guide makes visible. Working with budgets Budgets track spend and raise alerts, and are visible in Usage Analytics. --- ## Communicate with Agent Router users Enterprise Tier A system that changes silently is a system that loses trust. A new model becomes available and nobody hears about it; a maintenance window is scheduled and three teams find out by getting paged; a deprecation date passes and an application breaks because the integration team had no idea anything was changing. Most of these incidents are not about the change itself, but about the absence of a clear in-Console communication channel that reaches the people who need to know. The Admin Dashboard's Announcements surface is Agent Router's answer to that gap. Sending an email is rarely enough; emails get filtered, lost, or simply read by the wrong audience. Announcements are notifications that operators publish from the Admin Dashboard and that appear directly inside the Console for every developer who signs in. They carry a severity level that controls visual prominence, support markdown for richer formatting where it helps, and can be scheduled in advance. This guide covers when to use which severity level, how to write announcements that developers actually read, how scheduling works, and how the lifecycle of an announcement (creation, edit, archival) interacts with the audit log. **Persona:** Platform operator working in the Admin Dashboard. **Estimated time:** 5 to 10 minutes per announcement; ongoing as Agent Router evolves. ## When this guide applies Announcements are the right tool whenever a change affects what developers can do or see in Agent Router, and a quick, in-context heads-up beats relying on out-of-band communication. Typical cases: | Situation | Suggested severity | | :---- | :---- | | A new model has become available and developers can start using it | Info | | A feature has been released or improved | Info | | Scheduled maintenance is upcoming | Warning | | A model or provider is being deprecated and will be removed by a known date | Warning | | A partial outage is in progress and one provider is unavailable | Warning, or Critical if the impact is broad | | A security incident is active and immediate action is required | Critical | | Agent Router itself is unavailable or behaving unsafely | Critical | For cases where the audience is narrower than the entire developer base (a single team, a specific product line, or a particular environment), announcements are still useful, but the body of the announcement should make the intended audience explicit. There is no built-in audience-filtering mechanism; every Console user sees every active announcement. ## Outcomes By the end of this guide: - At least one announcement has been published and confirmed visible in the Console. - The three severity levels are clear, including how each one is rendered and what dismissal behaviour applies. - The mechanics of editing, scheduling, and archiving are exercised, so the announcement lifecycle is understood. - The relationship between announcements and the audit log is clear: every announcement-related action is captured automatically. ## Prerequisites - Administrator access to the Admin Dashboard, typically the `super_admin` role. - A concrete change or message to communicate. Announcements that exist primarily to be dismissed do not improve trust. ### Field reference The Announcements list presents each entry as a row with the following columns: | Column | Description | | :---- | :---- | | **Title** | The headline of the announcement | | **Severity** | Info, Warning, or Critical | | **Status** | Active (visible to Console users) or Archived | | **Created** | Date the announcement was created | | **Published** | Date the announcement became visible; may be a scheduled future date | The severity levels map to distinct visual treatments in the Console: | Severity | Visual treatment | Use cases | | :---- | :---- | :---- | | **Info** | Neutral or blue indicator | New model availability, feature updates, general notices | | **Warning** | Yellow or amber indicator | Scheduled maintenance, upcoming deprecations, partial outages | | **Critical** | Red indicator | Active outages, security incidents, urgent action required | The core lifecycle tasks (create, edit, schedule, and archive) are detailed in the steps below. ![Announcements](/img/screenshots/admin/announcements.jpeg) ## Step 1: choose the right severity The severity choice shapes how the announcement is rendered, how prominent it is, and whether developers can dismiss it for themselves. The three levels are not interchangeable. | Severity | Visual treatment | Dismissal behaviour | Right use cases | | :---- | :---- | :---- | :---- | | **Info** | Neutral or blue banner | Dismissible per-user | New model availability, feature updates, general notices | | **Warning** | Amber banner | Dismissible per-user, but reappears on new sessions | Scheduled maintenance, upcoming deprecations, partial outages | | **Critical** | Red banner | Cannot be dismissed until archived by an operator | Active outages, security incidents, urgent action required | A per-user dismissal applies only to the developer who triggered it; the announcement itself remains in effect organization-wide until an operator archives it. The non-dismissibility of Critical announcements is deliberate. It is also the property that makes Critical announcements expensive to use: developers cannot make the banner go away while it is active, which is the entire point during an incident but is genuinely annoying once the underlying issue is resolved. A Critical announcement that is left active longer than it needs to be undermines the severity model for the next time it matters. Archive it promptly. A useful internal rule of thumb: if an announcement does not need to interrupt a developer who is mid-task, it is not Critical. Warning covers the majority of operationally important communication. ## Step 2: write the announcement The Announcements surface uses a small set of fields that together carry the message and its lifecycle. 1. Sign in to the Admin Dashboard. 1. Open **Announcements** from the sidebar. 1. Review the existing announcement list. Each entry shows the title, severity, status (Active or Archived), and the creation and publication dates. 1. Click **Create Announcement**. 1. Fill in the fields: - **Title.** A concise headline, ideally short enough to read in a banner without truncation. Examples: `Scheduled maintenance on 15 February`, `Claude Sonnet 4.6 now available`, `OpenAI degraded service in EU region`. - **Body.** The full message. Markdown is supported, so bullet lists, links, and emphasis can be used where they help readability. Long bodies are rendered in a collapsible region. - **Severity.** The choice from Step 1. - **Publish Date.** Optional. Leave blank to publish immediately, or set a future date and time to schedule the announcement. 1. Click **Publish** for immediate publication, or **Schedule** if a future publish date has been set. A few drafting conventions that consistently land well: - **Lead with the practical impact.** "OpenAI requests will fail intermittently for the next two hours" tells the reader what they need to know before they finish the first sentence; "We are experiencing an issue" does not. - **Name the audience when it is not everyone.** "If you use the `gpt-4o-mini` model in production, please note..." surfaces relevance immediately. - **Include a what-to-do.** When something requires action, name the action. When nothing is required, say that explicitly. - **Set an expectation.** A maintenance announcement should name a start time, an end time, and a follow-up channel. An outage announcement should name a frequency for updates. ## Step 3: verify the announcement in the Console Active announcements appear as banners at the top of the Console interface. The most reliable way to confirm an announcement is doing what it should be doing is to look at it from a developer's perspective. 1. Open the Console in a separate browser session or window. 1. Sign in as a developer (the operator account works just as well for verification). 1. Confirm the announcement appears as a banner with the expected severity styling. 1. For Info and Warning announcements, confirm that the dismiss control is present and functional. 1. For Critical announcements, confirm that the banner has no dismiss control, only the styling that distinguishes it from less urgent messages. If the announcement does not appear, the most common causes are a future publish date that has not yet been reached, a typo in the body that broke the markdown rendering, or browser caching that is showing a stale Console state. A hard refresh resolves the caching case. ## Step 4: edit, schedule, and archive Announcements are not write-once. Each one can be revised, scheduled in advance, or archived when it has outlived its usefulness. ### Edit an active announcement 1. Click the announcement row to open its detail view. 1. Update the fields that need to change. Edits to body text, severity, or publish date are all supported. 1. Click **Save**. Changes propagate to active announcements immediately; developers who already have the Console open will see the updated banner on their next page load. The edit itself is captured in the audit log, so a record exists of what changed and who changed it. ### Schedule a future announcement The publish date field is the lever for scheduled announcements. Pre-writing a maintenance announcement two weeks in advance, with a publish date set to the day of the maintenance window, is a useful pattern: the writing happens when the calendar is open and details are fresh, and the announcement appears at exactly the right moment without further effort. Scheduled announcements show up in the Announcements list with a Published date in the future, distinguishing them clearly from Active announcements that are already visible. ### Archive an old announcement When an announcement is no longer relevant, it should be archived. Archiving removes the banner from the Console immediately but preserves the announcement in the Admin Dashboard list for historical reference. 1. Open the active announcement. 1. Click **Archive**. Archived announcements remain visible to operators through the announcements list, filterable by status. They can be restored to Active if circumstances change, a maintenance that was rescheduled rather than completed, for example. ## How announcements interact with the audit log Every state-modifying action on the Announcements surface generates an entry in [Audit Logs](/agent-router-enterprise/guides/operate-and-govern/audit-platform-activity/): | Event | Captured fields | | :---- | :---- | | `announcement.created` | Title, severity, body, publish date, actor | | `announcement.updated` | The changed fields, old and new values, actor | | `announcement.archived` | Actor and timestamp | The audit trail means that the question "did anyone communicate this change?" is always answerable after the fact. For incident retrospectives and compliance reviews, the announcement lifecycle is part of the auditable record alongside everything else captured in the log. ## What to do next - **Audit Agent Router activity**: announcements are part of the auditable activity surface and complement the per-event log. See [Audit Agent Router Activity](/agent-router-enterprise/guides/operate-and-govern/audit-platform-activity/). - **Manage models and providers**: announcements about new model availability typically follow a successful provisioning pass. See [Managing Models and Providers](/agent-router-enterprise/guides/operate-and-govern/provision-models-and-providers/). - **Run multiple Agent Router instances**: announcements are scoped per instance; coordinating cross-instance communication is its own discipline. See [Run Multiple Agent Router Instances](/agent-router-enterprise/guides/operate-and-govern/run-multiple-platform-instances/). Where to go next Audit Agent Router activity Every announcement action is captured in the auditable activity surface. Manage models and providers Announcements about new model availability follow a provisioning pass. --- ## Configure data residency Enterprise Tier Regulated data carries two obligations that are easy to state and easy to violate. The first is residency: data that falls under a regional regime (personal data subject to the GDPR in the EU, or data covered by a US contractual commitment) must stay inside an approved geography for the whole of its journey, including the moment it reaches an upstream AI provider. The second is retention: even when a request stays in-region, the provider on the receiving end may log prompts and completions for abuse monitoring or model improvement unless it has been configured not to. A residency boundary that the gateway honours on the way out is undone if the provider retains the payload afterwards. Tetrate Agent Router addresses both obligations at the layer where requests are routed. Residency is enforced by mapping a policy to provider entries that live only inside the approved region, so a request can never reach an out-of-region endpoint: not on the primary path, and not on any fallback or load-balanced alternative. No-retention is enforced one step further out, on the upstream provider itself, by applying whatever zero-data-retention or no-logging setting that provider supports and recording which setting was applied. This guide covers the operator-side work in the Admin Dashboard, the operator-facing application within Agent Router: selecting a residency region, binding it to routing so the boundary holds under failover, configuring upstream providers for no-retention where supported, verifying where a request was actually served, and documenting the result for audit. The compliance artifacts that sit alongside this configuration (SOC 2, ISO 27001, and the provider data-processing agreements that make the no-retention claim contractually real) are catalogued separately in [Compliance reference](/reference/compliance/). This guide configures the runtime behaviour; that page records the evidence. :::important The exact no-retention or zero-data-retention setting differs by provider, and some providers gate it behind a specific contract tier or enterprise agreement. The setting names, the tiers that unlock them, and the contractual basis for each must be confirmed with the Tetrate field team and the customer's legal stakeholders before the configuration is treated as compliant. This guide describes the mechanism, not a guarantee that any particular provider offers it. ::: **Persona:** Platform operator working in the Admin Dashboard, in coordination with compliance and legal stakeholders who own the residency and retention requirements. **Estimated time:** 30 to 60 minutes for a first pass against one region, depending on how many providers are in scope and how quickly provider-side retention settings can be confirmed. ## When this guide applies This guide is the right starting point in any of these situations: | Situation | Why this guide helps | | :---- | :---- | | A workload is bound to a single geography (for example EU-only or US-only) | Residency must be enforced at the routing layer so requests cannot reach an out-of-region endpoint | | Regulated data must not be retained by upstream providers | Provider-side no-retention settings have to be applied and recorded, not assumed | | A residency policy must survive provider failure | Fallback and load balancing must be constrained so failover never crosses the boundary | | An auditor or legal stakeholder asks where requests are served and what the provider does with them | The verification and documentation steps produce the answer | | Preparing a region-restricted evaluation environment | The settings applied in the POC need to be captured for the evaluation record | ## Outcomes By the end of this guide: - A data-residency region (for example US or EU) has been selected for the regulated workload. - Routing for that workload reaches only provider entries inside the approved region, on the primary path and on every fallback and load-balanced alternative. - Each upstream provider in scope has had its no-retention or zero-data-retention setting applied where the provider supports it, with the applied setting recorded. - The region in which a sample request was served has been verified by inspection rather than assumption. - The applied residency and retention settings are documented in a form an auditor can read. ## Prerequisites - Administrator access to the Admin Dashboard, typically the `super_admin` or `provider_admin` role. The role model is covered in [Manage roles and permissions](/agent-router-enterprise/guides/identity-and-access-management/manage-roles-and-permissions/). - A written residency requirement from the compliance or legal stakeholder: the approved region or regions, and the data classes the requirement covers. - Per-region provider entries already configured, one per subscription and region, each reporting a healthy connection. Laying out this structure is covered in [Connect provider subscriptions across clouds and regions](/agent-router-enterprise/guides/operate-and-govern/provision-models-and-providers/connect-provider-subscriptions-across-clouds/). This guide assumes those entries exist and builds the residency policy on top of them. - The provider-side retention setting for each provider in scope, confirmed with the field and legal teams: the setting name, the contract tier that unlocks it, and whether it is in effect for the account being used. - A baseline understanding of how the data plane reaches providers, covered in [Architecture overview](/product-architecture/architecture-overview/). ## Step 1: capture the residency and retention requirement Configuration that is not traceable to a stated requirement cannot be audited. The first step is to write the requirement down in a form the later steps can be checked against. 1. Record the approved region or regions for the workload: for example `EU` for a GDPR-bound workload, or `US` for a workload under a US-only commitment. 1. Record the data classes the requirement covers, so it is clear which routing configurations the policy must be applied to and which are out of scope. 1. Record the retention requirement separately from the residency requirement. The two are independent: a request can stay in-region and still be retained, and a no-retention provider can still serve from the wrong region. Both have to hold. 1. Identify, with the field and legal teams, which providers can satisfy each requirement. A provider with no in-region endpoint cannot serve a residency-bound workload; a provider with no no-retention setting cannot serve a no-retention workload, regardless of where it runs. The output of this step is a short statement (region, data classes, retention expectation, and the providers eligible to serve under it) that drives the policy built in the steps that follow. ## Step 2: select the residency region and its provider entries Residency is enforced by routing a workload only to provider entries that live inside the approved region. The per-region entries created in [Connect provider subscriptions across clouds and regions](/agent-router-enterprise/guides/operate-and-govern/provision-models-and-providers/connect-provider-subscriptions-across-clouds/) are the building blocks; this step selects the subset that sits inside the boundary. 1. Open **Providers Management** from the sidebar. 1. Identify the provider entries whose endpoint sits in the approved region. Where a naming convention encodes the region in the identifier (for example an entry named for the provider, cloud, and region together) the in-region entries are identifiable at a glance; otherwise the endpoint URL distinguishes them, since a region-specific endpoint carries its region in the host name. 1. Confirm each in-region entry reports a healthy status. A residency policy that points at an unhealthy entry has nowhere to send traffic and fails closed rather than crossing the boundary. 1. Record the exact set of entries that are inside the boundary. This set, and only this set, is what the workload's routing is permitted to reference. The discipline here is exclusion as much as selection. An out-of-region entry for the same provider and the same logical model is a correct entry for some other workload, but it must not appear anywhere in the residency-bound workload's routing: not as a primary, not as a fallback, and not as a load-balancing member. ## Step 3: enforce the boundary through routing A region selected in the abstract enforces nothing. The boundary holds only when the workload's routing references in-region entries exclusively. 1. For the routing configuration that serves the regulated workload, map the logical model only to the in-region provider entries recorded in Step 2. 1. Exclude every out-of-region entry from the configuration, even where it serves the same logical model and would be a valid choice for an unconstrained workload. 1. Keep residency-bound routing separate from unconstrained routing. A single configuration that mixes in-region and out-of-region entries cannot guarantee residency, because the routing layer is free to choose any member. Separation is what makes the guarantee inspectable. The composition of residency with fallback and load balancing is the part that most often goes wrong, and it has its own step below. The principle stated here carries forward to it: every backend the routing layer can reach for this workload, by any path, must sit inside the boundary. ## Step 4: configure upstream providers for no-retention Residency governs where a request is served; it says nothing about what the provider does with the payload afterwards. No-retention is configured on the provider side, and Agent Router's responsibility is to apply the setting where it exists and to record which setting was applied. 1. For each provider entry in the residency set, determine, with the field and legal teams, whether the provider offers a no-retention or zero-data-retention mode, and what unlocks it. Some providers expose it as an account-level or organisation-level setting; some require an enterprise agreement or a specific contract tier; some do not offer it at all. 1. Apply the setting through whichever surface the provider exposes it on. For some providers this is an account flag configured in the provider's own console; for others it is a contractual term that takes effect at the account level rather than a toggle. Agent Router routes to the provider endpoint; the retention posture of that endpoint is established on the provider side. 1. Record, for each provider, the exact setting that was applied, where it was applied, and the date. A claim of no-retention that cannot be traced to a named setting on a named account is not auditable. 1. Treat a provider that offers no no-retention mode as ineligible for the no-retention workload. It may still be a valid backend for workloads without a retention requirement, but it must not appear in the routing for one that has it, the same exclusion discipline as residency. :::important The presence of a no-retention setting in a provider's console is not the same as a contractual no-retention commitment. The setting names referenced here are illustrative; the binding ones for any given provider, and the agreement that makes them enforceable, must be confirmed with the field and legal teams and recorded in [Compliance reference](/reference/compliance/). ::: ## Step 5: compose residency with fallback and load balancing Fallback and load balancing exist to keep a workload serving when a single backend degrades. For a residency-bound workload they introduce a specific hazard: a failover that reaches for a healthy backend in the wrong region would breach the boundary at the worst possible moment, silently, under load. The composition has to be constrained so that this cannot happen. - **Fallback** walks an ordered list of backends when the primary fails. Every entry in that ordered list must sit inside the approved region. A fallback chain whose later entries point out of region will hold residency only until the primary fails, which is exactly when residency must not give way. Build the fallback chain from in-region entries only, and accept that if every in-region backend is unavailable the request fails rather than crossing the boundary. Failing closed is the correct behaviour for a residency requirement. - **Load balancing** distributes requests across several backends for the same logical model. Every member of the load-balancing set must sit inside the approved region. A set that includes an out-of-region member will route some fraction of traffic across the boundary by design, not by accident. The regional load-balancing mechanics are covered in [Load balance across regional deployments](/agent-router-enterprise/guides/operate-and-govern/load-balance-across-regional-deployments/); the residency constraint on top of them is that the member set never extends past the boundary. The rule that unifies both is the one stated in Step 3: every backend the routing layer can reach for this workload, by any path, must be in-region. Fallback and load balancing widen the set of reachable backends, so they widen the surface that has to be checked; they do not relax the constraint. ## Step 6: verify where a request was served A residency policy is only as trustworthy as the evidence that it held. Configuration can be correct in intent and wrong in effect; the verification step replaces assumption with an observed fact about where a sample request actually went. 1. Send a representative request through the residency-bound routing configuration. 1. Inspect the request record to confirm which provider entry served it. Request-level attribution, the provider entry and therefore the region that handled the request, is surfaced in the request logs; configuring and reading them is covered in [Configuring request logs](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/configuring-request-logs/). 1. Confirm the serving entry is one of the in-region entries recorded in Step 2. An entry outside that set, even for the correct provider and model, is a residency breach and means the routing configuration still references an out-of-region backend somewhere. 1. Exercise the failover path deliberately where the environment allows it, for example by draining or disabling the primary in-region entry, and confirm the request is served by another in-region entry or fails, never by an out-of-region one. Verifying the primary path alone leaves the fallback path untested, and the fallback path is where residency is most likely to leak. Verification by inspection, including a deliberate failover, is what turns a configured policy into a demonstrated one. For a POC evaluation, the served-region observation and the failover observation are the evidence that the residency criterion is met. ## Step 7: document the applied settings for audit The configuration is complete only when it is written down in a form a reviewer who was not present can follow. The documentation is the deliverable an auditor or legal stakeholder reads; the running configuration is what it describes. - Record the residency policy: the approved region, the data classes it covers, and the exact set of in-region provider entries the workload's routing references. - Record the no-retention posture per provider: the setting applied, where it was applied, the account or organisation it applies to, the date, and the contract basis confirmed with the legal team. - Record the verification result: the region a sample request was served from, the failover observation, and the date the check was run. - Note explicitly which providers were excluded from the workload and why: no in-region endpoint, no no-retention mode, or both. The exclusions are as much a part of the audit record as the inclusions, because they show the boundary was applied deliberately rather than by omission. The retention and purge of the request logs that produced the verification evidence are themselves a compliance concern, covered in [Manage log retention and purge](/agent-router-enterprise/guides/operate-and-govern/compliance-and-governance/manage-log-retention-and-purge/). The compliance artifacts that back the no-retention claim contractually are catalogued in [Compliance reference](/reference/compliance/). ## What to do next - **Manage log retention and purge**: the request logs used to verify residency carry their own retention obligation; configure how long they are kept and how they are purged. See [Manage log retention and purge](/agent-router-enterprise/guides/operate-and-govern/compliance-and-governance/manage-log-retention-and-purge/). - **Review compliance artifacts**: pair the runtime configuration with the SOC 2, ISO 27001, and data-processing-agreement evidence that backs it. See [Compliance reference](/reference/compliance/). - **Audit Agent Router activity**: once the residency policy is live, the audit and request-log surfaces are where ongoing adherence is observed. See [Configuring request logs](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/configuring-request-logs/). Where to go next Manage log retention and purge The request logs used to verify residency carry their own retention obligation. Export audit and policy decisions to a SIEM Deliver audit and decision records into the organisation's system of record. --- ## Export audit decisions to SIEM Enterprise Tier A security team rarely asks "what does the gateway log". The question they actually bring is "what did the gateway decide, and why": for a specific request, on a specific date, against a specific caller. When a prompt was blocked, a response was redacted, or a request was rerouted, the team needs a structured record that names the decision, the policy that drove it, the version of that policy in force at the time, the identity behind the request, and the model that was targeted. That record has to be queryable on its own and exportable into the tooling the security organisation already runs. Tetrate Agent Router records two related but distinct streams. The audit log captures state-modifying administrative actions (a model enabled, a provider edited, or a key revoked) and is browsed through the Admin Dashboard. Policy decisions are the inline, per-request verdicts the gateway reaches as traffic flows: allow, deny, modify, and redact. Both streams are structured, both carry a correlation identifier and a policy version where a policy was involved, and both can be exported to a customer security information and event management (SIEM) system. This guide covers why policy decisions warrant their own structured record, how that record relates to the audit log and its event schema, how the records are queried, and how they are exported to common SIEM targets: Amazon S3, Splunk, and Datadog among them. **Persona:** Platform operator working in the Admin Dashboard, in coordination with security and compliance stakeholders who own the destination SIEM. **Estimated time:** 30 to 45 minutes to choose an export path, configure a destination, and verify that records arrive. ## When this guide applies Exporting audit and policy-decision records is the right step in any of these situations: | Situation | What export provides | | :---- | :---- | | A security team must answer "what did the gateway decide for this request, and under which policy" | Structured per-request decision records carrying correlation ID, policy version, caller identity, model, and the action taken | | Compliance requires audit and policy events to land in the organisation's system of record | A delivery path into Amazon S3, Splunk, Datadog, or a comparable SIEM | | Decision records must be correlated with the rest of the application stack | A shared correlation identifier that joins gateway decisions to downstream telemetry | | Audit records must be retained beyond the in-Console window | Exported copies held under the SIEM's own retention policy | | A periodic attestation must show that every allow, deny, modify, and redact verdict is recorded and exportable | A documented export pipeline plus the structured records it carries | For browsing administrative events interactively rather than exporting them, the [Audit Agent Router activity](/agent-router-enterprise/guides/operate-and-govern/audit-platform-activity/) guide is the right surface. For the field-level event schema, see [Audit log events](/reference/audit-log-events/). ## Outcomes By the end of this guide: - The distinction between the audit log (administrative actions) and policy-decision records (inline per-request verdicts) is clear, along with the fields each carries. - An export path has been chosen, either the OpenTelemetry-based observability pipeline or a direct export to the SIEM, with the trade-offs understood. - A destination SIEM has been configured to receive records, with Amazon S3, Splunk, and Datadog covered as representative targets. - At least one policy decision and one audit event have been confirmed to arrive in the destination, located by correlation ID. - The retention of exported copies is understood as distinct from Agent Router's own retention window. ## Prerequisites - Administrator access to the Admin Dashboard, typically the `super_admin` role for the audit and export surfaces. - A destination SIEM, with its ingestion endpoint, transport, and credentials obtained from whoever owns it. For Amazon S3 this is a bucket and a write credential; for Splunk an HTTP Event Collector endpoint and token; for Datadog an intake endpoint and API key. - Confirmation that the destination endpoint is reachable from the deployment. This is a network reachability question rather than an Admin Dashboard one; where there is doubt, the operator who installed Agent Router is the right person to confirm. - At least one policy in force (a guardrail, a routing rule, or a redaction rule) so that policy-decision records exist to export. An Agent Router deployment with no policies configured produces audit events but no inline decision records. ## Step 1: understand what a policy-decision record carries A policy decision is the verdict the gateway reaches inline, while a request is in flight, before the response is returned to the caller. Four verdicts are recorded: | Verdict | Recorded when | | :---- | :---- | | Allow | The request satisfied every policy in force and passed through unchanged | | Deny | A policy blocked the request, and no response was generated from the upstream model | | Modify | A policy altered the request or routing before it reached the upstream model | | Redact | A policy removed or masked content from the request or the response | Each verdict is captured as a structured record carrying the fields a security review depends on: - A **correlation ID** that is unique to the request and shared with the request's trace and logs, so the decision can be joined to the rest of the request's telemetry. - The **policy version** in force when the decision was reached, so a verdict can be reproduced against the exact rule set that produced it rather than against whatever is current. - The **caller identity** the request was attributed to: the API key and, where available, the user behind it. - The **model** the request targeted. - The **action taken**: the verdict above, plus the policy that drove it. The reason these records are kept separately from the audit log is a difference in subject. The audit log answers "who changed Agent Router"; policy-decision records answer "what did Agent Router decide about this request". A security team investigating a blocked prompt needs the second, not the first. Keeping them distinct means a decision record can be queried by correlation ID without wading through administrative noise, and the two can still be joined where an investigation needs both: for example, when a deny verdict is traced back to a guardrail that an audit event shows was edited an hour earlier. ## Step 2: relate decision records to the audit log schema The audit log and the policy-decision stream share a structural shape, which is what makes them straightforward to correlate and to export through the same pipeline. 1. Review the audit event envelope in [Audit log events](/reference/audit-log-events/). Every audit entry is an `AuditLogEntry`: flat fields such as `timestamp`, `userId` / `userEmail` / `actorType`, `actionType`, `resourceType` / `resourceId`, `metadata`, and `correlationId` that joins related rows and other telemetry. 1. Map the policy-decision fields onto the same mental model. The verdict corresponds to `actionType`, the caller identity corresponds to the actor fields, the targeted model and policy correspond to the resource fields, and the decision stream's correlation ID plays the same join role as `correlationId` on the audit entry. 1. Note where the two streams meet. An audit event such as `UPDATE_GUARDRAIL` or `SET_GUARDRAIL_STATUS` carries the resource and a timestamp; a later deny decision carries the policy version that edit produced. Joined on policy and time, the two answer "the gateway denied this request because a guardrail had been tightened at this moment, by this administrator". The practical takeaway is that a single export pipeline can carry both streams, and a single correlation strategy (correlation ID for request-scoped joins, policy plus timestamp for decision-to-administration joins) covers the questions a security team brings. ## Step 3: choose between the OTLP path and a direct export Two export paths are available, and they are complementary rather than competing. | Path | How it works | Best when | | :---- | :---- | :---- | | OpenTelemetry pipeline | Decision and audit records ride the same OpenTelemetry (OTLP) stream the gateway already emits for traces and metrics, into a collector that fans out to the destination | An observability or SIEM pipeline already consumes OTLP, or records must be correlated with traces on a shared identifier | | Direct export | Records are delivered straight to the SIEM's own ingestion endpoint (an Amazon S3 bucket, a Splunk HTTP Event Collector, or a Datadog intake) without an intervening collector | The SIEM is the system of record and no trace correlation is required, or a collector is undesirable operational surface | The OpenTelemetry path reuses the export mechanics already described for telemetry; see [Export telemetry to an observability stack](/agent-router-enterprise/guides/observability-and-analytics/export-telemetry-to-an-observability-stack/) for the configuration of the OTLP endpoint, transport, and authentication, and [OpenTelemetry metrics](/reference/otel-metrics/) for the attributes carried on the stream. Because decision records share the same correlation identifier as traces, routing them through this path lets the SIEM join a verdict to the full request trace without additional work. The direct path is the simpler model when the SIEM is the only destination that matters. Records are formatted for the target and delivered to its endpoint, with no collector to operate. The cost is that trace correlation, if it is needed later, has to be reconstructed from the correlation ID rather than arriving pre-joined. Most deployments that already run an OpenTelemetry collector route through it; deployments whose SIEM is the single system of record favour the direct path. The choice is reversible; a destination can be moved from one path to the other without changing what the records contain. ## Step 4: configure the destination SIEM The configuration differs per target, but the shape is constant: an endpoint, a transport, a credential, and a confirmation that records are flowing. 1. Sign in to the Admin Dashboard with an administrator account. 1. Open the export configuration for audit and policy-decision records. 1. Select the export path chosen in Step 3, the OpenTelemetry pipeline or a direct export. 1. Enter the destination details for the target SIEM. Representative targets: | Target | What to provide | | :---- | :---- | | Amazon S3 | A bucket name, a region, a path prefix, and a write credential. Records are delivered as structured objects under the prefix, which suits archival and downstream batch ingestion | | Splunk | An HTTP Event Collector endpoint and token. Records arrive as structured events that Splunk indexes for search | | Datadog | An intake endpoint and an API key. Records arrive as structured logs that Datadog correlates with the rest of the stack | 1. Set the authentication for the target: a write credential for Amazon S3, a token for Splunk, or an API key for Datadog. 1. Confirm with security stakeholders which streams are in scope. Both the audit log and the policy-decision stream can be exported; some destinations take both, others only one. 1. Save the configuration and enable the export. Where the OpenTelemetry path is used, the endpoint, transport, and authentication are configured as for any OTLP destination, with the SIEM either receiving OTLP directly or sitting behind a collector that forwards to it. The mechanics are the same as for trace export, described in [Export telemetry to an observability stack](/agent-router-enterprise/guides/observability-and-analytics/export-telemetry-to-an-observability-stack/). ## Step 5: verify records arrive and are queryable An export that is configured but unverified cannot be attested to. Confirm the pipeline end to end before relying on it. 1. Generate a policy decision to test against. Issue a request that a policy in force will act on (one that a guardrail denies, or one a redaction rule masks) so that a non-trivial verdict is produced. 1. Note the correlation ID for that request from the in the Console surfaces. 1. In the destination SIEM, search for the record by correlation ID. Confirm the verdict, the policy version, the caller identity, the model, and the action taken are all present and match the request issued. 1. Confirm an audit event also arrives. Make a small administrative change (toggle a model, or edit a provider) and confirm the corresponding audit event appears in the destination with its actor, action, and resource intact. 1. Confirm the records are queryable in the way the security team needs (by correlation ID, by caller, by verdict, and by policy version) rather than only present in raw form. If records do not arrive while the in the Console surfaces show the decision and the audit event, the cause is usually the export path rather than Agent Router: an unreachable endpoint, a transport mismatch, or a credential the destination rejects. Each is checkable independently, the same way an OTLP export error is diagnosed in [Export telemetry to an observability stack](/agent-router-enterprise/guides/observability-and-analytics/export-telemetry-to-an-observability-stack/). ## Step 6: account for the retention of exported copies Once a record reaches the SIEM, it lives under the SIEM's retention policy, not Agent Router's. This separation is the point of exporting (long-horizon retention is delegated to the system built for it) but it has to be stated accurately. - Agent Router's own audit and decision records age out on the deployment's configured window, which is intended for active investigation rather than indefinite storage. - Exported copies persist for as long as the destination's policy keeps them. An Amazon S3 lifecycle rule, a Splunk index retention setting, or a Datadog retention tier governs the exported copy independently. - A purge of a record from Agent Router does not remove any copy already delivered to the SIEM. That copy ages out, or is purged, on the SIEM's own terms. The consequence for a right-to-erasure or subject-access response is that "the record has been deleted" is only true for the store it names. Removing a record from Agent Router and removing every exported copy from the SIEM are separate operations, each on its own clock. See [Manage log retention and purge](/agent-router-enterprise/guides/operate-and-govern/compliance-and-governance/manage-log-retention-and-purge/) for the Agent Router-side retention and purge workflow and how it accounts for external SIEM copies. ## What to do next - **Manage log retention and purge**: to govern how long Agent Router's own audit and request records live, and how purge accounts for exported copies. See [Manage log retention and purge](/agent-router-enterprise/guides/operate-and-govern/compliance-and-governance/manage-log-retention-and-purge/). - **Audit Agent Router activity**: to browse administrative events interactively alongside the exported stream. See [Audit Agent Router activity](/agent-router-enterprise/guides/operate-and-govern/audit-platform-activity/). - **Review the audit log event schema**: to map exported fields back to Agent Router's event vocabulary. See [Audit log events](/reference/audit-log-events/). - **Review the OpenTelemetry metric and span model**: to understand the attributes that travel with records on the OTLP path. See [OpenTelemetry metrics](/reference/otel-metrics/). Where to go next Manage log retention and purge Govern how long Agent Router records live and how purge accounts for exported copies. Audit log events Map exported fields back to Agent Router's event vocabulary. --- ## Compliance and Governance The gateway sits in the request path, which makes it the natural enforcement and evidence point for compliance requirements: where data is processed, how long records are kept, and how decisions reach the organization's audit tooling. Configure data residency and no-retention Pin processing to a region and keep providers from retaining prompts. Manage log retention and purge Set retention windows and purge records on demand. Export audit and policy decisions to a SIEM Stream audit events and guardrail decisions into existing security tooling. --- ## Manage log retention and purge Enterprise Tier Request logs are useful for debugging, cost attribution, and incident response, but every record retained is also a record that has to be governed. Prompt and response bodies routed through Agent Router Enterprise routinely carry the most sensitive data an organisation handles, so the longer those records persist, the larger the surface that a compliance review, a breach, or a subject-access request has to account for. Retention is therefore a control in its own right: it minimises the volume of stored sensitive data, it satisfies a documented retention policy rather than letting records accumulate by default, and it supports right-to-erasure obligations by providing a defined way to remove records on demand. This guide covers the retention side of logging in Agent Router: how long records are kept, how they are archived to long-term storage before removal where that is required, and how records are purged and the deletion verified. It does not re-document the logging modes themselves. The choice of how much request detail is captured in the first place (from metadata-only, through metadata plus prompt, through metadata plus prompt and response, down to zero-retention) is made in [Control what request data leaves the cluster](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/configuring-request-logs/), and that mode can be scoped per app or route and per data classification. Retention picks up from there: given whatever records a mode produces, this guide governs how long they live and how they are removed. **Persona:** Platform operator working in the Admin Dashboard, in coordination with compliance and data-governance stakeholders. **Estimated time:** 20 to 30 minutes to set a retention window, configure archival where required, and run a verified purge. ## When this guide applies Retention controls are relevant whenever the question shifts from "what is captured" to "how long is it kept and how is it removed". Typical cases: | Situation | What this guide provides | | :---- | :---- | | A retention policy specifies a maximum age for request records | A retention window so records age out automatically | | Sensitive records must be preserved in long-term storage before they are removed from the active store | An archival step that runs ahead of purge | | A right-to-erasure or subject-access request requires specific records to be deleted now | An on-demand purge workflow with deletion verification | | Different apps, routes, or data classifications carry different retention obligations | Scoping retention alongside the per-app and per-classification logging modes set in the request-logs guide | | A periodic compliance attestation must show that stored data is bounded and removable | A documented, tested purge workflow plus the audit trail it produces | ## Outcomes By the end of this guide: - A retention window has been set so request records age out automatically once they exceed the configured maximum age. - Where required, an archival path to long-term storage has been configured to run before records are purged. - A purge has been run on demand and the deletion confirmed against the active store. - The relationship between request-log retention, audit-trail retention, and any external SIEM copies is understood, so a claim about "where the data lives and for how long" can be made accurately. ## Prerequisites - Administrator access to the Admin Dashboard, typically the `super_admin` role for retention and purge controls. - A logging mode already chosen and active for the apps, routes, or classifications in scope. Retention governs whatever records a mode produces; if a route is set to zero-retention, no request records are stored for it and there is nothing to age out or purge. See [Control what request data leaves the cluster](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/configuring-request-logs/). - Agreement with compliance stakeholders on the retention window per classification, on whether archival is required before purge, and on the destination and retention of any archive. - Where archival is required, a long-term storage destination that is reachable from the deployment and governed by its own retention policy. ## Step 1: confirm the logging mode and its scope Retention only governs records that exist, so the first step is to confirm what each app, route, or classification in scope is configured to capture. A route running in metadata-only mode has no prompt or response bodies to age out; a route running in full mode has the largest record and the strongest case for a short retention window. 1. Review the active logging mode for each app, route, or data classification in scope. The mode and its scoping are configured in [Control what request data leaves the cluster](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/configuring-request-logs/). 1. Note which scopes capture prompt or response bodies. These carry the most sensitive content and usually warrant the shortest retention. 1. Note which scopes are set to zero-retention. No request records are stored for these, so they need no retention window and produce nothing to purge. 1. Record the resulting picture (scope, mode, and the sensitivity of what each scope stores) as the basis for the retention windows set in the next step. This mapping is what allows retention to be reasoned about per app and per classification rather than as a single global number. A scope that stores full bodies and a scope that stores metadata only can carry different retention windows even though they share the same store. ## Step 2: set a retention window so records age out A retention window defines the maximum age of a request record. Once a record exceeds that age, it is removed automatically without operator action, which keeps the active store bounded and enforces the retention policy continuously rather than relying on periodic manual cleanup. 1. Sign in to the Admin Dashboard with an administrator account. 1. Open the request-log retention settings. 1. Set the retention window for the scope being configured, the maximum age after which records are removed. Shorter windows minimise stored sensitive data; longer windows preserve more history for investigation. 1. Where the deployment supports per-scope retention, set a distinct window for each app, route, or classification that carries a different obligation, using the mapping from Step 1. The most sensitive scopes, those capturing prompt or response bodies, generally take the shortest windows. 1. Save the change. Once set, records age out on their own schedule. Aging out is continuous and applies going forward; it does not retroactively change the content of records already stored, only how long they survive. A change to the window takes effect for records evaluated after the change, so shortening a window can bring older records within range of removal at the next cycle. ## Step 3: archive records before purge where required Some retention policies require records to be preserved in long-term storage before they leave the active store, so that an auditable copy survives even after the operational record is removed. Where this applies, archival runs ahead of both automatic aging-out and on-demand purge. 1. Confirm with compliance stakeholders whether archival is required for the scope, and where the archive must be held. 1. Configure the long-term storage destination for the records in scope. The destination is governed by its own retention policy, which is typically longer than the active store's window. 1. Confirm that records are written to the archive before they are removed from the active store, so that no record in scope leaves the active store without an archived copy first existing. 1. Record the archive location and its retention policy alongside the active-store retention window, so the full lifecycle of a record (active store, then archive, then final removal) is documented in one place. Archival and active-store retention are separate controls with separate clocks. A record can be removed from the active store on the active window while its archived copy persists for far longer under the archive's own policy. Both windows belong in any statement about how long the data is retained. ## Step 4: run a purge and verify the deletion Automatic aging-out handles routine retention. A purge handles the on-demand case: a right-to-erasure request, a subject-access deletion, or any situation where specific records must be removed now rather than at their natural expiry. A purge is only complete once the deletion has been verified against the active store. 1. Identify the records to be purged: by scope, by time range, or by the subject the deletion request concerns. 1. Where archival is required for the scope, confirm that the archival step in Step 3 has already preserved any copy that policy requires before the records are removed. A purge removes records from the active store; it does not reach into the archive. 1. Run the purge for the identified records. 1. Verify the deletion. Confirm that the records no longer appear in the request-logs view for the scope and time range concerned, and that a query for the subject of an erasure request returns no remaining records in the active store. 1. Record the purge: what was removed, when, on whose authority, and the verification result. This record is what demonstrates that a right-to-erasure obligation was met. Verification is not optional. A deletion that is requested but not confirmed cannot be attested to, and the most common failure is assuming a purge completed without checking the store afterward. Test the purge workflow end to end (run it, then verify) before relying on it for a live erasure request. ## Step 5: account for the audit trail and external SIEM copies Request-log retention governs request records. It does not govern two other places the same activity may be recorded, and a complete retention statement has to account for all three. - **The audit trail has its own retention.** Administrative actions, including changes to retention settings and the purge operations run in Step 4, are recorded in the Agent Router's immutable audit log, which is retained under its own policy rather than the request-log window. The audit log cannot be edited or deleted, so a purge of request records does not remove the audit record that the purge occurred. See [Audit Agent Router activity](/agent-router-enterprise/guides/operate-and-govern/audit-platform-activity/) for the audit surface and [Audit log events reference](/reference/audit-log-events/) for the event vocabulary. - **Exported copies live under the SIEM's retention.** Where request or audit records are exported to an external SIEM, those copies are governed by the SIEM's retention policy, not Agent Router's. Purging a record from the active store does not remove any copy already delivered to the SIEM; that copy ages out, or is purged, on the SIEM's own terms. See [Export audit and policy decisions to a SIEM](/agent-router-enterprise/guides/operate-and-govern/compliance-and-governance/export-audit-and-policy-decisions-to-a-siem/). The practical consequence is that "the record has been deleted" is only true for the store it names. A right-to-erasure response that has to cover every copy must address the active store, the archive (Step 3), and any external SIEM separately, each on its own retention clock. ## What to do next - **Configure no-retention for the most sensitive scopes**: where a classification should store nothing at all, zero-retention removes the record at source rather than relying on a short window. See [Configure data residency and no-retention](/agent-router-enterprise/guides/operate-and-govern/compliance-and-governance/configure-data-residency-and-no-retention/). - **Export audit and policy decisions to a SIEM**: to retain audit records beyond the in-Console window and govern them centrally. See [Export audit and policy decisions to a SIEM](/agent-router-enterprise/guides/operate-and-govern/compliance-and-governance/export-audit-and-policy-decisions-to-a-siem/). - **Revisit the logging mode**: if a retention window keeps coming up too short for a scope, the underlying mode may be capturing more than the scope needs. See [Control what request data leaves the cluster](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/configuring-request-logs/). Where to go next Configure data residency and no-retention Where a scope should store nothing, zero-retention removes the record at source. Export audit and policy decisions to a SIEM Retain audit records beyond the in-Console window and govern them centrally. --- ## Bill AI spend back to the teams that incur it Enterprise Tier As long as AI spend arrives as one opaque org-wide invoice, the operator owns every question about it. Chargeback moves the accountability to the teams that incur the cost, and showback at least makes the cost visible to them; both start with the same artifact, a per-user and per-team consumption statement exported into whatever financial system the organisation already runs. The reports here read from the same Usage Analytics breakdowns used to set the ceilings in [Keep each team's spend inside an agreed ceiling](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/keep-team-spend-inside-a-ceiling/). Reporting presents spend; export moves that spend out of Agent Router for chargeback (billing each business unit for its own consumption) or showback (reporting that consumption without moving money). **Persona:** Platform operator working in the Admin Dashboard, in partnership with the finance and business owners who consume the reports. **Estimated time:** 15 to 20 minutes per reporting cycle once the budgets and the group-to-team mapping are in place. ## Outcomes By the end of this guide: - A user- or team-based consumption report has been produced and read against the budgets it is measured against. - That report has been exported in a form suitable for chargeback or showback in a downstream finance or billing system. ## Prerequisites - Administrator access to the Admin Dashboard, typically the `super_admin` or `billing_admin` role. - Per-user and per-team budgets set, so that a consumption report can be read against the ceiling it is measured against. See [Keep each team's spend inside an agreed ceiling](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/keep-team-spend-inside-a-ceiling/). - Group identity flowing in from the identity provider, so that per-team statements attribute spend to the right cost centre. See [Map Entra ID groups to business functions](/agent-router-enterprise/guides/identity-and-access-management/single-sign-on/map-entra-id-groups-to-business-functions/). ## Step 1: Produce user- and team-based consumption reports 1. Open **Usage → Analytics** and set the time range to the reporting period, the closed billing cycle for a monthly statement. 1. Select the **By User** breakdown for a per-person statement, or the team view for a per-business-function statement. To narrow or compare specific people or teams before you export, use the multi-select filters in [Compare user and team spend in Analytics](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/compare-user-and-team-spend-in-analytics/). 1. Sort by cost descending so that the largest consumers sit at the top, which is the order most reviews want to read in. 1. Review the figures against the budgets that were set for each user and team. A consumption report read next to the ceiling it was measured against is far more useful than either in isolation. A consumption report is a point-in-time statement of what was spent; a budget is the ceiling it is measured against. Presenting them together (spend, ceiling, and the gap between them, per user and per team) is what turns raw usage data into something a business owner can act on. For the combined investigation pattern, where an unexpected line item in a report is traced back to the change that caused it, the spot-the-anomaly-then-find-the-cause workflow in [Audit Agent Router activity](/agent-router-enterprise/guides/operate-and-govern/audit-platform-activity/) is the companion surface. ## Step 2: Export data for chargeback and showback Chargeback and showback both depend on getting the spend data out of Agent Router and into whatever financial system the organisation already runs. 1. Apply the time range and the user or team filter that match the chargeback period and the cost-centre structure. CSV export follows the same selection as the chart and table. 1. Use the **Export** function to download the filtered data, typically as CSV. 1. Load the exported file into the billing, finance, or reporting system that owns the chargeback or showback process. The exported rows carry the dimensions chargeback needs (user, cost, token totals, and the period they cover) in a form a downstream system can join to its own cost-centre records. Agent Router's role ends at producing accurate, attributable data; the allocation of that data to ledgers and the decision to charge or merely to show belongs to the finance process consuming it. For organisations that need always-on cost dashboards rather than a periodic export, see [Get alerted to cost spikes as they happen](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/get-alerted-to-cost-spikes/). ## What to do next - **Know what every app and project actually costs**: add an app or project axis to the report when the user and team dimensions do not match the cost centres finance bills against. See [Know what every app and project actually costs](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/know-what-every-app-and-project-costs/). - **Keep each team's spend inside an agreed ceiling**: the ceilings these reports are measured against. See [Keep each team's spend inside an agreed ceiling](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/keep-team-spend-inside-a-ceiling/). - **Audit Agent Router activity**: the analytics and audit surface this guide reads from and records against. See [Audit Agent Router activity](/agent-router-enterprise/guides/operate-and-govern/audit-platform-activity/). Where to go next Know what every app and project actually costs Add an app or project axis when user and team do not match the cost centres. Keep each team's spend inside an agreed ceiling The ceilings these consumption reports are measured against. --- ## Catch spend drift before the invoice does Enterprise Tier Hard limits stop catastrophes; they do not catch the slower drifts. An agent framework that starts making ten calls where a human would have made one stays comfortably under every rate limit while it doubles the bill. The control for drift is observation on a cadence: a usage review short enough to be sustainable and regular enough that a surprise is at most a week or two old when it surfaces. **Persona:** Platform operator working in the Admin Dashboard. **Estimated time:** 10 to 15 minutes per review; weekly is typical, daily for environments with high turnover. ## Outcomes By the end of this guide: - Usage Analytics is read on a regular cadence, with per-key attribution as the unit of analysis. - The by-key, by-user, by-model, and by-provider breakdowns are each mapped to the question they answer. - Anomalies found in the review have a follow-up path: drill-down, the audit trail, or the owning team. ## Prerequisites - Administrator access to the Admin Dashboard: typically the `super_admin` or `billing_admin` role. - API keys that follow the per-purpose convention established in [Onboard developers and issue keys](/agent-router-enterprise/guides/operate-and-govern/onboard-developers-and-issue-keys/). A single organization-wide key that everyone uses makes per-purpose attribution effectively impossible. ## Step 1: Run the review on a cadence 1. Open **Usage → Analytics** in the Admin Dashboard on a regular cadence: weekly is typical, daily for environments with high turnover. 1. Apply a time range that matches the budgeting cadence (last 7 days, last 30 days, last billing cycle). 1. Switch to the **By API Key** breakdown. 1. Sort by cost descending. 1. Review the top consumers. The shape of the list should match the operator's mental model of Agent Router; surprises in this view are the most common signal that something is worth investigating. ## Step 2: Read the breakdowns against their questions The breakdowns by user, by model, and by provider all support the same workflow at different levels of aggregation. **By User** answers "which team is spending the most"; **By Provider** answers "which contract is bearing the load"; **By Model** answers "which models are doing the actual work". A row that moved sharply between reviews is the drift signal this guide exists to catch: drill into the row to see the keys, models, and costs behind it, then take the finding to the owning team. To compare several people or teams side by side on the same chart, use the multi-select **User** and **Team** filters described in [Compare user and team spend in Analytics](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/compare-user-and-team-spend-in-analytics/). Both units of measure matter, and they answer different questions. Token totals describe the workload, how much a consumer is actually asking the models to do, and are stable across pricing changes. Cost translates that workload into money and is the unit finance thinks in. ## Step 3: Route findings to the right follow-up - A drift with a legitimate cause (a feature launch, a new team onboarded) feeds the next budget conversation; the ceilings are adjusted in [Raise a spend ceiling without interrupting delivery](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/raise-a-spend-ceiling/). - A drift with no known cause is investigated through the audit trail: the spot-the-anomaly-then-find-the-cause workflow in [Audit Agent Router activity](/agent-router-enterprise/guides/operate-and-govern/audit-platform-activity/) traces a spend change back to the configuration or key event behind it. - A runaway in progress, rather than a drift, is an incident: see [Contain a leaked key before it drains the budget](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/contain-a-leaked-key/). For longer-horizon reporting (chargeback statements, quarterly reviews, or compliance attestations), the Export function returns the underlying data as CSV; the reporting workflow is covered in [Bill AI spend back to the teams that incur it](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/bill-ai-spend-back-to-teams/). For always-on visibility with alerting instead of a manual cadence, see [Get alerted to cost spikes as they happen](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/get-alerted-to-cost-spikes/). Where to go next Get alerted to cost spikes as they happen Replace dashboard-watching with alert rules in the observability stack. Keep each team's spend inside an agreed ceiling Turn the baselines read here into budgets that track spend and alert. --- ## Choose the right cost control for each workload Enterprise Tier AI traffic has a habit of growing without anybody noticing until the invoice arrives. A research team starts a benchmark, leaves it running over a weekend, and bills three weeks of normal spend in two days. A new agent framework gets wired into a production path and starts making ten calls where a human would have made one. A leaked credential lands in a public repository and an attacker spends a few hours mining it before the security team notices. Each of these scenarios is preventable, but only with the right control in the right place, and the controls are not interchangeable. Agent Router's cost story is layered. There is no single "budgets" screen that owns the entire mechanism end-to-end; instead, it is the combination of three things: **rate limits**, which enforce ceilings on traffic inline at the gateway and are the only control that rejects requests; **budgets**, which track cumulative spend per user or team over a period and raise alerts as a ceiling is approached; and **Usage Analytics**, which is where spending against those ceilings becomes visible. The **API-key-per-purpose** convention established in [Onboard developers and issue keys](/agent-router-enterprise/guides/operate-and-govern/onboard-developers-and-issue-keys/) is what makes all three precise rather than blurry. This page maps each control to the situation it fits, so the guides that follow are applied to the right problem. **Persona:** Platform operator working in the Admin Dashboard, often in partnership with the developer teams that own specific API keys and the finance owners who set spend ceilings. **Estimated time:** 10 minutes to read; the linked guides carry their own estimates. ## Match the situation to the control | Situation | Control that fits | | :---- | :---- | | A production application's spend should not exceed a known monthly ceiling | A budget, with [monitoring on a cadence](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/catch-spend-drift-early/) | | A research or evaluation team should not run away with the bill | [A budget to track spend and alert](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/keep-team-spend-inside-a-ceiling/) plus [aggressive rate limits on dedicated keys for the inline hard stop](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/stop-runaway-workloads/) | | Each team's spend should be attributable for chargeback | Per-team keys with [chargeback statements per cycle](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/bill-ai-spend-back-to-teams/) | | Spend must be booked against an app or project that does not map to a user or team | [Caller-set attribution tags](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/know-what-every-app-and-project-costs/) | | Overall spend across all consumers needs to stay under a contractual ceiling | Sum of per-key limits, plus [alerting on the aggregate](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/get-alerted-to-cost-spikes/) | | A leaked credential needs to be bounded in damage even before it is revoked | [A rate limit at the consumer's known traffic level](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/stop-runaway-workloads/), then [containment](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/contain-a-leaked-key/) | For Bring Your Own Key (BYOK) consumers (developers who present their own provider credentials, as described in [Use your own provider credentials](/agent-router-service/guides/use-your-own-provider-credentials/)), spend is attributed to the BYOK provider account directly, not to Agent Router's own usage records. The budgeting concerns there are a matter for the team that owns the BYOK account; the controls on this page cover centrally managed traffic. ## Hard ceilings or soft monitoring There are two ways to use these mechanisms, and the right choice depends on the consumer. The distinction is not a setting on a single control: an inline hard stop comes from a **rate limit**, while a **budget** always operates as soft monitoring because it tracks spend and alerts rather than rejecting requests. | Strategy | What it does | Control that provides it | When to use it | | :---- | :---- | :---- | :---- | | **Hard ceilings** | Limits enforced inline; requests beyond the ceiling are rejected | Rate limits | Experimental or CI keys where a runaway should be cut off automatically; any path where unbounded throughput is unacceptable | | **Soft monitoring** | Spend tracked against a ceiling, with Usage Analytics watched actively and alerts wired to someone who acts | Budgets and Usage Analytics | Production workloads where a false-positive block would be worse than occasional overspend; accountability and forecasting against an agreed number | Most deployments end up using a mix. Production keys for critical applications rely on budgets for soft monitoring, because a sudden block cascade is the wrong failure mode for them. Research, evaluation, and CI keys add rate limits for a hard ceiling, because a runaway there is more annoying than a rejection. ## Rate limit or budget The two ceilings are easy to conflate, but they answer different questions and operate on different axes. | | Budget | Rate limit | | :---- | :---- | :---- | | Controls | Total spend over a period, tracked with alerts | Request or token throughput over a short window, flow control | | Unit | Money or cumulative tokens, per user or team | Tokens, requests, or parallel calls per window, per key, user, group, or model | | Question answered | "How much has this person or team spent this month, and am I near the ceiling?" | "How fast may this credential consume right now?" | | Enforcement | Alert only; requests continue to be served | Inline at the gateway; requests over the ceiling are rejected | | Failure it surfaces | A cumulative overrun, flagged as spend approaches the ceiling | A burst or runaway happening in real time | A rate limit set generously can still allow a month-long drift that quietly exhausts a budget; a budget can be intact while a single misconfigured key bursts hard enough to need a rate limit. The two are configured to complement each other: the rate limit blocks the fast runaway at the key, and the budget flags the slow accumulation at the user or team so someone can act on it. Most workloads warrant both. ## The cost-control guides Catch spend drift before the invoice does The regular usage review that surfaces slow cost creep while it is still cheap. Keep each team's spend inside an agreed ceiling Turn a number agreed with finance into a budget that tracks spend and alerts as the ceiling nears. See budget exhaustion coming in time to act Read budget burn, headroom, and the members driving it, in time to intervene. Raise a spend ceiling without interrupting delivery Raise the ceiling for a team that legitimately outgrew it, and keep the change auditable. Bill AI spend back to the teams that incur it Chargeback and showback statements finance can load into its own systems. Know what every app and project actually costs Caller-set tags add the app and project axis the user and team dimensions cannot express. Stop runaway workloads before they burn the budget Token, request, and concurrency limits that wall off retry storms and fan-out bursts. Get alerted to cost spikes as they happen Telemetry export into the observability stack, with alert rules instead of dashboard-watching. Contain a leaked key before it drains the budget The incident response: bound the damage, revoke the credential, pause the workload. Configuring request logs The logging modes behind the usage data every control on this page depends on. --- ## Compare user and team spend in Analytics Enterprise Tier The User and Team filters on the Analytics page (/usage) let you select more than one at a time, so you can compare people or teams side by side. The chart, the table, and the CSV export all follow your selection. **Persona:** Platform operator working in the Admin Dashboard. **Estimated time:** 5 to 10 minutes once you know who to compare. ## Outcomes By the end of this guide: - Several users or teams are selected in Analytics, and the chip shows either a name or a count such as **2 users**. - Side-by-side spend is visible on the chart (one bar color per user or team), or a deliberate **Group** choice is preserved. - Narrowing combinations (user within a team), **No team assigned**, shareable filter URLs, and the 50-selection limit are understood. ## Prerequisites - Administrator access to the Admin Dashboard: typically the `super_admin` or `billing_admin` role. - Enough usage in the chosen time range for the people or teams you intend to compare. ## Step 1: Filter by several users or teams 1. Open **Usage → Analytics** in the Admin Dashboard. 1. Open the **User** (or **Team**) filter and tick the people or teams you want. The list stays open while you pick, and your selections stay visible at the top even while you search. 1. Confirm the chip: it shows the name when one is selected, or a count like **2 users** when more are selected. 1. Confirm that the chart, the table, and **Download CSV** all follow your selection. ![Multi-select User filter on Analytics](/img/screenshots/admin/analytics-multi-user-select.jpeg) ![Multi-select Team filter on Analytics](/img/screenshots/admin/analytics-multi-team-select.jpeg) ## Step 2: Compare them on the chart - The moment you pick a **second** user or team, the chart automatically switches to one bar color per user (or per team) — that is usually what you want when comparing. - This only happens if you have not chosen a grouping yourself. If you pick something in the **Group** control (including **No grouping**), your choice sticks and is never changed for you. - With several users selected and **No grouping**, you see the individual API keys instead, and each key is labeled with its owner (for example **Ada · prod-key**) so you can tell whose key is whose. - Clicking a user's bar in the chart, or their row in the table, focuses on just that person. ![Two users compared on the Analytics cost chart](/img/screenshots/admin/analytics-user-compare-chart.jpeg) ## Step 3: Combine filters and share the view - User and team filters combine by narrowing: selecting Ada plus the Security team shows Ada's usage *within* the Security team. If she is not in it, the result is empty — that is expected. - To see one person compared against their teammates, filter by the **team** and group by **User**. - **No team assigned** is available in the Team filter for usage from people who are not in any team, and it can be combined with real teams. - Filters are part of the page URL, so you can share a link and the other person sees the same selection (names are resolved on their side based on their permissions). - Up to 50 users or teams can be selected; past that, the picker asks you to remove one before adding another. ## What to do next - **Catch spend drift before the invoice does**: run this comparison on a cadence as part of the weekly review. See [Catch spend drift before the invoice does](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/catch-spend-drift-early/). - **Bill AI spend back to the teams that incur it**: export the filtered selection for chargeback or showback. See [Bill AI spend back to the teams that incur it](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/bill-ai-spend-back-to-teams/). - **Keep each team's spend inside an agreed ceiling**: turn baselines you read here into budgets. See [Keep each team's spend inside an agreed ceiling](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/keep-team-spend-inside-a-ceiling/). Where to go next Catch spend drift before the invoice does Use multi-select compare as part of a regular usage review. Bill AI spend back to the teams that incur it Export the filtered user or team selection for finance. --- ## Control request log data Enterprise Tier Prompt and response bodies routed through the gateway routinely contain the most sensitive data an organisation handles: customer records, source code, internal documents, regulated personal data. For many operators, the question is not whether that content is useful in the dashboard, it plainly is, but whether it is permitted to leave the data-plane cluster at all. Data-residency rules, privacy commitments, and compliance regimes frequently require prompt and completion content to stay inside the customer's own infrastructure, while usage and cost still need to be visible centrally for billing and capacity planning. The **Request logs** setting governs exactly that boundary. It controls what reaches the Tetrate management plane in `request_log` records, so the amount of request detail that leaves the cluster becomes a deliberate operator decision rather than a fixed default. The richest setting forwards full prompt and response content for inspection in the dashboard; the most restrictive setting keeps that content out of the management plane entirely. The setting accepts three values: | Mode | What reaches the management plane | What's dropped | | ----- | ----- | ----- | | **Full** *(default)* | Envelope, counters, costs, headers, `llm_parameters`, request body, response body. | Nothing. | | **Metadata only** | Envelope, counters, costs, headers, `llm_parameters`. | `request_body`, `response_body`. | | **Off** | Nothing. | The entire `request_log` record. | Selecting Request logs mode: ![Request Logs Mode selection][Image1] Each mode is a strict superset of the one below it. **Full** forwards the complete record, including request and response bodies, which gives the dashboard's per-request views their full detail. **Metadata only** forwards the envelope, counters, costs, headers, and `llm_parameters` but omits the two body fields, so usage and cost stay visible while prompt and completion content remains inside the cluster. **Off** forwards nothing, suppressing the `request_log` record at the management plane altogether. The setting is system-wide for the deployment. Switching takes effect across the data plane within a few seconds of saving: no restart, no reinstall. **Persona:** Platform operator working in the Admin Dashboard, often in coordination with security or compliance stakeholders. **Estimated time:** 5 to 10 minutes to choose, set, and verify a mode; longer if an external collector is also being wired up. ## When this guide applies The setting is relevant whenever a decision has to be made about how much request content may cross the cluster boundary toward Tetrate. Typical cases: | Situation | Suggested mode | | :---- | :---- | | Default evaluation or development, where full per-request detail in the dashboard is wanted | Full | | Prompt and completion content must stay inside the cluster, but usage and cost visibility is still required centrally | Metadata only | | No `request_log` record may leave the cluster at all, for the strictest data-residency posture | Off | | Full content is required for observability, but in a self-managed stack rather than the dashboard | Metadata only or Off, paired with an external OTEL destination | ## Outcomes By the end of this guide: - The three modes are understood, including exactly what each one forwards and what it drops. - A mode has been set from the dashboard and confirmed active with a smoke test. - The audit trail for the change has been located. - Where required, the full record has been confirmed arriving at a self-managed OTEL collector while content is withheld from the management plane. ## What each mode affects The mode affects only the records stored at the management plane. Its effect on each dashboard surface and downstream path is shown below: | Surface | Full | Metadata only | Off | | ----- | ----- | ----- | ----- | | Dashboard **Request logs** view (per-request rows) | rows with bodies | rows, body panels empty | no rows | | Dashboard **Usage** view (tokens, costs over time) | populated | populated | populated | | Dashboard **Audit logs** view | populated | populated | populated | | Billing / transactions table | populated | populated | populated | | Customer-attached OTEL destination on the data plane | full record | **full record** (mode applies only to MP) | **full record** | Two things are never affected by this setting: 1. **Billing.** Transactions are written through a separate path. Every request still produces a transaction regardless of mode, so monthly billing remains accurate. 1. **Customer-attached destinations.** A self-managed OTEL receiver (see the External destination section below) receives the full record under every mode; the toggle scopes only what crosses the cluster boundary toward Tetrate. ## How to change the mode The mode is changed entirely from the dashboard. No command-line access, redeployment, or pod restart is required. 1. Sign in to the dashboard with an admin account. 1. Go to **Settings → Request logs**. 1. Pick a mode from the dropdown. The card shows a short description of what each mode stores. 1. Click **Save**. The save action: - Persists the new mode to system settings. - Pushes the change to the data plane via the existing self-heal channel. - Writes an audit log entry recording who changed the mode and to what value. Propagation takes a few seconds. Confirm it by sending a prompt through the gateway and checking the Request logs view (see the next section). ## How to verify the mode is active Because the change takes effect within seconds, the active mode is confirmed by sending a known prompt through the gateway and observing how it appears in the dashboard. This smoke test uses the dashboard alone; no `kubectl` is required. ### Setup once In the dashboard, go to **API Keys → Create key**. Copy the `sk-...` value. Note the data plane's gateway URL (under **Settings → Workspace**). ### Smoke test per mode 1. Set the mode in **Settings → Request logs**. 1. Send a prompt with a distinctive word: ```shell curl -X POST https:///v1/chat/completions \ -H "Authorization: Bearer sk-" \ -H "Content-Type: application/json" \ -d '{"model":"claude-haiku-4-5","messages":[{"role":"user","content":"Say MANGO once and stop."}],"max_tokens":50}' ``` Alternatively, send a prompt from the playground. 1. Wait ~10 seconds, then check the dashboard. Expected result by mode: | Mode | Request logs view | Usage view | | ----- | ----- | ----- | | Full | New row for MANGO. Detail panel shows prompt + response bodies. | MANGO's tokens + cost reflected. | | Metadata only | New row for MANGO. Detail panel shows tokens + headers; body panels say "No request body available" / "No response body available". | MANGO's tokens + cost reflected. | | Off | **No row** for MANGO. | MANGO's tokens + cost still reflected (comes from the transaction path). | Repeat for each mode under test, picking a different keyword each time (MANGO / PAPAYA / DURIAN / KIWI, etc.) so the rows are easy to spot. ### Audit log check After any change, **Audit logs** in the dashboard shows a new row: - **Resource type**: `system_settings` - **Resource ID**: `request_logs.mp_mode` - **Action**: `UPDATE` - **Body**: `{"mode":""}` - **Actor**: the acting user account - **Source IP**, **User agent**, **Correlation ID** populated ## External destination: sending the full record to your own backend To keep prompt and completion content inside a self-managed observability stack while still letting Tetrate see usage and costs, configure **Metadata only** or **Off** mode on the management-plane path and route the full record to a backend you operate. The data plane always emits the full record to any local destination, regardless of mode. The typical operator pattern is: - **Mode** = `metadata_only` or `off` (depending on what reaches Tetrate) - **Local destination** = a backend subscribed to the `requestlog` signal, as below There are two ways to attach the local destination. The **collector-routed path** — subscribing a custom backend to the `requestlog` signal in Helm values — is the supported primary path. The older **direct-attach** method (hand-setting OTLP environment variables on the egress container) still works and is documented below as legacy, with a [migration path](#migrating-from-direct-attach-to-the-collector-routed-path). ### Primary path: subscribe a custom backend to the `requestlog` signal In the data-plane Helm values (the file that carries `global.observability`), add a `backends.custom[]` entry subscribed to `requestlog`: ```yaml global: observability: enabled: true signals: requestlog: enabled: true # chart default backends: custom: - name: request-log-sink endpoint: otel-collector.observability.svc.cluster.local:4317 signals: [requestlog] # explicit subscription — an omitted list never includes requestlog tls: { insecure: true } # in-cluster plaintext hop; omit for TLS sendingQueue: { queueSize: 64 } ``` Apply with `helm upgrade` (or the normal chart-management flow). The subscription is the only switch — from it, the chart wires the whole path automatically: - The data plane's local OpenTelemetry collector renders a **dedicated request-log lane**: an OTLP/gRPC receiver on port 3501 (with a raised message-size cap) plus an OTLP/HTTP receiver on 3502, and a pipeline whose destinations are exactly the subscribed backends. - The gateway's egress container is injected with the emission configuration — `OTEL_LOGS_EXPORTER=otlp`, `OTEL_EXPORTER_OTLP_LOGS_ENDPOINT=http://otel-gateway.:3501`, `OTEL_EXPORTER_OTLP_LOGS_PROTOCOL=grpc` — so the module pushes request logs to the collector over **OTLP/gRPC** (the default transport on this path). These variables are chart-managed: do not set them by hand alongside a subscription (the render fails — see the migration section). Emission and the collector lane appear and disappear together: remove the subscription and, on the next upgrade, the receiver, the pipeline, and the injected variables are all gone. The endpoint shape, `protocol: otlphttp` for backends behind HTTP load balancers or with records past a gRPC size cap, per-backend TLS, header auth from the `otel-gateway-backend-credentials` Secret, and queue sizing (the `sendingQueue: { queueSize: 64 }` above is the recommended request-log sizing) are covered field-by-field in [Custom observability backends](/agent-router-enterprise/guides/observability-and-analytics/configure-custom-observability-backends/) — including how to fan the same records out to several destinations by adding more entries. ### Legacy path: direct-attach via container environment variables :::note Legacy This is the pre-fan-out mechanism: it supports exactly one destination, speaks OTLP/HTTP, and bypasses the data plane's local collector. Existing setups keep working unchanged, but new destinations should use the collector-routed path above. ::: Two environment variables on the egress proxy container: | Variable | Value | | ----- | ----- | | `OTEL_LOGS_EXPORTER` | `otlp` | | `OTEL_EXPORTER_OTLP_LOGS_ENDPOINT` | the full URL of the collector's OTLP/HTTP logs endpoint, **including** the `/v1/logs` path | For example, a collector listening at `https://collector.observability.svc.cluster.local:4318` requires the endpoint `https://collector.observability.svc.cluster.local:4318/v1/logs`. The `/v1/logs` suffix is mandatory on this variable. Per the OTLP exporter specification, the signal-specific `OTEL_EXPORTER_OTLP_LOGS_ENDPOINT` is used verbatim, and most collectors only respond on that path. Omitting it is the single most common misconfiguration. (Alternatively, set the base variable `OTEL_EXPORTER_OTLP_ENDPOINT` to the collector URL **without** a path; the exporter appends `/v1/logs` itself. On data planes released before July 2026, only the signal-specific variable works — the base form was posted verbatim and returned 404.) When `OTEL_EXPORTER_OTLP_LOGS_PROTOCOL` is **unset, the transport stays OTLP/HTTP** (`http/protobuf`) — this has always been the direct-attach behavior, and upgrades preserve it. An explicit `OTEL_EXPORTER_OTLP_LOGS_PROTOCOL=grpc` is honored on current data planes; with gRPC the endpoint carries no `/v1/logs` path and the URL scheme selects TLS (`http://` plaintext, `https://` TLS against system CAs). In the Helm values for the egress chart, the variables go on the egress container's env block: ```yaml envoyProxy: provider: kubernetes: envoyDeployment: container: env: - name: OTEL_LOGS_EXPORTER value: otlp - name: OTEL_EXPORTER_OTLP_LOGS_ENDPOINT value: https:///v1/logs ``` Apply with `helm upgrade`. The egress pod restarts and the new endpoint becomes the local destination. If the collector requires an auth header, add it via `OTEL_EXPORTER_OTLP_HEADERS` (`"Authorization=Bearer "`); for sensitive headers, bind from a Kubernetes Secret with `valueFrom.secretKeyRef` instead of inlining. ### Migrating from direct-attach to the collector-routed path The collector-routed path replaces hand-managed env with a declared backend: it fans out to any number of destinations, moves header credentials from env strings into a Kubernetes Secret, and adds collector-side queueing and byte-aware batching in front of your backend. Migration is a single values change: 1. **Add the backend entry** (as in the primary-path example above) pointing at the same collector. For OTLP/gRPC (`protocol: otlp`, the default) the endpoint is bare `host:port` — no scheme, no `/v1/logs`. To keep talking OTLP/HTTP to a collector that only listens on 4318, set `protocol: otlphttp` and use the full `http(s)://` URL **without** the `/v1/logs` suffix (the collector exporter appends signal paths itself). 1. **In the same change, remove the operator-set variables** from `envoyProxy.provider.kubernetes.envoyDeployment.container.env`: `OTEL_LOGS_EXPORTER`, `OTEL_EXPORTER_OTLP_LOGS_ENDPOINT`, and `OTEL_EXPORTER_OTLP_LOGS_PROTOCOL` if present. Move any `OTEL_EXPORTER_OTLP_HEADERS` auth to the backend's `auth.headers` with the `otel-gateway-backend-credentials` Secret (see [Custom observability backends](/agent-router-enterprise/guides/observability-and-analytics/configure-custom-observability-backends/#authheaders)). 1. Apply, and re-run the verification below. **The render fails on collision — by design.** As soon as any backend subscribes to `requestlog`, those three variables become chart-managed, and the chart refuses to render a values file that also sets any of them on the container env. `helm template` / `helm upgrade` aborts before anything rolls out, with an error naming the colliding key: ```text egress-helm: OTEL_LOGS_EXPORTER is chart-managed while the requestlog fan-out lane is active (global.observability.signals.requestlog.enabled plus a subscribed observability.backends.custom[] entry), but it is also set via envoyProxy.provider.kubernetes.envoyDeployment.container.env (legacy direct-attach). Remove the operator-supplied key and configure the destination as a custom backend instead — see the request-logs migration guide: https://docs.tetrate.ai/guides/ops/cost-and-quota/configuring-request-logs ``` The fix is exactly what the message says: delete the named key (and its siblings) from the container env, keeping the destination only as a `backends.custom[]` entry. There is no silent precedence — the chart never lets the two mechanisms configure the same variables at once. Conversely, while **no** backend subscribes to `requestlog`, operator-set variables pass through untouched, which is why existing direct-attach setups keep working until they opt in. (The same rule protects the guardrail-trigger lane's `AIDISCOVERY_OTEL_EXPORTER_OTLP_LOGS_ENDPOINT`.) What changes in behavior after migrating: | | Direct-attach (legacy) | Collector-routed | | :---- | :---- | :---- | | Destinations | Exactly one | Any number of subscribed backends | | Transport from the module | OTLP/HTTP (unset protocol) | OTLP/gRPC to the local collector; per-backend `otlp` or `otlphttp` onward | | Auth | Header string in container env | Per-backend headers from the `otel-gateway-backend-credentials` Secret | | Buffering | SDK exporter only | Plus a sized in-memory queue and byte-aware batching per backend in the collector | | Lifecycle | Hand-managed env | Rendered and removed with the subscription | ### Verifying the external destination The external destination is verified by setting the management-plane path to drop everything, then confirming the full record still arrives at the local collector. 1. Stand up (or reuse) an OTEL collector reachable from the data-plane cluster on the URL above. 1. Apply the Helm values change. Wait for the egress pod to roll. 1. Set the dashboard mode to **Off** (the strongest case: Tetrate gets nothing). 1. Send a test prompt via `curl` as above. 1. Within ~10 seconds, the collector logs a record with: - Body: the full request-log record as a plain JSON document — `request_id`, model, tokens, costs, and the request and response bodies inline. No decompression or decoding is needed; any OTLP-compatible backend can display and index it directly. - Attribute `event.type`: `request_log`, plus `gen_ai.*` and `tars.*` dimensions (`tars.request.id` joins these records with guardrail-trigger records shipped to the same stack). - Resource attribute `log.type`: `request_log` Request logs are the only records on this stream. Internal management-plane records (billing transactions, access-log forwards, metrics) never reach the local destination. :::note Older data planes Data planes released before July 2026 sent the local destination the management-plane transport envelope instead: records with body `Request log` or `ALS access log data export` and an opaque gzip-compressed `payload` attribute marked `json+gzip` / `protobuf+gzip`. If your collector shows those, upgrade the data plane; the record content was unreadable by standard OTLP backends. ::: 1. The dashboard's **Request logs** view shows no new row for this prompt (off mode at MP). The dashboard's **Usage** view does show the tokens + cost (transaction path). If the collector receives nothing, on the **collector-routed path**: - Confirm the emission variables were injected and the collector lane rendered: ```shell kubectl get envoyproxy -n -o yaml | grep -A1 OTEL_ kubectl get cm otel-gateway-config -n -o yaml | grep -A5 requestlog ``` If both are absent, one leg of the gate is off: `global.observability.enabled`, `signals.requestlog.enabled`, or the backend's explicit `signals: [requestlog]` subscription (an omitted `signals` list never includes `requestlog`). - Check the `otel-gateway` collector pod's logs for export errors toward your backend, and see the delivery-semantics section of [Custom observability backends](/agent-router-enterprise/guides/observability-and-analytics/configure-custom-observability-backends/) for queue-overflow and oversized-record failure modes. - Confirm the backend is reachable from the data-plane cluster's pod network. If it is outside the cluster (for example, behind a tunnel or external load balancer), test reachability with `curl` from a debug pod first. On the **legacy direct-attach path**: - Confirm `OTEL_EXPORTER_OTLP_LOGS_ENDPOINT` ends in `/v1/logs`. - Confirm `OTEL_LOGS_EXPORTER=otlp` is set; without it the local destination is disabled. - Check the egress pod's container logs for `OTEL SDK error` lines; they include the failing URL and error code. ## Frequently asked questions ### Does switching to `off` break monthly billing? No. Transactions are written through a separate code path and arrive regardless of mode. The Usage view and billing reports keep working. ### Does switching to `off` lose audit information about who ran which prompt? The `request_log` row is the per-request record at the management plane. Under `off` it is not stored at MP. If that detail is required, run `metadata_only` (envelope + tokens reach MP, bodies do not) or wire an external destination as above. ### Does switching to `metadata_only` retroactively strip bodies from rows already in the database? No. The setting affects only records produced from the moment it takes effect onward. Existing rows are unchanged. ### Can different workspaces have different modes? Not in this version. The setting is system-wide for the deployment. A per-workspace control is on the roadmap if customers need it. ### What happens when an unknown mode value is saved via a direct database write? The data plane treats anything it does not recognize as `full`, so a misconfigured value can never silently drop data. The dashboard only ever writes one of the three known values. ### Why is the body panel empty in the dashboard? Either the request genuinely had no body, the response was empty (for example, an error before a generation finished), or the mode when the request was processed was `metadata_only`. The dashboard does not currently distinguish these cases in the UI. [image1]: Where to go next Custom observability backends The full field reference for backends.custom[], more signals to subscribe, and the delivery guarantees behind the collector-routed path. Manage log retention and purge Govern how long the records each mode produces are kept and how they are removed. Configure data residency and no-retention Keep regulated content in-region and unretained at the provider. --- ## Contain a leaked key before it drains the budget Enterprise Tier A leaked credential lands in a public repository, and an attacker spends a few hours mining it before the security team notices. A runaway integration is doing the same damage with no attacker at all. Either way, money is leaving right now, and the response has two halves: bound the damage immediately, then cut the credential off entirely. **Persona:** Platform operator, usually paged by a spend alert, a usage-review surprise, or a security notification. **Estimated time:** Minutes; this is an incident runbook, not a project. ## Outcomes By the end of this guide: - A compromised or runaway key is revoked, immediately and irreversibly. - The role of a pre-set rate limit as a damage bound before revocation lands is understood. - A workload can be paused deliberately, short-term or long-term, without ambiguity about the mechanism. ## Prerequisites - Administrator access to the Admin Dashboard, with permission to manage API keys and users. - The key-revocation mechanics from [Onboard developers and issue keys](/agent-router-enterprise/guides/operate-and-govern/onboard-developers-and-issue-keys/). ## Revoke the key The fastest response is to revoke the key from the Admin API Keys surface; the mechanics are covered in [Onboard developers and issue keys](/agent-router-enterprise/guides/operate-and-govern/onboard-developers-and-issue-keys/). Revocation is immediate and irreversible: the very next request on the credential is rejected. Reissue a replacement key to the owning team once the incident is understood. ## Why the rate limit already mattered A rate limit set at the key's normal traffic level is the damage bound that holds before anyone is paged: even if the revocation is delayed by minutes or hours, the limit caps the worst-case burn in the interim to a multiple of normal traffic instead of an unbounded mining run. Setting limits proactively on every production key, not just experimental ones, is partly a cost story and partly a security story; the practice is covered in [Stop runaway workloads before they burn the budget](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/stop-runaway-workloads/). ## Pause a workload deliberately Not every containment is an emergency. For a planned pause (a maintenance window, a vendor escalation, or a budget freeze), the mechanism is the same: revoke the API key and reissue when the pause ends. There is no "disable temporarily" state on a key; revocation is the available mechanism, and the developer team coordinates with the operator on the timing. For longer-term pauses, the user account itself can be marked inactive in the Users surface, which prevents further activity until the account is reactivated. ## Close the loop After containment, two follow-ups keep the incident from repeating: confirm the revocation and any limit changes are visible in the audit trail ([Audit Agent Router activity](/agent-router-enterprise/guides/operate-and-govern/audit-platform-activity/)) with the incident reference attached, and check whether an alert would have caught the spike earlier than it was actually caught; if not, the missing rule is described in [Get alerted to cost spikes as they happen](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/get-alerted-to-cost-spikes/). Where to go next Stop runaway workloads before they burn the budget The proactive rate limits that bound the damage before the page arrives. Get alerted to cost spikes as they happen The alert rules that turn the next incident from hours into minutes. --- ## Get alerted to cost spikes as they happen Enterprise Tier The in-Console surfaces are sufficient for periodic review and for active investigation. They are not the right place for always-on visibility into spend across a large fleet of keys, because a dashboard nobody is watching catches nothing. The always-on answer is telemetry export: Agent Router's per-request cost dimensions flow into the observability stack the organisation already runs, where alert rules page someone the moment spend moves outside expectations. **Persona:** Platform operator, in partnership with the team that owns the observability stack. **Estimated time:** Depends on the observability stack; the export itself is covered in the developer-side telemetry guide. ## Outcomes By the end of this guide: - Agent Router's per-request cost metrics flow into an external observability stack over OpenTelemetry. - At least one alert rule fires on a spend anomaly rather than relying on someone watching a dashboard. - The division of labour is understood: enforcement stays inline at the gateway, oversight lives in the observability stack. ## Prerequisites - An OpenTelemetry-compatible observability stack (Grafana, Datadog, Honeycomb, or equivalent) and access to configure alert rules in it. - The telemetry export configured; the plumbing is covered in [Export telemetry to an observability stack](/agent-router-enterprise/guides/observability-and-analytics/export-telemetry-to-an-observability-stack/). ## Step 1: Export the cost dimensions Agent Router emits per-request metrics with the dimensions that cost oversight cares about: API key identifier, resolved model, provider, status, and token counts. Pulled into the observability stack over OpenTelemetry, these dimensions support cost-per-key, cost-per-team, and cost-per-model views on whatever cadence the platform refreshes. ## Step 2: Wire the alert rules The rules that earn their keep in practice: - A key's hourly cost exceeds a threshold derived from its baseline. - A key's traffic shape changes unexpectedly (request rate, token volume, or error rate moving sharply against its history). - Aggregate Agent Router spend crosses a contractual ceiling agreed with the provider or with finance. Each alert should route to someone who will act on it; an alert channel nobody reads is the dashboard problem in a different costume. The follow-up paths are the same as for a manual review: legitimate growth feeds a ceiling adjustment ([Raise a spend ceiling without interrupting delivery](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/raise-a-spend-ceiling/)), and an illegitimate spike is an incident ([Contain a leaked key before it drains the budget](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/contain-a-leaked-key/)). ## Step 3: Correlate across systems The export's third win is correlation: joining Agent Router spend to the rest of the application stack, so that a sudden cost spike can be tied to a specific application deployment, a specific feature release, or a specific incident. This is what the in-Console surfaces cannot do, and it is usually the difference between "spend went up" and "spend went up because Tuesday's release doubled the context size". The split is intentional. Rate limits are enforced inline by Agent Router, where the cost of action is low. Spending oversight, including budget alerts, lives in the observability platform, where the cost of integrating with the rest of the organisation's tooling has already been paid. Where to go next Export telemetry to an observability stack The OpenTelemetry plumbing behind the alert rules described here. Catch spend drift before the invoice does The manual review cadence these alerts complement but do not replace. --- ## Keep each team's spend inside an agreed ceiling Enterprise Tier Spend questions rarely arrive as questions about API keys. They arrive as questions about people and the parts of the business those people belong to. A finance partner wants to know when the data-science organisation is approaching the number agreed for the quarter, without anyone having to watch it by hand. This guide turns that agreed ceiling into a budget that tracks spend and raises an alert as the limit is approached or crossed: baseline the existing spend, create the budget at the right scope, and set the monthly limit that drives the alert. **Persona:** Platform operator working in the Admin Dashboard, implementing a ceiling agreed with finance or the relevant business owner. **Estimated time:** 20 to 30 minutes for an initial setup with one or two teams; 5 to 10 minutes per budget thereafter. ## Outcomes By the end of this guide: - Usage and cost, in both tokens and money, can be read per user and per team over a chosen period. - A monthly spend ceiling exists for a chosen team or person, at a scope that matches how the cost is owned. - Spend is tracked against that ceiling and an alert is raised as the budget is approached or crossed, so the operator and business owner can act in time. :::note Budgets alert, they do not block Agent Router budgets are an observability control, not an inline gateway block. Spend is evaluated on the management plane against aggregated request logs, so a budget raises alerts and shows burn against the ceiling but does **not** reject requests once the limit is reached. To cut traffic off inline, use a rate limit instead. See [Stop runaway workloads](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/stop-runaway-workloads/). ::: ## Prerequisites - Administrator access to the Admin Dashboard, typically the `super_admin` or `billing_admin` role. - API keys that follow the per-purpose convention, so that usage attributed to a user or team is precise rather than blurred across shared credentials. The convention is established in [Onboard developers and issue keys](/agent-router-enterprise/guides/operate-and-govern/onboard-developers-and-issue-keys/). - Group identity flowing in from the identity provider. Team budgets are only as meaningful as the team membership behind them; the mapping is covered in [Map Entra ID groups to business functions](/agent-router-enterprise/guides/identity-and-access-management/single-sign-on/map-entra-id-groups-to-business-functions/). - Agreement with finance or the relevant business owner on the ceiling itself and on what should happen when it is reached. A budget is a policy decision before it is a configuration; the operator implements the decision rather than originating it. ## Step 1: Read the baseline before setting the number Before a budget can be set sensibly, the existing consumption has to be visible at the same dimension the budget will use. 1. Sign in to the Admin Dashboard and open **Usage → Analytics**. 1. Select tab **Cost**. 1. Apply a time range that matches the budgeting cadence: last 30 days or the last billing cycle is the usual choice for a monthly budget. 1. Select the **Group by User** breakdown and review the ranked list. Each row carries the request count, token totals, and estimated cost for that user across all of their keys. 1. Select a user row to drill into the specific keys, models, and costs behind it. To compare several people or teams side by side before you set the number, see [Compare user and team spend in Analytics](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/compare-user-and-team-spend-in-analytics/). The team dimension is the aggregate of its members. Where the deployment surfaces a team breakdown directly, it is read the same way; where it does not, team spend is derived by summing the cost of the users mapped to that team, which is exactly why the group-to-team mapping has to be correct before team budgets carry any weight. A user counted in the wrong team is spend charged to the wrong cost centre. Size the ceiling from this baseline rather than guesswork: comfortably above normal consumption but well below any level that would be a problem. A budget for a team with no history is tightened rather than left loose; real traffic surfaces the true shape quickly, and relaxing a tight ceiling is a low-risk adjustment. ## Step 2: Choose the budget scope Open the **New budget** dialog from **Usage → Budgets** in the Admin Dashboard and select one of the three scopes. | Scope | What the limit covers | When it fits | | :---- | :---- | :---- | | **Whole team** | One shared limit pooled across all members of the team | A cost centre with a single ceiling over everyone charged to it, where one member's heavy month can be offset by another's light one | | **Each teammate** | The same limit applied to every member individually | A per-person allowance that should hold for each member regardless of what the rest of the team spends | | **One person** | A limit that applies to a single user | An individual whose consumption is bounded on its own, independent of any team | A pooled limit bounds the team as a unit and tolerates uneven consumption between members; a per-member limit holds each person to the same allowance and flags any single member who exceeds it. The two layers coexist: a team can carry an overall ceiling while individual members carry their own, tighter ceilings underneath it, and the most restrictive applicable limit governs. ## Step 3: Select the team or person The selection list below the scope reflects the scope chosen in Step 2: a team scope lists teams, and the single-user scope lists individual users. 1. Use the search box to filter the list by name when the list is long. The match count is shown alongside the field. 1. Select the target team or person from the list. Each team row carries its member count, so that the reach of a pooled or per-member limit is visible before the budget is created. ## Step 4: Set the monthly spend limit The spend limit is the ceiling itself, expressed in US dollars per month, because the conversation behind it was conducted in money. 1. Enter the amount in the **Spend limit** field, or select one of the preset amounts ($100, $500, $1,000, $5,000, or $10,000) as a starting point. 1. Confirm the reset cadence. The budget resets at the start of every calendar month in UTC, so a limit set mid-month still governs the remainder of that month against spend already recorded. ## Step 5: Understand what happens at the ceiling A budget tracks cumulative spend against the ceiling and raises an alert as the limit is approached or crossed. It does not stop traffic: requests continue to be served after the ceiling is reached, because spend is evaluated on the management plane against aggregated request logs rather than enforced inline at the gateway. A budget is therefore a forecasting and accountability tool that gives the operator and business owner time to act, not a hard contractual cap that cuts requests off automatically. Because the ceiling does not block, the alerting behind it has to be wired to someone who will act on it. When the budget is approached, the legitimate responses are to investigate the spend, raise the ceiling if the growth is agreed and funded, or accept the overspend for the remainder of the period. See [See budget exhaustion coming](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/see-budget-exhaustion-coming/) for reading the burn in time to intervene. When you need a genuine hard stop, that job belongs to a rate limit rather than a budget: a rate limit is enforced inline at the gateway and rejects requests over the ceiling as they arrive. Pair a budget with [aggressive rate limits on dedicated keys](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/stop-runaway-workloads/) for evaluation or experimentation workloads where a runaway must be cut off automatically. ## Step 6: Name the budget and create it 1. Enter a descriptive name in the **Name** field, for example `Engineering monthly cap`. The name identifies the budget in the admin list where every budget is reviewed together, so it should describe what the budget covers rather than restate the amount. 1. Review the policy summary, which confirms the scope, target, and limit about to be applied. 1. Select **Create budget** to save the policy. Once the budget exists, the reasoning behind the ceiling belongs somewhere durable alongside it. The audit trail in [Audit Agent Router activity](/agent-router-enterprise/guides/operate-and-govern/audit-platform-activity/) records that the budget was created and who created it; the business justification for the number belongs next to that record. ## What to do next - **See budget exhaustion coming**: read the burn against the ceiling created here, in time to intervene. See [See budget exhaustion coming in time to act](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/see-budget-exhaustion-coming/). - **Bill AI spend back to the teams that incur it**: turn the budgets into the statements finance consumes. See [Bill AI spend back to the teams that incur it](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/bill-ai-spend-back-to-teams/). - **Stop runaway workloads**: the inline blocking companion to this budget. A budget alerts on cumulative spend; a rate limit rejects requests at the gateway when a hard stop is required. See [Stop runaway workloads before they burn the budget](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/stop-runaway-workloads/). Where to go next See budget exhaustion coming Read burn, headroom, and the members driving it before the ceiling hits. Bill AI spend back to the teams that incur it Produce the per-user and per-team statements finance consumes. --- ## Know what every app and project actually costs Enterprise Tier "What did the support-bot initiative cost this quarter?" is a funding question, and the user and team dimensions cannot answer it: spend is grouped by who issued the request and which organisational unit they belong to, while the question runs along the application or project the cost should be booked against. Caller-set tags supply that missing axis, so per-initiative cost stops being an estimate and becomes a report. A tag is a label the caller sets on a request (an app, a project, or whatever cost dimension the business reports along) so that cost can be grouped beyond the fixed user and team breakdowns. This guide agrees and enforces the tag dimensions, then exports the grouped reports finance and team leads consume. It extends the reporting discipline in [Bill AI spend back to the teams that incur it](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/bill-ai-spend-back-to-teams/) along the tag axis. **Persona:** Platform operator working in the Admin Dashboard, in partnership with finance, business owners, and the developers who set tags at the source. **Estimated time:** 30 to 45 minutes to agree and wire the first tag dimensions; ongoing as workloads and cost centres evolve. ## Outcomes By the end of this guide: - A governed set of tag dimensions has been agreed with finance and the business owners, carried on requests, and constrained by policy where attribution must be trustworthy. - A consumption report has been grouped by a tag and read against the budgets the tagged workloads sit under. - A tag-grouped cost report has been exported for the finance process or team lead that owns that cost. ## Prerequisites - Administrator access to the Admin Dashboard, typically the `super_admin` or `billing_admin` role. - The reporting and export workflow already in use, since tag attribution extends it along a new axis. See [Bill AI spend back to the teams that incur it](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/bill-ai-spend-back-to-teams/). - Cooperation from the developers who own the workloads, because tags are set on the request at the source. The convention is documented for those teams in [Monitor traffic and usage](/agent-router-service/guides/monitor-traffic-and-usage/). ## Step 1: Attribute cost with caller-set tags 1. Agree the tag dimensions with finance and the business owners before any are set. A tag is only useful for chargeback if everyone producing requests and everyone reading reports uses the same set of names; an ungoverned tag space produces a long tail of near-duplicate labels that no report can sum across. 1. Establish how each tag is carried on the request, typically as request metadata the calling code attaches, alongside the credential that already identifies the user. The convention is documented for the teams that own the workloads in [Monitor traffic and usage](/agent-router-service/guides/monitor-traffic-and-usage/), so that the tags a report depends on are set consistently at the source. 1. Constrain the tags by policy where the attribution has to be trustworthy. A policy can require that a tag is present before a request is served and can restrict its value to an agreed set, so that a caller can neither omit the attribution nor book spend against an app or project that is not theirs. Without this, a tag is a hint rather than a billable fact. 1. Group a consumption report by a tag the same way a report is grouped by user or team, by app or by project, and read it against the budgets the tagged workloads sit under. Tag-based attribution coexists with the user and team dimensions rather than replacing them. A single request carries its user, its team membership, and whatever tags the caller set, so the same spend can be reported per person, per cost centre, and per project without re-running the workload. The fixed dimensions answer "who spent this"; the tags answer "what should it be booked against", and the two are most useful read together. Where the attribution feeds chargeback, the policy constraint is what makes it defensible. A report grouped by a tag that any caller could set to any value invites disputes about whose budget a line item belongs to; a report grouped by a tag Agent Router required and validated carries the same authority as the user dimension behind it. That a tagging or attribution policy changed, and who changed it, is recorded in the audit trail described in [Audit Agent Router activity](/agent-router-enterprise/guides/operate-and-govern/audit-platform-activity/), alongside the budget changes made when [setting the team ceilings](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/keep-team-spend-inside-a-ceiling/). ## Step 2: Export grouped cost reports for finance and team leads A tag dimension earns its keep the same way the user and team dimensions do, in a report finance and team leads actually consume. The [chargeback export](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/bill-ai-spend-back-to-teams/) carries the tag dimensions out of Agent Router alongside the user, cost, and token totals already covered. 1. Apply the time range, the tag grouping (by app or by project), and any user or team filter that match the chargeback period and the cost-centre structure. Multi-select User and Team filters work the same way here as in [Compare user and team spend in Analytics](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/compare-user-and-team-spend-in-analytics/). 1. Use the **Export** function to download the grouped data, typically as CSV, with one row per tag value over the period. 1. Hand the app- or project-grouped file to the finance process or the team lead who owns that cost, or load it into the billing system that owns the chargeback allocation. A tag-grouped export gives a team lead a statement of their own team's spend without exposing the wider organisation's figures, and gives finance a per-project breakdown that maps onto the cost centres a user or team view cannot express. Agent Router's role still ends at producing accurate, attributable rows; the grouped file joins to the same downstream ledgers as the user- and team-based exports, along the dimension the business reports against. ## What to do next - **Bill AI spend back to the teams that incur it**: the user- and team-based reporting this guide extends along the tag axis. See [Bill AI spend back to the teams that incur it](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/bill-ai-spend-back-to-teams/). - **Monitor traffic and usage**: the developer-side view where the tags this guide depends on are set at the source. See [Monitor traffic and usage](/agent-router-service/guides/monitor-traffic-and-usage/). - **Audit Agent Router activity**: where tagging-policy changes are recorded, alongside budget changes. See [Audit Agent Router activity](/agent-router-enterprise/guides/operate-and-govern/audit-platform-activity/). Where to go next Bill AI spend back to the teams that incur it The user- and team-based reporting this guide extends along the tag axis. Monitor traffic and usage The developer-side view where the tags this guide depends on are set. --- ## Raise a spend ceiling without interrupting delivery Enterprise Tier Sooner or later a team legitimately outgrows its ceiling: a feature launches, usage doubles for good reasons, and the budget that tracked last quarter's plan now fires alerts against this quarter's higher, agreed spend. A budget never blocks traffic, so nothing is interrupted, but a ceiling that no longer matches reality either floods the owner with false alarms or, raised carelessly, stops fine-grained cost control. The fix stays governed: the new number agreed, the change recorded, and the exception distinguishable from a silent giving-up on cost control. **Persona:** Platform operator working in the Admin Dashboard, responding to a budget alert or a request from the team or business owner. **Estimated time:** Minutes per adjustment, once the new ceiling is agreed. ## Outcomes By the end of this guide: - A budget's limit is raised in place, effective immediately against spend already recorded this period. - Tracking and alerting can be paused without discarding the budget while a ceiling is renegotiated. - An obsolete budget is retired and, when it should not remain even as inactive, permanently deleted; every change lands in the audit trail with its reasoning recorded nearby. ## Prerequisites - Administrator access to the Admin Dashboard, typically the `super_admin` or `billing_admin` role. - Agreement on the new ceiling with the finance or business owner who set the original. The operator implements the decision rather than originating it; an unagreed raise is the silent giving-up this guide exists to avoid. ## Decide before adjusting An approaching or crossed ceiling has three legitimate responses, and the adjustment is only one of them: raise the ceiling when the growth is agreed to be real and funded; investigate the cause when the growth is unexplained (the workflow in [Catch spend drift before the invoice does](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/catch-spend-drift-early/) and the audit trail behind it); or accept the overspend for the remainder of the period. The alert gives the operator and the business owner time to decide deliberately rather than having Agent Router decide for them. ## Raise the limit in place The budget detail view is where a budget is adjusted after creation. 1. Open the budget's detail view and locate the configuration rail on the left. 1. Edit the **Spend limit** in place. A change to the limit applies to the current period immediately, measured against spend already recorded, so the meter and its alerts reflect the new ceiling as soon as the limit saves. 1. Edit the **Budget name** in place if the change makes the old name misleading (for example, when a quarterly uplift becomes permanent). Because a budget alerts rather than blocks, traffic is never interrupted in either direction: requests continue to flow throughout, and raising the limit simply resets the burn and alerting against the new, higher ceiling. ## Pause tracking during a renegotiation The **Status** toggle on the detail view pauses tracking and alerting without discarding the budget. This is the right state while a ceiling is being renegotiated: the policy and its history stay in place, but a stale number stops firing misleading alerts against spend the business has already agreed to fund. Reactivate the budget once the new ceiling is set. ## Retire a budget **Deactivate policy** on the detail view (or via the Active/Inactive controls on the budgets list) retires the budget without removing it: the right end state for a cost centre that no longer exists or a per-person budget superseded by a team one. Retirement is preferable to a ceiling raised so high it never fires, because a budget that cannot fire reads as active governance while providing none. Inactive budgets stay under the **Inactive** filter and can be reactivated later. When the budget should not remain even as inactive—test budgets, superseded policies that will never return—remove it permanently after deactivation: 1. Deactivate the budget first. An active budget cannot be permanently deleted; the console refuses the attempt and asks you to deactivate it first. 1. Switch to the **Inactive** filter on **Usage → Budgets**, open the budget's **⋯** menu, and select **Delete permanently…**. 1. Confirm in the dialog. Permanent deletion cannot be undone. **Delete permanently…** appears only for callers with the budget **delete** permission. Deactivate and reactivate work exactly as before and do not require that permission. **Good to know** - The audit log entry is the remaining record of a permanently deleted budget. - Past alerts the budget triggered are kept; they no longer link to a budget. - Nothing changes for the normal flow: deactivating and reactivating budgets works the same as always. ## Keep the change accountable Every edit, pause, deactivation, and permanent deletion is captured in the audit trail described in [Audit Agent Router activity](/agent-router-enterprise/guides/operate-and-govern/audit-platform-activity/), with who made the change and when. The business justification for the new number belongs next to that record, the same discipline as at creation: a ceiling whose reasoning is recoverable a quarter later is a control; one whose reasoning is lost is a leftover. Where to go next See budget exhaustion coming The early-warning read that triggers this guide before the overrun lands. Audit Agent Router activity Where every budget change made here is recorded, with actor and timestamp. --- ## See budget exhaustion coming in time to act Enterprise Tier A budget that quietly overruns defeats its purpose, because it alerts rather than blocks: nothing stops the spend on its own, so someone has to read the burn and act. That is avoidable when budget burn is read on a cadence: the detail view shows how much of a ceiling is gone, how many days remain in the period, and which members are driving the spend, which is enough to intervene at 80% instead of explaining at 100%. This guide reads the budget detail view produced by [Keep each team's spend inside an agreed ceiling](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/keep-team-spend-inside-a-ceiling/). The two layouts described here correspond to the **Whole team** and **Each teammate** scopes chosen at creation; the single-user scope reads as a one-row case of the per-member view. **Persona:** Platform operator working in the Admin Dashboard, reviewing spend on a regular cadence with the finance and business owners who set the ceilings. **Estimated time:** A few minutes per budget, repeated each reporting cycle. ## Outcomes By the end of this guide: - A budget's current spend, remaining headroom, and time left in the period can be read from its detail view. - The pooled whole-team layout and the per-member each-teammate layout are each understood, along with the member status indicators. - Billed spend is distinguished from passthrough usage, and the reason passthrough is excluded from the ceiling is understood. ## Prerequisites - Administrator access to the Admin Dashboard, typically the `super_admin` or `billing_admin` role. - At least one budget already created. See [Keep each team's spend inside an agreed ceiling](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/keep-team-spend-inside-a-ceiling/). ## Open the budget detail Each budget opens into a detail view split into two parts: a configuration rail on the left and a spend overview on the right. The left rail is the same for every budget regardless of scope: - **Scope** is shown as a pill at the top: **Team total** for a pooled whole-team budget, **Team — per user** for a per-member each-teammate budget. - **Budget name** is the descriptive label set at creation. - **Spend limit** is the ceiling and its cadence: `monthly` for a pooled budget, `each member, monthly` for a per-member budget. - **Status** is an Active toggle, and **Audit** records when the budget was created. Changing any of these is covered in [Raise a spend ceiling without interrupting delivery](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/raise-a-spend-ceiling/). The right side is the spend overview, and its shape follows the scope. ## Billed spend and passthrough usage Both layouts lead with **billed** spend against the ceiling. That is the only figure the meter and alerts use. **Passthrough usage** appears only when members have run **Claude Code passthrough mode** (Max/Pro subscription through the gateway). The line is labeled **passthrough usage (not counted toward budget)** and shows an approximate dollar value (`≈`): the theoretical cost of those requests **if the user did not have the subscription plan**, priced at standard API rates. It is for scale and comparison only—it never draws down the budget. | Spend overview | When you see it | | :---- | :---- | | **Simpler view** | No Claude Code passthrough traffic: billed against the ceiling, progress bar, **Used** / **Remaining** / **Days left** only. | | **With passthrough** | Claude Code passthrough mode configured and used: the same billed meter, plus **passthrough usage (not counted toward budget) ≈ …**. On a team **per user** budget, each member row can show **billed** and **passthrough** side by side against that member’s cap. | BYOK (provider API keys) is a separate Step 3 path; it does not add the Claude Code passthrough line described here. Claude Code passthrough mode is covered in the [fast track evaluation](/agent-router-enterprise/fully-managed/fast-track/#your-real-traffic). ## Read a whole-team budget A whole-team budget pools one ceiling across every member, so its overview leads with a single meter. - The headline reads the billed spend against the ceiling for the period, for example `$260.75 of $5,000.00 · monthly`, with a progress bar toward the 100% mark. - **Used** is that spend as a percentage of the ceiling, **Remaining** is the headroom left, and **Days left** is the number of days before the period resets at the start of the next calendar month. - **Top contributors** ranks the members by their share of the team's spend this period. Each row shows the member's billed amount and their percentage of the team total; when Claude Code passthrough mode is in use, passthrough usage appears alongside. The pooled meter answers the question a cost-centre owner asks first: how much of the team's allocation is gone and how much time is left in the period. The pairing of **Used** and **Days left** is the early-warning read: 80% used with 20 days left is a conversation to have this week, not at the reset. The contributor ranking answers the follow-up: which members account for the spend. Because the cap is pooled, an individual's heavy month is absorbed into the team total rather than flagged on its own, so the ranking is for attribution. ## Read a per-member budget An each-teammate budget applies the same ceiling to every member individually, so its overview leads with the health of the membership rather than a single meter. - The headline counts how many members are within their limits, for example `7 of 7 members within limits`, with a breakdown into **over** (red), **near** (amber), and **healthy** (green). - **Total billed** is the summed billed spend across all members, **Total allotted** is the per-member cap multiplied by the member count, and **Per-member cap** is the ceiling each member carries. - The **Members** list shows one row per member, each with that member's billed spend against their own cap and the percentage consumed. When Claude Code passthrough mode is in use, each row also shows passthrough usage (`≈`) for scale. The list can be filtered with **Search members** and ordered with the **Spend** sort. The per-member layout is read for exceptions. The **near** (amber) rows are the early warning this guide exists for: members approaching their cap who can still be contacted before they cross it. The **over** rows need attention now; the healthy majority needs none. Because each member carries the same cap independently, one member exceeding their allowance has no effect on the others' meters. ## What to do next - **Raise a spend ceiling**: when the burn read here is legitimate and the ceiling no longer fits. See [Raise a spend ceiling without interrupting delivery](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/raise-a-spend-ceiling/). - **Catch spend drift early**: the wider usage review that puts a single budget's burn in context. See [Catch spend drift before the invoice does](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/catch-spend-drift-early/). - **Bill AI spend back to the teams that incur it**: turn the spend read here into statements finance consumes. See [Bill AI spend back to the teams that incur it](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/bill-ai-spend-back-to-teams/). Where to go next Raise a spend ceiling without interrupting delivery Act on the early warning: adjust the ceiling before the team overruns it. Bill AI spend back to the teams that incur it Turn the spend read here into statements for chargeback and showback. --- ## Stop runaway workloads before they burn the budget Enterprise Tier A single misbehaving consumer can degrade Agent Router for everyone and burn a month of budget in an afternoon. An agent framework caught in a retry loop hammers the gateway thousands of times a minute. A batch job opens hundreds of parallel connections and saturates the upstream provider's concurrency allowance. A new integration ships with no client-side throttling and sends traffic in bursts that the rest of the fleet then waits behind. Rate limits are the control that keeps any one consumer from consuming more than its share, so that Agent Router stays responsive under load, upstream providers are not pushed past their own quotas, and a runaway hits a wall at two to three times normal traffic instead of at three weeks of spend. Tetrate Agent Router exposes three distinct rate limits, configured in the Admin Dashboard. Tokens per minute (TPM) caps how much model work a consumer can drive; requests per minute (RPM) caps how often it can call regardless of payload size; and a maximum-parallel-requests limit caps how many calls it can have in flight at once. Each addresses a different failure mode, and they are usually combined. This guide covers what each limit controls, how to apply limits at the model, API key, user, and group scopes, how limits at different scopes compose, what a caller experiences when a limit is hit, and how to confirm whether limits are actually being reached. A rate limit is flow control, not a spend cap. It protects shared capacity by smoothing the rate of traffic; it does not stop a consumer from spending a budget over the course of a month. The two are complementary, and choosing between them is covered in [Choose the right cost control for each workload](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/choose-the-right-cost-control/); the spend caps themselves are set in [Keep each team's spend inside an agreed ceiling](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/keep-team-spend-inside-a-ceiling/). **Persona:** Platform operator working in the Admin Dashboard. **Estimated time:** 15 to 25 minutes for an initial pass; ongoing as workloads evolve. ## When this guide applies Rate limits are the right concern in any of these situations: | Situation | Limit type that fits | | :---- | :---- | | A retry loop or runaway agent is sending far more requests than a healthy client would | Requests per minute (RPM) | | A consumer drives large token volumes (long contexts and verbose completions) that strain upstream capacity | Tokens per minute (TPM) | | A batch job opens many connections at once and saturates the provider's concurrency allowance | Maximum parallel requests | | One team's traffic should never crowd out another team sharing the same deployment | A per-group limit across all of that group's keys | | A new or untrusted integration should be bounded before its real traffic shape is known | A deliberately tight limit at the key scope | | Spend over a month should be capped regardless of request rate | A budget, not a rate limit. See [Keep each team's spend inside an agreed ceiling](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/keep-team-spend-inside-a-ceiling/) | ## Outcomes By the end of this guide: - The three limit types (TPM, RPM, and maximum parallel requests) are understood, along with the failure mode each one addresses. - At least one rate limit is applied at the scope appropriate to the workload it protects. - The rule for how limits at different scopes compose, the most restrictive applicable limit wins, is understood. - The caller-side experience of a hit limit, an HTTP `429` response, is understood, along with the back-off behaviour applications are expected to implement. - A way to confirm whether limits are being hit is in place. ## Prerequisites - Administrator access to the Admin Dashboard: typically the `super_admin` role, or a role with permission to manage limits. - At least one provider and model already provisioned. See [Provision models and providers](/agent-router-enterprise/guides/operate-and-govern/provision-models-and-providers/). - For per-group limits, the groups that traffic is attributed to. Group membership is sourced from the corporate identity provider, as described in [Map Entra ID groups to business functions](/agent-router-enterprise/guides/identity-and-access-management/single-sign-on/map-entra-id-groups-to-business-functions/). - Coordination with the developer teams that own the affected keys. Limits are enforced inline, so a misconfigured limit produces visible production impact. ## Step 1: choose the limit type The three limits answer different questions about a consumer's traffic. Pick the one, or the combination, that matches the capacity being protected. | Limit | What it caps | The failure mode it addresses | | :---- | :---- | :---- | | Tokens per minute (TPM) | Total tokens processed per minute, input plus output | Heavy model work: long contexts and verbose completions that consume upstream throughput out of proportion to the request count | | Requests per minute (RPM) | Number of requests per minute, regardless of size | High call frequency: retry storms, tight polling loops, and chatty agents that make many small calls | | Maximum parallel requests | Number of requests in flight at the same moment | Concurrency exhaustion: batch jobs and fan-out patterns that open many simultaneous connections and saturate the provider's concurrency allowance | TPM and RPM are rate measures; they constrain traffic averaged over a window of time. Maximum parallel requests is a concurrency measure; it constrains an instantaneous count and is unconcerned with rate. A consumer can stay well under its RPM yet still exhaust the parallel-request limit by issuing a single large burst, and the reverse is equally possible. Because they catch different shapes of traffic, the limits are usually applied together. A reasonable default for a new workload is a moderate RPM to absorb retry storms, a TPM sized to the expected token volume, and a parallel-request limit that reflects how much fan-out the workload legitimately needs. The values are tightened or relaxed once real traffic reveals the workload's shape, the sizing approach is covered in [Step 4](#step-4-size-and-adjust-limits). ## Step 2: choose the scope The same three limit types can be applied at several scopes. The scope determines which traffic the limit is measured against. | Scope | What the limit governs | Typical use | | :---- | :---- | :---- | | Per model | All traffic to one model, across every consumer | Protecting a single upstream model or deployment from aggregate overload, independent of who is calling it | | Per API key | All traffic presented with one key | Bounding a specific application or integration to its expected envelope | | Per user | All traffic across every key a user owns | Holding an individual developer's total footprint in check, regardless of how many keys they hold | | Per group | All traffic across every member of a group | Reserving a fair share of capacity for a business function, so one team does not crowd out another | Per-key limits are the most precise and the most common, because a key usually maps to a single workload. Per-user and per-group limits sit above the key scope and govern aggregate footprint: a user's group is resolved from the identity provider at request time, so traffic is attributed to the right group automatically once the mapping in [Map Entra ID groups to business functions](/agent-router-enterprise/guides/identity-and-access-management/single-sign-on/map-entra-id-groups-to-business-functions/) is in place. Per-model limits are orthogonal to the others; they cap total load on an upstream model regardless of which consumer is driving it, and are the right tool when a particular model or deployment has a known capacity ceiling. Limits are set from the Admin Dashboard surface that manages the relevant entity: the model, the key, the user, or the group. The fields presented are the three limit types from [Step 1](#step-1-choose-the-limit-type); a limit left unset at a given scope is simply not enforced at that scope. Alongside the Admin Dashboard scopes, each API key also carries Console-side hourly token limits (Total, Input, and Output tokens per rolling hour), configured by the developer who owns the key. The operator-side decision about which keys should carry which limits is what makes that mechanism useful: the operator establishes the policy ("research keys cap at 100 K tokens per hour") and the developer applies it. The developer-side flow is documented in [Route requests across providers](/agent-router-service/guides/route-requests-across-providers/) under *Rate Limiting*. ## Step 3: understand how scopes compose A single request can fall under limits at more than one scope at once. A request made with a particular key, by a particular user, who belongs to a particular group, against a particular model is subject to the limits set at all four scopes simultaneously. The composition rule is straightforward: every applicable limit is evaluated independently, and the request is rejected if it would breach any one of them. The most restrictive applicable limit is therefore the one that takes effect. A generous per-group allowance does not loosen a tight per-key limit, and a generous per-key limit does not override a tight per-model ceiling. The limits do not add together, and a higher-scope limit does not raise a lower one. A worked example makes the rule concrete: - A group is allowed 10,000 RPM across all its members. - A key belonging to a member of that group is allowed 500 RPM. - The model that key targets is allowed 2,000 RPM across all consumers. Traffic on that key is held to 500 RPM, because the per-key limit is the most restrictive of the three that apply. If a second key in the same group also runs near its own limit, the two together are still held under the group's 10,000 RPM; and all consumers of the model together are held under the model's 2,000 RPM. Each ceiling is enforced at its own scope, and a request must satisfy all of them to pass. The practical consequence is that the tightest limit governs. When a limit appears not to be taking effect, the usual cause is a tighter limit at another scope firing first. The way to confirm which limit is firing is covered in [Step 5](#step-5-monitor-whether-limits-are-being-hit). ## Step 4: size and adjust limits Sizing is the part of this work that takes the most judgement. Two failure modes are worth avoiding. - Limits set too tight. Normal traffic hits the ceiling, healthy clients start receiving `429` responses, and the workload looks, from the consumer's side, as though Agent Router is failing. The cost of this failure mode is immediate and visible. - Limits set too loose. A runaway consumer is never actually constrained, and the limit becomes a number that never fires. The cost surfaces later, as overloaded upstreams or a degraded experience for other consumers sharing the same capacity. The dependable approach is to size each limit from observed traffic. The expected peak is read from the usage surface, see [Monitor traffic and usage](/agent-router-service/guides/monitor-traffic-and-usage/), the highest legitimate per-minute value over a representative window is taken as the baseline, and the limit is set comfortably above that baseline but well below any level that would constitute overload. Setting a limit at roughly two to three times the observed peak is a common starting point, adjusted for how bursty the workload is. For a brand-new workload with no traffic history, a deliberately tight limit is the safer starting point. Real traffic surfaces the true shape quickly, and the limit is relaxed once the baseline is known. Relaxing a limit that proved too tight is a low-risk adjustment; discovering that a generous limit allowed an overload is harder to recover from. Limits are adjusted from the same surface used to set them. A change takes effect on subsequent requests, with no restart and no downtime window. Because the change is felt by live traffic, an adjustment to a production key is best coordinated with the team that owns it. Some workloads are quiet most of the time and very loud occasionally: end-of-month batch runs and scheduled report generation. A limit sized for the quiet baseline will fire on the burst. The cleanest answer is usually to isolate the bursty work on its own key with its own, more generous limit, which keeps the steady-state key tight and the usage attribution clean. The operational complication of issuing a second key is typically smaller than the complication of explaining why a single key's traffic shape is irregular. The patterns for those workloads are covered in [Run batch and long-running jobs](/agent-router-service/guides/run-batch-and-long-running-jobs/). ## Step 5: monitor whether limits are being hit A limit that never fires and a limit that fires constantly are both worth knowing about: the first may be set too loosely to matter, and the second is likely throttling healthy traffic. Both are visible in Agent Router's usage and traffic surfaces. The signal to watch for is the rate of `429` responses, broken down by the scope the limit is set on. A workflow that pairs well with this guide: 1. Open the usage and traffic view. See [Monitor traffic and usage](/agent-router-service/guides/monitor-traffic-and-usage/). 1. Apply a time range that matches the cadence of the workload under review. 1. Break the traffic down by the scope the limit is set on: by key, by user, by group, or by model. 1. Look at the proportion of requests returning `429`. A small, occasional fraction during peaks is normal; a sustained high fraction indicates the limit is too tight for the legitimate workload. 1. Correlate the `429` rate with the limit type. A spike concentrated in one limit type (a parallel-request limit firing while RPM stays clear, for instance) points to the specific control that needs adjustment. When a limit is firing more than expected, the choices are to raise it if the traffic is legitimate, see [Step 4](#step-4-size-and-adjust-limits), or to address the consumer if the traffic is not, by working with the owning team or, in the case of a suspected-compromised key, following [Contain a leaked key before it drains the budget](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/contain-a-leaked-key/). ## What a caller experiences when a limit is hit When a request would breach an active limit, the gateway rejects it with an HTTP `429 Too Many Requests` response rather than forwarding it upstream. The rejection is immediate and applies only to the request that crossed the threshold; once the window advances or in-flight requests drain, traffic flows again. A `429` is therefore a transient, recoverable signal, not a hard failure, though to a client that does not handle it, it presents the same way as an error. Applications are expected to back off and retry rather than fail or retry immediately: - Retry the request after a short delay rather than treating the `429` as fatal. - Use exponential back-off with jitter (increasing the delay on each successive `429`, with a small random offset) so that many clients hitting the limit at once do not retry in lockstep and re-create the burst. - Respect any retry-after guidance the response carries, where the client library surfaces it. - Cap the number of retries so that a sustained limit does not turn into an unbounded retry loop, which is itself a source of the RPM pressure the limit exists to contain. Most current AI client SDKs implement back-off of this kind by default, so a well-behaved application typically experiences a hit limit as added latency rather than as a visible error. The patterns matter most for custom integrations and for high-concurrency batch workloads, which are covered in [Run batch and long-running jobs](/agent-router-service/guides/run-batch-and-long-running-jobs/). ## What to do next - Set spend caps to complement the flow control established here. See [Keep each team's spend inside an agreed ceiling](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/keep-team-spend-inside-a-ceiling/). - Confirm group attribution is correct before relying on per-group limits. See [Map Entra ID groups to business functions](/agent-router-enterprise/guides/identity-and-access-management/single-sign-on/map-entra-id-groups-to-business-functions/). - Watch the effect of the limits on live traffic. See [Monitor traffic and usage](/agent-router-service/guides/monitor-traffic-and-usage/). - Apply the back-off and isolation patterns to high-concurrency workloads. See [Run batch and long-running jobs](/agent-router-service/guides/run-batch-and-long-running-jobs/). Where to go next Keep each team's spend inside an agreed ceiling The spend cap that complements the flow control established here. Contain a leaked key before it drains the budget The incident response when the runaway is not legitimate traffic. --- ## Govern MCP server access Enterprise Tier New MCP servers appear frequently: some open-source, some commercial, some assembled internally to wrap an organisation-specific tool. For Agent Router's operator, this is a governance problem: anything that calls itself an MCP server can request to be used, and the operator is the one who decides what is allowed to reach developers in the first place. The Admin Dashboard's MCP surface is built around exactly that decision. **MCP Servers** is the catalogue of every server Agent Router has been told about: what URL it lives at, what kind of authentication it expects, and what tools it exposes. **MCP OAuth Clients** holds the OAuth client configurations needed for servers that authenticate users through an OAuth provider rather than a static token. Together they form the catalogue from which developers assemble profiles in the Console; nothing reaches the developer side that has not first been admitted to this catalogue. This guide covers the recurring work of curating that set: adding new servers, configuring the authentication each one requires, retiring servers that have outlived their usefulness, and keeping OAuth client credentials in step with the upstream providers. **Persona:** Platform operator working in the Admin Dashboard, with the MCP catalogue as the unit of governance. **Estimated time:** 15 to 25 minutes for an initial catalogue setup; ongoing thereafter as the MCP ecosystem grows. ## When this guide applies This guide is the right starting point in any of these situations: | Situation | What it covers | | :---- | :---- | | Introducing a new MCP server to Agent Router | Adding the server to the catalogue with the correct authentication model | | Configuring an OAuth provider for one or more MCP servers | Setting up the OAuth client used by every server that delegates auth to that provider | | Retiring an MCP server that is no longer supported | Removing the server cleanly and understanding the impact on existing profiles | | Rotating OAuth client credentials on schedule or after a security event | Editing the OAuth client without disrupting active sessions where possible | | Auditing the catalogue to confirm what is currently exposed | Reviewing the server list, authentication types, and which servers map to which OAuth clients | For the developer-side flow (assembling profiles, choosing the security type on the profile URL, and connecting from Claude Code, Cursor, or VS Code), the *Aggregate MCP Servers into a Profile* developer guide is the right surface. That guide assumes the catalogue this one builds. ## Outcomes By the end of this guide: - At least one MCP server is present in the catalogue with a correctly configured authentication type. - For any OAuth-authenticated servers, the corresponding OAuth client configuration exists and is mapped correctly. - The mechanics of editing, retiring, and rotating both servers and OAuth clients are understood. - The relationship between server-level authentication (which this guide governs) and profile-level authentication (which the developer guide governs) is clear. ## Prerequisites - Administrator access to the Admin Dashboard, typically the `super_admin` or `mcp_admin` role. - For each MCP server to be introduced: its endpoint URL, a description of the tools it exposes, and an understanding of what authentication it requires. - For OAuth-authenticated servers: a client ID, client secret, authorization URL, and token URL from the upstream OAuth provider. These are typically obtained by registering an application with the provider in question. ## Step 1: decide what authentication the server requires Authentication is the first decision because everything else flows from it. Agent Router supports three models: | Auth type | What the server expects | When to use it | | :---- | :---- | :---- | | **Open Access** | No authentication on requests | Internal servers on trusted networks; rare in production | | **Bearer Token** | A static bearer token in the `Authorization` header | Servers with a simple shared-secret model | | **OAuth** | An OAuth flow with per-user consent | Third-party servers that grant access on behalf of a specific user | The right answer is usually determined by the server itself rather than by operator preference, so the server's documentation is the place to start. The most common mistake is to assume Bearer Token will work for an OAuth-only service; the service rejects the static token and the failure looks, from inside the gateway, like a generic upstream error. For Open Access and Bearer Token servers, Step 2 is sufficient. For OAuth servers, Step 3 has to happen first; the OAuth client must exist before the MCP server can reference it. ## Step 2: add the server to the catalogue The MCP server entry carries the endpoint, the authentication settings, and the description that developers see when assembling profiles. 1. Sign in to the Admin Dashboard. 1. Open **MCP Servers** from the sidebar. ![MCP Servers management](/img/screenshots/admin/mcp-servers.jpeg) 1. Review the current catalogue. Each entry surfaces the columns relevant for governance: | Column | What it captures | | :---- | :---- | | Server Name | The display name shown in the Console MCP catalogue | | Description | A summary of what the server provides | | Auth Type | Open Access, Bearer Token, or OAuth | | Tools | The number of tools the server exposes | | Status | Whether the server is active and available in the Console catalogue | 1. Click **Add Server** at the top of the list. 1. Fill in the configuration: - **Name.** A descriptive label that explains what the server does. The name appears in the Console catalogue, so a phrase like `GitHub Issues`, `Internal Wiki Search`, or `Browser Automation` is more useful than a generic label. - **URL.** The server's endpoint. - **Description.** A short sentence on the server's purpose and capabilities. - **Auth Type.** The authentication model chosen in Step 1. 1. If the auth type is **Bearer Token**, paste the token into the field that appears. 1. If the auth type is **OAuth**, select an existing OAuth client from the dropdown. If no suitable client exists, save the partial configuration after creating one in Step 3, or jump to Step 3 first. 1. Click **Save**. The server appears in the catalogue immediately. Developers see the new entry the next time they open the MCP catalogue in the Console; existing profiles are unaffected. ### Inspect the tools the server exposes Once a server is in the catalogue, its tool list becomes visible: 1. Click the server row to open the detail view. 1. Scroll to the **Tools** section. 1. Review every tool the server exposes, with its name and description. This is the operator's last opportunity to evaluate what Agent Router is about to expose. Servers that present surprisingly broad tool surfaces, a "file system" server that turns out to expose arbitrary shell execution for example, are worth pausing on before they are made generally available. Where the tool list contains capabilities that should not be exposed, the right action is to either deselect those tools at the profile level (developers cannot opt in to tools that operators have not approved) or to choose a different server entirely. ## Step 3: configure an OAuth client (only for OAuth-authenticated servers) For servers that delegate user authentication to an OAuth provider, the corresponding client configuration has to exist in the **MCP OAuth Clients** screen before the server can reference it. 1. Open **MCP OAuth Clients** from the sidebar. ![MCP OAuth Clients](/img/screenshots/admin/mcp-oauth-clients.jpeg) 1. Review the existing clients. Each row surfaces the columns relevant for governance: | Column | What it captures | | :---- | :---- | | Client Name | A descriptive label for the OAuth client configuration | | Client ID | The OAuth client identifier registered with the provider | | Authorization URL | The authorization endpoint users are redirected to | | Token URL | The endpoint used to exchange authorization codes for access tokens | | Mapped Servers | The MCP servers currently mapped to the client | 1. Click **Add OAuth Client**. 1. Fill in the required fields: | Field | What it carries | | :---- | :---- | | Client Name | A descriptive label, used internally to identify the configuration | | Client ID | The identifier issued by the OAuth provider | | Client Secret | The secret issued by the OAuth provider, stored encrypted at rest | | Authorization URL | The full URL of the provider's authorization endpoint, for example `https://auth.example.com/authorize` | | Token URL | The full URL of the provider's token endpoint, for example `https://auth.example.com/token` | | Scopes | An optional space-separated list of OAuth scopes to request | 1. Click **Save**. 1. Return to **MCP Servers**, open the relevant server, set its auth type to **OAuth**, and select the newly-created client. The end-to-end flow that follows, when a developer adds the server to a profile and a Console user reaches the profile URL, is: 1. The Console user is redirected to the **Authorization URL** with the configured client ID and scopes. 1. The user authenticates and grants consent at the OAuth provider. 1. The provider redirects back to Agent Router with an authorization code. 1. Agent Router exchanges the code for an access token using the **Token URL** and the client secret. 1. The access token is used for subsequent requests to the MCP server, on behalf of the consenting user. This delegated-consent model is what makes OAuth the right choice for third-party services that grant access per user. The bearer-token model, by contrast, gives every consumer the same level of access regardless of who they are. ## Step 4: maintain the catalogue over time The catalogue is not write-once. Servers are added and retired; OAuth clients have their secrets rotated; URLs change when upstream services are migrated. ### Edit a server 1. Open the server's row in **MCP Servers**. 1. Update the fields that need to change: name, URL, description, auth type, bearer token, or mapped OAuth client. 1. Save. Changes take effect immediately for new MCP profile connections. Connections that are mid-session at the moment of the change continue with the previous configuration until they next re-authenticate. ### Edit an OAuth client OAuth client credentials are the most security-sensitive part of this surface. Rotation typically happens on a schedule defined by either the OAuth provider or the organisation's own security policy. 1. Open the OAuth client's row in **MCP OAuth Clients**. 1. Update the fields. The most common edit is a new client secret after a scheduled rotation. 1. Save. Every MCP server mapped to that OAuth client picks up the change automatically. Users with currently-valid access tokens continue to use those tokens until they expire; refreshing or obtaining a new access token uses the new credentials. ### Remove an MCP server A removal cleanly takes a server out of circulation: 1. Open the server's row. 1. Click **Delete** or **Remove Server**. 1. Confirm. The server disappears from the Console MCP catalogue immediately. Profiles that already include the server may stop functioning depending on what tool calls they were about to make. Scanning the developer-side surfaces before removing a server that is in active use, or coordinating the removal through [Communicate with Agent Router Users](/agent-router-enterprise/guides/operate-and-govern/communicate-with-platform-users/), is usually worth the trouble. ### Remove an OAuth client OAuth clients carry a protection that MCP servers do not: a client cannot be deleted while any MCP server still references it. 1. Open the OAuth client's row. 1. Click **Delete**. 1. If any servers still map to the client, the deletion is refused. Unmap each server (either by changing its auth type or by mapping it to a different client) before retrying. This refusal is deliberate; it prevents the orphaned-server state where a server's OAuth flow points at a configuration that no longer exists. ## How server auth and profile auth relate The two layers of authentication in the MCP world are sometimes confused, so it is worth being explicit. The Admin Dashboard governs **server-level authentication**: how the gateway authenticates to the upstream MCP server on the user's behalf. The Console governs **profile-level authentication**: how the MCP client (Claude Code, Cursor, VS Code) authenticates to the profile URL exposed by the gateway. Both layers operate, both have to be correct for a tool call to succeed, and they are configured in entirely different screens. The developer-side mechanics of profile authentication are covered in [Aggregate MCP Servers into a Profile](/agent-router-service/guides/aggregate-mcp-servers-into-a-profile/). ## What to do next - **Audit Agent Router activity**: review the events generated by additions and removals to the MCP catalogue. See [Audit Agent Router Activity](/agent-router-enterprise/guides/operate-and-govern/audit-platform-activity/). - **Communicate with Agent Router users**: announce a new server, deprecate an old one, or coordinate a maintenance window. See [Communicate with Agent Router Users](/agent-router-enterprise/guides/operate-and-govern/communicate-with-platform-users/). - **Configure SSO**: the SSO setup that governs Admin Dashboard access also touches the MCP OAuth surface for some deployments. See [Configure SSO](/agent-router-enterprise/guides/identity-and-access-management/single-sign-on/sso-configuration-guide/). The MCP catalogue curated in this guide remains in place for subsequent guides. Where to go next Communicate with Agent Router users Announce a new server, deprecate an old one, or coordinate a maintenance window. Audit Agent Router activity Review the events generated by additions and removals to the MCP catalogue. --- ## Guides for Platform Operators Enterprise Tier Written for the platform operator persona: the administrator or IT operations engineer who manages models, providers, users, and organization-wide settings from the Admin Dashboard. Each guide is organised around a single operational goal rather than a single screen. Operator setup has two layers. First, understand **projects**—the isolation boundary that owns models, keys, and gateways. A default project is created during onboarding; the concept is covered in [Key concepts → Projects](/product-architecture/key-concepts/#projects) and the operator tasks live under [Project and data plane management](/agent-router-enterprise/guides/operate-and-govern/project-and-data-plane-management/). Second, [provision models and providers](/agent-router-enterprise/guides/operate-and-govern/provision-models-and-providers/) into the organisation catalog, then grant those models to a project so its gateway can route to them. The remaining guides can be read in any order as the need arises. Goals Project and data plane management Create isolated projects, grant them models, provision gateways, and operate the data planes that host them. Provision models and providers Configure provider connections and enable the models developers route to, including AWS Bedrock and provisioning via API. Onboard developers and issue keys Confirm users, assign roles, and manage organization-wide API keys. Govern MCP server access Curate the MCP server catalogue and configure OAuth clients. Audit Agent Router activity Investigate activity and spend using audit logs and usage analytics. Communicate with Agent Router users Publish in-Console announcements with the appropriate severity. Configure SSO Connect an OIDC identity provider. Run multiple Agent Router instances Operate and reconcile more than one Agent Router instance. Working with budgets Enforce spend limits with per-key rate limits and monitoring. Control what request data leaves the cluster Govern how much request detail reaches the management plane. --- ## Load-balance across regions Enterprise Tier A single model is rarely served from a single place. The same model is frequently available from several regional deployments at once (an Azure OpenAI deployment in West Europe and another in East US, a self-hosted model running in two Kubernetes clusters on different continents, or a managed provider exposed through endpoints in three regions for capacity reasons). Treating those deployments as separate backends works, but it leaves throughput, latency, and resilience on the table. Tetrate Agent Router addresses this by letting several regional deployments of the same model sit behind one logical model and balancing requests across them at the data plane, using the Kubernetes Gateway API inference extension: an InferencePool that groups the regional endpoints, paired with an Endpoint Picker that chooses among them per request using live backend signals rather than fixed weights. **Persona:** Platform operator working in the Admin Dashboard and the Kubernetes data plane. **Estimated time:** 30 to 60 minutes, depending on how many regional backends are involved and whether a maintenance drain is rehearsed during the same session. ## When this guide applies This guide is relevant when the same model is reachable through more than one regional deployment and the operator wants Agent Router to use all of them: | Situation | What it covers | | :---- | :---- | | One model is available from two or more regional deployments and only one is currently in use | Grouping the deployments into a single InferencePool behind one logical model | | A single deployment's rate limit or capacity caps aggregate throughput | How pooled endpoints raise the ceiling and how the Endpoint Picker spreads load to stay under per-endpoint limits | | Callers are spread across geographies and latency varies by origin | How proximity-aware and load-aware selection lowers tail latency | | A regional incident takes down one deployment and traffic must stay on the survivors | Health-aware selection and how it relates to fallback policies | | A region needs to be taken offline for maintenance without dropping requests | Draining a region and confirming the redistribution | For a model served from a single place, this guide does not apply; there is nothing to balance, and a plain backend definition is sufficient. The pattern becomes worthwhile only once a second deployment of the same model exists. ## Outcomes By the end of this guide: - Several regional deployments of one model have been grouped into a single InferencePool exposed as one logical model. - The Endpoint Picker's role in per-request selection across the pooled endpoints is understood, along with the live signals it acts on. - The interaction between in-pool balancing, fallback policies, and traffic splitting is clear, including which layer acts first. - A region has been drained for maintenance, and the resulting shift in distribution has been observed. ## Prerequisites - Administrator access to the Admin Dashboard, typically the `super_admin` role. - An Enterprise deployment, since pooling regional backends operates against the Kubernetes data plane. The pool definition lives in the data plane manifests, edited through whatever GitOps or manual process the deployment uses; the logical model that fronts it is managed in the Admin Dashboard. The single-versus-multi management split is described in [Run multiple Agent Router instances](/agent-router-enterprise/guides/operate-and-govern/run-multiple-platform-instances/). - At least two regional deployments of the same model, each reachable from the cluster where the data plane runs. Connecting provider subscriptions in more than one region or cloud is covered in [Connect provider subscriptions across clouds](/agent-router-enterprise/guides/operate-and-govern/provision-models-and-providers/connect-provider-subscriptions-across-clouds/). - The Kubernetes context required to apply InferencePool and Endpoint Picker resources to the data plane. ## Step 1: decide which deployments belong in one pool A pool only makes sense for deployments that are interchangeable from the caller's point of view. The defining test is whether a request served by any member of the pool returns an equivalent result. Two Azure OpenAI deployments of the same model version in different regions pass this test; the same model from two different providers does not, because the response shape, behaviour, and identifiers diverge; that is a fallback or traffic-splitting concern, not a pool. Three considerations decide pool membership: - **Model equivalence.** Every endpoint in the pool should serve the same model and, where it matters, the same model version. A pool that mixes versions silently turns load balancing into an uncontrolled A/B test; keep version experiments to the traffic-splitting mechanism described in [Apply advanced routing rules](/agent-router-service/guides/apply-advanced-routing-rules/). - **Regional spread.** The deployments should fail independently and sit at different distances from the caller population. Two endpoints in the same availability zone add capacity but little resilience; two endpoints in different regions add both. - **Capacity and limits.** Each endpoint carries its own rate limit and capacity. The aggregate ceiling of the pool is the sum of the members' limits, which is the throughput argument for pooling, but only if the Endpoint Picker spreads load rather than saturating one member first. The output of this step is a short list: one logical model name, and the set of regional endpoints that will stand behind it. ## Step 2: group the regional endpoints into an InferencePool Agent Router balances across endpoints using the Kubernetes Gateway API inference extension. Rather than a static list of weighted backends, the regional deployments are grouped into an **InferencePool**, a set of endpoints serving the same model, and an **Endpoint Picker** chooses among them on every request. This is the same dynamic-selection mechanism described in [Apply advanced routing rules](/agent-router-service/guides/apply-advanced-routing-rules/); here it is applied deliberately across regional deployments rather than across replicas of a single deployment. The InferencePool references the endpoints and names the Endpoint Picker that scores them. The exact resource shape depends on the inference-extension version installed with the data plane; the manifest below is illustrative of the structure rather than a literal field reference: ```yaml apiVersion: inference.networking.x-k8s.io/v1alpha2 kind: InferencePool metadata: name: chat-model-regional spec: # The endpoints that serve this model in each region. # Selector and endpoint wiring follow the inference-extension # version installed with the data plane. selector: app: chat-model extensionRef: # The Endpoint Picker that scores candidates per request. name: chat-model-endpoint-picker ``` Apply the pool and its Endpoint Picker through the data plane's normal change process: 1. Define the InferencePool that groups the regional endpoints identified in Step 1. 1. Reference the Endpoint Picker that will score those endpoints per request. 1. Apply the manifests to the cluster through the GitOps or manual workflow the deployment uses. 1. Confirm the pool reports its member endpoints as registered before any traffic is routed to it. Once the pool is healthy, it is exposed to the rest of Agent Router as a single logical model. ## Step 3: front the pool with one logical model Callers should not address regions directly. The pool is presented to developers as one logical model in the Admin Dashboard catalogue, and the data plane resolves that name to the pool. The logical-model pattern is the same one used for version virtualisation in [Apply advanced routing rules](/agent-router-service/guides/apply-advanced-routing-rules/), a stable name the application speaks, resolved by the gateway, applied here so that a single name fans out across regions. 1. In the Admin Dashboard model catalogue, confirm the logical model that maps to the InferencePool is enabled. 1. Verify that the logical model resolves to the pool rather than to any single regional endpoint. 1. Confirm developers reference only the logical model name in the `model` field of their requests. With this in place, a request for the logical model is handed to the InferencePool, and the Endpoint Picker decides which regional endpoint serves it. No application or API-key change is needed when regions are later added to or removed from the pool; the logical name stays constant. ## Step 4: understand how the Endpoint Picker distributes requests The Endpoint Picker is what makes pooling more than a round-robin. On each request it evaluates live metrics for every candidate endpoint and routes to the one with the best capacity and cache affinity, which lowers latency and raises throughput compared with static weights. Three signals drive the decision: | Signal | What it captures | | :---- | :---- | | KV-cache usage | Memory pressure on each endpoint. Heavily loaded endpoints are deprioritised. | | Queue depth | Number of pending requests on each endpoint. Less-loaded endpoints are preferred. | | Prefix cache scoring | How well an endpoint's cache matches the request's prompt prefix. Better matches reduce latency. | Two consequences follow from this for regional pools specifically: - **Throughput.** Because the picker steers away from endpoints that are filling up, the pool absorbs more aggregate load before any single endpoint hits its rate limit. The effective ceiling approaches the sum of the members' limits rather than the limit of whichever endpoint was chosen first. - **Latency.** An endpoint that is geographically distant or already saturated tends to report higher queue depth and weaker cache affinity for local traffic, so the picker naturally favours the closer, warmer endpoint for a given caller, without any explicit geographic rule. Selection operates only within the pool. The Endpoint Picker never routes outside the set of endpoints in the InferencePool; widening or narrowing the candidate set is a change to pool membership, not to the picker. ## Step 5: layer fallback and traffic splitting correctly In-pool balancing and the routing patterns from the developer guides are not alternatives; they stack, and the order in which they act matters. - **In-pool balancing acts first.** For a request routed to the logical model, the Endpoint Picker selects among the healthy regional endpoints inside the pool. This is the inner loop, and it handles the common case: one endpoint is busier or further away, so another serves the request. - **Fallback acts second.** A fallback policy, as described in [Improve resilience with fallbacks](/agent-router-service/guides/improve-resilience-with-fallbacks/), is the outer loop. The pool is treated as a single backend in the fallback chain. Only when the pool as a whole cannot serve a request (every regional endpoint is unhealthy or rate-limited) does the gateway walk to the next backend in the chain, which is typically a different provider or a different model class. In-pool balancing absorbs single-region trouble; cross-provider fallback absorbs whole-pool trouble. - **Traffic splitting sits alongside.** A weighted split, as covered in [Apply advanced routing rules](/agent-router-service/guides/apply-advanced-routing-rules/), selects which logical backend a request is sent to. When the selected backend is a pooled logical model, the Endpoint Picker then balances within it. A split can therefore send a percentage of traffic to a regional pool and the remainder elsewhere, with intra-pool balancing applied to the pool's share. The mental model is three nested layers: traffic splitting chooses the logical backend by weight, fallback orders the logical backends for failure, and the Endpoint Picker balances among the physical endpoints inside a pooled backend. Each layer is configured independently and the boundaries do not blur; the picker never crosses a pool edge, and fallback never reaches inside a pool to pick an endpoint. ## Step 6: rely on health-aware selection and drain a region for maintenance The Endpoint Picker's signals double as a health filter. An endpoint that stops responding, returns errors, or reports saturation is deprioritised or excluded automatically, so a regional incident shifts load onto the survivors without operator action and without a fallback event at the chain level. This is the difference between in-pool resilience and chain-level fallback: a single failed region is handled silently inside the pool, whereas fallback is reserved for the case where the whole pool is unavailable. Planned maintenance uses the same machinery deliberately. Draining a region means removing its endpoint from the pool's eligible set so that in-flight requests complete while new requests are steered elsewhere: 1. Identify the regional endpoint to be taken offline. 1. Remove or cordon that endpoint in the InferencePool through the data plane's change process, so the Endpoint Picker stops selecting it for new requests. 1. Allow in-flight requests on the drained endpoint to complete rather than terminating them abruptly. 1. Confirm the remaining endpoints absorb the redistributed load and that aggregate latency and error rate stay within the expected band. 1. Perform the maintenance, then return the endpoint to the pool and confirm the Endpoint Picker resumes selecting it. Because callers only ever address the logical model, a drain and a restore are both invisible to the application; the only observable effect is a shift in which region serves each request. ## Step 7: observe the distribution A balancing policy that is not observed cannot be tuned. Request Logs and usage analytics record the resolved backend for each request, which for a pooled logical model reveals which regional endpoint actually served it. 1. In the Console, open **Monitoring → Request Logs**. 1. Filter to traffic for the logical model that fronts the pool. 1. Inspect the resolved backend per request to confirm requests are spread across the regional endpoints rather than concentrated on one. 1. During a drain, confirm the drained region stops appearing as a resolved backend while the survivors pick up its share. The healthy steady state is a spread across endpoints that tracks caller geography and per-endpoint load, not a perfectly even split, since the Endpoint Picker optimises for latency and capacity rather than for equal counts. A sudden collapse onto a single endpoint, or a cluster of chain-level fallback events, indicates that one or more regional endpoints have become unhealthy and is worth correlating against the provider's regional status. Richer filtering by time range, status, and resolved backend is covered in [Monitor traffic and usage](/agent-router-service/guides/monitor-traffic-and-usage/). ## What to do next - **[Connect provider subscriptions across clouds](/agent-router-enterprise/guides/operate-and-govern/provision-models-and-providers/connect-provider-subscriptions-across-clouds/)**: add further regional deployments, including across cloud providers, before extending a pool. - **[Improve resilience with fallbacks](/agent-router-service/guides/improve-resilience-with-fallbacks/)**: wrap the regional pool in a cross-provider fallback chain so whole-pool outages are absorbed too. - **[Apply advanced routing rules](/agent-router-service/guides/apply-advanced-routing-rules/)**: combine the pool with logical-name virtualisation and weighted splits for staged rollouts. - **[Run multiple Agent Router instances](/agent-router-enterprise/guides/operate-and-govern/run-multiple-platform-instances/)**: distinguish balancing within one instance from operating separate regional instances of Agent Router itself. - **[Monitor traffic and usage](/agent-router-service/guides/monitor-traffic-and-usage/)**: extend the distribution checks here into ongoing dashboards with latency and cost tracked per resolved backend. Where to go next Connect provider subscriptions across clouds Add further regional deployments, including across clouds, before extending a pool. Run multiple Agent Router instances Distinguish balancing within one instance from operating separate regional instances. --- ## Manage configuration as code Enterprise Tier Agent Router configuration (the policies, budgets, model catalogue, pricing, and API keys that govern how requests are routed and what they cost) has to live somewhere, and the choice of where determines who can change it, how a change is reviewed, and whether the change can be reproduced on a second instance. Agent Router Enterprise exposes the same configuration through three surfaces: the Admin Dashboard UI, the admin API, and infrastructure as code (IaC) via Helm. Most deployments use the three together, each for the kind of change it suits, rather than choosing one. **Persona:** Platform operator or platform engineer responsible for how Agent Router is configured and how that configuration is reviewed and promoted across environments. **Estimated time:** 20 to 30 minutes to read; longer to design and adopt an IaC workflow for a specific deployment. ## When this guide applies This guide is relevant in any of these situations: | Situation | What it covers | | :---- | :---- | | A non-engineer needs to read or adjust budgets and policies | The Admin Dashboard UI surfaces for budgets, policies, the model catalogue, pricing, and API keys | | The baseline configuration has to be reviewed and version-controlled before it reaches production | The IaC path via Helm, and why it is preferred for the baseline | | The same configuration has to be reproduced across staging, production, and regional instances | Promotion of versioned configuration across environments | | A routing or policy change has to take effect without taking Agent Router down | The dynamic configuration model | | An auditor or security reviewer asks who changed what and when | The audit record that covers every administrative change regardless of surface | | UI edits and IaC runs are starting to overwrite each other | The recommended split of ownership between the two | ## Outcomes By the end of this guide: - The three configuration surfaces (the Admin Dashboard UI, the admin API, and IaC via Helm) are understood, along with the kind of change each suits. - The reasons to manage the baseline configuration as code (review, version control, and reproducibility) are clear. - It is clear that policy and routing changes take effect dynamically, without a redeploy of the data plane. - It is clear that every administrative change is versioned and recorded in the audit log, regardless of which surface made it. - A division of ownership between UI-driven and IaC-driven changes has been chosen, with a rule for keeping the two from overwriting each other. ## Prerequisites - Administrator access to the Admin Dashboard (typically the `super_admin` role) for the UI and audit surfaces. - For the admin API: an API credential with administrative scope, issued from the Admin Dashboard. - For the IaC path: a Helm toolchain, a version-control repository, and, for Enterprise deployments, the Kubernetes context that hosts the data plane. The Kubernetes resources the installer renders are covered in [Retrieving data plane resources](/agent-router-enterprise/self-hosted-data-plane/data-plane-installation/k8s-resources-for-gitops/). - A working model catalogue, so that policies and budgets have something to act on. Provisioning is covered in [Provision models and providers](/agent-router-enterprise/guides/operate-and-govern/provision-models-and-providers/) (Admin Dashboard) and [Provision models via API](/agent-router-enterprise/guides/operate-and-govern/provision-models-and-providers/provision-models-via-api/) (Catalog API). ## Step 1: choose the right surface for the change The first decision for any configuration change is which surface to make it through. The three surfaces read and write the same underlying configuration, so the choice is about review, reproducibility, and who is making the change, not about capability. | Surface | Best for | Audience | | :---- | :---- | :---- | | Admin Dashboard UI | Day-to-day adjustments: reading a budget, tightening a policy, enabling a model, rotating a key | Operators, and non-engineers who need to read or adjust budgets and policies without touching code | | Admin API | Scripted or automated changes, integration with internal tooling, bulk operations | platform engineers and automation | | IaC (Helm) | The baseline configuration that has to be reviewed, versioned, and reproduced across instances | platform engineers | The UI is the surface a non-engineer can use. Budgets and policies are presented as forms rather than as code, so a finance owner can read what a budget is set to, and a security owner can read what a policy enforces, without reading a manifest. Where their permissions allow it, the same forms let them adjust those values in place. The day-to-day adjustments that follow a single decision (raising one team's budget, adding one model to the catalogue, or rotating one key) are fastest through the UI. The admin API exposes the same operations programmatically. It is the surface for automation: a script that provisions a new team's keys and budget in one pass, an integration that syncs the model catalogue from an internal source of truth, or a bulk policy update across many keys. IaC is the surface for the baseline: the configuration that defines what Agent Router is before any day-to-day adjustment. The next step covers why. ## Step 2: understand why the baseline is managed as code Any single configuration value can be set through the UI. The reason teams move the baseline into IaC is not capability; it is the discipline that version control brings. - **Review.** An IaC change is a pull request. The proposed change to a policy, a budget, or a routing rule is visible as a diff, can be commented on, and is approved by a second person before it is applied. A UI change has no equivalent gate; it takes effect when the operator selects **Save**. - **Version control.** The repository is the history of how the configuration reached its current state. Every change has an author, a timestamp, and a commit message explaining why. Reverting a change is a revert of a commit, not a reconstruction from memory. - **Reproducibility.** The same configuration definition can be applied to staging, then to production, then to a new regional instance, and produce the same result each time. Reproducing a UI-built configuration on a second instance means clicking through the same forms again and hoping nothing was missed: the configuration drift that the multi-instance guide warns about. See [Run multiple Agent Router instances](/agent-router-enterprise/guides/operate-and-govern/run-multiple-platform-instances/). **Helm** renders and applies the Kubernetes resources for the data plane itself: the Controller, the Agent Router gateway, and their supporting objects. This is the installation-time layer, covered in [Retrieving data plane resources](/agent-router-enterprise/self-hosted-data-plane/data-plane-installation/k8s-resources-for-gitops/), where the rendered manifest can be committed to a GitOps repository and applied by Argo CD or Flux rather than applied directly. ## Step 3: rely on dynamic configuration for policy and routing changes A configuration change does not require the data plane to be rebuilt or redeployed. Policies, routing rules, fallback chains, traffic-splitting weights, budgets, and the model catalogue are read by the gateway from the management plane and take effect dynamically. The mechanism is the same regardless of which surface made the change: 1. The change is written through the UI, the admin API, or an IaC apply. 1. The management plane stores the new configuration. 1. The Controller in the data plane reconciles the change and pushes the updated configuration to the Agent Router gateway. 1. The gateway begins enforcing the new configuration after a brief propagation period, without dropping in-flight requests and without a restart. The practical consequence is that a routing change (shifting traffic from one backend to another, adding a fallback, or tightening a budget) is a configuration operation, not a deployment. A Helm change that alters the data plane release itself is the exception: changing the gateway image or its replica count is a deployment, because it changes the running pods rather than the configuration they read. The line is the same as the deployment-mode boundary described in [Run multiple Agent Router instances](/agent-router-enterprise/guides/operate-and-govern/run-multiple-platform-instances/): configuration that lives in the management plane is dynamic, and infrastructure that lives in the data plane manifests is deployed. ## Step 4: confirm that every change is versioned and audited Two independent records cover configuration change, and a complete picture uses both. - **The IaC repository** is the version history for everything managed as code. The diff, the author, the timestamp, and the review are all in the version-control system. This record exists only for changes made through the IaC path. - **The audit log** records every administrative change to Agent Router regardless of the surface that made it: a UI edit, an API call, or an IaC apply all produce an audit entry. Each entry captures who made the change, what was changed, and when. This is the record that answers an auditor's question for changes that did not go through IaC, and it is the cross-check that confirms an IaC apply did what its diff claimed. Reading and exporting the audit log is covered in [Audit Agent Router activity](/agent-router-enterprise/guides/operate-and-govern/audit-platform-activity/). For deployments that need a single timeline across both records, exporting the audit stream to a central SIEM, alongside the IaC repository history, gives one place to answer "who changed this policy, and why". The two records are complementary, not redundant. The repository explains the intent behind a change and carries the review; the audit log proves the change reached the running system and catches anything changed outside the IaC path. ## Step 5: divide ownership between the UI and IaC The UI and IaC write the same configuration, so a value set in one can be overwritten by the other. An IaC apply that defines a budget will reset that budget to its declared value, discarding an interim UI adjustment, on the next run. Avoiding this is a matter of deciding, per category of configuration, which surface owns it. A workable split for most deployments: - **IaC owns the baseline.** Providers, the model catalogue structure, the standing set of policies, the default budgets, and the data plane release are defined in code and promoted through review. These are the things that have to match across instances and that benefit most from version control. - **The UI owns day-to-day adjustments** that are deliberately not in the baseline: a one-off budget increase for a team running an experiment, an urgent policy tightening during an incident, a key rotation. These are the changes that need to happen in minutes, by whoever is on hand, and that do not need to be reproduced on another instance. The rule that keeps the two from fighting is to not manage the same value in both places. A budget that the IaC definition declares should be changed in that definition, not in the UI; otherwise the next apply silently reverts the UI change. A budget that the team has decided to manage by hand should be left out of the IaC definition entirely, so that no apply touches it. When a value that started as a hand-managed UI adjustment becomes permanent, it is promoted into the IaC baseline by a deliberate change to the code, and from that point it is owned by IaC. For Enterprise deployments running a GitOps reconciler such as Argo CD or Flux, the reconciler will actively revert drift from the committed state on its own schedule. Anything under the reconciler's management must not be edited in the UI at all, because the reconciler will undo it; the UI's scope on those deployments is limited to the categories deliberately excluded from the GitOps repository. ## What to do next - **Provision models and providers**: the model catalogue and provider connections are the baseline that an IaC workflow manages first. See [Provision models and providers](/agent-router-enterprise/guides/operate-and-govern/provision-models-and-providers/) for the Admin Dashboard, or [Provision models via API](/agent-router-enterprise/guides/operate-and-govern/provision-models-and-providers/provision-models-via-api/) for the Catalog API sequence. - **Work with budgets**: budgets are a primary candidate for the UI-versus-IaC ownership split; designing them is covered in [Working with budgets](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/choose-the-right-cost-control/). - **Audit Agent Router activity**: the audit log that records every administrative change across all three surfaces. See [Audit Agent Router activity](/agent-router-enterprise/guides/operate-and-govern/audit-platform-activity/). - **Retrieve data plane resources**: the Helm-rendered manifest that the GitOps path commits and reconciles. See [Retrieving data plane resources](/agent-router-enterprise/self-hosted-data-plane/data-plane-installation/k8s-resources-for-gitops/). - **Run multiple Agent Router instances**: where reproducibility across instances pays off, and where configuration drift is most costly. See [Run multiple Agent Router instances](/agent-router-enterprise/guides/operate-and-govern/run-multiple-platform-instances/). Where to go next Provision models and providers The model catalogue and provider connections an IaC workflow manages first. Provision models via API Automate provider, credential, model, and project assignment through the Catalog API. Audit Agent Router activity The audit log that records every administrative change across all three surfaces. --- ## Onboard developers and issue keys Enterprise Tier Once models and providers are provisioned, the next operator task is to make Agent Router reachable to the people who will actually consume it. In small deployments this is a manageable thing to do one user at a time; in larger ones, it becomes the recurring background work of running Agent Router: new joiners, leavers, role changes, and the occasional emergency revocation. The Admin Dashboard makes both ends of that spectrum tractable through two surfaces: Users, a organization-wide view of registered accounts and their roles, and API Keys, a organization-wide view of every API key regardless of which user issued it. This guide covers the typical onboarding flow, the role model that governs what each user can do, the developer-side counterpart of API key issuance, and the recurring operational work of identifying stale keys and revoking compromised ones. **Persona:** Platform operator working in the Admin Dashboard, with occasional excursions into the Console. **Estimated time:** 10 to 20 minutes for a single onboarding pass; ongoing thereafter. ## When this guide applies This guide is the right place to start in any of these situations: | Situation | What it covers | | :---- | :---- | | A new developer or team needs platform access | User registration confirmation, role assignment, and an initial API key | | A developer has changed role and now needs admin access (or has lost it) | Role assignment through the Users surface | | An API key has been leaked or is suspected to be compromised | Immediate organization-wide revocation through the Admin API Keys surface | | Periodic security hygiene, identifying keys that have not been used in months | The Last Used filter and organization-wide sort on the API Keys surface | | A leaver needs to have every credential they hold removed | The combination of user-level audit and per-user API key filter | For self-service key creation by individual developers, the most common day-to-day onboarding pattern once an account exists, the Console's own API Keys page is the right surface, and developers manage it themselves. This guide covers the administrative side of the same activity. ## Outcomes By the end of this guide: - The Users surface has been opened, exercised, and at least one user's role has been reviewed or changed. - The organization-wide API Keys view has been used to locate keys by owner and status. - The mechanics of revocation are understood, including what changes from the developer's perspective when their key stops authenticating. - The relationship between SSO-driven role mapping and manual role edits is clear, so the two are not used in conflict. ## Prerequisites - Administrator access to the Admin Dashboard, typically the `super_admin` or `user_admin` role. - For SSO-managed deployments, the SSO and role-mapping configuration is already in place. See [Configure SSO](/agent-router-enterprise/guides/identity-and-access-management/single-sign-on/sso-configuration-guide/) and the [SSO role mapping guide](/agent-router-enterprise/guides/identity-and-access-management/single-sign-on/sso-role-mapping-guide/) for the foundational setup. On Auth0 evaluation tenants, create accounts in Auth0 first; users appear in Agent Router after their first SSO sign-in (see [Team prep](/product-architecture/product-evaluation/auth0-for-evaluation/#team-prep-add-teammates-in-auth0)). - For non-SSO deployments, an out-of-band registration channel exists so that new users can land in Agent Router's user table in the first place. ## Step 1: confirm the user is registered The Users surface is the operator-side ledger of who exists in Agent Router. Every account that has ever signed in through SSO, or been provisioned through whichever mechanism the deployment uses, appears here. 1. Sign in to the Admin Dashboard. 1. Open **Users** from the sidebar. ![Users management](/img/screenshots/admin/users.jpeg) 1. Review the columns surfaced for each user: | Column | What it captures | | :---- | :---- | | Name | The user's display name | | Email | The user's email address | | Role | The user's currently assigned Agent Router role | | Organization Unit | The organisational group, when configured | | Last Sign-In | The timestamp of the user's most recent authentication | | Status | Active or Inactive | 1. Use the search box to find the new user by name or email. The role and status filters narrow the list when the user base has grown large. Results update in real time as the query is typed or a filter is selected. 1. If the user is missing entirely, the cause is upstream: either the user has not yet signed in for the first time (no Agent Router record exists until first sign-in), or the SSO configuration is not yet exposing them to Agent Router. The fix is at the identity-provider level rather than in the Admin Dashboard. The user's first sign-in is what creates the record. Once present, the row is editable. Clicking a user's row opens the detail view, which carries more than the list columns expose. The full profile adds the registration date alongside the name, email, role, organisation unit, and last sign-in timestamp. An activity summary on the same view reports recent API key usage and request counts. A dedicated Activity tab lists the user's recent actions (API key creation, model usage, and request history), which is the most direct way to understand an individual user's consumption patterns before a role change or an offboarding. ## Step 2: assign or verify the role The role determines what the user can see and do across Agent Router. The role model is intentionally small, so the assignment decision rarely takes long. | Role | Permissions | | :---- | :---- | | `super_admin` | Full platform administration: everything in the Admin Dashboard | | `model_admin` | Manage model routes and backends | | `provider_admin` | Manage upstream AI providers | | `mcp_admin` | Manage MCP profiles and the MCP server catalogue | | `user_admin` | Manage users and organisation membership | | `billing_admin` | View and manage billing and usage | | `user` | Console access only; standard developer permissions | To assign or change a role: 1. Click the user's row to open the detail view. 1. Click **Edit Role** or the role dropdown. 1. Select the new role. 1. Click **Save**. The change takes effect on the user's next sign-in or next API request, depending on which surface they reach first. :::important For deployments that use SSO-driven role mapping, the role assigned by claim mapping wins on every login. Manual role edits to a user whose role is also managed by SSO mapping are not durable; the next login will overwrite the manual change to whatever the IdP claim resolves to. Manage those users' roles through the identity provider, not through the Admin Dashboard. Manual edits are appropriate only for users who are not covered by the mapping. The precedence rules are documented in the [SSO role mapping guide](/agent-router-enterprise/guides/identity-and-access-management/single-sign-on/sso-role-mapping-guide/). ::: ## Step 3: understand how developers issue their own keys Most API keys in Agent Router are created by developers themselves, in the Console, against their own user record. The operator-side surface watches and governs the resulting key set rather than driving its creation. The developer-side flow, for context: 1. The developer signs in to the Console. 1. They navigate to **Settings → API Keys**. 1. They click **Create API Key** and supply a descriptive name. 1. The full key value is displayed once. The developer copies it into a secrets manager or environment variable. 1. The key is presented at the gateway as a bearer token thereafter. The naming convention developers adopt at this step matters operationally. An Agent Router deployment whose keys are named after their purpose (`checkout-service-prod`, `rag-experiment-staging`, `cursor-local-john`) is much easier to audit than an Agent Router deployment whose keys are named `key1`, `key2`, and `test`. Encouraging the convention through documentation and onboarding materials is more effective than enforcing it after the fact. For the deeper developer-side reference, see [Route Requests Across Providers](/agent-router-service/guides/route-requests-across-providers/). ## Step 4: audit and revoke keys organization-wide The Admin Dashboard's **API Keys** page is the organization-wide view that the Console's per-user view does not provide. Every key in Agent Router appears here, regardless of which user issued it. 1. Open **API Keys** from the sidebar. ![Admin API keys](/img/screenshots/admin/api-keys.jpeg) 1. Review the columns surfaced for each key: | Column | What it captures | | :---- | :---- | | Key Name | The descriptive name the developer chose | | Owner | The user who created the key | | Key Prefix | The first characters of the key (such as `sk-...abc`) for identification | | Created | The creation date | | Last Used | The timestamp of the most recent request authenticated by the key | | Status | Active or Revoked | 1. Use the search box to find keys by name, owner, or key prefix, and the status and owner filters to narrow the list. Results update in real time as the query is typed or a filter is selected. Clicking a key's row opens its detail view, which exposes the full metadata: name, owner, creation date, last used timestamp, status, and the routing configuration associated with the key. The same view carries the key's usage history, which is the quickest way to confirm recent activity before deciding whether to revoke. The distinction between the Admin and Console views is worth keeping clear: | Feature | Admin API Keys | Console API Keys | | :---- | :---- | :---- | | Scope | All keys across all users | Only the signed-in user's keys | | Ownership | Owner column visible | All keys belong to the current user | | Actions | View and revoke any key | Create, configure, and revoke own keys | | Audience | Agent Router administrators | Individual developers | To revoke a key: 1. Click the key's row to open its detail view. 1. Click **Revoke**. 1. Confirm. The key stops authenticating immediately. Any request, in-flight or future, that presents the revoked key receives an authentication error. The revocation event is logged in the audit log; the cause and the operator who performed the revocation are visible there. See [Audit Agent Router Activity](/agent-router-enterprise/guides/operate-and-govern/audit-platform-activity/) for the audit surface. Revocation is permanent from the key's perspective. There is no un-revoke; if the developer needs a replacement, they create a new key in the Console. This is deliberate: a revoked credential cannot be silently re-enabled. ### Finding stale keys The Last Used column makes it easy to find keys that have not been touched in a long time. A common periodic-hygiene workflow: 1. Sort the key list by **Last Used** in ascending order. Keys with no recent activity surface at the top. 1. Review the entries against expectations. A test key from six months ago, the credential of a leaver who has not been offboarded, or a CI key for a job that has since moved to another platform: all of these are candidates for revocation. 1. Revoke the keys that no longer have a legitimate consumer. Stale credentials are one of the most common sources of long-term security exposure on AI platforms. The cost of revoking a key that turns out to still be needed is small; the developer creates a new one in a minute. The cost of leaving an unused key alive indefinitely is whatever the worst-case use of that credential would be. ## Step 5: handle leavers When a developer leaves the organisation, both the user record and any credentials they hold need to be deactivated. The two are independent and have to be handled separately. 1. In **Users**, locate the leaver's account. 1. Confirm or update their status. In an SSO-managed deployment, deactivation in the identity provider propagates on next sign-in attempt; in non-SSO deployments, the status is set directly here. 1. Switch to **API Keys** and filter by the leaver's email as owner. 1. Revoke each remaining key. The order matters: revoking keys first ensures that any automated process still presenting one of those credentials is cut off, even if the user record itself is not yet fully deactivated. Doing it in the reverse order leaves a brief window in which the user is gone but their credentials are not. ## What to do next - **Govern MCP server access.** The parallel onboarding flow for the MCP catalogue. See [Govern MCP Server Access](/agent-router-enterprise/guides/operate-and-govern/govern-mcp-server-access/). - **Audit Agent Router activity.** Review the events generated by the onboarding and revocation work performed in this guide. See [Audit Agent Router Activity](/agent-router-enterprise/guides/operate-and-govern/audit-platform-activity/). - **Working with budgets.** Attach spending limits to the keys issued through this flow. See [Choose the right cost control for each workload](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/choose-the-right-cost-control/). The user records and API keys handled in this guide remain in place for subsequent guides. Where to go next Govern MCP server access The parallel onboarding flow for the MCP server catalogue. Working with budgets Attach spending limits to the keys issued through this flow. --- ## Plan HA and disaster recovery Enterprise Tier A gateway that sits in front of every AI request is a shared dependency for everything behind it. When the gateway is unavailable, no application that routes through it can reach a model, and the failure is visible to every team at once. Planning for high availability (HA) and disaster recovery (DR) is therefore a requirement for a production deployment of Tetrate Agent Router. It determines whether a regional outage degrades the service or takes the whole organisation's AI traffic down with it. Agent Router is built to be operated this way. The data plane, the customer-managed Controller and Proxy, scales horizontally rather than vertically, so capacity and redundancy are added by adding replicas. Several regional data planes can be run active-active behind regional load balancing, so the loss of one region degrades rather than stops the service. Routing is health-aware, so unhealthy backends and unhealthy regions are taken out of rotation without operator intervention. What Agent Router does not do automatically is decide how much loss is acceptable, prove that a restore actually works, or confirm that the deployment meets its agreed throughput targets. Those are planning and validation tasks, and they are the subject of this guide. This guide covers why HA and DR planning matters for a production gateway, the building blocks Agent Router provides, how to set recovery point objective (RPO) and recovery time objective (RTO) targets, how to write and test a DR runbook with at least one restore, and how to load-test the deployment against agreed throughput and concurrency targets while recording p95 latency and error rate. **Persona:** Platform operator or site reliability engineer (SRE) responsible for the production deployment. **Estimated time:** Half a day to draft the targets and the runbook; a separate scheduled window for the load test and the restore test, since both involve live infrastructure. ## When this guide applies This guide is relevant whenever the deployment is heading toward, or already in, production use: | Situation | What it covers | | :---- | :---- | | A production deployment is being designed and resilience targets have not yet been agreed | Defining RPO and RTO and choosing a regional topology | | The deployment is single-region and a regional failure would stop all AI traffic | The multi-region active-active option and health-aware failover | | Capacity has been sized on paper but never validated under load | Load testing against agreed throughput and concurrency targets | | A DR plan exists on paper but no restore has ever been performed | Writing and testing a DR runbook | | An evaluation criterion requires documented and tested RPO/RTO and load-test evidence | The full sequence below, which produces that evidence | For a short-lived proof-of-concept that will never carry production traffic, the planning here can be scoped down, but the method does not change, only the targets do. ## Outcomes By the end of this guide: - The building blocks Agent Router provides for HA (horizontal scaling, multi-region active-active operation, and health-aware routing) are understood, along with the guides that configure them. - RPO and RTO targets have been agreed with the field team and written down. - A DR runbook covering configuration and state backup, failover, and restore has been drafted. - At least one restore has been performed against the runbook, and the result has been recorded. - A load test has been run against agreed throughput and concurrency targets, and p95 latency and error rate have been captured. ## Prerequisites - Administrator access to the Admin Dashboard for the deployment, and the Kubernetes context for each data-plane region. - An understanding of the deployment topology described in [Architecture Overview](/product-architecture/architecture-overview/): which components are customer-managed and which are Tetrate-hosted. - The capacity-planning method in [Sizing and Scale](/reference/sizing-and-scale/), which supplies the throughput and concurrency figures the load test validates. - Agreement from the field and subject-matter-expert (SME) team on the throughput, concurrency, RPO, and RTO targets; these are engagement-specific and cannot be assumed. - A GitOps or manifest workflow for the data plane, as described in [Kubernetes Resources for GitOps](/agent-router-enterprise/self-hosted-data-plane/data-plane-installation/k8s-resources-for-gitops/), so that configuration can be restored from source rather than reconstructed by hand. ## Step 1: understand the building blocks Agent Router provides Three solution capabilities underpin every HA and DR plan. The plan combines them; it does not invent resilience Agent Router does not already offer. A fourth, optional capability covers failing over **gateways** within a project. - **Horizontal scaling of the data plane.** Both the Controller and the gateway proxy add capacity by adding replicas behind a load balancer rather than by enlarging a single instance. Running more than one replica of each is what makes the data plane survive the loss of an individual pod or node. The replica-count method (measure per-replica throughput, divide peak concurrency by it, then add headroom for failover and rolling upgrades) is described in [Sizing and Scale](/reference/sizing-and-scale/). - **Multi-region active-active operation.** Several regional data planes can serve the same traffic at once, each sized for its own share of peak concurrency plus headroom to absorb a failed peer. Running regions active-active rather than active-passive means there is no cold standby to spin up during an incident; the surviving regions are already serving traffic. Coordinating configuration across regional instances is covered in [Run Multiple Agent Router Instances](/agent-router-enterprise/guides/operate-and-govern/run-multiple-platform-instances/). - **Health-aware routing and failover.** Agent Router selects backends and regional deployments per request using live health signals, taking unhealthy endpoints out of rotation automatically. Pooling regional deployments of one model and observing how traffic redistributes when a region is drained is covered in [Load-Balance Across Regional Deployments](/agent-router-enterprise/guides/operate-and-govern/load-balance-across-regional-deployments/). - **Optional multi-gateway DNS failover (Enterprise).** By default a project has one gateway. When high availability across gateway runtimes is required, Agent Router Enterprise can attach additional like-configured gateways to the same project; the operator fronts them with customer-managed DNS health checks against each member's membership probe. This fails over **gateways**, not model backends, and is distinct from regional InferencePool load balancing. See [Gateway sets and DNS-level failover](/agent-router-enterprise/guides/operate-and-govern/project-and-data-plane-management/gateway-sets-and-dns-failover/). A resilient deployment uses the first three: enough replicas per region to survive node loss, enough regions to survive a regional outage, and health-aware routing to make backend failover automatic rather than manual. Add the fourth only when the project itself needs interchangeable gateway members behind a shared client-facing name. ## Step 2: define RPO and RTO targets Recovery point objective (RPO) and recovery time objective (RTO) are the two numbers that turn "Agent Router should be resilient" into a testable requirement. - **RPO** is the maximum acceptable amount of data loss, measured as a span of time. An RPO of one hour means that, after a failure, the deployment may be restored to a state no older than one hour, so configuration and state must be backed up at least that often. For a gateway, the state at risk is the routing rules, policies, API keys, and user configuration, not the in-flight requests themselves. - **RTO** is the maximum acceptable time to restore service after a failure. An RTO of fifteen minutes means service must be back within fifteen minutes of the failure being detected. Agent Router's architecture shapes what these targets attach to. The management plane is Tetrate-hosted and stores the routing rules, policies, and user configuration; the data plane is customer-managed and processes traffic. A regional data-plane failure in an active-active topology is absorbed by the surviving regions, so its effective RTO is bounded by how quickly health-aware routing sheds the failed region rather than by any manual restore. A loss of customer-managed configuration, by contrast, is governed by how recently that configuration was backed up (which is the RPO) and how quickly it can be reapplied (which is the RTO). Targets are agreed with the field team, not chosen unilaterally, because they trade directly against cost: a tighter RPO means more frequent backups, and a tighter RTO usually means more standing capacity. Record the agreed numbers alongside the deployment's other non-functional requirements so the rest of the plan can be measured against them. ## Step 3: back up configuration and state A restore is only possible if there is something to restore from. Two categories of state are backed up. 1. Capture the data-plane configuration from its source of truth. When the data plane is managed through GitOps as described in [Kubernetes Resources for GitOps](/agent-router-enterprise/self-hosted-data-plane/data-plane-installation/k8s-resources-for-gitops/), the manifests in version control are themselves the backup; the cluster can be rebuilt from the repository. Confirm that every manifest the running deployment depends on is committed, and that no configuration has been applied out of band that would be lost in a rebuild. 1. Capture Agent Router configuration that lives in the management plane: routing rules, policies, API keys, and user records. Export or snapshot this on a schedule no longer than the agreed RPO, so that a restore lands within the data-loss budget. The backup cadence is driven by the RPO from Step 2: if the RPO is one hour, the most stale backup at any moment must be no more than one hour old. Store backups outside the region they protect, so that a regional incident does not take the backup with it. ## Step 4: write the DR runbook A DR runbook is the document an on-call operator follows during an incident, when there is no time to work things out. It covers three procedures. - **Failover.** The steps to shift traffic away from a failed region or component. In an active-active topology, health-aware routing performs most of this automatically; the runbook records what the operator confirms (that traffic has shifted, that the surviving regions are within capacity) and any manual action needed if automatic failover does not fully cover the failure. - **Restore.** The steps to rebuild a failed region or reapply lost configuration from the backups taken in Step 3: reapplying the GitOps manifests, restoring the management-plane configuration snapshot, and verifying the rebuilt deployment serves traffic correctly. - **Verification.** The checks that confirm service is genuinely healthy after failover or restore, not merely reachable: a representative request succeeds end to end, error rate has returned to baseline, and the affected region is back in rotation. Write each procedure as numbered, ordered steps that name the exact commands, surfaces, and expected results. The test of a good runbook is whether an operator who did not write it can follow it under pressure. ## Step 5: test the runbook with at least one restore A runbook that has never been exercised is untested. Restore tests routinely surface a missing backup, a manifest that was never committed, or a step that assumed access the on-call operator does not have. A documented RPO and RTO are only credible once at least one restore has actually been performed against them. 1. Schedule a restore test in a non-production environment, or in a maintenance window where a controlled failure is acceptable. Coordinating disruptive tests against the right instance is covered in [Run Multiple Agent Router Instances](/agent-router-enterprise/guides/operate-and-govern/run-multiple-platform-instances/). 1. Simulate the failure the runbook is written for: for example, by tearing down a region's data plane or by starting from an empty cluster. 1. Follow the restore procedure from Step 4 exactly as written, without improvising. Where a step does not work as documented, the runbook is corrected, not worked around. 1. Time the restore from simulated failure to verified-healthy service, and compare it against the RTO. Confirm that the restored state is within the RPO: no configuration newer than the last backup was expected to survive. 1. Record the outcome: the date, the scenario, the measured restore time, whether RPO and RTO were met, and any runbook corrections made. This record is the evidence that the DR plan has been tested. Repeating the restore test on a schedule, and after any significant change to the deployment, keeps the runbook from drifting out of date as Agent Router evolves. ## Step 6: load-test against agreed throughput and concurrency targets Capacity planned on paper is an estimate until it is validated under load. A load test confirms that the deployment sustains the agreed throughput and concurrency while keeping latency and error rate within bounds, and it is the evidence that horizontal scaling actually delivers the planned capacity. 1. Agree the targets with the field team: the peak requests per second, the peak concurrent in-flight requests, and the workload mix: token distribution and the streaming-versus-non-streaming split. The method for deriving these figures, and the reasons concurrency rather than monthly average drives sizing, are in [Sizing and Scale](/reference/sizing-and-scale/). 1. Build a load profile that reproduces the agreed mix against a representative set of models, rather than a single trivial request repeated, so that the test exercises the same per-request cost the real workload will. 1. Drive the load against the deployment, ramping up to the agreed peak and holding it long enough for the data plane to reach steady state under sustained concurrency. 1. Record the results that the criterion calls for: the 95th-percentile (p95) request latency and the error rate at the agreed peak. Because provider inference dominates end-to-end latency, separate the gateway's own contribution from provider time using the latency metric families described in [Sizing and Scale](/reference/sizing-and-scale/), so that a latency result is not misattributed to the gateway when it originates upstream. 1. Confirm horizontal scaling under the same test: add Agent Router replicas and verify that sustained throughput rises and per-replica load falls, demonstrating that capacity is added by scaling out rather than by enlarging a single instance. A load test passes when the deployment holds the agreed peak with p95 latency and error rate within the agreed bounds, and the result is recorded alongside the RPO/RTO evidence as the deployment's validated capacity. ## What to do next - **Validate sizing assumptions.** The load test in Step 6 depends on agreed throughput, concurrency, and peaking-factor figures. Confirm the open capacity questions before committing any number. See [Sizing and Scale](/reference/sizing-and-scale/). - **Coordinate the regional fleet.** Multi-region active-active operation is administered per instance, and configuration parity across regions is an operational discipline in its own right. See [Run Multiple Agent Router Instances](/agent-router-enterprise/guides/operate-and-govern/run-multiple-platform-instances/). - **Rehearse a regional drain.** Exercising health-aware failover by draining a region and observing the redistribution complements the restore test in Step 5. See [Load-Balance Across Regional Deployments](/agent-router-enterprise/guides/operate-and-govern/load-balance-across-regional-deployments/). - **Optional: gateway sets and DNS-level failover.** When a project needs interchangeable Enterprise gateway members behind customer-managed DNS, see [Gateway sets and DNS-level failover](/agent-router-enterprise/guides/operate-and-govern/project-and-data-plane-management/gateway-sets-and-dns-failover/). Where to go next Run multiple Agent Router instances Coordinate the regional fleet and keep configuration parity across active-active instances. Gateway sets and DNS-level failover Optional Enterprise HA: attach additional gateways and fail over with customer-managed DNS. --- ## Authenticate gateway requests with an external IdP Enterprise Tier Gateway authentication lets callers that already hold a token from your own identity provider (IdP) reach a project's gateways, without being issued an Agent Router API key first. You configure one JWT provider for the project and describe how its token claims map to an Agent Router principal. The configuration is project-scoped, so it applies to every gateway in the project at once. Authentication is configured per project because a project is the shared security boundary: the model catalogue, MCP servers, and providers are authored once for the project and inherited by all of its gateways, and identity works the same way. A single provider therefore lets requests fail over between the project's gateways under one identity boundary. This sits alongside API keys rather than replacing them: project API keys continue to authenticate on the `Authorization` header, while IdP tokens are read from a separate, dedicated header, so both can be used against the same gateway at the same time. ## How access control differs from project API keys Project API keys and IdP authentication gate access in fundamentally different ways, and it is worth being deliberate about the distinction: - **Project API keys are _per-user_ grants.** A user is provisioned into Agent Router (typically synced from your SSO), granted membership of the project, and issues a key. Access is an explicit grant you manage in the user directory, and the gateway recognises the user because it already knows them. - **IdP authentication is _attribute-based_, and the gateway holds no directory of your external users.** You do not pre-grant individual people. Admission is delegated to your IdP (only identities it will mint a token for can get in) and scoped by the **identity-mapping rules** you define here: a token is admitted only if its claims match a rule (for example an email domain or a group), and a just-in-time rule provisions a principal on first sight, up to its cap. There is no per-user allow-list synced from the user directory; your IdP's membership and the rule's claim pattern are the access boundary. In practice this means you control *who can reach the gateway* by managing membership and group assignment **in your IdP** (together with the rules below), not by granting individual users in Agent Router. To admit or remove a whole population, adjust the IdP group or the rule; to cut off a single external identity after the fact, disable its provisioned principal. Your IdP stays the single source of truth for external-user lifecycle, which is also how the underlying Envoy Gateway JWT authentication is designed to work. **Persona:** Platform operator or project administrator working in the Admin Dashboard, often in coordination with the identity team that owns the upstream provider. **Estimated time:** 15 to 30 minutes for an initial configuration, depending on how quickly the IdP's issuer, audiences, and JWKS endpoint can be confirmed. ## Outcomes By the end of this guide: - One JWT identity provider is configured for the project and applies to all of its gateways. - Verified token claims resolve to an Agent Router principal, either a pre-registered one or one provisioned just in time within a capped project. - A request carrying a valid IdP token is accepted, and requests carrying forged or expired tokens are rejected. - Existing project API keys continue to work unchanged on the same gateways. ## Prerequisites - Membership of the project, with permission to manage its gateway configuration. - The project already has a gateway provisioned, so there is an endpoint to authenticate against. See [Provision a gateway for a project](/agent-router-enterprise/guides/operate-and-govern/project-and-data-plane-management/provision-a-gateway/). - From the IdP, the values Agent Router validates every token against: the issuer (`iss`), the audiences (`aud`) the token carries, and either the JWKS URL where the public signing keys are published or the inline JWKS itself. - A way for your callers to place the IdP's token on a request header other than `Authorization` (the default is `x-tars-idp-token`). ## Step 1: Open the Authentication tab 1. Use the project switcher at the top of the application to select the project the provider is for, for example `bedrock-team`. 1. Open the project and select the **Authentication** tab. 1. If no provider is configured yet, select **Configure authentication**. Otherwise, select **Edit provider** on the existing provider card. Everything configured here applies to the currently selected project. Selecting the wrong project configures authentication for gateways you did not intend. ## Step 2: Configure the JWT identity provider Enter the provider's details and save it. Agent Router uses these to verify every incoming IdP token. 1. Give the provider a **Name** that is recognisable later, for example `Okta production`. 1. Set the **Issuer** to the `iss` value the IdP stamps on its tokens, for example `https://okta.example.com/oauth2/default`. 1. Choose a **JWKS source**. Select **Remote URL** and enter the endpoint where the IdP publishes its public signing keys, or select **Inline JWKS** and paste the key set directly. Remote keys are refreshed on a schedule. 1. Add the **Audiences** the token must carry, one per entry. A token whose `aud` does not include one of these values is rejected. 1. Leave the **Extraction header** as `x-tars-idp-token` unless your callers place the token elsewhere. It must not be `Authorization`, which is reserved for project API keys. 1. Set **Enabled** to on to begin verifying tokens as soon as the provider is saved, then save. Once saved, the provider card shows the configuration and confirms it has been reconciled to every gateway in the project. ### Provider field reference | Field | Description | | :---- | :---- | | Name | A label to recognise the provider across the dashboard. Does not affect verification. | | Mechanism | The verification mechanism. `JWT` is the only mechanism in this phase. | | Issuer | The `iss` claim every token is validated against. | | JWKS source | Where Agent Router fetches the public signing keys: a remote URL (refreshed on a schedule) or an inline key set. | | Audiences | The `aud` values a token must carry. A token missing all of them is rejected. | | Extraction header | The request header the IdP token is read from. Defaults to `x-tars-idp-token`; must not be `Authorization`. | | Enabled | Whether the provider verifies tokens. Turning it off stops accepting IdP tokens without deleting the configuration. | ## Step 3: Map verified claims to principals A verified token is only accepted once a mapping rule resolves it to a principal, the identity that usage, budgets, and policy attach to. Rules are evaluated top-down and the first match wins, so order the more specific rules first. 1. On the provider, select **Add rule**. 1. Set the **Claim name** to the token claim to match on, for example `email` or `groups`. 1. Optionally set a **Match template**, a shell-style glob such as `*@example.com`. Leave it blank to match any value of the claim. 1. Choose a **Target**. Select **Pre-registered** to resolve to a principal that already exists, or **JIT provision** to create one on first sight. 1. For a JIT rule, choose the **JIT project** the new principal is created in and set a **Principal cap**, the maximum number of identities that project will auto-create before further new tokens are rejected. 1. Set the **Principal type**, then save the rule. Repeat for each claim pattern you want to admit. A common shape is one JIT rule per trusted email domain or directory group, with a final pre-registered rule for known service identities. ### Mapping-rule field reference | Field | Description | | :---- | :---- | | Claim name | The JWT claim the rule matches on, for example `sub`, `email`, or `groups`. | | Match template | Optional shell-style glob the claim value must match. Blank matches any value. | | Target | `Pre-registered` resolves to an existing principal; `JIT provision` creates one on first match. | | JIT project | The project a just-in-time principal is created in. JIT rules only. | | Principal cap | The maximum number of identities the JIT project auto-creates before new tokens are rejected. JIT rules only. | | Principal type | The kind of principal the rule produces, for example an external identity or a service account. | :::info Tokens are verified in full A mapping rule runs only after the token itself is verified: its signature, issuer, audience, and expiry are checked against the configured JWKS. Claim headers a client sends are never trusted on their own, and the IdP token is never forwarded to an upstream model provider. ::: ## Step 4: Test and enable Confirm the provider before relying on it. 1. From a client that holds a token from the IdP, send a request to the project's gateway with the token on the extraction header, for example: ```bash curl https:///v1/models \ -H "x-tars-idp-token: " ``` 1. Confirm the request is accepted and that its usage is attributed to the expected principal in the project's request logs. 1. Confirm a request carrying a forged, expired, or wrong-audience token is rejected, and that a request with the IdP token placed on `Authorization` is refused rather than treated as an API key. 1. Confirm an existing project API key still authenticates on the `Authorization` header, unchanged. If verification is not behaving as expected, re-check the issuer, audiences, and JWKS source against the IdP, and confirm the token is on the extraction header rather than `Authorization`. ## Configure with the admin API The same provider and rules can be managed programmatically through the admin API, which is useful for configuration as code. The provider resource is created against the project: ```bash curl https:///admin/v1/gateway-auth-policies \ -H "X-API-Key: " \ -H "Content-Type: application/json" \ -d '{ "projectId": "bedrock-team", "name": "Okta production", "mechanism": "jwt", "issuer": "https://okta.example.com/oauth2/default", "jwksUri": "https://okta.example.com/oauth2/default/v1/keys", "audiences": ["agent-router"], "extractionHeader": "x-tars-idp-token", "enabled": true }' ``` Mapping rules are then added to the returned policy, and the same operations are available through the Agent Router SDKs (`gatewayAuthPolicy` in the TypeScript and Python SDKs, `GatewayAuthPolicy()` in Go). See the [SDK reference](/reference/sdk/) for language quickstarts. ## What to do next - **Issue a project-scoped API key**: the other credential a gateway accepts, on the `Authorization` header, alongside IdP tokens. See [Issue a project-scoped API key](/agent-router-enterprise/guides/operate-and-govern/project-and-data-plane-management/issue-a-project-api-key/). - **Configure SSO**: the separate, organisation-wide identity flow for signing operators into the Admin Dashboard itself. See [Configure SSO](/agent-router-enterprise/guides/identity-and-access-management/single-sign-on/sso-configuration-guide/). Where to go next Issue a project-scoped API key The API-key credential a gateway accepts alongside IdP tokens. Configure SSO Organisation-wide identity for signing operators into the dashboard. --- ## Create and delete a project Enterprise Tier A project is the isolation boundary for a team's models, keys, Model Context Protocol (MCP) servers, policy, and members. The Create Project wizard creates the project and can grant members, models, and MCP servers in the same pass; models must already be enabled in the organisation catalog ([Provision models and providers](/agent-router-enterprise/guides/operate-and-govern/provision-models-and-providers/)). On its own the project is configuration only—selecting a data plane so the platform provisions the gateway that makes it callable follows in the next guide. When a project is no longer needed, delete it from the project's page after clearing anything that blocks removal. Every wizard step except Details is optional. A project can be created empty and given members, models, and MCP servers later from its own pages: members in [Manage project members and access](/agent-router-enterprise/guides/operate-and-govern/project-and-data-plane-management/manage-project-members/), MCP in [Grant MCP servers and profiles to a project](/agent-router-enterprise/guides/operate-and-govern/project-and-data-plane-management/scope-mcp-to-a-project/), and the endpoint in [Provision a gateway for a project](/agent-router-enterprise/guides/operate-and-govern/project-and-data-plane-management/provision-a-gateway/). **Persona:** Platform operator working in the Admin Dashboard. **Estimated time:** 5 to 10 minutes per project to create; a few minutes to delete once blockers are cleared. ## Outcomes By the end of this guide: - A project exists as an isolated configuration and access boundary. - The members who should work in the project are added, with the creator as admin. - The models the project may call are granted, with their providers granted automatically. - The MCP servers the project may use are granted. - You can delete a project when it is no longer needed, after clearing live keys, data planes, gateways, and project settings. ## Prerequisites - Access to the Admin Dashboard with permission to create projects. The **Create Project** button is only visible with project write permission. Deleting a project requires permission to manage projects so the **Delete project** option appears. - The models the project will use already enabled in the organisation's catalog. Adding provider credentials and enabling models is covered in [Provision models and providers](/agent-router-enterprise/guides/operate-and-govern/provision-models-and-providers/). ## Step 1: Open the Create Project wizard 1. In the Admin Dashboard, open **Projects** and select **Create Project**. ![Projects page with the Create Project button](/img/screenshots/admin/create-project-list.jpeg) The wizard walks through five steps—Details, Members, Models, MCP servers, and Review. Only Details is required; the catalog steps (Members, Models, MCP servers) may be skipped and granted later. Some steps may not appear at all. The wizard shows a catalog step only if the account can browse that catalog (the user directory, the model catalog, or the MCP server catalog). The project can still be created without those steps; someone with the missing permissions adds the grants afterwards. ## Step 2: Name the project (Details) 1. Give the project a name that identifies the team, application, or environment it represents, for example `bedrock-team`. 1. Review the generated **Project ID**. It is suggested from the name and can be edited, and must use only lowercase letters, digits, and hyphens. The ID becomes the project's path, shown under the field as `/projects/`. 1. Optionally add a description. ![Details step with name, project ID, and description](/img/screenshots/admin/create-project-wizard-details.jpeg) The name is an optional display name; the Project ID is the required, permanent identifier. ## Step 3: Add members (optional) 1. Search for users by name or email, or browse by team, and select the people who should have access. 1. Optionally select a whole team to add all of its members at once. ![Members step with users and teams selected](/img/screenshots/admin/create-project-wizard-members.jpeg) The creator becomes the project admin automatically; everyone selected joins as a member. Selecting a team adds its current members individually—users who join the team later are not added to the project. Roles are adjusted later in [Manage project members and access](/agent-router-enterprise/guides/operate-and-govern/project-and-data-plane-management/manage-project-members/). ## Step 4: Grant models (optional) 1. Pick the models the project may call, grouped by provider. Select all models from a provider in one click, or search across all models. ![Models step with providers and models selected](/img/screenshots/admin/create-project-wizard-models.jpeg) Providers are granted automatically based on the models picked—there is no separate provider-grant step. Model access remains explicit: a model that is not granted cannot be reached through the project's gateway and is refused at request time. This explicit grant is the mechanism behind per-project model isolation. Only models already enabled in the organisation catalog appear here. ## Step 5: Grant MCP servers (optional) 1. Pick the MCP servers the project may use from the organisation's catalog. ![MCP servers step with servers selected](/img/screenshots/admin/create-project-wizard-mcp.jpeg) Granting servers here scopes them to the project; building a profile that exposes them to clients is covered in [Grant MCP servers and profiles to a project](/agent-router-enterprise/guides/operate-and-govern/project-and-data-plane-management/scope-mcp-to-a-project/). ## Step 6: Review and create 1. Check the summary of members, models, and MCP servers. Expand a section to remove individual items before creating. 1. Select **Create project**. ![Review step summarising models, members, and MCP servers](/img/screenshots/admin/create-project-wizard-review.jpeg) If any grant fails during creation—for example one provider errors—the project is still created, and a warning lists what did not apply. Re-apply the missing grants from the project's pages afterwards. ## Delete a project To remove a project you no longer need, open the project's page, select the **⋯** menu in the header, then **Delete project**. You need permission to manage projects to see this option. The default project created during onboarding cannot be deleted. A confirmation dialog opens and shows what will happen **before** anything is removed. ![Delete project confirmation dialog](/img/screenshots/admin/delete-project-dialog.jpeg) ### When something is blocking the delete If the project cannot be deleted yet, the dialog says so and lists exactly what to clear first: - **Live API keys** — revoke or delete them from the project first. See [Issue a project-scoped API key](/agent-router-enterprise/guides/operate-and-govern/project-and-data-plane-management/issue-a-project-api-key/). - **Data planes** — remove them from the project first. - **Gateways** — delete the project's gateway first. See [Manage multiple gateways on a data plane](/agent-router-enterprise/guides/operate-and-govern/project-and-data-plane-management/manage-gateways-on-a-data-plane/). - **Settings** (rate limit or global fallback) — these have no other screen, so the dialog provides a one-click **Clear settings** control. Once everything listed is cleared, the delete control unlocks. ### When nothing is blocking the delete The dialog previews the impact—for example that deletion will remove members and model assignments—so you know what goes away with the project. Tick the **I understand** checkbox, then select **Delete**. ### What gets removed The project and everything attached to it are removed in one step: members, model assignments, provider and MCP server grants, and gateway links. Removed members lose their project permissions, and the project's API keys disappear from the keys list. Shared organisation resources are not deleted. Models, providers, MCP servers, and users themselves remain; only the project's grants and memberships are removed. Deletion is permanent—there is no undo. The project's name becomes available again right away, so a new project can reuse it. ## What to do next A project with granted models is still configuration with nowhere to send traffic. It becomes usable only once you select a data plane on the project and the gateway is auto-provisioned. - **Provision a gateway for a project**: give the project a live inference URL by selecting a data plane. See [Provision a gateway for a project](/agent-router-enterprise/guides/operate-and-govern/project-and-data-plane-management/provision-a-gateway/). - **Manage project members and access**: adjust membership and roles after creation. See [Manage project members and access](/agent-router-enterprise/guides/operate-and-govern/project-and-data-plane-management/manage-project-members/). Where to go next Provision a gateway for a project Give the project a live inference URL by selecting a data plane so the gateway is provisioned. Manage project members and access Add the users who will use the project, and assign their roles. --- ## Deploy and register a data plane Enterprise Tier A data plane is the layer that provisions and manages gateways inside a Kubernetes cluster the customer controls. Adding a data plane is how capacity is placed in a new region, cloud, or environment. This guide provisions a data plane from the management plane, installs the credential it uses to connect, deploys it, and confirms it has registered. A data plane connects to the management plane using a *data plane credential*, which authenticates its calls back to the management plane for configuration, lookup, telemetry, and status. The connection is outbound only: the data plane reaches out to the management plane, and no inbound connection into the cluster is required. Once connected, the data plane receives the desired configuration for any gateways assigned to it and reconciles them. **Persona:** Platform operator or platform engineer with access to both the Admin Dashboard and the target Kubernetes cluster. **Estimated time:** 30 to 60 minutes, depending on cluster readiness. ## Outcomes By the end of this guide: - A data plane is provisioned and visible in the management plane. - Its data plane credential is installed in the target cluster. - The data plane is deployed, has connected to the management plane, and is available to provision gateways on. ## Prerequisites - Administrator access to the Admin Dashboard, with data-plane operations permissions. - A Kubernetes cluster to host the data plane, in any cloud or on-premises, meeting the platform requirements in [Prerequisites](/agent-router-enterprise/self-hosted-data-plane/prerequisites/). - Familiarity with the data plane installation procedure in [Data plane installation](/agent-router-enterprise/self-hosted-data-plane/data-plane-installation/). ## Step 1: Provision the data plane 1. In the Admin Dashboard, provision a data plane, giving it a name and any labels that describe where it runs, such as region, environment, or team. A newly provisioned data plane exists in the management plane before any software is deployed. At this stage it is a record awaiting a deployment to connect to it. ## Step 2: Install the data plane credential 1. When the data plane is provisioned, the management plane produces a data plane credential and presents it for download. 1. Install the credential in the target cluster, where the deployed data plane will read it to authenticate back to the management plane. The credential identifies the customer and the data plane it belongs to. Credentials support rotation, so a credential can be replaced without recreating the data plane. ## Step 3: Deploy the data plane to the cluster 1. Deploy the data plane components into the Kubernetes cluster, following [Data plane installation](/agent-router-enterprise/self-hosted-data-plane/data-plane-installation/). 1. Confirm the deployment can reach the management plane over its single outbound connection. ## Step 4: Confirm the data plane has registered 1. Confirm the data plane connects and appears as reachable in the management plane. The management plane uses the data plane's regular check-in as a health signal, so a connected data plane shows as available and one that stops checking in is flagged as unreachable. 1. Confirm the data plane is now selectable when placing a project on a data plane. Once registered, the data plane receives the desired state for any gateway assigned to it and materialises the runtime for each. Running several data planes from one management plane is described in [Run multiple platform instances](/agent-router-enterprise/guides/operate-and-govern/run-multiple-platform-instances/). ## What to do next - **Provision a gateway for a project**: put the new data plane to work by hosting a project's gateway on it. See [Provision a gateway for a project](/agent-router-enterprise/guides/operate-and-govern/project-and-data-plane-management/provision-a-gateway/). - **Manage multiple gateways on a data plane**: operate the gateways this data plane provisions. See [Manage multiple gateways on a data plane](/agent-router-enterprise/guides/operate-and-govern/project-and-data-plane-management/manage-gateways-on-a-data-plane/). Where to go next Manage multiple gateways on a data plane Review and assign the gateways this data plane provisions and manages. Run multiple platform instances Operate several data planes from a single management plane. --- ## Gateway sets and DNS-level failover Enterprise Tier A project can attach more than one gateway, and every attached gateway serves the identical project configuration by construction. That turns a set of member gateways into interchangeable failover targets: point customer-managed DNS at the member URLs, health-check each member's membership probe endpoint, and a client that moves from one member to another encounters the same authentication, guardrails, and catalogs it left behind. This guide covers the membership model, the probe contract, a reference DNS failover pattern, and the runbook for detaching a member safely. :::info Availability Multi-gateway projects require an Agent Router Enterprise release that includes the multi-gateway projects capability, and the deployment itself must declare the Enterprise product line (`DEPLOYMENT_PRODUCT=enterprise`, derived from the installation charts; the gate fails closed, so a deployment that does not declare it behaves as Agent Router Service). On earlier releases a project maps to exactly one gateway and this guide does not apply. This surface does not exist on Agent Router Service (SaaS). ::: Agent Router provides the consistency guarantee, the membership lifecycle, per-member visibility, and per-data-plane request attribution. It ships **no traffic steering**: there is no built-in load balancer, GSLB, or failover executor in front of a project's gateways. Failover runs entirely under the operator's control, at the DNS (or equivalent routing) layer the operator already owns. This is deliberate — the DNS name clients call is the operator's artifact, and the failover policy (active/passive, weighted, geo) is an infrastructure decision Agent Router does not make. **Persona:** Platform operator working in the Admin Dashboard and the organisation's DNS provider (the examples use AWS Route 53). **Estimated time:** 30 to 45 minutes to set up the reference pattern for one project; 10 to 15 minutes to run the detach runbook. ## Outcomes By the end of this guide: - The membership model is understood: like-configured members, the four membership states, the zero-gateway state, and per-data-plane request attribution. - Customer-managed DNS failover is configured against the member URLs, health-checked on the membership probe endpoint. - The project's endpoint URL is set, so developers are given the client-facing name rather than a single member's URL. - A member has been detached safely: DNS shifted, drain verified through attribution, and the terminal state interpreted correctly. - The one documented limitation — per-gateway rate limits — is understood and communicated to application teams. ## Prerequisites - Administrator access to the Admin Dashboard, with project and data-plane operations permissions. - A project with at least one gateway attached and ready. See [Provision a gateway for a project](/agent-router-enterprise/guides/operate-and-govern/project-and-data-plane-management/provision-a-gateway/). - Control of the DNS zone that clients resolve, and a DNS provider that supports health-checked records (Route 53 failover or weighted records, or equivalent). ## Concepts: a project is a set of like-configured gateways ### Identical by construction A project is the single configuration authority; its attached gateways are interchangeable members. Every member receives the identical project configuration — authentication and authorization, guardrails, and the model and MCP catalogs — generated from the same project state. Per-member variance is limited to identity, URL, name, and labels. Divergence of *desired* configuration between members is not reconciled after the fact; it is unrepresentable — there is no per-member override surface at all. Delivery, though, is by data-plane-initiated polling: at any instant the guarantee is **identical desired configuration plus a bounded convergence window**, during which members can serve different revisions of the same project configuration. Each member's configuration version and sync state are surfaced on the **Gateways** tab. For security review this means the project, not the gateway, is the unit of review: no failover event can reach a boundary that was never part of the project's configuration, because there is no member on which a different rule set could exist. Reviewing the project's policy once covers every member it will ever have, including members attached after the review. The convergence window is the one nuance: after a security-*tightening* change, a client failing over to a member still catching up can observe the previous (looser) boundary until that member converges. Confirm every member reports the new configuration version before treating a tightened boundary as enforced fleet-wide — a check that is meaningful only for attached, reachable members with management-plane connectivity. Treat an empty configuration-version column as convergence **unconfirmed**, not as converged. ### Membership states Each member is in exactly one state, shown in the **Status** column of the project's **Gateways** tab: | State | Meaning | Operational significance | | :---- | :---- | :---- | | `syncing` | The member is converging on the project's current configuration version. | A freshly attached member that has never converged serves nothing and probes red — it cannot receive traffic prematurely. A previously ready member that is catching up to a config change keeps serving its last-known-good configuration and probes green. | | `ready` | The member has confirmed convergence on the project's current configuration version. | A valid traffic and failover target. Advertised in the project view with its configuration version and last-seen time. | | `detached — zeroized` | Detach completed and the gateway acknowledged the final, emptied configuration: routes, credentials, and catalogs removed. | Teardown is confirmed. The listener may still be up; the probe is red (a true no-route 404 — the probe path is exempt from gateway authorization). Client traffic on the drained hostname is rejected `403 unknown_host` if the data plane hosts any other serving gateway, and gets a no-route 404 only when it hosts none — expect 403s in client logs during a drain on a shared data plane. Nothing further is required. | | `detached — unreachable` | Detach was requested but the gateway never acknowledged the zeroized configuration; the management plane revoked its identity as the backstop. | The gateway can no longer fetch project configuration or authenticate to the management plane, but its local teardown is unconfirmed — a partitioned-but-reachable gateway may still be serving its last-known configuration and answering its probe green. Remove its DNS record manually and verify decommissioning (see the runbook below). | A member is promoted from `syncing` to `ready` only on confirmed convergence, so no client ever reaches a half-configured gateway: until a new member is ready, its probe stays red and health-checked DNS never routes to it. ### Zero gateways is a valid state A project with zero attached gateways is a fully valid, fully configured project — keys, models, guardrails, and members are all retained. The Gateways tab shows a "nothing serving" state, which is a description, not an error. Use it deliberately: - **Stage before capacity exists.** Configure a project completely — policy reviewed, keys issued — before any gateway is attached. Attaching the first gateway later materializes the already-reviewed configuration. - **Drain to zero safely.** Detaching the last member leaves the project intact and re-attachable; nothing about the project is torn down with its capacity. ### Per-data-plane request attribution Every request is attributed to the **data plane** that served it — the reporting data plane's identity, stamped server-side when its telemetry reaches the management plane — and the usage and request-log surfaces are filterable by data plane. Attribution is data-plane granularity, not per-gateway: a data plane can host gateways from several projects, and gateways co-located on one data plane share an attribution bucket. This is what makes failover auditable and drains verifiable: - An audit trail survives failover — the record shows which data plane served each request, not just which project. When each member runs on its own data plane, that resolves the record to the member. - An incident can be localized: filter usage or request logs by data plane and compare the members' data planes. - A drain can be verified empirically before a detach — this is the backbone of the runbook below — **provided the member's data plane hosts no other serving gateway, from any project**. A shared data plane's counts include every co-located gateway; give a member a dedicated data plane if drain verification matters. Two caveats. Attribution follows the data plane: migrating a gateway to a different data plane splits its request history across the two data-plane buckets. And requests recorded before data-plane attribution existed appear under an unattributed `""` bucket rather than any data plane's. ## The membership probe endpoint Every member gateway exposes a health endpoint whose meaning is *ready member of this project* — not process liveness. This is the endpoint DNS health checks must target. The endpoint, on the member's own hostname: ``` GET https:///healthz/membership ``` The scheme, host, and port are the member's URL exactly as configured on the gateway — the same listener your clients use. No separate port, no admin interface. **Standard HTTPS port (443) only:** hostname matching for the gateway's host-stamped routes (the probe included) is port-exact, and health checkers send `Host: host:port` for a non-standard port — so a member URL on a non-standard port is outside this contract. Front a non-443 listener with a standard-port endpoint before pointing DNS health checks at it. Green and red, exactly as specified: > - **Green (healthy):** HTTP `200` with body `{"status":"ready","gateway":""}` (`Content-Type: application/json`). The UUID identifies **which** member answered — useful when DNS fronts several members. > - **Red (unhealthy):** any non-2xx response — concretely HTTP `404` — or a connection failure/timeout. There is no "degraded" body; absence of the green answer **is** the signal. **Semantics: "ready member of this project" — not process liveness.** The probe is green if and only if the gateway currently holds the project's materialized, non-zeroized configuration: > | Membership state | Probe | Why | > |---|---|---| > | `ready` (config synced) | **200 green** | The probe route is materialized together with the member's project routes. | > | `syncing`, never synced (fresh attach) | **404 red** | Nothing materialized yet — the probe route does not exist until the first successful config rollout. | > | `syncing`, previously synced (catching up to new config) | **200 green** | The member still serves its last-known-good config; it remains a valid failover target. | > | detached — zeroized | **404 red** | Detach removes the member's config **and its probe route**; the listener stays up, so the probe goes red (the probe path is exempt from gateway authorization, so it is a true no-route 404) and DNS health checks drain the member. Client traffic on the drained hostname gets `403 unknown_host` when the data plane hosts any other serving gateway, and a no-route 404 only when it hosts none — expect 403s in client logs during the drain window on a shared data plane. | > | detached — unreachable (from the DP's side) | **red** once the zeroize lands or the gateway is down — but **may stay GREEN** while the gateway is partitioned from the management plane yet still client-reachable | The probe is data-plane-local configuration; a zeroize that never reaches the DP cannot turn it red. DNS will NOT drain this member automatically: remove the member's DNS record manually and verify decommissioning (see the detach runbook below). | > | Gateway/listener down | **red** (connection failure) | Handled by the health checker's own timeout. | The properties behind the table, summarised from the contract: - **Answered by the data plane itself.** The probe is pure gateway configuration (a native direct-response route) — no management-plane round-trip, no backend, no AI processing. It keeps answering during a full management-plane outage; a dashboard or management-plane failure can never fail your DNS health checks. The flip side is the detached-unreachable row above: because the probe is data-plane-local, a zeroize that never reaches the gateway cannot turn its probe red. - **Red by construction on delivered teardown.** The probe route exists only as part of the member's materialized project configuration and is removed with it when the zeroized configuration lands — there is no separate health state machine that can drift out of sync with what the gateway actually serves. - **Host-scoped to the member.** The probe answers only on the member's configured hostname — never on another member's hostname sharing the same data plane. A member with no URL configured has no probe (red; it cannot be in a DNS rotation anyway). On a hostname change, the probe on the **old** hostname goes red as soon as the new configuration rollout begins, and re-greens on the **new** hostname once that rollout is ready — a stalled rollout can never leave the old hostname green. - **Unauthenticated for `GET`/`HEAD` only.** Quoting the contract's security claim precisely: "DNS health checkers cannot send API keys, so the data plane exempts exactly `GET`/`HEAD` on this path from gateway API-key authentication. Any other method — and any other path — still requires an API key (`401` otherwise), so the probe cannot be used as an unauthenticated request surface." - **What the probe does not tell you:** provider/upstream health, spend or rate-limit state, or per-request data-path success. A ready member of a project with no assigned models still probes green — it is faithfully serving its (empty) configuration. The probe answers "is this a ready member", nothing more. - **The probe, not the console, is the DNS go-signal.** Probe-green can briefly trail the `ready` state shown on the Gateways tab (typically under a minute) — the console reports confirmed convergence while the probe route finishes materializing on the data plane. A member that shows `ready` with a still-red probe is converging, not broken; add it to a DNS rotation only once its probe answers green. Two consequences worth calling out: - **Zeroized members drain themselves; unreachable ones do not.** When a detach completes — the gateway acknowledges the zeroized configuration — the probe route is removed with the config, the health check fails, and any health-checked DNS record pointing at the member goes unhealthy automatically. But a member that lands in `detached — unreachable` while still client-reachable (partitioned from the management plane, gateway process up) keeps answering its probe green: DNS will **not** drain it. This is why the runbook below shifts DNS *first* — the DNS shift is the drain; the probe going red is the automatic backstop only for the zeroized terminal. - **A newly attached member cannot receive traffic early.** Its probe stays red until first convergence, so health-checked DNS never routes to it half-configured. Verify a member by hand at any time: ```bash $ curl -s https://ai-gw-usw2.payments.examplebank.com/healthz/membership {"status":"ready","gateway":"1c9f4a6e-8d21-4f0b-9b3a-5e7d2c840a17"} ``` ## Reference pattern: DNS-level failover The pattern is: one client-facing DNS name the operator owns, resolving to member gateway URLs, with per-member health checks on the membership probe. The examples use Route 53; any DNS provider with health-checked failover or weighted records works the same way. The running example is a project `payments-assist` with two ready members: | Gateway | URL | Data plane | | :---- | :---- | :---- | | `gw-use1-payments` | `https://ai-gw-use1.payments.examplebank.com` | `dp-prod-use1` | | `gw-usw2-payments` | `https://ai-gw-usw2.payments.examplebank.com` | `dp-prod-usw2` | Clients call one name: `ai.payments.examplebank.com`. That name is the operator's artifact: the operator creates it, owns it, and Agent Router never resolves or routes on it. Agent Router can be told the name so it shows developers the right URL — see [Step 3](#step-3-tell-developers-the-name) — but that is a label, not configuration. Whatever produces the name is the operator's choice. This guide's example is a health-checked failover record, which is the common case; a load balancer or global traffic manager fronting the members works identically as far as Agent Router is concerned, because Agent Router is not involved either way. ### Step 1: Create a health check per member For each member, create a health check against the membership probe on the member's own hostname: 1. In Route 53, create an **HTTPS** health check with domain `ai-gw-use1.payments.examplebank.com`, port `443`, and path `/healthz/membership`. Count **only 2xx** as healthy, and enable **string matching** on `"status":"ready"` — an edge-level redirect (3xx) must never be mistaken for green. 1. Repeat for `ai-gw-usw2.payments.examplebank.com`. 1. Route 53 knobs: **fast interval** (10s; standard is 30s) and **failure threshold 3**; the request timeout is fixed by Route 53, not configurable. Expect roughly **30 seconds** to mark a member unhealthy, plus the record TTL for clients to drain to the survivor. The health check must target the member's standard HTTPS port: the probe contract is 443-only (see above), so front any non-443 listener with a standard-port endpoint before health-checking it. Check the member's hostname directly, never the shared client-facing name — a health check on `ai.payments.examplebank.com` would follow the failover itself and observe nothing useful. On providers other than Route 53, use the tightest check interval offered, require 2xx (never 2xx/3xx), and enable response-body matching on `"status":"ready"` where supported. ### Step 2: Create the failover records Active/passive with Route 53 failover routing: 1. Create a record `ai.payments.examplebank.com` → `ai-gw-use1.payments.examplebank.com`, routing policy **Failover**, type **Primary**, associated with the `gw-use1-payments` health check. 1. Create a second record `ai.payments.examplebank.com` → `ai-gw-usw2.payments.examplebank.com`, routing policy **Failover**, type **Secondary**, associated with the `gw-usw2-payments` health check. 1. Keep the record TTL short (30 to 60 seconds): the TTL, not the health check, usually dominates total failover time for clients that respect it. Active/active is the same shape with **Weighted** records instead — one weighted, health-checked record per member — and degrades to the surviving members when a health check fails. TLS note: clients connect to `ai.payments.examplebank.com` but the record resolves to a member; the certificate presented on each member's listener must cover the name clients dial (a SAN for the client-facing name, or terminate TLS at an edge the operator controls). This is edge infrastructure the operator owns, like the DNS name itself. ### Step 3: Tell developers the name Nothing so far tells developers the name exists. Left alone, the developer console shows them one member's URL — because until now that is the only URL it has — and an application that hardcodes it is pinned to that member and will not fail over. Everything in Steps 1 and 2 is then bypassed by the client. Set the project's **endpoint URL** to the client-facing name: 1. In the admin dashboard, open the project, go to the **Gateways** tab, and use the **Project endpoint** card. 1. Enter the name clients call — `https://ai.payments.examplebank.com` — and save. The developer console then shows that name as the project's Base URL, in code samples, in API-key setup, and in generated client configuration, in place of any member URL. Clearing the field reverts to showing a member URL. Two things this field is **not**: - **It is not routing configuration.** Agent Router stores it and displays it. It does not resolve it, does not health-check it, and it never reaches a gateway: routes, authorization, and the membership probes are all still keyed on member hostnames, exactly as in Steps 1 and 2. Setting it changes nothing about how traffic is served. - **It is not verified.** It names infrastructure the operator owns and Agent Router cannot reach, so the only check applied is that the value is a well-formed `http://` or `https://` URL. If the record is wrong, Agent Router will faithfully show the wrong URL. The console labels it as administrator-provided for exactly this reason. Because Agent Router does not route on the name, the members must accept it themselves. **Members accept it already.** Each member gateway runs in its own cluster, as its own install, and a gateway that is the only one in its install serves its routes **without a hostname match** — it answers on whatever `Host` arrives. A client dialling `ai.payments.examplebank.com` reaches whichever member DNS resolved to and is served normally. Authorization is scoped the same way. This is what makes the whole pattern work: no edge rewriting, no per-name configuration on the Agent Router side, nothing to enable. That leaves one genuine operator-side requirement: **TLS**. Clients dial the client-facing name, so whatever terminates TLS in front of each member must present a certificate valid for it. Health checks dial the member's *own* hostname, so that name must be covered too — two names per member, from two different callers. How much work this is depends entirely on how the certificates are named: - **A wildcard on the shared domain** (`*.payments.examplebank.com`) covers every member hostname and the client-facing name at once. Adding a member later is DNS and an attach; **no certificate work on the existing members**. This is the shape to aim for — but it is only available when the client-facing name and the member hostnames live under the **same domain**. If the client-facing name is published from a different zone than the members (common when member hostnames sit in per-account infrastructure zones and the client-facing name sits in a corporate zone kept for certificate management), no single wildcard covers both and the multi-SAN shape below is the correct one, not a fallback. - **A multi-SAN certificate (explicit domain list)** carries each name explicitly: every member's certificate lists the member's own hostname *and* the client-facing name. Retrofitting the client-facing name onto existing members is a reissue per member — managed certificates are generally immutable, so a name cannot be added to an issued certificate, only reissued. Once every member's certificate carries the client-facing name, adding a member later costs exactly one new certificate (the new member's own); existing members are untouched. See the [worked example](#worked-example-acm-multi-san-certificates-on-albs) below. :::tip Decide the client-facing name before the second member exists Under an explicit domain list, adding the shared name to each member's certificate after the fact is a reissue per member, on your PKI's timeline, while the failover configuration is half-built. Getting the name onto every certificate up front makes it one coordinated change — and it has to be decided anyway, since [Step 3](#step-3-tell-developers-the-name) publishes it to developers. ::: Certificates are also typically **regional**: each member's load balancer needs one issued in its own region, so even a wildcard is requested once per region rather than shared across them. #### Worked example: ACM multi-SAN certificates on ALBs This example is the multi-SAN branch made concrete: TLS terminates at an AWS **Application Load Balancer** in front of each member (an `ingressClassName: alb` Ingress, as installed by the [AWS data-plane installation guide](/agent-router-enterprise/self-hosted-data-plane/data-plane-installation/installation-guide-aws/#appendix-a-tls-via-acm--https-listener)), certificates come from **AWS Certificate Manager (ACM)**, and the client-facing name lives in a *different* zone than the member hostnames, so a wildcard is not an option. The naming chain, extending the running example: ```none ai-payments.apps.examplebank.net client-facing FQDN, corporate zone (what clients dial) CNAME → ai.payments.examplebank.com Route 53 failover record (Step 2) → healthy member's hostname ai-gw-use1... (primary) / ai-gw-usw2... (secondary) CNAME → that member's regional ALB k8s-tarsdata-...us-east-1.elb.amazonaws.com ``` Which names each certificate must carry follows from who dials what. CNAME resolution never rewrites what the client sends: a client dialling `ai-payments.apps.examplebank.net` presents that name in SNI and `Host` no matter how many CNAME hops resolution took, so the ALB it lands on — **either** region's, depending on failover state — must present a certificate valid for it. Health checks dial the member's own hostname (Step 1), so that name must be on the member's certificate too. The intermediate Route 53 record name appears in neither SNI nor `Host`; it needs a SAN only if operators dial it directly. Per regional ALB, that is one ACM certificate with (at least) two SANs: | ALB | SANs required | | :---- | :---- | | us-east-1 (fronts `gw-use1-payments`) | `ai-gw-use1.payments.examplebank.com`, `ai-payments.apps.examplebank.net` | | us-west-2 (fronts `gw-usw2-payments`) | `ai-gw-usw2.payments.examplebank.com`, `ai-payments.apps.examplebank.net` | Request each in its own region — ACM certificates are regional, and an ALB can only use certificates from its region: ```shell aws acm request-certificate \ --region us-east-1 \ --domain-name ai-gw-use1.payments.examplebank.com \ --subject-alternative-names ai-payments.apps.examplebank.net \ --validation-method DNS ``` Repeat for `us-west-2` with the other member hostname. Every SAN must complete validation before ACM issues the certificate — including the shared client-facing name, in **each** region's request. Names served from a private-only zone cannot complete public DNS validation; publish the validation CNAMEs from a public zone, fall back to email validation, or import a certificate from your own PKI. This is a common reason the client-facing name is deliberately placed in a publicly validatable corporate zone while member hostnames stay in private infrastructure zones. Attach each certificate to its region's Ingress (`alb.ingress.kubernetes.io/certificate-arn`, per the [installation guide's TLS appendix](/agent-router-enterprise/self-hosted-data-plane/data-plane-installation/installation-guide-aws/#appendix-a-tls-via-acm--https-listener)). One more ALB-specific check: an ALB is layer 7 and routes by `Host`. A dedicated ALB whose Ingress has no `host:` rule forwards every name and needs no change, but if the Ingress uses host-based rules — typical when the ALB is shared — each member's Ingress must match **both** names, or failed-over clients get the ALB's 404 instead of the gateway: ```yaml rules: - host: ai-gw-usw2.payments.examplebank.com # health checks + operators http: { ... } - host: ai-payments.apps.examplebank.net # clients, whenever DNS points here http: { ... } ``` :::warning The failure hides until failover Route 53 health checks dial member hostnames and do **not** validate the certificate, and in steady state clients only ever reach the primary. A secondary whose certificate is missing the client-facing SAN therefore looks completely healthy — green checks, working member URL — until the first real failover, at which point every client gets a TLS hostname-mismatch error. Verify the SAN set on **every** member's certificate, secondary included, before the failover rehearsal: ```shell openssl s_client -connect :443 \ -servername ai-payments.apps.examplebank.net /dev/null \ | openssl x509 -noout -ext subjectAltName ``` Run it against each regional ALB hostname directly (this bypasses the failover record, so the secondary is reachable regardless of failover state) and confirm the output lists both the member hostname and the client-facing name. Then rehearse the failover itself, as Step 4 requires. ::: Adding a third member later is one new certificate — the new member's hostname plus the client-facing name, in the new member's region — and no reissue on existing members, provided the client-facing name went onto every certificate up front (the tip above). Where TLS terminates depends on how the data plane is exposed — for a self-hosted deployment on AWS, see [Gateway installation on EKS](/agent-router-enterprise/self-hosted-data-plane/gateway-installation/gateway-installation-guide-aws/). Health checks still target member hostnames, never the client-facing name — Step 1 already covers why. The membership probe is host-scoped to the member's own hostname even though routes are not, so probing a member always tells you about that member specifically. :::note One-gateway-per-cluster is the shape this assumes Members are clones of one another in separate clusters — that is what makes them failover targets, and it is what the managed platform installs. Two gateways sharing a single install is not a failover topology (a lost cluster takes both), and in that shape each gateway's routes become matched to its own hostname, so a request carrying the client-facing name would be rejected. Keep one gateway per cluster and the question does not arise. Running one gateway per cluster is also what makes drain verification authoritative — see [Per-data-plane request attribution](#per-data-plane-request-attribution). ::: ### Step 4: Confirm the failover semantics What this pattern gives, and what it deliberately does not: - **A newly attached member cannot receive traffic early.** Its probe is red until first convergence, so its record stays unhealthy — attach order and DNS order do not need coordinating. - **A member detached to `zeroized` drains automatically.** The delivered zeroize removes the probe route; the health check fails; DNS steers away. A member that lands in `detached — unreachable` does **not** self-drain — its probe can stay green while the gateway is partitioned from the management plane but still client-reachable — which is one more reason the runbook below shifts DNS before detaching. - **A member catching up on a config change keeps serving.** `syncing` after a previous sync probes green and serves last-known-good configuration — a routine config rollout never triggers failover. - **Failover is invisible to clients beyond the URL resolution.** API keys, model access, MCP access, and guardrail behaviour are identical on every converged member, by construction; a member catching up on a rollout serves the project's last-known-good revision until it converges (see the convergence note under [Identical by construction](#identical-by-construction)). See the rate-limit limitation below for the one per-member exception. - **No steering from Agent Router.** If the health checks, records, or TTLs are wrong, Agent Router will not compensate. Test the failover (take the primary's data plane offline in a rehearsal window and watch the secondary take traffic) before relying on it. Richer alarm-driven variants — for example a CloudWatch alarm on the health-check status feeding notifications, or a Lambda that shifts weights on custom conditions — layer on top of the same probe without any change on the Agent Router side. ## Runbook: detach a member gateway Detaching is how a member is decommissioned: the member is actively emptied — routes, credentials, and catalogs removed — so a decommissioned gateway cannot keep serving project traffic once the emptied configuration lands. The `detached — unreachable` terminal below is the exception that keeps verification on the operator. Detach is not delete: the gateway record survives and the project is untouched. Run the steps in this order; each verifies the one before it. ### Step 1: Shift DNS away from the member (act) 1. In the DNS provider, remove the member's record, zero its weight, or demote it from the failover set, so `ai.payments.examplebank.com` no longer resolves to it. 1. Wait out the record TTL so cached resolutions expire. Do this *before* detaching. A detach that completes to `zeroized` would drain DNS by itself via the probe, but during the health-check detection window clients would receive no-route 404s from the zeroized member — and a detach that lands `detached — unreachable` would not drain DNS at all (its probe can stay green). Shifting DNS first makes the detach invisible and is the only drain that works in both terminals. ### Step 2: Verify the drain through attribution (verify) Request attribution is per **data plane**, not per gateway (see [Per-data-plane request attribution](#per-data-plane-request-attribution)). This verification is authoritative only when the member being detached has a **dedicated data plane** — one hosting no other serving gateway from any project. On a shared data plane the counts include every co-located gateway; fall back to the organisation's own edge metrics for the member's hostname. 1. In the **Gateways** tab, start the detach on the member's row and read the confirmation dialog's traffic warning before confirming. It shows the request count for the member's data plane over the trailing hour — "*gateway*'s data plane served *N* requests in the last hour" — and notes that counts are per data plane: gateways sharing a data plane share a count. 1. Mind the window: the count covers a fixed trailing 60 minutes, so for up to an hour after the DNS shift it still includes pre-shift traffic. A non-zero count immediately after Step 1 does **not** mean the drain failed. Either wait out the window (TTL expiry plus the remainder of the hour) before treating the count as a verdict, or query the request-log stats surface with an explicit `start_time` after the TTL expired: ```bash $ curl -sS "https://management.example.com/v1/request-logs/stats?group_by=dataplane&metric=count&start_time=2026-07-22T14:30:00Z" \ -H "Authorization: Bearer ${ADMIN_API_KEY}" ``` Each entry is keyed by the reporting data plane's identity; add `dataplane_id=` to filter to the member's data plane. Requests recorded before data-plane attribution existed group under the empty key `""` — treat that bucket as unattributable, never as the member's. 1. The dialog count is best-effort. If it reads that recent traffic "couldn't be determined", the stats surface was unavailable — treat the drain as **unverified** and verify via the stats query above (or edge metrics) before detaching. 1. A count that stays non-zero after the window has passed, on a dedicated data plane, means Step 1 is incomplete — stop and find who is still calling the member URL directly (stragglers with cached DNS or hardcoded member URLs show up here and nowhere else). ### Step 3: Detach the member (act) 1. In the **Gateways** tab, choose **Detach** on the member's row and confirm. 1. The member leaves the ready set; a final, emptied configuration revision is made available to the gateway, and the management plane revokes the member's identity so it can never resync, rejoin, or fetch future configuration. Step 4 spells out exactly what that revocation does and does not stop. Detach is idempotent: confirming a detach on a member that is already detached reports that it was already detached and changes nothing. ### Step 4: Interpret the terminal state (verify) The member lands in one of two terminal states, shown on its recently-detached row: - **`detached — zeroized`** — the gateway acknowledged the emptied configuration. Teardown is **confirmed**: routes, credentials, and catalogs are removed from the runtime; its probe is red; project traffic to it gets no-route 404s. Nothing further is required before decommissioning the runtime. - **`detached — unreachable`** — the gateway never acknowledged. The management plane has revoked the gateway's identity, so it can never resync, rejoin, or fetch future configuration — but its local state is **unconfirmed**, and revocation cannot reach a gateway that does not poll: client API keys are validated in-process on the gateway against its last-synced material, and the provider credentials embedded in that material remain usable until rotated at the provider. Assume a partitioned-but-reachable gateway is still serving its last-known configuration to anyone who can reach its URL directly — **and still answering its membership probe green**, so DNS health checks will not drain it automatically. Close the gap manually: confirm the member's DNS record is removed (Step 1 already did this if the runbook ran in order), verify through the hosting infrastructure that the runtime is actually down (or its last-known config torn down), and rotate the project's provider credentials if the host cannot be verified. Only then treat the member as decommissioned. ## Documented limitation: rate limits are per-gateway The identical-boundary guarantee covers authentication, authorization, guardrails, and catalogs. It does **not** cover rate state: token allowances are enforced per gateway, and counters are not shared between members. Concretely: a client that fails over from `gw-use1-payments` to `gw-usw2-payments` arrives with **fresh token-per-hour allowances** on the new member. A workload near its allowance on one member effectively doubles its headroom by moving; during an active/active weighted setup, a project's aggregate consumption can approach the per-gateway allowance multiplied by the member count. Budgets are different: they are evaluated at the management plane against fleet-aggregated request-log spend — accounting accrues as logs reach the management plane, not in real time on the request path — so membership neither fragments nor resets budget accounting. What to do about it today: - Size per-gateway token allowances with the member count in mind if aggregate consumption is the concern. - Use per-data-plane attribution to watch aggregate consumption across members after a failover event (one data plane per member makes the buckets per-member). - Tell application teams that a failover resets their rate-limit window — relevant to any client-side backoff logic keyed to limit responses. Treat token allowances as a per-member property when sizing and when reasoning about failover. ## Behaviour during a management-plane outage Member gateways are autonomous with respect to the management plane: - **Members keep serving.** Each member continues serving its last-known configuration while the management plane is unreachable. Data-path traffic does not depend on the management plane. - **Probes stay green.** The membership probe is answered by the gateway's own data plane with no management-plane round-trip, so DNS health checks — and therefore the failover pattern above — keep working through a full management-plane outage. - **Detach effects are deferred — and do not drain via the probe.** A detach issued while a member cannot reach the management plane takes effect on reconnect: the zeroized configuration lands and identity revocation bites when the gateway next contacts the management plane. Until then the member keeps serving its last-known configuration and its membership probe **stays green** — DNS health checks will not steer away from it. Shift DNS away from the member first, exactly as the runbook orders, and treat the member as `detached — unreachable` with the manual-verification posture until reconnect. ## What to do next - **Provision a gateway for a project**: attach additional members, and the attach/detach mechanics, cap, and zero-gateway state. See [Provision a gateway for a project](/agent-router-enterprise/guides/operate-and-govern/project-and-data-plane-management/provision-a-gateway/). - **Migrate a gateway or upgrade its runtime**: move a member between data planes without changing its URL. See [Migrate a gateway or upgrade its runtime](/agent-router-enterprise/guides/operate-and-govern/project-and-data-plane-management/migrate-a-gateway/). - **Plan high availability and disaster recovery**: place gateway sets inside the broader HA/DR topology. See [Plan high availability and disaster recovery](/agent-router-enterprise/guides/operate-and-govern/plan-high-availability-and-disaster-recovery/). - **Load balance across regional deployments**: balancing *model backends* across regions, which composes with — and is distinct from — failover across *gateways*. See [Load balance across regional deployments](/agent-router-enterprise/guides/operate-and-govern/load-balance-across-regional-deployments/). Where to go next Provision a gateway for a project Attach members to a project, with the gateway cap and the zero-gateway state. Plan high availability and disaster recovery Place gateway sets inside the broader resilience topology. --- ## Project and data plane management Enterprise Tier # Project and data plane management These guides cover creating isolated projects, provisioning the gateways that serve them, deploying the data planes that host those gateways, and moving runtimes between data planes without disruption. Models must be enabled in the organisation catalog before you can grant them to a project; that catalog work is covered in [Provision models and providers](/agent-router-enterprise/guides/operate-and-govern/provision-models-and-providers/). Three objects underpin every guide in this section. A short definition of each follows; the full treatment is in [Key concepts](/product-architecture/key-concepts/#projects) and [Planes and core components](/product-architecture/planes-and-components/). - A **project** is the logical boundary that owns models, API keys, Model Context Protocol (MCP) servers and profiles, routing and policy, budgets, and members. It is also the isolation boundary: each project sees and serves only its own configuration. - A **gateway** is a single inference endpoint (one customer-facing URL) provisioned by a data plane and mapped to exactly one project. By default a project gets one gateway when you select a data plane; on Enterprise you can optionally attach additional gateways for high availability (see [Gateway sets and DNS-level failover](/agent-router-enterprise/guides/operate-and-govern/project-and-data-plane-management/gateway-sets-and-dns-failover/)). A project has no endpoint until at least one gateway is provisioned for it. - A **data plane** is the layer, deployed in a Kubernetes cluster, that provisions and manages gateways. One data plane can host many gateways, and one management plane can manage many data planes. Set up a project Create and delete a project Create the project with the wizard, granting members, models, and MCP servers in one pass—or delete a project when it is no longer needed. Provision a gateway for a project Give the project a live inference URL by selecting a data plane, which provisions the project's gateway. Manage project members and access Add members, assign roles, and keep project access separate from data-plane operations. Issue a project-scoped API key Create a key that works only against one project's gateway. Grant MCP servers and profiles to a project Scope MCP servers and profiles to a project under the same isolation as models. Operate data planes Deploy and register a data plane Provision a data plane, install its credential, and connect it to the management plane. Manage multiple gateways on a data plane Review the gateways a single data plane provisions and manages after projects select it. Migrate a gateway or upgrade its runtime Move a gateway between data planes, or upgrade it in place, without changing its URL. Gateway sets and DNS-level failover Run a project on several like-configured gateways and fail over between them with customer-managed DNS. --- ## Issue a project-scoped API key Enterprise Tier An API key is the credential an application presents when it calls a gateway. Keys are created in the context of a project, so a key belongs to one project and works only against that project's gateway. This guide creates a project-scoped key and explains the isolation it carries. Because keys are project-scoped, they are part of what keeps projects isolated from one another: a key issued for one project is refused on any other project's gateway, and it can reach only the models that project has been granted. Creating a key is done from the project context, so the project must be selected first. **Persona:** Platform operator or project administrator working in the Admin Dashboard. **Estimated time:** 5 minutes per key. ## Outcomes By the end of this guide: - An API key exists that is scoped to a single project. - The key's scope is understood: it authenticates only against that project's gateway. - The one-time nature of the key value is understood, so it is captured securely at creation. ## Prerequisites - Membership of the project, with permission to create keys. - The project already has a gateway provisioned, so the key has an endpoint to authenticate against. See [Provision a gateway for a project](/agent-router-enterprise/guides/operate-and-govern/project-and-data-plane-management/provision-a-gateway/). ## Step 1: Select the project 1. Use the project switcher at the top of the application to select the project the key is for, for example `bedrock-team`. 1. Confirm the application has re-scoped to that project before creating the key. Everything done in the application, including key creation, applies to the currently selected project. Selecting the wrong project produces a key that will not work against the intended gateway. ## Step 2: Create the API key 1. Open **API Keys** and select **Add API Key**. 1. Name the key so its purpose is recognisable later, then create it. 1. Copy the key value immediately. It is shown once, so it must be captured at this point and stored securely. ## Step 3: Understand the key's scope The key is bound to the project it was created in. - It authenticates only against that project's gateway hostname. Presented to another project's gateway, it is refused with `403`. - It can call only the models granted to the project. A request for a model the project does not have returns `404`. This scoping is automatic and needs no extra configuration. The same isolation applies to MCP: a profile reached with a key from another project, or on another project's gateway, is refused. Organisation-wide key practices, including rotation and revocation, are covered in [Onboard developers and issue keys](/agent-router-enterprise/guides/operate-and-govern/onboard-developers-and-issue-keys/). ## What to do next - **Provision a gateway for a project**: the endpoint this key authenticates against, including how to repoint existing clients. See [Provision a gateway for a project](/agent-router-enterprise/guides/operate-and-govern/project-and-data-plane-management/provision-a-gateway/). - **Grant MCP servers and profiles to a project**: extend the project with tools reached using this same key. See [Grant MCP servers and profiles to a project](/agent-router-enterprise/guides/operate-and-govern/project-and-data-plane-management/scope-mcp-to-a-project/). Where to go next Grant MCP servers and profiles to a project Expose tools through the project, reached with this project key. Onboard developers and issue keys The organisation-wide view of users, roles, and key lifecycle. --- ## Manage multiple gateways on a data plane Enterprise Tier A data plane is shared provisioning capacity: one data plane can provision and manage many gateways at once, each serving a single project. This guide covers operating that shared capacity: reviewing the gateways a data plane manages, placing more projects on it so their gateways are auto-provisioned, and understanding how the data plane reconciles each one independently. Each gateway on a data plane has its own customer-facing URL and serves exactly one project, so a request to a gateway URL always resolves to a single project's configuration. The data plane provisions the runtime for each gateway and keeps it aligned with the desired configuration the management plane publishes. Because gateways are reconciled independently, they can be added, changed, and recovered without affecting one another. **Persona:** Platform operator with data-plane operations access, working in the Admin Dashboard. **Estimated time:** 10 to 15 minutes. ## Outcomes By the end of this guide: - The gateways a data plane manages can be reviewed, with their URL, owning project, configuration version, and health. - Additional gateways appear on the data plane when more projects select it. - The independence of gateway reconciliation is understood: one gateway's problem does not block the others. ## Prerequisites - Administrator access to the Admin Dashboard, with data-plane operations permissions. - A data plane deployed and registered. See [Deploy and register a data plane](/agent-router-enterprise/guides/operate-and-govern/project-and-data-plane-management/deploy-a-data-plane/). ## Step 1: Review the gateways a data plane manages 1. In the Admin Dashboard, open **Data planes** and select the data plane. 1. Review the gateways it manages. For each gateway, the view shows the customer-facing URL, the owning project, the configuration version applied, and the gateway's health. The management plane tracks whether the desired configuration version is applied per gateway, so a gateway that has not yet caught up with a recent change is visible as such. ## Step 2: Assign more gateways to the data plane Gateways appear on a data plane when their projects select that data plane. There is no top-level **Gateways** menu for creating gateways separately, and gateways are not created on the data plane directly. 1. On a project, select this data plane so its gateway is auto-provisioned here, following [Provision a gateway for a project](/agent-router-enterprise/guides/operate-and-govern/project-and-data-plane-management/provision-a-gateway/). 1. Confirm the new gateway appears among those the data plane manages. A single data plane can host many projects' gateways this way, which is how several teams share one deployment while each keeps its own isolated endpoint. ## Step 3: Understand independent reconciliation The data plane reconciles each gateway on its own. 1. Note that a failure affecting one gateway's assignment does not block the other gateways the data plane manages. 1. Use the per-gateway health and configuration-version indicators to isolate a problem to a single gateway rather than the whole data plane. Spreading traffic across several data planes, rather than several gateways on one, is covered in [Load balance across regional deployments](/agent-router-enterprise/guides/operate-and-govern/load-balance-across-regional-deployments/). That pattern is about capacity and backends across regions. Separately, when a **single project** optionally attaches several like-configured gateways for high availability behind customer-managed DNS, see [Gateway sets and DNS-level failover](/agent-router-enterprise/guides/operate-and-govern/project-and-data-plane-management/gateway-sets-and-dns-failover/) — that is not the same as many projects sharing one data plane. ## What to do next - **Understand guardrail enforcement on shared capacity**: when several projects' gateways share one data plane, each project's guardrails apply only to its own traffic, and gateway attachment health determines what is enforced. See [Guardrails on shared data planes](/agent-router-enterprise/guides/operate-and-govern/safety-and-data-protection/guardrails-on-shared-data-planes/). - **Migrate a gateway or upgrade its runtime**: move a gateway to another data plane, or upgrade it in place, without changing its URL. See [Migrate a gateway or upgrade its runtime](/agent-router-enterprise/guides/operate-and-govern/project-and-data-plane-management/migrate-a-gateway/). - **Plan high availability and disaster recovery**: topology and recovery planning across data planes. See [Plan high availability and disaster recovery](/agent-router-enterprise/guides/operate-and-govern/plan-high-availability-and-disaster-recovery/). - **Optional: gateway sets and DNS-level failover**: attach additional members to one project for DNS failover. See [Gateway sets and DNS-level failover](/agent-router-enterprise/guides/operate-and-govern/project-and-data-plane-management/gateway-sets-and-dns-failover/). Where to go next Migrate a gateway or upgrade its runtime Move a gateway between data planes, or upgrade in place, with a stable URL. Gateway sets and DNS-level failover Optional Enterprise HA for several members of the same project. --- ## Manage project members and access Enterprise Tier Membership decides who can work inside a project and what they can do there. Because a project is the isolation boundary, its member list also controls who can see its models, keys, and Model Context Protocol (MCP) configuration. This guide covers adding members, the roles available, and the deliberate separation between project access and data-plane operations access. Access in Agent Router is layered. Project access governs the logical configuration of a single project; data-plane access governs the physical infrastructure that runs gateways. The two are kept separate so that application teams can manage their own project without being able to touch runtime capacity, and infrastructure operators can manage capacity without being able to change a project's models or policy. **Persona:** Platform operator or project administrator working in the Admin Dashboard. **Estimated time:** 5 to 10 minutes per project. ## Outcomes By the end of this guide: - The users who should work in a project are added as members with an appropriate role. - The difference between project access and data-plane operations access is understood. - Members can find and switch into the project using the project switcher. ## Prerequisites - Administrator access to the project, or organisation-level administrator access. - The project already created. See [Create and delete a project](/agent-router-enterprise/guides/operate-and-govern/project-and-data-plane-management/create-a-project/). - The user accounts to be added already present in the organisation. Onboarding users is covered in [Onboard developers and issue keys](/agent-router-enterprise/guides/operate-and-govern/onboard-developers-and-issue-keys/). ## Step 1: Add members to the project 1. Open the project's **Members** view. 1. Add each user who should use the project. 1. Assign a role: **member** for those who consume the project, or **admin** for the project owners who manage its configuration and membership. Membership is per project. A user added to one project gains access to that project only, and a user may belong to several projects without any relationship between them. ## Step 2: Confirm access through the project switcher Members find and move between their projects using the project switcher at the top of the application. 1. Confirm the added user can open the project switcher and see the project in the list. 1. Note that the whole application re-scopes to the selected project: its API keys, models, and MCP views all reflect the project currently in context. A user who does not see a project in the switcher is not a member of it, and access is granted by adding them in Step 1. ## Step 3: Keep project and data-plane access separate Project roles do not grant infrastructure permissions. - Project admins manage the project: members, models, MCP, policy, budgets, and agent configuration. - Data-plane operators manage the physical runtime: gateway provisioning, capacity, health, and rollout, covered in [Deploy and register a data plane](/agent-router-enterprise/guides/operate-and-govern/project-and-data-plane-management/deploy-a-data-plane/) and [Manage multiple gateways on a data plane](/agent-router-enterprise/guides/operate-and-govern/project-and-data-plane-management/manage-gateways-on-a-data-plane/). Granting a user project access does not give them data-plane operations access, and the reverse is also true. Audit records distinguish logical project changes from physical data-plane and gateway operations, so the two kinds of activity remain separately attributable. Audit review is covered in [Audit platform activity](/agent-router-enterprise/guides/operate-and-govern/audit-platform-activity/). ## What to do next - **Issue a project-scoped API key**: give members a credential for the project's gateway. See [Issue a project-scoped API key](/agent-router-enterprise/guides/operate-and-govern/project-and-data-plane-management/issue-a-project-api-key/). - **Onboard developers and issue keys**: the organisation-wide view of users and roles. See [Onboard developers and issue keys](/agent-router-enterprise/guides/operate-and-govern/onboard-developers-and-issue-keys/). Where to go next Issue a project-scoped API key Create a key that members use against the project's gateway. Audit platform activity Where project and data-plane changes are recorded and attributed. --- ## Migrate a gateway or upgrade its runtime Enterprise Tier A gateway's runtime can be upgraded in place or moved to a different data plane, for example to relocate a project to another region or cloud, or to retire an old cluster. Because the gateway's URL and its project mapping stay fixed through the change, the migration is transparent to the applications calling it. This guide covers the migration sequence and what keeps it non-disruptive. The gateway is a durable record: its customer-facing URL and its mapping to its project do not change when the underlying runtime is upgraded or moved. (A project may have other gateway members when optional HA is configured; this guide moves one gateway's runtime, not the project's membership set.) What changes is the data plane that provisions the runtime behind it. The sequence is designed so that a working runtime is always serving traffic, and cutover happens only once the replacement is ready. **Persona:** Platform operator with data-plane operations access, working in the Admin Dashboard and coordinating the routing or DNS change. **Estimated time:** 30 to 60 minutes, plus DNS propagation and drain time. ## Outcomes By the end of this guide: - A gateway's runtime is upgraded in place, or its serving data plane is changed, without changing the gateway's URL. - The project-to-gateway mapping is unchanged throughout. - In-flight requests are drained rather than dropped during cutover. ## Prerequisites - Administrator access to the Admin Dashboard, with data-plane operations permissions. - For a migration between data planes, a target data plane already deployed and registered. See [Deploy and register a data plane](/agent-router-enterprise/guides/operate-and-govern/project-and-data-plane-management/deploy-a-data-plane/). - Control of the routing or DNS layer that resolves the gateway's hostname, so cutover can be performed. ## Step 1: Provision the target runtime 1. Start the upgrade in place, or begin a migration by having the target data plane provision a runtime for the gateway. 1. During a migration, allow both the source and target data planes to run a runtime for the gateway concurrently. Running both runtimes at once means the existing endpoint keeps serving throughout preparation, so there is no window in which the gateway is unavailable. ## Step 2: Cut over once the target reports ready 1. Wait until the target runtime reports ready. 1. Cut over at the routing or DNS layer, pointing the gateway's hostname at the target runtime. Cutover is a routing change, not a change to the gateway record. The customer-facing URL is the same before and after, so applications need no reconfiguration and no new keys. ## Step 3: Drain and retire the source runtime 1. Allow the source runtime to drain its in-flight requests rather than terminating it immediately. 1. Retire the source runtime once it has drained. Draining is what makes the cutover non-disruptive: requests already in progress on the old runtime complete, while new requests land on the target. ## What stays fixed Throughout an upgrade or migration: - The customer-facing URL does not change. - The project-to-gateway mapping does not change. - Existing API keys continue to work, because they are scoped to the project and the gateway, not to the runtime behind them. This stability is the point of separating the durable gateway from the runtime that serves it. Distributing projects across data planes by region or cloud, which migration makes possible, is covered in [Load balance across regional deployments](/agent-router-enterprise/guides/operate-and-govern/load-balance-across-regional-deployments/). ## What to do next - **Manage multiple gateways on a data plane**: operate the gateways on the source and target data planes. See [Manage multiple gateways on a data plane](/agent-router-enterprise/guides/operate-and-govern/project-and-data-plane-management/manage-gateways-on-a-data-plane/). - **Plan high availability and disaster recovery**: use migration as part of a broader resilience plan. See [Plan high availability and disaster recovery](/agent-router-enterprise/guides/operate-and-govern/plan-high-availability-and-disaster-recovery/). Where to go next Load balance across regional deployments Distribute projects and traffic across data planes by region or cloud. Manage multiple gateways on a data plane Operate the gateways on the source and target data planes. --- ## Provision a gateway for a project Enterprise Tier A gateway is a single inference endpoint, one customer-facing URL, provisioned by a data plane and mapped to exactly one project. Selecting a data plane on the project is the step that turns configuration into something applications can call: the platform provisions the project's first gateway automatically. There is no separate top-level Gateways menu and no create-then-select flow for that default path. This guide places the project on a data plane, sets its URL, and confirms the gateway is ready. Attaching additional gateways for high availability is optional and covered later in this page. A project owns the logical configuration (models, keys, policy); a gateway is the physical capacity that serves it. The two are separate on purpose: the project's setup is decoupled from the runtime that runs it, so the endpoint can later be upgraded or moved between data planes without the project's configuration or its URL changing. Choosing which data plane hosts the project is the event that drives that data plane to provision and configure the gateway runtime. **Persona:** Platform operator working in the Admin Dashboard, with a data plane already available to provision on. **Estimated time:** 10 to 20 minutes, plus reconciliation and DNS propagation. ## Outcomes By the end of this guide: - The project is placed on a chosen data plane, and a gateway is auto-provisioned for it with the customer-facing URL applications will call. - By default the project has one gateway; each gateway still serves exactly one project. On Enterprise, additional gateways can optionally be attached later for high availability (see [Optional: attach additional gateways for HA](#optional-attach-additional-gateways-for-ha)). - The gateway has reconciled and reports ready, and its URL is shown on the project. ## Prerequisites - A project that already has its models granted. See [Create a project](/agent-router-enterprise/guides/operate-and-govern/project-and-data-plane-management/create-a-project/). - At least one data plane deployed and connected to the management plane. See [Deploy and register a data plane](/agent-router-enterprise/guides/operate-and-govern/project-and-data-plane-management/deploy-a-data-plane/). - Control of the DNS name that will resolve to the data plane ingress, if the endpoint is to be reachable by hostname outside the cluster. ## Step 1: Select a data plane on the project 1. In the Admin Dashboard, open **Projects** and select the project. 1. On the project, select the data plane that should host this project's gateway. 1. Set the customer-facing URL that developers will call, for example `https://bedrock-team.acme.example.com`. Routing is by hostname, so each project's gateway carries its own hostname. Selecting the data plane provisions the **first** gateway automatically. There is no separate create step in a top-level **Gateways** menu for that default path. Additional gateways are attached later only when you configure high availability (see [Optional: attach additional gateways for HA](#optional-attach-additional-gateways-for-ha)). The data plane is shared provisioning capacity: it can provision and manage many gateways at once. Choosing which data plane hosts the gateway is how a project is placed in a particular region or cloud. ## Step 2: Confirm the gateway is ready 1. Watch the gateway's status as the data plane reconciles the desired configuration. 1. Confirm the gateway reports ready and its URL is shown on the project. Once ready, requests sent to the gateway's hostname, authenticated with a key issued for the project, resolve to that project's configuration and nothing else. Cross-project requests are refused: a key from another project returns `403`, and a model the project has not been granted returns `404`. ## Repoint existing clients If applications previously called a single shared endpoint on the data plane and are moving onto project gateways, the change is small and does not require new keys. 1. Keep existing keys: the same key continues to work, so nothing needs to be re-issued. 1. Repoint each application's base URL (or Host) from the shared endpoint to the project's gateway hostname. 1. Confirm the project is wired up: its gateway has a URL, DNS resolves that hostname to the data plane ingress, and the models the application uses are granted to the project. Calling a project's models over the OpenAI-compatible API is covered on the developer side in [Make an API call](/agent-router-service/quickstarts/make-an-api-call/). ## Optional: attach additional gateways for HA Most projects run with the single gateway provisioned in Steps 1 and 2. Attaching more members is an **optional** Enterprise high-availability step: it is not required for a working project. :::info Availability Attaching more than one gateway to a project requires an Agent Router Enterprise release that includes multi-gateway projects, and the deployment must declare the Enterprise product line (`DEPLOYMENT_PRODUCT=enterprise`, chart-derived; the gate fails closed). This surface does not exist on Agent Router Service (SaaS). ::: When HA is needed, a project can attach several gateways at once. Every attached gateway receives the identical project configuration by construction — authentication, guardrails, and catalogs cannot diverge between members — which is what makes the members safe failover targets for one another. The full membership model, the health-probe contract, and the DNS failover pattern are covered in [Gateway sets and DNS-level failover](/agent-router-enterprise/guides/operate-and-govern/project-and-data-plane-management/gateway-sets-and-dns-failover/). ### Attach 1. In the Admin Dashboard, open the project and its **Gateways** tab. 1. Choose **Attach gateway** and select an existing unattached gateway, or provision a new one on a target data plane (provisioning remains the only way a gateway is created). 1. The new member appears with status `syncing` while it converges on the project's configuration. It is promoted to `ready` — and starts answering its membership probe — only once convergence is confirmed, so it cannot receive health-checked traffic half-configured. The **Gateways** tab lists each member with its URL, status, health, configuration version, and last-seen time, so fleet consistency is verifiable at a glance. ### Detach 1. Shift any client traffic away from the member first (via the DNS layer that fronts the members), and verify the drain through request attribution for the member's **data plane**. Attribution is per data plane, not per gateway: the count is only the member's own when no other serving gateway shares its data plane. The full verification procedure, including this caveat, is in the failover guide's runbook. 1. Choose **Detach** on the member's row. The confirmation shows the request count for the member's data plane over the trailing hour — a fixed window, so traffic from before the DNS shift still counts toward it for up to an hour. 1. The member is actively emptied — routes, credentials, and catalogs removed — and lands in a terminal state: `detached — zeroized` (teardown confirmed) or `detached — unreachable` (credentials revoked; verify decommissioning manually). Detach is not delete: the gateway record survives, and the project is untouched. The ordered runbook, with the terminal-state interpretation, is in [Gateway sets and DNS-level failover](/agent-router-enterprise/guides/operate-and-govern/project-and-data-plane-management/gateway-sets-and-dns-failover/#runbook-detach-a-member-gateway). ### The gateway cap The number of gateways a project can attach is capped — **10 by default, configurable per deployment**. Attaching beyond the cap fails with a clear at-cap error rather than silently queueing; raising the cap is a deployment-configuration change, not a project setting. ### Zero gateways is a valid state A project with no attached gateways remains a fully valid, fully configured project: keys, models, policy, and members are all retained, and the Gateways tab shows a "nothing serving" state. Use it to stage a project's configuration before capacity exists, or to drain a project to zero without tearing anything down. Attaching a gateway later (re)materialises the existing configuration. ## What to do next - **Issue a project-scoped API key**: create the credential developers use against this gateway. See [Issue a project-scoped API key](/agent-router-enterprise/guides/operate-and-govern/project-and-data-plane-management/issue-a-project-api-key/). - **Grant MCP servers and profiles to a project**: expose tools through the same gateway. See [Grant MCP servers and profiles to a project](/agent-router-enterprise/guides/operate-and-govern/project-and-data-plane-management/scope-mcp-to-a-project/). - **Gateway sets and DNS-level failover**: optional Enterprise HA — attach additional gateways and fail over with customer-managed DNS. See [Gateway sets and DNS-level failover](/agent-router-enterprise/guides/operate-and-govern/project-and-data-plane-management/gateway-sets-and-dns-failover/). Where to go next Issue a project-scoped API key Create a key that works only against this project's gateway. Manage multiple gateways on a data plane Review the gateways a single data plane manages after projects select it. --- ## Grant MCP servers and profiles to a project Enterprise Tier A project controls which Model Context Protocol (MCP) servers it can use and exposes them to AI clients through MCP profiles. MCP is a standard for connecting AI clients to external tools and data sources. Scoped to a project, MCP falls under the same isolation as models: a project sees only its own servers, and its profiles are reachable only through its own gateway with its own keys. An MCP server is an adapter that exposes a tool or data source to AI clients; an MCP profile aggregates one or more servers behind a single endpoint that clients connect to once. Granting MCP to a project makes the project the boundary for tools in exactly the way it is the boundary for models. This guide grants servers to a project and builds a profile on the project's gateway. **Persona:** Platform operator or project administrator working in the Admin Dashboard. **Estimated time:** 15 to 25 minutes for the first profile. ## Outcomes By the end of this guide: - The MCP servers a project may use are granted to it. - A profile that bundles the chosen servers is published on the project's gateway. - The profile's isolation is understood: it is reachable only with the project's key, through the project's gateway. ## Prerequisites - Membership of the project, with permission to manage its MCP configuration. - A gateway provisioned for the project, so the profile has an endpoint. See [Provision a gateway for a project](/agent-router-enterprise/guides/operate-and-govern/project-and-data-plane-management/provision-a-gateway/). - The MCP servers available in the organisation's catalog. Cataloguing and governing servers is covered in [Govern MCP server access](/agent-router-enterprise/guides/operate-and-govern/govern-mcp-server-access/). ## Step 1: Grant MCP servers to the project 1. Use the project switcher to select the project. 1. Open **MCP Servers** and enable the servers this project may use. A project can see and use only the servers granted to it. A server that is not granted is neither listed for the project nor callable through it, which is what keeps one project's tools separate from another's. ## Step 2: Build an MCP profile A profile bundles one or more of the project's granted servers into a single endpoint an MCP client connects to. 1. In the project, open **My MCP Profiles** and select **Add MCP Profile**, then name it. 1. Select **Add Server**, choose from the servers granted to the project, and enable the tools wanted from each. 1. Set the profile's authentication, either API key or OAuth. The profile is published on the project's gateway, at a path under the project's own hostname. Securing MCP credentials and identity is covered in [Secure MCP secrets and identity](/agent-router-enterprise/guides/operate-and-govern/secure-mcp-secrets-and-identity/). ## Step 3: Confirm profile isolation The profile inherits the project's isolation. - It is reachable only through the project's gateway hostname. Called on another project's gateway, the path is not served and returns `404`. - It accepts only the project's keys. A key from another project is refused with `403`. The developer-side workflow of connecting a client such as Claude Code, Cursor, or an MCP-capable editor to the profile URL is covered in [Aggregate MCP servers into a profile](/agent-router-service/guides/aggregate-mcp-servers-into-a-profile/). ## What to do next - **Govern MCP server access**: the organisation-wide catalog and access controls behind per-project grants. See [Govern MCP server access](/agent-router-enterprise/guides/operate-and-govern/govern-mcp-server-access/). - **Secure MCP secrets and identity**: how profile credentials and identity are protected. See [Secure MCP secrets and identity](/agent-router-enterprise/guides/operate-and-govern/secure-mcp-secrets-and-identity/). Where to go next Govern MCP server access The organisation-wide catalog and controls behind per-project grants. Aggregate MCP servers into a profile The developer-side view of connecting a client to a profile. --- ## AWS Bedrock access with workload identity This guide configures the Tetrate Agent Router data plane on AWS Elastic Kubernetes Service (EKS) to call AWS Bedrock with the pod's IAM identity, using either IAM Roles for Service Accounts (IRSA) or EKS Pod Identity, instead of static access keys. ## Overview Serving Bedrock without static AWS keys takes two actions on an installed EKS data plane: 1. **Bind the identity.** `tare dataplane identity bind` attaches an AWS IAM role to the data plane's egress workload through the cluster (IRSA or EKS Pod Identity). 1. **Register a credential-less provider.** Create the AWS Bedrock provider in the Admin Dashboard with no access keys; the data plane signs Bedrock requests with the pod's role. Binding places the IAM role in the AWS SDK default credential chain of the data plane's egress pod. A Bedrock provider registered without keys is then reconciled to a region-only authentication policy on the data plane, and every Bedrock request is signed (SigV4) with the identity the cluster provides. No AWS keys are stored in the cluster or in the management plane. Plan for 10 to 15 minutes, assuming the data plane is already installed. ## Choose a mechanism Both mechanisms end in the same place: the egress pod holds an IAM role that is allowed to invoke Bedrock. They differ in how the role is trusted and bound: | | IRSA | EKS Pod Identity | | :- | :- | :- | | Role trust | Cluster OIDC provider, scoped to the ServiceAccount | `pods.eks.amazonaws.com` | | Binding | ServiceAccount annotation (`eks.amazonaws.com/role-arn`) | Pod-identity association (no annotation) | | Cluster prerequisite | IAM OIDC provider registered | `eks-pod-identity-agent` addon installed | | One role, many clusters | No | Yes | | Injected marker | `AWS_ROLE_ARN` / `AWS_WEB_IDENTITY_TOKEN_FILE` | `AWS_CONTAINER_CREDENTIALS_FULL_URI` | | `--cluster` / `--region` flags | Only with `--create-iam` | Always | IRSA is the default (`--type irsa`). Prefer EKS Pod Identity when one role should serve several clusters, or when the cluster already runs the pod-identity agent. ## Prerequisites - **Data plane installed and connected** to its management plane. See the [AWS installation guide](/agent-router-enterprise/self-hosted-data-plane/data-plane-installation/installation-guide-aws/), which also covers [installing the tare CLI](/agent-router-enterprise/self-hosted-data-plane/data-plane-installation/installation-guide-aws/#step-2-install-the-tare-cli). - **kubectl context** pointing at the EKS cluster. - **A live AWS session for the target account** when using `--create-iam`, established with `aws sso login`, `aws configure`, or `AWS_*` environment variables. Confirm with: ```shell aws sts get-caller-identity ``` The session needs permission to create IAM roles and policies (and, for Pod Identity, to manage EKS pod-identity associations). - **Per-mechanism cluster prerequisite.** `bind` detects a missing prerequisite and prints the exact fix command, but it can also be prepared up front: - IRSA: the cluster's IAM OIDC provider is registered: ```shell eksctl utils associate-iam-oidc-provider --cluster --approve ``` - EKS Pod Identity: the `eks-pod-identity-agent` addon is installed: ```shell aws eks create-addon --cluster-name --addon-name eks-pod-identity-agent --region ``` ## Step 1: Bind the data plane identity `tare dataplane identity bind` creates the IAM role, its trust relationship, and a `bedrock:InvokeModel` policy (with `--create-iam`), binds the data plane's signing ServiceAccount, waits for the rollout, and confirms credentials were injected into the egress pod. Before applying anything, a read-only preflight confirms the ServiceAccount exists and the current kubectl context is allowed to patch the resources the bind touches. ### IRSA ```shell # Create the role + trust + Bedrock policy, then bind: tare dataplane identity bind --type irsa --create-iam --cluster --region # Or bind an IAM role you already created: tare dataplane identity bind --type irsa --role-arn arn:aws:iam:::role/ ``` ### EKS Pod Identity The pod-identity association is cluster-scoped, so `--cluster` and `--region` are always required: ```shell tare dataplane identity bind --type eks-pod-identity --create-iam --cluster --region ``` ### Preview and verify Both commands are read-only and apply nothing to the cluster: ```shell # Preview the planned actions: tare dataplane identity bind --type --create-iam --cluster --region --dry-run # Verify an existing binding: tare dataplane identity bind --check --type ``` ### Defaults | Flag | Default | | :- | :- | | `--type` | `irsa` | | `--service-account` | `egress` | | `--namespace` | `tars-dataplane` | | `--role-name` (with `--create-iam`) | `-bedrock-dp` | | `--wait` | on | Run `tare dataplane identity bind --help` for the full flag reference. In non-interactive contexts (CI), add `--yes` to skip the confirmation prompt. ## Step 2: Register a credential-less Bedrock provider In the Admin Dashboard, create an AWS Bedrock provider with **no access keys**: on the provider create form, turn on **Use IRSA / EKS Pod Identity (no static keys)**. The AWS key fields disappear and the provider is created without a secret; the identity bound in Step 1 signs its requests. Follow the [Provision AWS Bedrock models](/agent-router-enterprise/guides/operate-and-govern/provision-models-and-providers/provisioning-aws-bedrock-models/) guide for the full provider-registration walkthrough. Two fields matter especially here: - **Region** must be the region the IAM role can reach Bedrock in, the same `` used in Step 1. - **API base URL** is pre-filled as `https://bedrock-runtime..amazonaws.com` from the region and must not be left empty. ## Step 3: Verify end-to-end 1. Confirm the binding and credential injection: ```shell tare dataplane identity bind --check --type ``` 1. Send a normal chat request through the gateway to a Bedrock model. A completion means the data plane signed the request with the pod's IAM role, with no static keys involved. ## Troubleshooting - **A cluster prerequisite is missing.** `bind` reports the missing prerequisite together with the exact command that fixes it (see [Prerequisites](#prerequisites)); run the command and re-run `bind`. - **The rollout stalls.** With `--wait` (the default), `bind` reports the first not-ready pod and its reason instead of timing out silently. - **Checking the injected credentials by hand.** The signing container image is distroless, so `kubectl exec ... -- env` cannot work. Read the pod spec instead, where the injected marker appears on the egress pod's containers (`AWS_ROLE_ARN` and `AWS_WEB_IDENTITY_TOKEN_FILE` for IRSA; `AWS_CONTAINER_CREDENTIALS_FULL_URI` for Pod Identity): ```shell kubectl get pods -n tars-dataplane -l gateway.envoyproxy.io/owning-gateway-name=egress -o yaml | grep -A2 AWS_ ``` `tare dataplane identity bind --check` performs the same check. - **A "secret NotFound" warning appears in the data plane logs.** For a credential-less provider, a warning mentioning `AISECRET_` and "secret NotFound" is **expected**: there is intentionally no secret, and processing proceeds with the pod identity. - **Requests to Bedrock fail with authentication errors.** Verify the provider's **API base URL** is `https://bedrock-runtime..amazonaws.com` and its **Region** matches both the URL and the region the IAM role was created for. Where to go next Provision AWS Bedrock models The full provider-registration walkthrough, including the credential-less pod-identity mode. Data plane installation for AWS Install the data plane on EKS, from cluster prerequisites to a connected gateway. --- ## Connect providers across clouds Larger organisations rarely consume an AI provider through a single subscription. Existing vendor contracts are spread across cloud accounts, data-residency obligations pin certain workloads to certain geographies, and capacity headroom is often easier to secure by spreading the same model family across several regional deployments. Tetrate Agent Router models this fan as a set of independent provider entries, each subscription in each region configured as its own provider with its own endpoint and credentials, so that the same logical model can be mapped to several regional entries and routing has more than one place to send a request. The routing layer that consumes this structure is configured separately in Load balance across regional deployments; this guide stops at the point where the regional entries exist, are healthy, and are mapped to the logical models that routing will reference. **Persona:** Platform operator working in the Admin Dashboard. **Estimated time:** 30 to 45 minutes for a first pass across two or three clouds, depending on how many subscriptions and regions are involved and whether private connectivity is in scope. ## When this guide applies This guide is the right starting point when more than one subscription to the same provider, or to equivalent models across clouds, has to be represented in Agent Router: | Situation | Why a separate provider entry is warranted | | :---- | :---- | | Existing contracts with more than one cloud vendor | Each contract carries its own account, endpoint, and credentials, and has to be represented independently | | A model offered in several regions of the same cloud | Each regional endpoint is a distinct connection with its own latency and capacity profile | | Data-residency obligations | A workload bound to a geography must reach a provider endpoint in that geography, not a default global one | | Latency-sensitive traffic | Routing to the nearest regional endpoint reduces round-trip time; the nearer endpoints must exist as entries first | | Capacity headroom and quota limits | Spreading load across several regional subscriptions raises the aggregate quota available to a single logical model | If the goal is a single subscription to a single provider, the broader [Provision models and providers](/agent-router-enterprise/guides/operate-and-govern/provision-models-and-providers/) guide is the simpler starting point. This guide assumes the multi-subscription case. ## Outcomes By the end of this guide: - One provider entry exists per subscription and region, each with its own endpoint and credentials, and each reporting a healthy connection status. - A naming convention is in place that keeps regional entries distinguishable at a glance. - At least one logical model is mapped to several regional provider entries, so the routing layer has more than one backend to choose among. - Where compliance requires it, regional entries dial a private endpoint rather than a public one, keeping provider traffic on the cloud backbone. - Each connection has been verified, so routing can be configured against entries known to be reachable. ## Prerequisites - Administrator access to the Admin Dashboard, typically the `super_admin` or `provider_admin` role. The role model is covered in [Manage roles and permissions](/agent-router-enterprise/guides/identity-and-access-management/manage-roles-and-permissions/). - Valid credentials for each subscription to be connected. A separate set of credentials is expected per cloud account and, in most cases, per region. - The endpoint URL for each regional subscription. For cloud-hosted providers this is usually region-specific (the region appears in the host name) and differs from one regional entry to the next. - For private connectivity, the network-level configuration (the private endpoint, route, or service connection) already in place in each cloud environment. The Admin Dashboard configures only the URL the gateway dials; it does not provision the underlying network. The architecture is described in [Architecture Overview](/product-architecture/architecture-overview/). - A decision, taken before configuration begins, about which model families are to be mapped across which regions. Agent Router does not infer this; it is an explicit operator choice. ## Step 1: decide which subscriptions warrant a separate entry Before anything is configured, the set of provider entries to be created should be enumerated. Agent Router treats each provider entry as a single endpoint reached with a single set of credentials, so the rule is straightforward: wherever the endpoint or the credentials differ, a separate entry is required. This typically produces one entry for each of the following: - Each cloud vendor under contract: a subscription on one cloud and an equivalent subscription on another are always separate entries, because the credentials and endpoints have nothing in common. - Each region of a cloud-hosted provider: a model served from one region and the same model served from another reach different endpoints, even when the underlying account is shared. - Each account where contracts are split for billing or governance reasons, even within one cloud and region. The output of this step is a short inventory: provider, cloud, region, account, and the reason the entry exists (contract, residency, latency, or capacity). That inventory drives the entries created in Step 3 and the model mapping in Step 4. ## Step 2: settle on a naming convention Because several entries will refer to the same underlying provider, the entry names are the only thing that keeps them distinguishable in the provider list, in routing configuration, and in usage analytics. A convention agreed before the entries are created avoids a later rename pass. A workable convention encodes provider, cloud, and region in the identifier, with the region last so that related entries sort together. For example: ```text anthropic-aws-us-east-1 anthropic-aws-eu-west-1 anthropic-gcp-europe-west4 openai-azure-westus openai-azure-swedencentral ``` The constraints are the same as for any provider identifier (lower case, beginning with a letter), and the display name carries a human-readable form alongside it, for example `Anthropic (AWS, us-east-1)`. The value of the convention is operational rather than cosmetic: when a connection goes unhealthy or a region is drained for maintenance, the entry to act on is identifiable without cross-referencing. ## Step 3: configure one provider entry per subscription and region Each entry from the Step 1 inventory is configured as an independent provider. The mechanics of adding a provider (the fields, the credential entry, and the save-and-verify cycle) are the same as for a single provider and are covered in detail in [Provision models and providers](/agent-router-enterprise/guides/operate-and-govern/provision-models-and-providers/); for provider-specific endpoint and authentication detail, see [Provision AWS Bedrock models](/agent-router-enterprise/guides/operate-and-govern/provision-models-and-providers/provisioning-aws-bedrock-models/). The points specific to the multi-region case are these: 1. Add a provider entry for each row in the inventory, naming it according to the convention from Step 2. 1. Set the endpoint URL to the region-specific endpoint for that subscription. The region usually appears in the host name, and two entries for the same provider in different regions differ only in this value. 1. Enter the credentials belonging to that subscription. Credentials are not shared across regional entries even when the same vendor issued them; each entry carries its own. 1. Set the region field, where the provider type exposes one, to match the region in the endpoint. A mismatch between the stated region and the endpoint is a common cause of a connection that authenticates but routes to the wrong geography. 1. Save the entry and let Agent Router verify it before moving to the next. Verifying one entry at a time makes a failure easy to attribute to the subscription that caused it. Where credentials are supplied by the developer rather than held by Agent Router, under the Bring Your Own Key (BYOK) pattern, the per-region structure still applies, but the credential handling differs; see [Use your own provider credentials](/agent-router-service/guides/use-your-own-provider-credentials/). ## Step 4: map one logical model to several regional entries A logical model is the name developers route to, a model family such as a given Claude or GPT model, independent of which regional subscription ultimately serves the request. Mapping that logical model across several regional entries is what gives the routing layer a choice of backends. 1. Identify the logical model to be served from more than one region, drawn from the inventory in Step 1. 1. For each regional provider entry that offers an equivalent model, enable that model so it is exposed through the entry. Model enablement is described in [Provision models and providers](/agent-router-enterprise/guides/operate-and-govern/provision-models-and-providers/). 1. Confirm that the same logical model name is now associated with each of the intended regional entries. The set of regional entries backing one logical model is the pool the routing layer will distribute across. 1. Repeat for each logical model that is to span regions. The mapping defined here is the structure; it does not by itself decide how requests are spread across the regional entries. Weighting, ordering, and failover among them are configured in [Load balance across regional deployments](/agent-router-enterprise/guides/operate-and-govern/load-balance-across-regional-deployments/). Equivalence across clouds is an operator judgement: a model offered by one vendor and a comparable model from another are treated as one logical model only if the organisation has decided they are interchangeable for the traffic in question. For models that are self-hosted rather than vendor-served, the same mapping approach applies once the self-hosted endpoint is configured; see [Provision custom and self-hosted models](/agent-router-enterprise/guides/operate-and-govern/provision-models-and-providers/provision-custom-and-self-hosted-models/). ## Step 5: keep compliance-bound traffic on private connectivity Where a subscription serves traffic subject to data-residency or network-isolation requirements, the regional entry should dial a private endpoint rather than a public one, so that provider traffic stays on the cloud backbone and never crosses the public internet. 1. Confirm the private connection for that region is already provisioned at the cloud level: AWS PrivateLink, Azure Private Link, or GCP Private Service Connect, depending on the cloud. 1. In the corresponding provider entry, set the endpoint URL to the private-link address rather than the public host name. 1. Save and re-verify the entry. A private endpoint that resolves only inside the cloud network will fail verification if the data plane cannot reach it, which surfaces a misconfigured route or DNS record before any traffic depends on it. The Admin Dashboard configures only the URL the gateway dials. The private endpoint itself (DNS, routing, and network policy) is configured beforehand in the cloud environment, and the value entered here references the result of that work. The architectural detail is in [Architecture Overview](/product-architecture/architecture-overview/). Private and public entries can coexist: a region under a residency obligation can use a private endpoint while another region, not so bound, uses a public one. ## Step 6: verify each connection's health A regional entry is only useful to routing once it is reachable. Before the routing layer is pointed at the pool of entries built in Step 4, each entry should be confirmed healthy. 1. Open the provider list and review the status of every entry created in this guide. Each should report an active, healthy connection and a recent successful verification timestamp. 1. For any entry reporting an unhealthy connection, open it and re-check the endpoint URL, the region, and the credentials in turn. A regional endpoint pasted from the wrong region, or credentials from the wrong account, are the usual causes when one entry among several fails while its siblings succeed. 1. For private-endpoint entries, treat a verification failure as a network-reachability question first, whether the data plane can resolve and reach the private address, before suspecting the credentials. 1. Once every entry in the pool is healthy, the logical models mapped in Step 4 are ready for the routing layer to distribute across. Connection health is re-checked periodically and on each configuration change, so an entry that is healthy now but loses its endpoint or has its credentials expire will surface as unhealthy in the provider list without manual prompting. ## What to do next - **Load balance across regional deployments**: with the regional entries healthy and the logical models mapped, the next step is configuring how requests are distributed and how failover behaves across the pool. See [Load balance across regional deployments](/agent-router-enterprise/guides/operate-and-govern/load-balance-across-regional-deployments/). - **Use your own provider credentials**: where developers supply their own subscription credentials rather than relying on centrally held ones, the BYOK flow complements the per-region structure built here. See [Use your own provider credentials](/agent-router-service/guides/use-your-own-provider-credentials/). - **Provision custom and self-hosted models**: to extend the regional pools with models served from self-hosted endpoints rather than vendor subscriptions. See [Provision custom and self-hosted models](/agent-router-enterprise/guides/operate-and-govern/provision-models-and-providers/provision-custom-and-self-hosted-models/). The regional provider entries and logical-model mappings created here remain in place as the structure the routing layer is built on. Where to go next Load balance across regional deployments Configure how requests are distributed and how failover behaves across the pool. Provision custom and self-hosted models Extend the regional pools with models served from self-hosted endpoints. --- ## Provision models and providers This guide covers the catalog layer in the Admin Dashboard: connect providers and enable models so there is something to grant to a project. Providers carry credentials and connectivity; models carry which of those providers' offerings appear in the catalog. A default project exists from onboarding ([Key concepts → Projects](/product-architecture/key-concepts/#projects)); after enablement here, grant models in Create a project and grant models. Recurring work—credential rotation, retiring models, adding capacity, and private-link endpoints—is also covered below. To automate the same work, see Provision models via API. **Persona:** Platform operator working in the Admin Dashboard. **Estimated time:** 20 to 30 minutes for an initial provisioning pass, depending on how many providers and models are involved. ## When this guide applies This guide is the right starting point in any of these situations: | Situation | Why this guide helps | | :---- | :---- | | First-time setup of a fresh deployment | The catalog must have enabled models before any project can be granted access to them | | Adding a new AI provider relationship (a contract with a vendor, or a self-hosted endpoint) | The provider must be configured and verified before any of its models can be enabled | | Adding new model versions from an existing provider | Agent Router exposes only the models that have been explicitly enabled, not everything the provider supports | | Rotating provider credentials on a schedule or after a security event | A clean credential rotation is the recurring operational task that this layer requires | | Retiring a model or provider as part of a sunset or contract change | Disabling at the provider or model level cleanly removes capability from the developer-facing surface | ## Outcomes By the end of this guide: - At least one provider is configured with healthy credentials and an **Active** connection status. - At least one model from that provider is enabled and visible to developers in the Console Model Catalog. - The mechanics of credential rotation and provider removal are understood, so the recurring operational work can be performed safely. - Where private-link connectivity is appropriate, the configuration that keeps model traffic on the cloud backbone has been considered. ## Prerequisites - Administrator access to the Admin Dashboard, typically the `super_admin` or `provider_admin` role. The exact role model is covered in [Manage roles and permissions](/agent-router-enterprise/guides/identity-and-access-management/manage-roles-and-permissions/). - Valid credentials for at least one upstream AI provider: an API key, an Azure deployment endpoint, or whatever the provider expects. - For private-link setups, the network-level configuration (the private endpoint, route, or service connection) is already in place in the cloud environment. The Admin Dashboard configures only the endpoint URL that the gateway dials; it does not provision the underlying network. ## Orient in the Admin Dashboard The Admin Dashboard is reached at the deployment's dashboard URL, for example `https://dashboard.poc.tetrate.ai/`, and is entered with administrator credentials or corporate SSO. The default landing screen is the **Home** page, which presents organization-wide metrics. The [Console](/agent-router-service/guides/route-requests-across-providers/) serves developers who consume models through the gateway; the Admin Dashboard is where what is available, to whom, and under what conditions is configured. Models and providers enabled here are what developers subsequently access in the Console. ### Who uses the Admin Dashboard Three operator personas typically interact with the Admin Dashboard, each with a distinct emphasis: | Role | Typical tasks | | :---- | :---- | | Agent Router administrators | Enable and disable models, manage provider credentials, configure SSO | | IT operations | Monitor Agent Router health, review audit logs | | Security leads | Review user access, audit administrative actions, manage API keys | ### Sidebar navigation The Admin Dashboard sidebar is organised into the following sections: | Section | Purpose | | :---- | :---- | | Home | The landing page; displays organization-wide metrics and quick links to common administrative tasks | | Models Management | Browse the model registry, enable or disable models, filter by provider, and view context windows and pricing tiers | | Providers Management | Manage connections to AI providers, view provider status, configure credentials, and add new connections | | MCP Servers | Manage the catalogue of Model Context Protocol (MCP) servers available to Console users, configure authentication types, and view the tools each server exposes | | MCP OAuth Clients | Create and manage OAuth client configurations for MCP servers that require OAuth-based authentication | | Usage → Analytics | Organization-wide usage metrics across all users and API keys, with breakdowns by user, model, provider, and API key, time-range filtering, and cost analysis | | Usage → Budgets | Spend ceilings per user or team, with billed vs passthrough spend overview | | Announcements | Publish organization-wide notifications visible to Console users, with severity levels and scheduled publication dates | | Users | View and manage all registered users, including roles, last sign-in timestamps, and activity | | API Keys | Admin-level view of all API keys across Agent Router (key owners, creation dates, and last-used timestamps) with the ability to revoke keys | | Audit Logs | Tamper-resistant chronological history of all administrative actions, filterable by user, action type, resource, and time range | | Settings | Configure OIDC single sign-on with the corporate identity provider, and view or configure deployment instance settings | ### Read the Home dashboard The Home page refreshes its metrics on each load and reflects the current state of Agent Router. Five organization-wide metrics are displayed: | Metric | Description | | :---- | :---- | | Total Users | Number of registered users across Agent Router, regardless of current activity status | | Total API Keys | Number of API keys created, both active and revoked, giving a sense of total credential issuance | | Total Requests | Cumulative count of requests routed through the gateway; **Usage → Analytics** provides time-range breakdowns | | Active Models | Number of models currently enabled for Console users; disabled models do not appear in the Console | | Provider Status | Summary of connected AI providers and whether each connection is healthy and reachable | The Home page also provides shortcuts to the most common administrative tasks: jumping to Models Management to enable or disable models, to Users to review registered accounts, to **Usage → Analytics** for organization-wide consumption data, and to Audit Logs to inspect recent administrative actions. ## Step 1: configure the provider Providers are the upstream services the gateway is allowed to reach. Each provider entry carries the credentials and endpoint information for one external AI service. 1. Sign in to the Admin Dashboard. 1. Open **Providers Management** from the sidebar. 1. Review the existing provider list. Each entry shows the provider name, current status (Active or Inactive), the number of models the provider exposes, and the timestamp of the last successful health check. ![Providers Management](/img/screenshots/admin/providers-management.jpeg) 1. Click **Add Provider** at the top of the list. 1. Select the provider type from the dropdown. The supported set is broad and grows over time: | Provider | Models exposed | | :---- | :---- | | OpenAI | GPT-4o, GPT-4o mini, o1, o3, and other OpenAI models | | Anthropic | Claude Opus, Sonnet, and Haiku families | | Google | Gemini 2.0 Flash, Gemini 2.5 Pro, and other Google AI models | | Azure OpenAI | Azure-hosted OpenAI models with enterprise Azure credentials | | Mistral | Mistral Large, Medium, and other Mistral models | | Cohere | Command R, Command R+, and Cohere embeddings | | Together | Open-source models hosted on Together AI infrastructure | | Perplexity | Perplexity search-augmented models | 1. Enter the required credentials. The exact fields depend on the provider: a single API key for OpenAI and Anthropic; an endpoint URL and a key for Azure OpenAI; deployment-specific values for self-hosted backends. 1. Click **Save**. 1. Agent Router verifies the connection. A successful verification flips the provider's status to **Active**; a failure leaves it at **Inactive** with diagnostic detail in the row. A common cause of an immediate **Inactive** status is a credential pasted with leading or trailing whitespace. The provider-side API rejects the key, Agent Router reports the failure, and the cause is invisible unless the credential is re-entered carefully. Re-pasting from a known-good source is the fastest first attempt at recovery. ### Field reference: provider list Each entry in the provider list carries the following columns: | Column | Description | | :---- | :---- | | **Provider Name** | The provider identifier | | **Status** | Active (connected and healthy) or Inactive (disconnected or credentials invalid) | | **Models** | Number of models available through this provider | | **Last Verified** | Timestamp of the most recent successful connection check | Provider status is checked periodically and on each configuration change. An **Inactive** status may indicate expired API keys, network issues, or provider-side outages; the **Last Verified** timestamp shows when the connection was last confirmed healthy. ### Private provider connectivity For deployments where compliance or performance requirements demand that provider traffic stay on the cloud backbone rather than traversing the public internet, the endpoint URL field can be set to a private-link address: Azure Private Link, AWS PrivateLink, or GCP Private Service Connect. The data plane reaches the provider over the private connection without ever crossing the public internet. Agent Router configures only the URL it dials. The underlying private endpoint (DNS, routing, and network policy) is configured at the cloud level beforehand, and the value entered here references the result of that work. For the architectural detail on this setup, see [Architecture Overview](/product-architecture/architecture-overview/). ## Step 2: enable models for the provider Once a provider's connection is healthy, the models it exposes appear in **Models Management**. They are not, however, immediately visible to developers; enablement is an explicit decision. 1. Open **Models Management** from the sidebar. 1. Review the model list. Each row shows the model identifier, the provider, the context window size, the pricing tier, and the current enabled/disabled status. 1. Use the search box to find a model by name, or the provider filter to narrow the list. Both work in real time. 1. Click any model row to open its detail view and review the full model information (provider, context window size, pricing tier, and current enabled/disabled status), then close the detail view to return to the list. 1. Locate a model that should be exposed and toggle its **Status** switch to enabled. ![Models Management](/img/screenshots/admin/models-management.jpeg) A newly-enabled model appears in the Console Model Catalog immediately. Developers whose routing configurations target it can start sending requests right away. ### Field reference: model list Each row in the model list carries the following columns: | Column | Description | | :---- | :---- | | **Model Name** | The model identifier (e.g., `gpt-4o`, `claude-sonnet-4-20250514`, `gemini-2.0-flash`) | | **Provider** | The AI provider that serves the model | | **Context Window** | Maximum token context length supported by the model | | **Pricing Tier** | Cost classification for the model | | **Status** | Whether the model is enabled or disabled for Console users | For larger provisioning passes (enabling everything from a new provider, retiring everything from a deprecated one), the bulk-action surface is significantly faster than per-row toggling: 1. Select multiple models using the checkboxes on the left of each row. 1. Open the **Bulk Actions** dropdown at the top of the list. 1. Choose **Enable Selected** or **Disable Selected**. 1. Confirm the operation when prompted. The selected models all update at once. :::tip Curating the enabled set is the most important governance lever in this guide. A system that enables every model from every provider exposes developers to a confusing menu, complicates Reference and Usage Analytics, and makes it harder to enforce model-specific policy later. A system that enables only the models the organisation has actually decided to support is calmer, easier to audit, and easier to operate. Default to the smaller list. ::: Disabling a model does not delete it from the registry. The model can be re-enabled at any time, and re-enablement is immediate. Requests that target a disabled model receive a clear error indicating the model is unavailable, which makes the disabled state easy to observe both from the Admin Dashboard and from developer-side traffic. ### Configure model-level auto-fallback A model's detail view exposes an **Autofallback** setting. When it is enabled and one or more fallback models are chosen, the gateway attaches that fallback chain to the model itself: if the model's primary provider returns a recoverable error, requests fail over to the chosen fallback model automatically. This is the operator-side, catalogue-level counterpart to the per-key fallback policy developers configure in the Console. The distinction between the two is important, because it determines whose traffic the fallback protects: - **Model-level auto-fallback** (this setting, in the Admin Dashboard) applies to the model across **every router in the project**. It is the right tool for organisation-wide resilience that should not depend on each developer configuring their own key. - **Per-key fallback** (configured developer-side in the Console, under [Improve resilience with fallbacks](/agent-router-service/guides/improve-resilience-with-fallbacks/)) applies only to the single API key it is attached to. A request sent with any other key does not inherit it, and falls through to the model's global route instead. To enable it: 1. Open the model's detail view from **Models Management**. 1. Turn on **Autofallback**. 1. Choose one or more fallback models, in priority order. 1. Save. The fallback model must itself be **enabled** in the catalogue (Step 2); a fallback entry that points at a disabled or unlisted model is skipped, leaving the chain shorter than intended. When the fallback provider differs from the primary — for example a same model served from a second region — confirm the fallback model resolves to the correct upstream identifier before relying on it, by sending a test request that forces the failover path (see Step 3). ## Step 3: verify end-to-end through a developer-side request The provisioning is most useful when its result is visible from where developers actually work. A short end-to-end check confirms that the model is reachable from the gateway and that the credentials flow correctly. 1. Open the Console (a separate sign-in to the developer surface). 1. Create or reuse an API key with a routing configuration that targets the newly-enabled model. The mechanics are covered in [Route Requests Across Providers](/agent-router-service/guides/route-requests-across-providers/). 1. Send a test request through the gateway. 1. Confirm a successful response. 1. Return to the Admin Dashboard. 1. Open **Usage → Analytics**. Within a short delay, the test request appears in the organization-wide usage view, attributed to the model and provider just provisioned. If the request fails with an authentication error on the upstream side, the provider's credentials are the most likely cause; if the request fails because the model is unavailable, the enablement step has been missed or has not yet propagated. Both failure modes are distinguishable in the Console's Request Logs. ## Step 4: operate the provisioned layer Provisioning is not a one-shot activity. Provider credentials rotate, models are released and retired, and the supported provider set itself changes over time. The Admin Dashboard supports the recurring operations from the same screens used for the initial setup. ### Rotate provider credentials Provider credentials need to be rotated for the usual reasons: scheduled rotation, suspected compromise, contract migration to a new account. 1. Open **Providers Management**. 1. Click the provider row to open its configuration. 1. Update the credential fields with the new value. 1. Click **Save**. 1. Confirm that the status remains **Active** after re-verification. The rotation takes effect on the next request. In-flight requests that were authorised with the old credential complete with it; subsequent requests use the new one. There is no service restart and no downtime window. ### Remove a provider configuration Retiring a provider (because a contract has ended, because the relationship is being consolidated onto another account, or because the provider itself has been deprecated) is a single action. 1. Open the provider's configuration row. 1. Click **Delete** or **Remove Provider**. 1. Confirm. Every model previously served by that provider becomes unavailable immediately. Any routing configuration that targeted those models will fall through to its configured fallback, if one exists, or fail with an unavailable-model error otherwise. Before removing a provider that is actively serving traffic, scan the Console-side routing configurations and reroute or retire any that depend on it. ### Add new model versions When a provider releases a new model version, the model appears in **Models Management** automatically once the provider's catalogue is refreshed. The decision to enable the new version is a separate, explicit step; Agent Router never enables new models on the operator's behalf. A typical introduction: 1. Locate the new model in the list. 1. Review its context window and pricing tier. 1. Toggle its status to enabled. 1. Coordinate with the developer side, through internal communication or through the [Communicate with Agent Router Users](/agent-router-enterprise/guides/operate-and-govern/communicate-with-platform-users/) flow, so that developers know the model is available. 1. Optionally pair the introduction with a canary rollout on the developer side, using the patterns from *Apply Advanced Routing Rules* in the developer guides. ## What to do next With the catalog in place, grant models to a project, then onboard keys and apply spend controls as needed. The provisioned models and providers remain in place for subsequent guides. Where to go next Create a project and grant models Grant catalog models to a project so its gateway can route to them. Provision models via API Automate provider, credential, model, and project assignment through the Catalog API. Onboard developers and issue keys Make the provisioned models reachable to developer accounts. Provision AWS Bedrock models Configure AWS Bedrock as an AI model provider with region-specific endpoints. --- ## Custom and self-hosted models Most provisioning work in Tetrate Agent Router starts with vendor models: the GPT, Claude, and Gemini families served through their respective providers. A point arrives, though, where the organisation's own models matter as much as the vendor catalogue: a fine-tuned model on internal data, a small self-hosted model for thousands of cheap classification calls, or an embedding model for search and retrieval-augmented generation. Agent Router treats a custom or self-hosted model the same way it treats a vendor model, as a model served by a provider, with the difference only in where the provider points. As long as that endpoint speaks the OpenAI-compatible protocol, the convention Agent Router's routing layer expects, the model behaves like any other entry in the catalogue once it is enabled. This guide covers registering such a model, the specific case of embedding models, how to version a model so that developers are insulated from churn, and how to confirm the result from the developer side. **Persona:** Platform operator working in the Admin Dashboard. **Estimated time:** 15 to 25 minutes per model, plus any upstream work to stand up or secure the endpoint itself. ## When this guide applies This guide is the right reference in any of these situations: | Situation | Why this guide helps | | :---- | :---- | | A self-hosted or fine-tuned LLM needs to be routed through the gateway | The model is registered as a custom provider pointing at the endpoint that serves it | | A small language model is wanted for cheap, low-latency tasks alongside frontier models | The small model is provisioned exactly like any other model and selected through routing rules | | A search or RAG feature needs an embedding model | Embedding models are provisioned through the same provider-and-model layer, with their own verification path | | A model is being upgraded and developers should not have to change their code | A stable logical name pinned to an explicit version absorbs the change | | An embedding model is being replaced | The re-embedding implications have to be understood before the swap, not after | For provisioning vendor models and the general mechanics of the provider-and-model layer, see [Provision models and providers](/agent-router-enterprise/guides/operate-and-govern/provision-models-and-providers/). For pointing a provider at an endpoint reached over a private cloud connection, see [Connect provider subscriptions across clouds](/agent-router-enterprise/guides/operate-and-govern/provision-models-and-providers/connect-provider-subscriptions-across-clouds/). ## Outcomes By the end of this guide: - A custom or self-hosted model that exposes an OpenAI-compatible endpoint is registered as a provider and enabled as a model. - An embedding model is provisioned and distinguished from chat-completion models in the catalogue. - A versioning scheme is in place, with explicit versions pinned underneath a stable logical name, so that model upgrades do not break developer code. - The new model is exposed to the intended audience and no wider. - The model has been confirmed reachable through a developer-side request. ## Prerequisites - Administrator access to the Admin Dashboard, typically the `super_admin` or `provider_admin` role. The role model is covered in [Provision models and providers](/agent-router-enterprise/guides/operate-and-govern/provision-models-and-providers/). - An endpoint that serves the model and speaks the OpenAI-compatible protocol: a self-hosted inference server, a fine-tuned deployment, or a third-party host. The endpoint must be reachable from the data plane. - The credentials the endpoint expects, typically a bearer token or API key. Endpoints that require no authentication are supported but are appropriate only on a trusted private network. - The logical model name the endpoint accepts in the request body. Most OpenAI-compatible servers expect a `model` field, and the value entered during provisioning must match what that server recognises. - For endpoints reached over a private cloud connection, the network-level configuration is already in place. See [Connect provider subscriptions across clouds](/agent-router-enterprise/guides/operate-and-govern/provision-models-and-providers/connect-provider-subscriptions-across-clouds/). ## Step 1: decide what is being added and why The provisioning steps are the same for every custom model, but the decision that precedes them differs by intent. Three intents are common, and naming the intent up front makes the later choices (audience, versioning, and routing) straightforward. | Intent | What it serves | What to keep in mind | | :---- | :---- | :---- | | Self-hosted or fine-tuned LLM | A model the organisation operates for control, data residency, or domain tuning | Capacity and availability are now the organisation's responsibility; the gateway routes to it but does not run it | | Small language model | High-volume, latency-sensitive, or cost-sensitive tasks: classification, extraction, routing decisions | Best paired with routing rules that send only the appropriate traffic to it, rather than exposing it as a general-purpose model | | Embedding model | Search, clustering, and retrieval-augmented generation | Returns vectors rather than text; provisioned the same way but verified and versioned differently (see Step 4) | A self-hosted model rarely replaces the vendor catalogue. It sits alongside it, and the value comes from developers being able to choose between them through routing configuration rather than through separate integrations. The provisioning work below makes the model available; the selection logic lives on the developer side and is covered in [Apply advanced routing rules](/agent-router-service/guides/apply-advanced-routing-rules/) and [Route requests across providers](/agent-router-service/guides/route-requests-across-providers/). ## Step 2: register the endpoint as a provider A custom model is reached through a provider entry whose endpoint points at the model's host rather than at a vendor API. The provider carries the connectivity and credentials; the model entry, configured in the next step, carries the visibility decision. 1. Sign in to the Admin Dashboard. 1. Open the providers surface from the sidebar. 1. Start a new provider entry. 1. Give the provider a clear identifier and display name that signal it is custom: for example, an identifier such as `acme-internal-llm` and a display name such as `Acme Internal LLM (self-hosted)`. A name that distinguishes the entry from the vendor providers prevents later confusion in the catalogue and in analytics. 1. Set the endpoint to the model's OpenAI-compatible base URL. The base URL is the address the data plane dials; Agent Router appends the standard OpenAI-compatible paths to it, so the value entered is the root of the API rather than a specific route. 1. Select the authentication method the endpoint expects and enter the credential. A bearer token or API key is the common case. Where the endpoint sits behind a private cloud connection, the base URL is the private-link address rather than a public one; see [Connect provider subscriptions across clouds](/agent-router-enterprise/guides/operate-and-govern/provision-models-and-providers/connect-provider-subscriptions-across-clouds/). 1. Save the provider. Agent Router verifies the connection and reports the result on the provider entry. A connection that fails to verify is most often a credential pasted with surrounding whitespace, a base URL that includes a trailing path Agent Router also appends (producing a doubled route), or an endpoint not reachable from the data plane's network. Each of these is distinguishable from the diagnostic detail on the provider entry, and re-entering the value from a known-good source is the fastest first attempt at recovery. :::note The OpenAI-compatible protocol is a convention, not a guarantee. Some self-hosted servers implement only a subset of it. A model that completes chat requests but rejects, for example, streaming or function calling will surface those gaps to developers at request time rather than during provisioning. Confirming which capabilities the endpoint actually supports before exposing it widely avoids surprises downstream. ::: ## Step 3: enable the model and choose its logical name Once the provider verifies, the model it serves is registered and enabled. For a custom endpoint, the operator supplies the model identity rather than selecting it from a discovered vendor catalogue. 1. Open the models surface from the sidebar. 1. Add a model entry against the provider just created. 1. Set the model identifier to the logical name developers will call. This is the value that appears in the `model` field of a developer's request, and it is the contract between the developer and Agent Router. A clear, stable name such as `acme-internal-llm` is preferable to one that encodes a version or a hostname. 1. Set the upstream model name to the value the endpoint itself recognises, if Agent Router distinguishes the two. The endpoint may expect a different string in its own request body than the logical name developers use; mapping the logical name to the upstream name at this layer is what lets the developer-facing name stay stable across upstream changes. 1. Record the model's characteristics where Agent Router captures them: context window and whether the model serves chat completions or embeddings. Accurate metadata keeps the catalogue and analytics meaningful and helps developers choose the right model. 1. Enable the model. A newly enabled model becomes selectable to developers in line with the audience rules covered in Step 5. Until then it exists in the catalogue but is not yet reachable by the intended consumers. ## Step 4: provision an embedding model An embedding model is provisioned through the same provider-and-model layer, but three things differ and each one matters. - An embedding model returns vectors, not text. It is called through the embeddings route of the OpenAI-compatible protocol rather than the chat-completions route, and it is useful only to features that consume vectors: semantic search, clustering, and retrieval-augmented generation. It is not a substitute for a chat model and should be labelled clearly so that developers do not select it by mistake. - The vector dimension is a fixed property of the model. Every vector an embedding model produces has the same length, and that length is part of the contract with whatever vector store holds the results. A vector store provisioned for one dimension cannot hold vectors of another. The dimension is therefore worth recording alongside the model so that developers and operators alike can see it without inspecting a response. - Embedding output is not portable across models. Vectors from one embedding model are not comparable to vectors from another, even when the dimension happens to match. This is what makes the versioning discipline in Step 4a more than a convenience for embedding models; it is a correctness requirement. To provision an embedding model, register its endpoint as a provider as in Step 2, then add a model entry as in Step 3, marking the model as an embedding model where Agent Router captures the distinction. Developers consume the result through the embeddings route; the developer-side mechanics are covered in [Generate embeddings](/agent-router-service/guides/generate-embeddings/). ### Step 4a: version models behind a stable logical name Models change. Vendors release new versions, self-hosted deployments are retrained, and endpoints move. The goal of versioning is that none of this churn reaches developer code. The pattern that achieves it is the same for chat and embedding models: - Pin explicit versions. Where a model has a version (a vendor's dated revision, an internal training run, or a tag on a self-hosted image), provision it under a model identifier that names the version explicitly. An explicit version is reproducible: a request routed to it today behaves the same as a request routed to it next month. - Expose a stable logical name as an alias. Alongside the pinned versions, expose one logical name that developers call (`acme-internal-llm` rather than `acme-internal-llm-2026-04`) and point it at the version the organisation currently considers current. Developers code against the stable name; the operator moves what it points at. An upgrade becomes a single operator action with no developer-side change. - Keep the previous version enabled during a transition. Retiring the old version the instant the alias moves leaves no fallback if the new version misbehaves. Keeping both enabled for a window allows a clean cut-over and an equally clean roll-back. For embedding models the alias carries one extra obligation. Because vectors are not portable across models, moving an embedding alias to a new model silently invalidates every vector already stored against the old one. New text is embedded with the new model and compared against vectors produced by the old one, and the comparison is meaningless. Changing the embedding model therefore implies re-embedding the corpus: the stored vectors are regenerated with the new model before, or as part of, the cut-over. Plan the re-embedding as part of the upgrade rather than discovering the need for it after search quality has degraded. A practical sequence is to provision the new embedding model under an explicit version, re-embed the corpus into a separate index, and move the alias only once the new index is populated and verified. ## Step 5: expose the model to the right audience A registered, enabled model is governed by the same visibility rules as any other model in the catalogue. Exposing a custom or self-hosted model to the right audience, and no wider, is the same governance lever described for vendor models in [Provision models and providers](/agent-router-enterprise/guides/operate-and-govern/provision-models-and-providers/). - A small or experimental model is often best exposed to a single team or a pilot group before any broader release, so that its behaviour and cost are understood on real traffic first. - A self-hosted model with finite capacity should be exposed only to the audience that capacity can serve. The gateway routes whatever traffic it is given; it does not protect an undersized endpoint from being overwhelmed. - An embedding model should be exposed to the teams building search or retrieval features and not offered as a general option, both to avoid misuse as a chat model and to keep the embedding dimension stable for the consumers that depend on it. Where the model should reach developers, pair the exposure with a notification so that the audience knows it is available and under what name. Where a model is being introduced as a cheaper or faster alternative to an existing one, the cut-over is usually best done gradually on the developer side using the patterns in [Apply advanced routing rules](/agent-router-service/guides/apply-advanced-routing-rules/). ## Step 6: verify from the developer side Provisioning is confirmed when the model answers a request sent the way a developer would send it. The check differs slightly between a chat model and an embedding model. 1. Open the Console and use, or create, an API key whose routing configuration targets the new model by its logical name. The mechanics are covered in [Route requests across providers](/agent-router-service/guides/route-requests-across-providers/). 1. For a chat model, send a short completion request and confirm a coherent text response. 1. For an embedding model, send a short text through the embeddings route and confirm that a vector of the expected dimension is returned. A response whose length does not match the recorded dimension points to the wrong model being targeted or the metadata being incorrect. 1. Return to the Admin Dashboard and open the usage surface. Within a short delay, the test request appears, attributed to the new model and its provider. A request that fails with an upstream authentication error points to the provider credential; one that fails because the model is unavailable points to a missed enablement step or an audience rule that excludes the test key; one that reaches the endpoint but is rejected as an unsupported operation points to the OpenAI-compatible gap noted in Step 2. All three are distinguishable from the Console request logs. ## What to do next - **Route traffic to the new model deliberately.** A custom model earns its place through the routing rules that send the right requests to it. See [Apply advanced routing rules](/agent-router-service/guides/apply-advanced-routing-rules/) and [Route requests across providers](/agent-router-service/guides/route-requests-across-providers/). - **Build on an embedding model.** Once an embedding model is provisioned, the developer-side workflow for producing and consuming vectors is covered in [Generate embeddings](/agent-router-service/guides/generate-embeddings/). - **Connect endpoints over a private cloud link.** Where the model's endpoint should be reached without crossing the public internet, see [Connect provider subscriptions across clouds](/agent-router-enterprise/guides/operate-and-govern/provision-models-and-providers/connect-provider-subscriptions-across-clouds/). - **Return to the provisioning baseline.** For the general model-and-provider mechanics this guide builds on, see [Provision models and providers](/agent-router-enterprise/guides/operate-and-govern/provision-models-and-providers/). Where to go next Provision models and providers The general model-and-provider mechanics this guide builds on. Connect provider subscriptions across clouds Reach a model endpoint over a private cloud connection without crossing the public internet. --- ## Provision models via API The Admin Dashboard is the usual surface for connecting providers and enabling models, but automation needs the same outcome without clicking through forms. The Catalog API is that path: upsert a provider, store its credential, upsert a model with pricing and capabilities, assign the model to a project so a project gateway can route to it, then verify with a read-back and an inference call. This guide walks that sequence end to end with illustrative HTTP requests. For the dashboard equivalent, see Provision models and providers. :::tip Two API surfaces This guide uses the [Management API](/reference/api/) (Catalog and project assignment) to provision configuration. Live prompts use the [Gateway APIs](/reference/supported-apis/) on the data plane (`/v1/chat/completions`, `/v1/models`, and related paths). The final verification step switches to that gateway surface; the provisioning steps do not. ::: **Persona:** Platform operator or platform engineer automating catalog provisioning against the management API. **Estimated time:** 15 to 25 minutes once you have an admin API key, a provider credential, and the customer and project identifiers for assignment. ## When this guide applies This guide is the right reference in any of these situations: | Situation | Why this guide helps | | :---- | :---- | | A script or pipeline must provision providers and models without the Admin Dashboard | The Catalog write RPCs are the programmatic surface for the same configuration | | An internal source of truth syncs the model catalogue into Agent Router | Upsert and assign map cleanly onto a repeatable reconciliation loop | | You need pricing and capability fields set at create time, not only toggled later in the UI | UpsertModel accepts pricing and capabilities in one request | | You have the OpenAPI or SDK reference but no ordered how-to | This guide sequences the calls and shows how to verify the result | If you prefer the Admin Dashboard, use [Provision models and providers](/agent-router-enterprise/guides/operate-and-govern/provision-models-and-providers/) instead. For managing the broader configuration baseline as code, see [Manage configuration as code](/agent-router-enterprise/guides/operate-and-govern/manage-configuration-as-code/). ## Outcomes By the end of this guide: - A provider record exists with a stored credential. - A model record exists under that provider, with pricing and capabilities set. - The model is assigned to a project so the project gateway can route to it. - You have confirmed the configuration with Catalog read-backs and a gateway inference call. ## Prerequisites - An API credential with administrative scope for Catalog writes, issued from the Admin Dashboard or an equivalent control-plane path. Bearer auth on every request below. - The management API base URL for the deployment (the host that serves `/v1/catalog/...`). Replace `https://management.example.com` in the examples with that host. - Upstream provider credentials (API key, bearer token, or whatever the provider expects) and the provider's API base URL. - The customer and project identifiers used when assigning a model to a project. - For the final inference check: a **client** API key minted for that project (CreateClientWithKey), not a CreateApiKey user token. See [Which API key for which surface](/reference/api/#which-api-key-for-which-surface). The `tare api catalog` CLI exposes the same operations as flags. This guide uses HTTP so the request bodies are explicit; either surface is valid. ## Step 1: upsert the provider Create or fully replace the provider record with [UpsertProvider](/reference/api/catalog/agentrouter-catalog-v-1-catalog-service-upsert-provider/). The `id` is a canonical slug: lowercase alphanumeric only (`[a-z0-9]+`), no hyphens. ```bash curl -sS -X POST "https://management.example.com/v1/catalog/providers" \ -H "Authorization: Bearer ${ADMIN_API_KEY}" \ -H "Content-Type: application/json" \ -d '{ "id": "acmeopenai", "displayName": "Acme OpenAI", "baseUrl": "https://api.openai.com/v1", "supportedAuthSchemes": ["bearer"] }' ``` Required fields are `id`, `displayName`, and `baseUrl`. `supportedAuthSchemes` is optional (for example `["bearer"]`). ## Step 2: set the provider credential Store or rotate the upstream credential with [SetProviderCredential](/reference/api/catalog/agentrouter-catalog-v-1-catalog-service-set-provider-credential/). The plaintext is never returned; the provider's `credential_suffix` reflects the last four characters after a successful write. ```bash curl -sS -X POST "https://management.example.com/v1/catalog/providers/acmeopenai/credential" \ -H "Authorization: Bearer ${ADMIN_API_KEY}" \ -H "Content-Type: application/json" \ -d '{ "credential": "'"${PROVIDER_API_KEY}"'" }' ``` Credential rotation uses the same call with the new value. In-flight requests that already held the old credential complete with it; subsequent requests use the new one. ## Step 3: upsert the model (pricing and capabilities) Create or fully replace the model with [UpsertModel](/reference/api/catalog/agentrouter-catalog-v-1-catalog-service-upsert-model/). Pricing fields are decimal USD strings. Capabilities declare what the model supports (for example `chat` or `embeddings`); that is the API counterpart of choosing a model mode in the UI. ```bash curl -sS -X POST "https://management.example.com/v1/catalog/models" \ -H "Authorization: Bearer ${ADMIN_API_KEY}" \ -H "Content-Type: application/json" \ -d '{ "providerId": "acmeopenai", "name": "gpt-4o", "upstreamModel": "gpt-4o", "inputPerMillion": "2.50", "outputPerMillion": "10.00", "cacheReadPerMillion": "1.25", "maxCostPerRequest": "", "maxContextTokens": 128000, "capabilities": ["chat"] }' ``` Required fields are `providerId` and `name`. `upstreamModel` defaults to `name` when empty. Capture the model id returned in the response; you need it for project assignment in the next step. For an embedding model, set `capabilities` to include `embeddings` and verify with the embeddings endpoint instead of chat completions. ## Step 4: assign the model to a project :::note Projects are the isolation boundary A **project** owns the models, API keys, and gateway URL an application calls. Catalog enablement alone does not make a model visible on a project gateway—you must assign it. A default project is created during onboarding and is enough for a first verification; creating additional projects and granting models in the Admin Dashboard is covered in [Create a project and grant models](/agent-router-enterprise/guides/operate-and-govern/project-and-data-plane-management/create-a-project/). For the full concept, see [Key concepts → Projects](/product-architecture/key-concepts/#projects). ::: Grant access with [AssignModelToProject](/reference/api/catalog/agentrouter-catalog-v-1-catalog-service-assign-model-to-project/). The call is idempotent: re-assigning the same model is a no-op. ```bash curl -sS -X POST \ "https://management.example.com/v1/customers/${CUSTOMER_ID}/projects/${PROJECT_ID}/models" \ -H "Authorization: Bearer ${ADMIN_API_KEY}" \ -H "Content-Type: application/json" \ -d '{ "modelId": "'"${MODEL_ID}"'" }' ``` `modelId` is the catalog model id from the UpsertModel response. Without this assignment, a project-scoped gateway may not list or route to the model even though the catalog record exists. In the Admin Dashboard, enablement and project scoping are the UI counterparts of the upsert-plus-assign sequence. ## Step 5: verify Confirm configuration first, then traffic. ### Read-back 1. Fetch the provider: `GET /v1/catalog/providers/{id}` ([GetProvider](/reference/api/catalog/agentrouter-catalog-v-1-catalog-service-get-provider/)). Confirm `baseUrl` and that `credential_suffix` is set. 1. Fetch the model: `GET /v1/catalog/models/{id}` ([GetModel](/reference/api/catalog/agentrouter-catalog-v-1-catalog-service-get-model/)). Confirm pricing, `maxContextTokens`, and `capabilities`. 1. List project models: `GET /v1/customers/{customer_id}/projects/{project_id}/models` ([ListProjectModels](/reference/api/catalog/agentrouter-catalog-v-1-catalog-service-list-project-models/)). Confirm the new model appears. ```bash curl -sS "https://management.example.com/v1/catalog/providers/acmeopenai" \ -H "Authorization: Bearer ${ADMIN_API_KEY}" curl -sS "https://management.example.com/v1/catalog/models/${MODEL_ID}" \ -H "Authorization: Bearer ${ADMIN_API_KEY}" curl -sS \ "https://management.example.com/v1/customers/${CUSTOMER_ID}/projects/${PROJECT_ID}/models" \ -H "Authorization: Bearer ${ADMIN_API_KEY}" ``` ### Inference The steps above used the **management** (Catalog) API. End-to-end verification switches to the **gateway** inference surface: a different host and a **project client** key (CreateClientWithKey), not the admin credential used for Catalog writes. Gateway paths, formats, and auth are documented in [Gateway APIs](/reference/supported-apis/). Use the gateway base URL for the deployment (for Fully Managed, often `https://api.router.tetrate.ai/v1`). First list the models routable for that client key with [`GET /v1/models`](/reference/supported-apis/#models-api-v1models). Each returned `id` is a value you can pass as `model` on an inference call. Confirm the model you just provisioned and assigned appears in the list. ```bash curl -sS "https://api.router.tetrate.ai/v1/models" \ -H "Authorization: Bearer ${CLIENT_API_KEY}" ``` Then send a Chat Completions request ([`POST /v1/chat/completions`](/reference/supported-apis/#chat-completions-api-v1chatcompletions)) using one of those `id` values: ```bash curl -sS -X POST "https://api.router.tetrate.ai/v1/chat/completions" \ -H "Authorization: Bearer ${CLIENT_API_KEY}" \ -H "Content-Type: application/json" \ -d '{ "model": "gpt-4o", "messages": [{"role": "user", "content": "ping"}] }' ``` A successful response confirms credentials, catalog configuration, and project assignment end to end. If the model is missing from `GET /v1/models`, the usual causes are a missed upsert or project assignment. An upstream authentication error on the completion call usually points at the provider credential. For more gateway call shapes, see [Make an API call](/agent-router-service/quickstarts/make-an-api-call/). ## What to do next Use the dashboard for the same catalog and grant work when you prefer UI, or stay on the API and IaC path below. Catalog method detail is in [CatalogService](/reference/api/catalog/agentrouter-catalog-v-1-catalog-service/). Where to go next Provision models and providers The Admin Dashboard workflow for providers and model enablement. Create a project and grant models Grant catalog models to a project, then provision its gateway and keys. Manage configuration as code How the admin API fits UI and IaC ownership of the catalogue. --- ## Provision AWS Bedrock models This guide covers configuring AWS Bedrock as an AI model provider in Tetrate Agent Router. The procedure covers: - Configuring the AWS Bedrock API endpoint - Setting authentication credentials - Defining the AWS region - Enabling or disabling the provider - Marking the provider as a first-party provider Plan for 5 to 10 minutes, plus any AWS IAM setup if credentials do not already exist. Bedrock supports two authentication modes: - **Static keys** — an AWS access key pair stored as the provider secret. - **Workload identity (no static keys)** — for a self-hosted data plane on AWS EKS whose pod IAM identity has been bound with `tare dataplane identity bind`; the provider is created credential-less. See [Bedrock access with workload identity](/agent-router-enterprise/guides/operate-and-govern/provision-models-and-providers/bedrock-workload-identity/). ## Prerequisites - Administrator access to the Admin Dashboard, typically the `super_admin` or `billing_admin` role. - One of: - **Static keys**: a valid AWS access key pair (`AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY`) with permission to access AWS Bedrock, or - **Workload identity**: a data plane on EKS already bound to an IAM role per [Bedrock access with workload identity](/agent-router-enterprise/guides/operate-and-govern/provision-models-and-providers/bedrock-workload-identity/). ## Configuration guide ### Step 1: Add provider 1. Log in to the Admin Dashboard. 1. In the left-hand navigation panel, select **Catalog → Providers**. 1. Click **+ Add provider**. ### Step 2: Configure provider details - **Provider identifier** (required): must be all lower-case and start with a letter. Example: `my-aws-bedrock`. - **Display name** (optional): a recognizable label. Example: `AWS Bedrock BYOK`. ### Step 3: Configure authentication method - **Auth method** (required): select `AWS` from the drop-down. - **API base URL** (required): `https://bedrock-runtime..amazonaws.com`. The form pre-fills this from the selected region and keeps it in sync when the region changes; only edit it deliberately. It must not be left empty. ### Step 4: Set the AWS region - **Region** (required): set to the AWS region used in the base URL. The region must match both the Bedrock deployment region and the region in the base URL. ### Step 5: Provide authentication credentials Choose one of the two authentication modes: **Static keys:** - **Provider secret**: provide `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` in the corresponding fields. Both values are required together. - The credentials must have permission to access AWS Bedrock. - Use the **Rotate** button to update credentials later. **Workload identity (no static keys):** - Turn on **Use IRSA / EKS Pod Identity (no static keys)**. The AWS key fields are hidden and the provider is created without a secret; the data plane authenticates to Bedrock with its pod IAM role. - This mode requires the data plane's identity to be bound first — see [Bedrock access with workload identity](/agent-router-enterprise/guides/operate-and-govern/provision-models-and-providers/bedrock-workload-identity/). - The toggle is available when creating a provider. To switch an existing static-key provider to workload identity, create a new provider with the toggle on. ### Step 6: Enable the provider - **First-party provider** (optional): toggle on to give the provider first-party visibility in dashboard listings and search results. - **Enable provider** (required to use the configuration): toggle on. This controls whether the provider is available for use. ### Step 7: Save the configuration - Click **Add provider**. ## Example configuration With static keys: ```text Display Name: AWS Bedrock API Base URL: https://bedrock-runtime.us-east-1.amazonaws.com Auth Method: AWS Region: us-east-1 AWS_ACCESS_KEY_ID: AWS_SECRET_ACCESS_KEY: First-Party Provider: Enabled Enable Provider: Enabled ``` With workload identity: ```text Display Name: AWS Bedrock API Base URL: https://bedrock-runtime.us-east-1.amazonaws.com Auth Method: AWS Region: us-east-1 Use IRSA / EKS Pod Identity (no static keys): Enabled First-Party Provider: Enabled Enable Provider: Enabled ``` ## AWS account requirements Before the configuration is saved, confirm the following: - AWS Bedrock is enabled in the AWS account. - The selected region supports the required Bedrock models. - The IAM identity used — the static-key credentials, or the IAM role bound to the data plane — carries permissions such as: ```json { "Effect": "Allow", "Action": [ "bedrock:InvokeModel", "bedrock:InvokeModelWithResponseStream" ], "Resource": "*" } ``` - The API base URL and region match. ## Conclusion Once configured and saved, Agent Router can authenticate with AWS Bedrock and route model requests through the specified endpoint. Where to go next Bedrock access with workload identity Bind the EKS data plane's pod IAM identity so Bedrock needs no static keys. Connect provider subscriptions across clouds Represent Bedrock subscriptions in several regions as independent provider entries. --- ## Run multiple Agent Router instances Enterprise Tier A single Agent Router instance is fine for the first phase of any deployment. The complications start later. A staging environment is created so that risky configuration changes can be tested before they touch production. A second region is added because data residency requirements demand local processing. A subsidiary acquires its own instance because the parent's contract does not cover them. A research team needs a sandbox that they can break without affecting the rest of the organisation. Each of these is a perfectly reasonable reason to run more than one Agent Router instance, and each of them brings the same set of operational questions: which is which, what is configured where, how do the configurations stay in step where they should, and how do they stay independent where they should not. The Admin Dashboard's **Instances** surface is the entry point for that work. It surfaces the configuration that defines what an instance is: the gateway URL, the Console and Admin Dashboard URLs, the deployment mode (SaaS or Enterprise), the enabled feature set, and the current software version. It is not a multi-instance management surface in the strict sense; each instance is administered through its own Admin Dashboard. What the screen provides is the per-instance ground truth that makes coordinated multi-instance operation possible. This guide covers how to read that ground truth, the common multi-instance topologies Agent Router supports, and the operational disciplines that keep multiple instances coherent without accidentally making them entangled. **Persona:** Platform operator working in the Admin Dashboard, often across more than one instance. **Estimated time:** 10 to 15 minutes per instance for initial orientation; longer for setting up the operational disciplines that span instances. ## When this guide applies This guide is relevant in any of these situations: | Situation | What it covers | | :---- | :---- | | Setting up a separate staging environment alongside production | The per-instance configuration model and the disciplines that keep two instances coherent | | Operating multiple production instances for regulatory or regional reasons | The independence model between instances and the cross-instance audit posture | | Onboarding a subsidiary or business unit with its own instance | The verification step that confirms a fresh instance is configured as expected | | Coordinating a Agent Router upgrade across several instances | The version field on the Instances surface and the rolling-update considerations | | Investigating why one instance behaves differently from another | The configuration drift the Instances surface makes visible | For single-instance deployments, this guide is mostly informational; the same surface still applies, but the multi-instance disciplines do not. The configuration model is worth understanding either way, because every deployment has at least one instance. ## Outcomes By the end of this guide: - The Instances surface has been opened and the configuration fields are understood. - The deployment mode of the current instance (SaaS or Enterprise) is clear, along with what that mode controls. - The conditions under which fields are editable are clear, and any editable settings have been verified. - The cross-instance disciplines (configuration parity, version coherence, audit segregation) are understood for deployments that need them. ## Prerequisites - Administrator access to the Admin Dashboard, typically the `super_admin` role. - For multi-instance work: administrator access to each instance, since each one is administered separately. - For Enterprise deployments: the underlying Kubernetes context required to make data-plane-level changes. Configuration that lives in the Admin Dashboard is editable from inside Agent Router; configuration that lives in the data plane manifests is editable through whatever GitOps or manual process the deployment uses. ## Step 1: open the instances surface and orient The Instances surface is the canonical source of truth for what a given instance is. 1. Sign in to the Admin Dashboard. 1. Open **Settings → Instances** from the sidebar. 1. Review the fields surfaced for the instance: | Field | What it captures | | :---- | :---- | | Instance Name | The identifier for this deployment, distinguishing it from any others | | Deployment Mode | SaaS (both planes hosted by Tetrate) or Enterprise (self-hosted data plane in customer Kubernetes; management plane Tetrate-hosted) | | Gateway URL | The base URL of the AI gateway proxy endpoint | | Console URL | The URL where developers reach the Console | | Admin URL | The URL where operators reach the Admin Dashboard | | Enabled Features | Agent Router capabilities active for this instance | | Version | The current software version of the deployment | 1. Confirm that every value matches the documented expectation for this instance. A staging instance whose gateway URL accidentally points at production, or a production instance whose enabled-feature list has drifted from the rest of the fleet, are both reliable sources of "why is this weird" investigations weeks later. The Instance Name is the single most important field for multi-instance operation. Naming conventions that include the environment (`prod`, `staging`, `dev`), the region (`us-east-1`, `eu-central`), and the business unit (`platform`, `research`, `subsidiary-name`) make the rest of the operational discipline considerably easier. A flat name like `agent-router-2` becomes ambiguous immediately. ## Step 2: understand the deployment mode The deployment mode is the most consequential field on the screen because it determines who manages what. | Mode | Who manages infrastructure | Who manages configuration | Update model | | :---- | :---- | :---- | :---- | | **SaaS** | Tetrate, end-to-end (both planes) | Customer, through the Admin Dashboard | Managed by Tetrate; the customer is notified through release notes | | **Enterprise** | Tetrate for the management plane; customer for the data plane in their Kubernetes cluster | Customer, through the Admin Dashboard and through data plane manifests | Customer controls data-plane upgrade timing; the upgrade flow is documented in the installation materials | A SaaS instance is appropriate when the customer wants a fully hosted Agent Router deployment and does not require the data plane to live inside their own network boundary. An Enterprise instance is appropriate when data sovereignty, integration with internal networking, or fine-grained control over data-plane update timing makes a self-hosted data plane worthwhile — the management plane remains Tetrate-hosted in both modes. Most large organisations end up with Enterprise; smaller pilots and certain customer-specific PoCs use SaaS. The deployment mode also determines what is editable on the Instances surface. SaaS instances generally expose a smaller surface for direct edit because the underlying infrastructure is managed; Enterprise instances expose more for the customer-managed data plane, while the management plane stays Tetrate-operated. Where a setting is editable, subject to the deployment mode and the operator's permissions, it is changed in place: 1. Select **Edit** next to the configurable setting. 1. Update the value and select **Save**. Changes may require a brief propagation period before they take effect across Agent Router. ## Step 3: verify the gateway and Console URLs The URL fields are where misconfiguration causes the most user-visible damage. A wrong gateway URL means requests do not reach Agent Router; a wrong Console URL means developers cannot find the surface they need; a wrong Admin URL means the operator team can be locked out of their own instance. 1. Confirm the **Gateway URL** matches what is documented in the team's integration guides. This is the URL that every developer-side integration (the `base_url` in their SDK calls) has to point at. 1. Confirm the **Console URL** is reachable from the network where developers actually work. A Console URL that resolves only inside a corporate VPN is a usability problem if developers are expected to use the Console from home. 1. Confirm the **Admin URL** is reachable from where operators actually work. The same VPN consideration applies. These three URLs are the most-shared facts about an instance. Including them on the team's internal wiki, in Agent Router's runbook, and in the Console Dashboard itself (where the Gateway URL is also surfaced) keeps the misconfiguration-driven incident rate low. ## Step 4: review the enabled feature set Different deployment tiers, contracts, and customer commitments expose different features. The Enabled Features section is the authoritative list of what is active on this instance. Typical entries include: - **MCP Profile support.** The MCP catalogue and profile mechanism described in [Govern MCP Server Access](/agent-router-enterprise/guides/operate-and-govern/govern-mcp-server-access/). - **Bring Your Own Key (BYOK).** The developer-side BYOK surface described in [Use Your Own Provider Credentials](/agent-router-service/guides/use-your-own-provider-credentials/). - **OpenTelemetry export.** The telemetry export described in [Export Telemetry to an Observability Stack](/agent-router-enterprise/guides/observability-and-analytics/export-telemetry-to-an-observability-stack/). - **Usage analytics.** The analytics surface described in [Audit Agent Router Activity](/agent-router-enterprise/guides/operate-and-govern/audit-platform-activity/). Features not listed are not available on this instance, either because the deployment tier does not include them or because they have been disabled deliberately. Attempts to use a non-enabled feature surface as either a missing screen, an inactive control, or a clear "not available on this instance" message, not as a generic error. If a feature that the team expects is missing from the list, the resolution is typically a contractual or deployment-tier change rather than a configuration toggle. ## Step 5: apply multi-instance discipline For single-instance deployments, the work ends at Step 4. For multi-instance deployments, the disciplines that follow are what keep the fleet coherent. ### Configuration parity for environments Production, staging, and development instances should match each other in configuration shape, even if they differ in scale or upstream credentials. A change made in staging should be reflected in production once it is verified, and vice versa. The Instances surface itself does not enforce parity; the operator team's process does. A typical pattern is to keep a written checklist of configuration that is supposed to match across environments (provider list, enabled feature set, SSO configuration shape, MCP catalogue) and to run it as part of any cross-environment change. Drift is most often discovered at the worst possible moment, during an incident, so periodic checks pay for themselves. ### Configuration independence for separate tenants For instances that belong to different business units, different regions, or different regulatory boundaries, the goal is the opposite: keep them independent. Each instance has its own Admin Dashboard, its own user records, its own credentials, its own audit log, and its own data plane. Cross-instance leakage of any of these is a serious problem, not an inconvenience. Two disciplines help here: - **Naming.** An instance name that includes the tenant identifier (a region, a business unit, an environment) makes cross-instance confusion in tooling, logs, and team conversations dramatically less likely. - **Access.** Operator access to each instance should reflect the boundary the instance represents. A `super_admin` on the EU instance should not automatically be a `super_admin` on the US instance; SSO and role mapping make this easy to enforce if both instances point at the same IdP, but the role-claim mapping has to be configured per instance. ### Version coherence The Version field captures the current software version. In Enterprise deployments, the operator controls when each instance upgrades; in SaaS, Tetrate controls the schedule. For multi-instance fleets, the typical pattern is to upgrade staging first, run for a few days to confirm nothing has surprised the team, then promote to production. The Version field across instances makes the current state of the rollout immediately visible: staging on the new version, production still on the old one, both visible on their respective Instances screens. ### Audit segregation Each instance has its own audit logs and usage analytics, both described in [Audit Agent Router Activity](/agent-router-enterprise/guides/operate-and-govern/audit-platform-activity/). The two surfaces do not span instances; an event recorded on the US instance does not appear in the EU instance's log, and vice versa. For investigations that span instances, the operator team has to either correlate manually across both surfaces or, more sustainably, export both audit streams to a centralised SIEM via the OpenTelemetry path described in the developer-side observability guide. ### Field reference The Instances surface displays the following configuration for each instance: | Field | Description | | :---- | :---- | | Instance Name | The identifier for this deployment instance | | Deployment Mode | SaaS (both planes hosted by Tetrate) or Enterprise (self-hosted data plane in customer Kubernetes; management plane Tetrate-hosted) | | Gateway URL | The base URL of the AI gateway proxy endpoint | | Console URL | The URL where developers access the Console | | Admin URL | The URL where operators access the Admin Dashboard | | Enabled Features | Agent Router capabilities active for this instance | | Version | The current software version of the deployment | Three routine tasks operate against these fields: - **View instance configuration.** Open the Instances surface from Settings and review the deployment details (all URLs, the deployment mode, and the enabled-feature list) to confirm the instance is configured as expected. - **Verify gateway URLs.** Check the Gateway URL to confirm the proxy endpoint that developers and API clients send requests to, then verify it matches the URL documented in the team's integration guides and surfaced in the Console Dashboard. - **Review enabled features.** Check the Enabled Features section against Agent Router capabilities expected for the deployment tier. Capabilities not listed are unavailable for the tier or have been disabled. ## What to do next - **Audit Agent Router activity.** The audit posture differs per instance; multi-instance audit work is the natural next step. See [Audit Agent Router Activity](/agent-router-enterprise/guides/operate-and-govern/audit-platform-activity/). - **Configure SSO.** SSO is configured per instance, with the considerations described here. See [Configure SSO](/agent-router-enterprise/guides/identity-and-access-management/single-sign-on/sso-configuration-guide/). - **Working with budgets.** Budgets are also per-instance and need to be designed in step with how traffic actually distributes across the fleet. See [Choose the right cost control for each workload](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/choose-the-right-cost-control/). Where to go next Audit Agent Router activity The audit posture differs per instance; multi-instance audit work is the natural next step. Configure SSO SSO is configured per instance, with the considerations described here. --- ## About guardrails Enterprise Tier The Tetrate Agent Router gateway inspects large language model (LLM) requests and responses in flight. Guardrails decide what is allowed to pass. A guardrail is a collection of rules; each rule performs one check and is executed by a provider. Providers score the content, and the gateway decides what to do with the score. This page explains that model: what each object is, how the pieces connect, and how a single rule is evaluated. For how guardrails are created and rolled out, see Configure guardrails. For field-level lookup, including every setting, all 23 check types, the five provider kinds, and the built-in templates, see the guardrails reference. For where guardrail evaluation sits on the request path, see Guardrails architecture. The five objects Guardrail A named policy: a collection of rules plus shared settings (failure mode, evaluation timeout, blocked response). Rule One check type plus an action, run by a chosen provider against input, output, or both. Provider The engine that executes a check and returns a score. Each provider supports only some check types. Check type A name from a fixed vocabulary (pii for personally identifiable information, toxicity, jailbreak, and others): what a rule looks for. Pattern A named, reusable regular expression in a shared catalog that rules reference by name. ## What guardrails are The Guardrails module sits in the request path. It buffers the request body, understands the provider format (Anthropic Messages, AWS Bedrock, OpenAI Chat and Responses), and parses it into a conversation. Before that request reaches the model, and again when the response comes back, it runs guardrails over the content. Guardrails inspect LLM traffic. Standalone Model Context Protocol (MCP) profile requests are also proxied by the gateway but do not pass through content guardrails: they are governed by identity, access control, and audit instead. Buffering has one consequence worth knowing before an output rule is enforced. An enforcing rule on the output stage holds the model's response until evaluation completes, which de-streams the response. Rules in monitor mode are evaluated asynchronously and leave streaming intact. The latency implications are covered in [Guardrails best practices and FAQ](/reference/best-practices/guardrails-best-practices-and-faq/). Every guardrail evaluation ends in one of three enforcement outcomes: - **`block`:** reject the request or response outright, returning HTTP `403` with a message. - **`redact`:** rewrite the content to mask the offending spans, then let it continue. - **`monitor`:** record what would have happened, but let the content pass unchanged. A guardrail is the policy object being configured. On its own it enforces nothing: enforcement comes from the rules it contains. A guardrail groups rules that share a few policy-level settings, most importantly the **failure mode**. If a rule cannot be evaluated because a provider is down, fail-close treats it as blocked and fail-open lets it pass. Scope: a project-level policy. Guardrails are bound to a project, not to the organisation. The Admin Dashboard's top-left selector switches between an Organisation scope and individual projects (for example default and mvp); the Rules, Patterns, and Guardrail Providers pages all operate inside the currently selected project. A guardrail authored under one project does not apply to another: each project carries its own set of policies. The Admin Dashboard groups guardrails into four project-scoped surfaces: **Overview** reports trigger activity across the project, **Rules** holds the guardrails and their rules, **Patterns** is the reusable regular expression catalog, and **Guardrail Providers** holds the configured detection engines. ![The project selector open, showing an Organisation entry above a Projects section listing the default and mvp projects](/img/screenshots/guides/operate-and-govern/safety-and-data-protection/02-project-scope-selector.png) *The top-left selector: an Organisation scope sits above the Projects list. Guardrail policies live under a project, not organisation-wide, so switching projects manages a different set.* ## Creating a guardrail Two entry points exist on the Rules page, both scoped to the current project. - **Add guardrail (start empty).** The policy is named, its type, category, severity, failure mode, evaluation timeout, and blocked-response message are set, and rules are then added one at a time. This path gives control over every field. - **Create from template (start curated).** A ready-made guardrail is cloned from a catalog with its rules, providers, and thresholds already wired, then tailored. This path reaches a baseline for a known regime faster. Selecting a template clones a fully formed guardrail into the project, where it is then edited like any other policy. Every field of the clone remains editable, and no configuration is hidden. The catalog is grouped by domain and filterable by type (Security, Compliance, Policy, Custom, Evaluator) and severity; the 14 templates it ships, covering data privacy, healthcare, regulatory compliance, security, and content policy, are listed in the [guardrails reference](/reference/guardrails/#built-in-guardrail-templates). The creation procedure itself, including validation in monitor mode before enforcement, is covered in [Configure guardrails](/agent-router-enterprise/guides/operate-and-govern/safety-and-data-protection/configure-custom-guardrails/). Providers have templates too. The Guardrail Providers page carries a provider catalog of pre-built configurations to clone when bootstrapping an instance, alongside "Your Providers": the concrete, operator-configured instances that rules reference by name. ## How the pieces connect ```mermaid flowchart LR G["Guardrail(policy)"] --> R["Rules[]"] R --> C["Rule:checkType, action, mode,providerId, configuration, threshold"] C --> P[Provider] P -->|"score 0.0 to 1.0"| D{"score ≥ threshold?"} D -->|yes| A["block / redact / monitor"] D -->|no| N["Rule passes"] style D fill:#ff5500,color:#fff ``` Three constraints govern how the objects relate: - **The check-type vocabulary is fixed and owned by the system.** A rule names one of 23 check types, all of which are listed in the [guardrails reference](/reference/guardrails/#check-types). - **A provider declares which check types it can execute.** A rule is valid only if its provider supports its check type, verified when configuration loads rather than at request time. This is why the rule form offers only the checks the selected provider can run. - **A rule carries a configuration blob and a threshold.** The configuration holds the check's parameters. The provider scores; the gateway compares `score ≥ threshold` to decide whether the rule was violated, then applies the rule's action. The clean split: providers score, the gateway decides. A provider never chooses to block or redact and never compares against a threshold. It only returns a normalised score between 0.0 and 1.0 and, for redaction, a sanitised copy of the text. All policy (thresholds, actions, fail-open and fail-close, chaining redactions between providers) lives in the gateway. This is why the same provider can serve many rules at many thresholds at once. ## Rules A rule is the unit of enforcement. It answers five questions: - **What to look for.** The **Check type**, which the rule's provider must support. - **Where to look.** **Run on** selects the input stage, the output stage, or both. Which kind of content an evaluation ran against (text, image, document, tool call, thinking, or context) is an attribute of the evaluation rather than a setting on the rule, and appears on the resulting trigger record. - **Who evaluates it.** The **Provider** instance, referenced by its UUID. - **How sensitive to be.** The **Threshold**, between 0.0 and 1.0. - **What to do on a hit.** The **Action** (`block` or `redact`) and the **Mode** (`enforce` applies the action, `monitor` records the would-be action without applying it). The parameters of the check itself live in the rule's configuration, whose shape depends on the check type and the provider. Every field, with its permitted values, is tabulated in the [guardrails reference](/reference/guardrails/#rule-level-settings). ### The rule form changes with the provider and check type A rule's configuration is not fixed. Two dropdowns at the top of the rule editor reshape everything beneath them. 1. **Choosing a provider filters the check-type list.** Selecting Tetrate Guardrails narrows the menu to exactly the checks that provider supports (PII, toxicity, jailbreak, banned topics, relevance, sentiment, and others), never one it cannot run. The dropdown reflects the same validation applied when configuration loads. 1. **The provider and check-type pair decides the Configuration fields.** The rest of the rule stays constant: Run on (input, output, or both), Action (block or redact), Mode (enforce or monitor), an "Edit as JSON (advanced)" escape hatch, and an optional triggered response. The body of the form differs every time. | Provider | Check type | Configuration the form shows | | :---- | :---- | :---- | | `regexp` | any (for example PII) | **Patterns** (one or more, from the pattern library) and **Redaction format**, which supports a `{pattern_name}` placeholder such as `{ssn}_REDACTED`. | | `builtin` | PII or category | **Category name**, **Minimum severity**, **Keywords** (each with a severity), **Blocked keywords**, **Target subjects**, **Flagged terms**, **Allowlist phrases**, and **Detection patterns** (inline regular expressions, or entries from the library). | | `tetrate` | PII | Nothing to configure: detection is handled entirely by the provider's built-in machine-learning (ML) model. | | `tetrate` | `banned_topics` | **Banned topics** (a list, matched semantically rather than as literal strings) and **Similarity threshold** (0.0 to 1.0, or the provider default). | ![The Add Rule form with Provider set to regexp and Check type set to PII, where the Configuration section shows a Patterns picker and a Redaction format field](/img/screenshots/guides/operate-and-govern/safety-and-data-protection/04-add-rule-regexp.png) *`regexp` with PII: the configuration is a pattern picker plus a redaction format.* ![The Add Rule form with Provider set to builtin, where the Configuration section shows Category Name, Minimum Severity, Keywords, Blocked Keywords, Target Subjects, Flagged Terms, Allowlist Phrases, and Detection Patterns](/img/screenshots/guides/operate-and-govern/safety-and-data-protection/05-add-rule-builtin.png) *`builtin`: the same rule shell, but a rich multi-field configuration of keywords, blocked keywords, target subjects and flagged terms, allowlist phrases, and detection patterns.* ![The Add Rule form with Provider set to Tetrate Guardrails and the Check type dropdown open, listing Banned Substrings, Banned Topics, Bias, Code, Gibberish, Hate Speech, Jailbreak, Language, PII, Prompt Injection, Regexp, Relevance, Secrets, Self Harm, Sentiment, Sexual Content, Toxicity, and Violence](/img/screenshots/guides/operate-and-govern/safety-and-data-protection/06-add-rule-tetrate-checktypes.png) *`tetrate`: the check-type dropdown is scoped to what this provider supports (its ML checks). The Admin Dashboard does not allow a provider to be paired with a check it cannot run.* The check type names what to look for, the provider decides how, and together they determine which controls the form shows. All of it is stored as the rule's `configuration` blob and forwarded verbatim to the provider as that check's parameters. ### How a single rule is evaluated 1. **Route to the provider.** The gateway groups every active rule by its `providerId` and hands each provider all of its checks in one call. 1. **The provider scores the content.** For each check it returns a normalised score between 0.0 and 1.0, plus a sanitised copy of the text if, and only if, it transformed anything. 1. **The gateway compares to the threshold.** A score at or above the threshold is a violation. Below the threshold, the rule passes. 1. **The action is applied.** `block` returns immediately. `redact` swaps in the sanitised text and re-runs any providers with rules still pending against the new content. `monitor` logs and continues. On failure. If a provider errors, the guardrail's failure mode decides: fail-close (the default) treats the rule as blocked, and fail-open skips it. Providers never retry, because deterministic fail-open and fail-close behaviour is the gateway's job. ## Patterns A pattern is a named regular expression, for example `us-ssn` mapped to `\b\d{3}-\d{2}-\d{4}\b`. Patterns live in a catalog shipped to the provider as `{ patterns: [ { name, pattern } ] }`. Rules do not have to embed regular expressions; they can point at a catalog entry by name, so one canonical definition of a social security number or an API key is reused across many rules and stays consistent. Two consumers reference the catalog: - The `regexp` and `tetrate` providers take a rule parameter `patternNames` (a list of catalog names) plus an optional `redactionFormat`. - The `builtin` provider's detection patterns can each set a `patternName` that resolves to a catalog regular expression instead of an inline `pattern`. The catalog tolerates bad input: a single malformed entry is logged and skipped rather than taking the whole provider offline, and a rule referencing a name missing from the catalog simply drops that reference, which is also logged. Catalog regular expressions are compiled as authored, carrying their own flags, whereas regular expressions written inline on a rule are compiled case-insensitively. ![The Patterns page: a table of built-in named regular expression patterns such as uae_phone, sg_nric, self_harm_suicide, br_cpf, and fr_nir, with display name, category, and type Builtin](/img/screenshots/guides/operate-and-govern/safety-and-data-protection/08-patterns-library.png) *The Patterns library ships more than 80 built-in named patterns grouped by category (EU, Singapore, Brazilian, and UAE PII; Protected Class for Fair Lending; Dangerous Content for Crisis). Entries can be filtered to built-in only, searched by name, or created with New pattern. These are the entries a rule points at by name.* ## Guardrail providers Providers are the engines that execute checks. A provider is an instance with its own UUID, and a rule points at that instance rather than at a kind, so several instances of the same kind can coexist in a project. Three kinds ship with Agent Router, ordered below by cost and capability; two further kinds, Azure AI Content Safety and a generic external service, are operator-configured integrations described in the [guardrails reference](/reference/guardrails/#detection-engines-providers). All of them implement the same contract, so a rule looks identical regardless of which one runs it: only the provider reference changes. What differs is how they detect and which check types they can execute. | Provider (`kind`) | Tier | How it detects | Latency | | :---- | :---- | :---- | :---- | | **regexp** | local | Pure pattern matching against the named-pattern catalog. | Near zero, in process | | **builtin** | local | A multi-layer keyword-and-regex detection pipeline. | Near zero, in process | | **tetrate** | ML service | Embeddings, classifiers, and natural language inference (NLI) entailment via Tetrate's guardrails engine. | Network, remote inference | ### The regexp provider The `regexp` provider resolves the `patternNames` on a rule against the catalog and tests them against the content. A match scores `1.0` and no match scores `0.0`. If a `redactionFormat` is set, matched spans are rewritten for redact rules. `regexp` advertises support for every check type. Because detection is only pattern matching, the check type is a label describing what is being detected, and the behaviour is the same whether the rule is tagged `pii` or `secrets`. The referenced patterns define the check. **Provider config:** `patterns` (the named-regex catalog). **Per-rule parameters:** `patternNames`, `redactionFormat`. ### The builtin provider `builtin` is the most configurable local provider. For a given rule it runs a fixed five-layer pipeline over the content. Unlike a stop-at-first-hit filter, every layer runs and every matched span is redacted into a single sanitised output. A match anywhere scores `1.0`. As with `regexp`, the check type is mostly a label: the same pipeline powers `toxicity`, `banned_topics`, `secrets`, and the rest. What distinguishes one rule from another is its parameters. | Property | What it means | | :---- | :---- | | `keywords` | Category keywords, each tagged with a severity (high, medium, or low). Matches only if severity is at or above the rule's `minSeverity`. The general keyword layer. | | `blockedKeywords` | Zero-tolerance keywords. Always matched and redacted, severity ignored, and never bypassed by the allowlist. | | `targetSubjects` and `flaggedTerms` | The conditional layer. A subject and a flagged term only trigger when they co-occur in the same sentence, producing far fewer false positives than a bare keyword. | | `allowlistPhrases` | An escape hatch. If any allowlist phrase appears, the conditional, pattern, and keyword layers are suppressed. Blocked keywords still fire. | | `detectionPatterns` | Regular expressions: each is an inline `pattern` or a catalog `patternName`, plus per-pattern exceptions and a severity band. | | `minSeverity` | The severity floor for the keyword and detection-pattern layers. | #### The five layers in evaluation order 1. **Blocked keywords (zero tolerance):** checked first and redacted immediately. The allowlist cannot suppress this layer. 1. **Allowlist (gate):** if a phrase is present, layers 3 to 5 are skipped for this content. 1. **Conditional:** for each sentence, if a `targetSubject` and a `flaggedTerm` both appear, both are redacted. 1. **Detection patterns:** inline or catalog regular expressions match the working text, honouring exceptions and the severity floor. 1. **Category keywords:** keywords at or above `minSeverity` are matched and redacted. Why subject and term matters. With targetSubjects: [customer] and flaggedTerms: [password], the rule only triggers when both concepts land in the same sentence. "Please reset the customer's password before the demo." fires. "I changed my password to something stronger." is ignored. The lone mention of "password" never fires, which is the false-positive suppression a plain keyword list cannot provide. `builtin` also has one specialised check that skips the generic pipeline: `ban_competitors`. It takes `brandSelf` and a list of `competitors`, matches competitor names (word-boundary matching for single words, phrase matching for multi-word names), and ignores any competitor mention that overlaps the configured brand, so the brand "Acme Cloud" does not trip an "Acme" competitor rule. **Provider config:** `patterns` (optional catalog, referenced by `detectionPatterns[].patternName`). **Per-rule parameters:** `keywords`, `blockedKeywords`, `targetSubjects`, `flaggedTerms`, `allowlistPhrases`, `detectionPatterns`, `minSeverity`, `failOpen`, and for `ban_competitors`: `brandSelf`, `competitors`. ### The tetrate provider The `tetrate` provider, backed by Tetrate's semantic router, detects meaning rather than literal strings. It routes each check to the right model behind a single endpoint: - **Embedding and signal evaluation** for toxicity, hate speech, violence, self-harm, sexual content, language, secrets, banned substrings, and regex signals. - **A security classifier** (prompt-guard) for jailbreak and prompt injection. - **A PII model** that returns detected entities and a masked copy of the text. - **NLI entailment** (an auxiliary service) for banned topics, sentiment, bias, code, gibberish, and relevance. Some checks cascade across layers. Jailbreak first tries the fast security classifier, then falls back to NLI entailment to catch roleplay-style attacks such as "developer mode". Prompt injection cascades three ways: configured `known_patterns` matched locally, then the ML classifier, then an NLI fallback, short-circuiting at the first layer that fires. In each case the provider returns a score, and the threshold decision stays with the gateway. Because several checks call a remote model, the provider needs an `endpoint` (required) and a `timeout` (required). Checks backed by NLI or the classifier additionally need an `auxEndpoint`; without it, those specific checks return an error rather than silently passing. **Provider config:** `endpoint` (required), `auxEndpoint`, `timeout` (required), `patterns` (catalog). **Representative per-rule parameters:** `banned_topics` takes `topics`; `language` takes `allowed_languages` and `block_unknown`; `banned_substrings` takes `substrings` and `case_sensitive`; `secrets` takes `secret_types`, `custom_patterns`, and `entropy_analysis`; `prompt_injection` takes `known_patterns` and `ml_detection`; `toxicity` and `bias` take `categories`; `regexp` takes `patternNames` and `redactionFormat`. ## Which provider handles which check type A rule is valid only if its provider supports its check type, enforced when configuration loads and mirrored in the UI by the provider-scoped check-type dropdown. The full matrix, covering all 23 check types and every provider kind, is in the [guardrails reference](/reference/guardrails/#provider-support-by-check-type). Four properties of that matrix are worth carrying into it: - `regexp` accepts every check type, but only because a regular expression is check-type-agnostic. The check type is a label, and the referenced patterns are the real definition. - `builtin` covers the keyword-shaped and regex-shaped checks. It does not implement the three inference-backed checks (`gibberish`, `relevance`, `sentiment`) or `factual_consistency`. - `tetrate` is the only shipped provider that reads meaning, and the inference-backed checks are exclusively its own. It does not implement the four checks that are keyword-shaped or policy-shaped by nature: `compliance`, `ban_competitors`, `ban_code`, and `protected_material`. - `factual_consistency` is implemented by none of the three shipped providers, other than nominally by `regexp`. Groundedness checking requires an operator-configured provider that implements it, such as Azure AI Content Safety. Where to go next Configure guardrails Create project policies and rules, and validate in monitor mode before enforcing. Configure guardrail providers Choose regexp, builtin, or tetrate for each check type. Guardrails reference Every setting, all 23 check types, the five provider kinds, and the built-in templates. Protect requests with guardrails How developers see blocks and redactions in application code. Guardrails on shared data planes How enforcement stays per-project when one data plane serves several projects. Guardrails architecture Where guardrail evaluation sits on the request path through the gateway. Guardrails API reference Request and response detail for the guardrail and rule management API. --- ## Configure guardrails Enterprise Tier A guardrail is a named policy for a project: a collection of rules plus shared settings such as failure mode, evaluation timeout, and a blocked-response message. Each rule names a check type (what to look for), a provider (which engine scores the content), an action (block or redact), and a mode (enforce or monitor). Providers return a normalised score; the gateway compares that score to the rule's threshold and applies the action. This guide covers creating guardrails from scratch or from a template, adding rules, using the Patterns library, and validating with monitor mode before enforcement. **Persona:** Platform operator working in the Admin Dashboard, often alongside security and compliance stakeholders who own the underlying policy. **Estimated time:** 20 to 40 minutes for a first guardrail, including time spent validating in monitor mode; less thereafter once the pattern is familiar. ## When this guide applies | Situation | What it covers | | :---- | :---- | | Creating a project baseline safety policy | Add guardrail, or clone from the template catalog | | Redacting PII or secrets before traffic reaches a model | A redact rule with the `regexp`, `builtin`, or `tetrate` provider | | Blocking jailbreaks or prompt injection | A block rule with `jailbreak` or `prompt_injection` | | Piloting a new rule without disrupting live traffic | Monitor mode before promote to enforce | | Reusing a canonical regex across many rules | The Patterns library | For how the scoring engines differ and which check types each supports, see [Configure guardrail providers](/agent-router-enterprise/guides/operate-and-govern/safety-and-data-protection/configure-vendor-guardrails/). For the developer-side view of blocked and redacted traffic, see [Protect requests with guardrails](/agent-router-enterprise/guides/operate-and-govern/safety-and-data-protection/protect-requests-with-guardrails/). ## Outcomes By the end of this guide: - At least one guardrail exists in the selected project with type, category, severity, failure mode, and timeout set. - At least one rule is attached with a supported check type, provider, action, and mode. - The guardrail has been validated in monitor mode against live traffic before enforcement. - Project scope, patterns, and the template catalog are understood. ## Prerequisites - Administrator access to the Admin Dashboard with permission to manage guardrails for a project. - An active project selected. Guardrails are bound to a project, not the organisation. - A written statement of the policy the guardrail is meant to enforce. - For pattern-based rules: the specific terms or catalog pattern names to match, ideally reviewed with the policy owner. ## Enabling guardrails By default, new guardrails are NOT enabled. If you wish to use a newly created guardrail, make sure to enable it. ## Step 1: select the project Guardrails, patterns, and guardrail providers all operate inside the currently selected project. A guardrail authored under one project does not apply to another. 1. Sign in to the Admin Dashboard. 1. Use the top-left selector to choose the project (not Organisation scope). ![Project selector open: Organisation above the Projects list, with Default selected](/img/screenshots/guides/operate-and-govern/safety-and-data-protection/02-project-scope-selector.png) ## Step 2: open Rules and choose how to create the guardrail 1. Open **Guardrails → Rules**. ![Rules page with a guardrail expanded to show nested rules (provider, check, action, mode)](/img/screenshots/guides/operate-and-govern/safety-and-data-protection/01-rules-guardrails-list.png) Two entry points create a guardrail in the current project: - **Add guardrail**: start empty. Name the policy, set type, category, severity, failure mode, evaluation timeout, and blocked-response message, then add rules one at a time. - **Create from template**: clone a curated guardrail from the catalog (rules already wired) and tailor it. ![Create from template catalog grouped by domain, with type and severity filters](/img/screenshots/guides/operate-and-govern/safety-and-data-protection/03-create-from-template.png) Templates are grouped by domain (for example data privacy, healthcare, and compliance) and filterable by type and severity. Cloning copies a fully formed guardrail into the project; edit it afterward like any other policy. ## Step 3: set guardrail-level settings When adding a guardrail from scratch, supply: | Setting | Meaning | | :---- | :---- | | Name | Display name; use a phrase that states the policy | | Type | Broad purpose (for example Security, Compliance, Policy, Custom) | | Category | Domain context (for example data privacy, healthcare, government, custom) | | Severity | Risk level of violations this policy catches | | Failure mode | If a provider cannot evaluate a rule: **fail-close** (default) treats it as blocked; **fail-open** lets traffic pass | | Evaluation timeout (ms) | Maximum time to wait for evaluation before the failure mode applies | | Triggered response | Optional message returned when a block fires | ![New Guardrail form with Name, Description, Type, Category, Severity, Failure mode, Evaluation timeout in milliseconds, and Triggered response](/img/screenshots/guides/operate-and-govern/safety-and-data-protection/10-new-guardrail-form.png) Creating the guardrail does not by itself change live traffic until rules are active and set to enforce mode. ## Step 4: add a rule A rule is the unit of enforcement. It answers what to look for, where to look, who evaluates it, and what to do on a hit. | Field | Meaning | | :---- | :---- | | Name / description | Optional labels for the rule | | Active | Whether the rule is evaluated | | Provider | Engine that scores the check (`regexp`, `builtin`, or `tetrate`, or another configured provider) | | Check type | Canonical safety check (for example `pii`, `toxicity`, `jailbreak`, `banned_topics`) | | Run on | Input, output, or both. Leaving it unset means **both**, so a check the provider supports on only one stage needs the stage set explicitly | | Action | `block` or `redact`: what enforcement would do | | Mode | `enforce` applies the action; `monitor` records the would-be action without applying it | | Configuration | Structured fields for the selected provider and check-type pair, with an **Edit as JSON (advanced)** panel underneath for the stored form | | Triggered response | Optional message shown when this rule fires | Provider and check type are chosen first because they determine everything below them. The check-type list is scoped to the selected provider, so an unsupported pair cannot be created; the full matrix of provider, check type, and stage is in the [guardrails reference](/reference/guardrails/#provider-support-by-check-type), and an unsupported combination submitted directly to the API is rejected at save time with an error naming the supported set. Changing either dropdown replaces the Configuration section with the fields that pair needs, and switching provider re-seeds it: `regexp` starts as `{"regex":{"patternNames":[]}}`, `tetrate` as `{}`, and `builtin` opens its structured form. ![Add Rule form with Provider regexp and Check type Pii, showing a Patterns picker and a Redaction format field](/img/screenshots/guides/operate-and-govern/safety-and-data-protection/04-add-rule-regexp.png) *`regexp` with `pii`: the configuration is a pattern picker plus a redaction format, which accepts the `{pattern_name}` placeholder.* ![Add Rule form with Provider builtin, showing Category Name, Minimum Severity, Keywords, Blocked Keywords, Target Subjects, Flagged Terms, Allowlist Phrases, and Detection Patterns](/img/screenshots/guides/operate-and-govern/safety-and-data-protection/05-add-rule-builtin.png) *`builtin`: the same rule shell with a multi-field configuration covering each layer of its detection pipeline.* ![Add Rule form with Provider Tetrate Guardrails and the Check type dropdown open](/img/screenshots/guides/operate-and-govern/safety-and-data-protection/06-add-rule-tetrate-checktypes.png) *`tetrate`: the check-type dropdown is scoped to the ML checks this provider can run.* ![Add Rule form with Provider Tetrate Guardrails and Check type Banned Topics, showing a Banned topics list and a Similarity threshold field](/img/screenshots/guides/operate-and-govern/safety-and-data-protection/07-add-rule-tetrate-banned-topics.png) *`tetrate` with `banned_topics`: a list of topics matched semantically rather than literally, plus a similarity threshold.* Whatever the form shows is stored as the rule's configuration and forwarded to the provider as that check's parameters. The **Edit as JSON (advanced)** panel exposes that stored object for cases the fields do not cover. **Providers score; the gateway decides.** A provider never chooses to block or redact on its own: it returns a normalised score, and for redaction a sanitised copy of the text. The gateway applies the action when mode is enforce. A block returns an error code. A redact swaps in sanitised text and may re-run providers with rules still pending against the new content. Monitor mode logs the would-be action and lets the content pass unchanged. ## Step 5: use the Patterns library A pattern is a named regular expression (for example a national identifier format) stored in a shared catalog. Rules can reference patterns by name instead of embedding regexes, so one canonical definition is reused across many rules. ![Patterns library of built-in named regex entries](/img/screenshots/guides/operate-and-govern/safety-and-data-protection/08-patterns-library.png) The library ships many built-in entries grouped by category (82 in a typical deployment). Filter to built-in only, search by name, or add a custom pattern. A rule on the `regexp` provider selects patterns from this library in its **Patterns** field; the `builtin` provider's detection patterns can each name a catalog entry instead of carrying an inline regex. ## Step 6: validate in monitor mode before enforcing There is no separate test-content surface that dry-runs a rule against pasted samples. Validation is monitor mode against live traffic: 1. Create or edit the rule with the intended action (`block` or `redact`) and set **mode** to **monitor**. 1. Allow representative traffic through the project. 1. Review whether the rule would have fired at an acceptable rate (false positives and misses). 1. Only then switch mode to **enforce**. Fail-close remains the default when a provider errors; set fail-open deliberately on paths where availability must dominate safety. See [Guardrails best practices and FAQ](/reference/best-practices/guardrails-best-practices-and-faq/). ## How evaluation works for one rule 1. The gateway groups active rules by provider and hands each provider its checks. 1. The provider scores the content and may return a sanitised copy if it transformed anything. 1. The gateway compares the score to the threshold. 1. On a violation: enforce applies block or redact; monitor records the would-be action and continues. If a provider errors, the guardrail's failure mode decides: fail-close treats the rule as blocked; fail-open skips it. ## What to do next - **[Configure guardrail providers](/agent-router-enterprise/guides/operate-and-govern/safety-and-data-protection/configure-vendor-guardrails/):** the three engines, the builtin pipeline, and the check-type matrix. - **[Detect and redact sensitive data](/agent-router-enterprise/guides/operate-and-govern/safety-and-data-protection/detect-and-redact-sensitive-data/):** PII and secrets workflows built on these rules. - **[Detect and block prompt injection](/agent-router-enterprise/guides/operate-and-govern/safety-and-data-protection/detect-and-block-prompt-injection/):** jailbreak and injection checks. - **[Protect requests with guardrails](/agent-router-enterprise/guides/operate-and-govern/safety-and-data-protection/protect-requests-with-guardrails/):** developer-side handling of blocks and redactions. - **[Guardrails best practices and FAQ](/reference/best-practices/guardrails-best-practices-and-faq/):** rollout and investigation. Where to go next Configure guardrail providers Choose regexp, builtin, or tetrate for each rule. Protect requests with guardrails How developers see blocks and redactions. --- ## Configure guardrail providers Enterprise Tier A guardrail is a named policy: a collection of rules plus shared settings such as failure mode and a blocked-response message. Each rule names a check type (what to look for) and a provider (the engine that scores the content). Providers return a normalised score between 0.0 and 1.0; the gateway compares that score to the rule's threshold and applies the rule's action. This guide covers the three providers that ship with Tetrate Agent Router, how they differ, and which check types each one can run. Creating guardrails and rules is covered in [Configure guardrails](/agent-router-enterprise/guides/operate-and-govern/safety-and-data-protection/configure-custom-guardrails/). **Persona:** Platform operator working in the Admin Dashboard. **Estimated time:** 15 to 25 minutes to review the provider ladder and confirm which engines are enabled for the project. ## When this guide applies | Situation | What it covers | | :---- | :---- | | Choosing which engine should score a new rule | Comparing `regexp`, `builtin`, and `tetrate` on latency and capability | | Understanding why a check type is missing from the rule form | The provider-scoped check-type list | | Tuning a keyword or pattern-based detection | The choices that matter on the `builtin` provider | | Wiring the ML provider for jailbreak, toxicity, or semantic topics | `tetrate` endpoint, timeout, and auxiliary endpoint requirements | | Confirming a rule can use a given check type | Where the provider-by-check-type support matrix lives, and the cases that surprise | ## Outcomes By the end of this guide: - The three shipped provider kinds and their place on the cost-and-capability ladder are clear. - When to use subject-plus-term matching instead of a bare keyword list is clear. - The required configuration for the `tetrate` provider is known. - A rule is only created with a check type its provider supports. ## Prerequisites - Administrator access to the Admin Dashboard with permission to manage guardrails for a project. - An active project selected in the Admin Dashboard. Guardrail providers are project-scoped; Rules, Patterns, and Guardrail Providers all operate inside the currently selected project. - Familiarity with how guardrails and rules are created, covered in [Configure guardrails](/agent-router-enterprise/guides/operate-and-govern/safety-and-data-protection/configure-custom-guardrails/). ## Step 1: open Guardrail Providers for the project 1. Sign in to the Admin Dashboard and select the project whose providers are to be managed. 1. Open **Guardrails → Guardrail Providers**. ![Guardrail Providers page listing regexp, builtin, and Tetrate Guardrails instances](/img/screenshots/guides/operate-and-govern/safety-and-data-protection/09-guardrail-providers.png) Providers are **instances** with UUIDs. Rules reference a provider by its id. The page may also show a **Provider catalog** of pre-built configurations that can be cloned; **Your Providers** lists the concrete instances rules can use. Three kinds ship by default: `regexp`, `builtin`, and `tetrate` (displayed as Tetrate Guardrails). Where an external guardrail service is configured in addition to these kinds, it appears as another provider instance under the same page. This guide focuses on the three that ship in the box. ## Step 2: choose a provider on the cost-and-capability ladder All three kinds implement the same contract: they score content and, for redaction, may return a sanitised copy of the text. They never choose to block or redact themselves, because that decision stays with the gateway. What differs is *how* they detect and *which check types* they can execute. | Provider (`kind`) | Tier | How it detects | Latency | | :---- | :---- | :---- | :---- | | **regexp** | Local | Pure pattern matching against the named-pattern catalog | Near zero; in-process | | **builtin** | Local | A multi-layer keyword-and-regex detection pipeline | Near zero; in-process | | **tetrate** | ML service | Embeddings, classifiers, and NLI entailment via Tetrate's semantic router | Network hop; remote inference | Stack rules so cheaper providers evaluate first when the same content can be caught by a local check. A pattern match can short-circuit evaluation before a remote ML call runs. Operational ordering guidance is in [Guardrails best practices and FAQ](/reference/best-practices/guardrails-best-practices-and-faq/). ## Step 3: configure a regexp provider The `regexp` provider tests named patterns from the Patterns catalog against the content, scoring `1.0` on a match and `0.0` otherwise. Choose it when the content has a predictable shape (email addresses, national identifiers, API key formats) and in-process latency matters. The mechanism is described in [About guardrails](/agent-router-enterprise/guides/operate-and-govern/safety-and-data-protection/about-guardrails/#the-regexp-provider). Because detection is only pattern matching, the check type is a label for what the operator intends to catch, and the referenced patterns define the real check. **Per-rule parameters:** pattern names from the library, and an optional redaction format (for example `{ssn}_REDACTED`). ## Step 4: configure a builtin provider The `builtin` provider is the most configurable local engine. Each rule runs a fixed five-layer pipeline of blocked keywords, an allowlist gate, subject-and-term co-occurrence, detection patterns, and category keywords. Every layer runs, and a match anywhere scores `1.0`. The layers, their evaluation order, and what each one matches are described in [About guardrails](/agent-router-enterprise/guides/operate-and-govern/safety-and-data-protection/about-guardrails/#the-builtin-provider). Three choices matter most when configuring a rule on this provider: - **Reserve blocked keywords for zero-tolerance terms.** That layer is checked first and the allowlist cannot suppress it. - **Prefer subject-plus-term matching to a bare keyword list** where a word is only sensitive in context. Target subjects `[customer]` with flagged terms `[password]` fire on "Please reset the customer's password before the demo" and ignore "I changed my password to something stronger", which a plain keyword list cannot do. - **Set the minimum severity deliberately.** It is the floor for the keyword and detection-pattern layers, so one keyword list can serve several rules at different sensitivities. ## Step 5: configure a tetrate provider The `tetrate` provider detects meaning rather than literal strings, routing each check to an embedding model, a security classifier, a PII model, or natural language inference (NLI) entailment behind a single endpoint. Which check reaches which model, and which checks cascade across layers, is described in [About guardrails](/agent-router-enterprise/guides/operate-and-govern/safety-and-data-protection/about-guardrails/#the-tetrate-provider). ## Step 6: confirm which provider supports which check type A rule is valid only if its provider supports its check type. The Admin Dashboard mirrors this by scoping the check-type dropdown to the selected provider, so the quickest confirmation is to select the provider in the rule form and look for the check type in the list. The full matrix is in the [guardrails reference](/reference/guardrails/#provider-support-by-check-type). Three cases account for most surprises: - `regexp` accepts every check type, because a regular expression is check-type-agnostic. - `gibberish`, `relevance`, and `sentiment` are available on `tetrate` only. - `compliance`, `ban_competitors`, `ban_code`, and `protected_material` are available on `builtin`, and nominally on `regexp`, but not on `tetrate`. ## What to do next - **[Configure guardrails](/agent-router-enterprise/guides/operate-and-govern/safety-and-data-protection/configure-custom-guardrails/):** create a guardrail, add rules that reference these providers, and validate in monitor mode. - **[Protect requests with guardrails](/agent-router-enterprise/guides/operate-and-govern/safety-and-data-protection/protect-requests-with-guardrails/):** the developer-side view of how blocks and redactions appear. - **[Guardrails best practices and FAQ](/reference/best-practices/guardrails-best-practices-and-faq/):** rollout order, fail-open versus fail-closed, and investigation. Where to go next Configure guardrails Create policies and rules that reference these providers. Guardrails best practices and FAQ Roll out safely with monitor mode and deliberate failure modes. --- ## Detect and block prompt injection Enterprise Tier Prompt injection is the manipulation of a model's behaviour through instructions smuggled into the text it processes, causing it to ignore its system prompt, leak data, or take actions it should not. It is a distinct threat from the harmful-content categories that guardrails address. A content guardrail asks whether a piece of text is unsafe in itself; prompt-injection detection asks whether a piece of text is trying to subvert the model that reads it. The two concerns overlap rarely and must be handled separately. The threat arrives by more than one path. A direct jailbreak is an injection in the user's own prompt: the familiar "ignore previous instructions" family of attacks, role-play framings, and encoded instructions intended to escape the system prompt. Indirect injection is more dangerous because it does not require a hostile user. It hides instructions in content the model is given to work with: a passage pulled in by retrieval-augmented generation (RAG), a web page summarised on the user's behalf, or the output of a tool the model called. When a Model Context Protocol (MCP) server returns text and that text is fed back to the model, any instructions buried in it are read with the same authority as the rest of the context. A trusted user, asking an ordinary question, can trigger an attack that was planted in a document or a tool response long before. Tetrate Agent Router applies injection detection inline in the same filter path as its other guardrails, using check types such as `prompt_injection` and `jailbreak` (often scored by the `tetrate` provider, which can cascade from known patterns to a classifier to NLI). This guide covers turning that detection on for a project, choosing what happens on a detection (block or redact in enforce mode, or monitor mode for a pilot), and wiring the resulting trigger record into an alert. **Persona:** Platform operator working in the Admin Dashboard, often alongside the security stakeholders who own the threat model. **Estimated time:** 20 to 40 minutes for an initial configuration, including time spent testing. ## When this guide applies This guide is the right starting point in any of these situations: | Situation | What it covers | | :---- | :---- | | Defending against direct jailbreak attempts in user prompts | Enabling detection on inbound prompts and the patterns it covers | | Defending against instructions hidden in retrieved RAG context | Applying detection to retrieved context before it reaches the model | | Defending against instructions returned by a tool or MCP server | Applying detection to tool and MCP responses when that text re-enters the model path | | Choosing what should happen when an injection is detected | Comparing block, redact, and monitor mode | | Alerting when an injection is detected | The trigger record written on a detection and how it reaches an alert | For the harmful-content side of safety, and for how injection detection composes with it, see [Configure guardrails](/agent-router-enterprise/guides/operate-and-govern/safety-and-data-protection/configure-custom-guardrails/) and [Configure guardrail providers](/agent-router-enterprise/guides/operate-and-govern/safety-and-data-protection/configure-vendor-guardrails/). Because tool and MCP outputs are a primary injection vector, the way MCP access is governed and aggregated is directly relevant: see [Govern MCP server access](/agent-router-enterprise/guides/operate-and-govern/govern-mcp-server-access/) and [Aggregate MCP servers into a profile](/agent-router-service/guides/aggregate-mcp-servers-into-a-profile/). ## Outcomes By the end of this guide: - Injection detection is enabled and applied to at least inbound prompts and tool or MCP responses on the model path. - The patterns the detection covers, common jailbreak framings and indirect-injection attempts, are understood, along with its limits. - The action on a detection (block or redact) and the mode (enforce or monitor) are chosen deliberately for each surface. - The trigger record written on a detection is understood, including where its content is delivered, and an alert is configured against it. - The relationship between injection detection, content guardrails, and data-loss prevention (DLP) is clear. ## Prerequisites - Administrator access to the Admin Dashboard with permission to manage guardrails for the project, typically the `super_admin` role. The read-only `guardrail-reader` role can review detections but cannot configure them. - At least one provider configured with a healthy connection and at least one model enabled, so detection has live traffic to act on. Provisioning is covered in [Provision models and providers](/agent-router-enterprise/guides/operate-and-govern/provision-models-and-providers/). - For coverage of indirect injection through tools, an understanding of which MCP servers are reachable from Agent Router and which profiles expose them. See [Govern MCP server access](/agent-router-enterprise/guides/operate-and-govern/govern-mcp-server-access/) and [Aggregate MCP servers into a profile](/agent-router-service/guides/aggregate-mcp-servers-into-a-profile/). - A few representative injection samples for the testing step: both a direct jailbreak prompt and an indirect attempt embedded in document-like or tool-output-like text. ## Step 1: understand where detection is applied Injection detection inspects text at the points where untrusted instructions can enter the model's context. Three surfaces matter, and they are not interchangeable. All three sit on the **input stage**, because all three are text travelling towards the model. What separates them is the kind of content the text arrives as, which the gateway tracks as the evaluation's **content type**. | Surface | Content type on the evaluation | Threat it addresses | | :---- | :---- | :---- | | Inbound prompt | `text` | Direct jailbreaks: instructions the user supplies to escape the system prompt | | Retrieved context | `context` | Indirect injection planted in documents, knowledge bases, or pages the retrieval step pulls in | | Tool and MCP response | `tool` | Indirect injection returned by a tool, a primary vector, because tool output is read with the same authority as the rest of the context | One input-stage rule therefore covers all three surfaces; there is nothing to configure per surface. The content type is not a field on the rule form, it is an attribute of each evaluation, which matters when reading trigger records in Step 6, because that attribute is what says which surface fired. The full list of content-type values is in the [guardrails reference](/reference/guardrails/#rule-level-settings). The inbound prompt is the surface most operators think of first, but it is the least sufficient on its own. A direct jailbreak requires a hostile user; indirect injection does not. The retrieved-context and tool-response surfaces are where a trusted user, asking an ordinary question, can be turned into the delivery mechanism for an attack that was planted elsewhere. Detection that runs only on prompt text leaves the indirect paths open. Tool and MCP responses warrant particular attention. When Agent Router mediates a model's call to an MCP server, the server's response re-enters the model's context as authoritative text. A compromised or untrusted MCP server, or a legitimate one returning data an attacker controls, can therefore inject instructions without ever touching the user's prompt. Detection on this surface inspects that returning text before the model acts on it, which is to say when the text rides the LLM request path, not when a client calls an MCP profile endpoint (`/mcp/*`) directly. Standalone MCP profile traffic is governed by identity, access control, and audit, not by content guardrails or injection detection. Which MCP servers a given profile may reach is itself a control, covered in [Govern MCP server access](/agent-router-enterprise/guides/operate-and-govern/govern-mcp-server-access/). ## Step 2: understand what the detection covers Injection detection recognises the patterns characteristic of an attempt to subvert the model, rather than the harmful-content categories a guardrail addresses. The patterns fall into two broad families. - Direct jailbreak framings: instructions that try to override the system prompt or the model's role. Common forms include explicit override phrasing ("ignore previous instructions", "disregard your rules"), role-play and persona framings that ask the model to assume an unrestricted identity, and instructions encoded or obfuscated to slip past naive matching. - Indirect-injection patterns: imperative instructions appearing where only data is expected. Text retrieved by RAG or returned by a tool is meant to be information for the model to use, not commands for it to follow. Instructions embedded in that text (directing the model to exfiltrate context, call a tool, or change its behaviour) are the signature of an indirect attack. Two limits are worth stating plainly, because treating detection as absolute leads to misplaced confidence. - Detection is heuristic and probabilistic. It raises the cost of a successful injection; it does not reduce it to zero. Novel phrasing and adversarial obfuscation will sometimes evade it, and benign text will sometimes resemble an attack. - Detection is one layer. It is most effective combined with least-privilege tool access, scoped MCP profiles, and the content and DLP controls described in Step 6, so that an injection that evades detection still cannot reach a high-value action or exfiltrate sensitive data. ## Step 3: choose the action and mode on a detection When an injection is detected, the rule's **action** and **mode** decide what happens. A rule carries one action, applied to every input-stage surface it inspects, so the choice is made once per rule rather than per surface. | Action / mode | What happens on a detection | When it fits | | :---- | :---- | :---- | | Block (enforce) | The request is rejected with HTTP `403`, or the offending content is withheld, and the model does not act on it | The detection is high-confidence, and proceeding is unacceptable: the default for direct jailbreaks on inbound prompts | | Redact (enforce) | Offending spans are stripped or masked and processing continues with the cleaned content | Indirect injection in a retrieved passage or tool response, where the surrounding data is still wanted but the embedded instruction must not be followed | | Monitor mode | The content passes unchanged, but the would-be action is recorded | Establishing a baseline rate before enforcing, or observing a low-confidence surface without disrupting traffic | Block is the safest response where a detection means the interaction itself should not proceed, which is the usual case for a clear jailbreak in a user's prompt. Redact suits the indirect surfaces better: a retrieved document or a tool response often contains legitimate data alongside an injected instruction, and discarding the whole response would break the task, so removing only the instruction preserves the useful content. Because one rule cannot apply block to prompt text and redact to tool output, that tension is resolved either by accepting one action across all surfaces or by splitting the policy across two rules on different check types, for example blocking on `jailbreak` and redacting on `prompt_injection`. Monitor mode is the right starting point in either case; it produces the same trigger record without changing what callers or models experience, which makes it the natural pilot for Step 5. Prefer the `prompt_injection` and `jailbreak` check types on the `tetrate` provider when semantic detection is required; see [Configure guardrail providers](/agent-router-enterprise/guides/operate-and-govern/safety-and-data-protection/configure-vendor-guardrails/). ## Step 4: create the rules There is no separate injection-detection object. Injection detection is a guardrail like any other, distinguished only by its check types, so it is created through the same Guardrails surface. The full creation procedure is in [Configure guardrails](/agent-router-enterprise/guides/operate-and-govern/safety-and-data-protection/configure-custom-guardrails/); what follows is the injection-specific reading of it. 1. Sign in to the Admin Dashboard and select the **project** whose traffic the detection should cover. 1. Open **Guardrails → Rules** and choose **Add guardrail**. Name it for its purpose (for example "Prompt injection baseline") so its intent is clear in later review, and set type to Security. 1. Set the guardrail-level failure mode deliberately. Fail-close is the default and the right choice here: if the provider that scores injection checks cannot be reached, an unevaluated prompt should not reach the model by default. 1. Choose **Add rule**, and set the **Provider** first, because it scopes the check types on offer. Semantic detection requires `tetrate`, which cascades from known patterns to a jailbreak classifier to NLI entailment; `regexp` and `builtin` can only match phrasings someone has written down in advance. See [Configure guardrail providers](/agent-router-enterprise/guides/operate-and-govern/safety-and-data-protection/configure-vendor-guardrails/). 1. Set the **Check type** to `prompt_injection` for smuggled instructions, or `jailbreak` for the direct "ignore previous instructions" family. Most baselines carry both, since they detect different things. 1. Set **Run on** to **Input**. That one setting covers all three surfaces from Step 1. 1. Set the **Action** and **Mode** as decided in Step 3, which for an initial rollout means monitor on every rule. 1. Save the rule, then add a second rule the same way for the other check type. Both `prompt_injection` and `jailbreak` are input-stage checks on the ML providers, which matches the threat: injection is something that arrives, not something a model emits. A rule set to run on the output stage will not detect it. Detection takes effect on subsequent requests in the project. Requests already in flight complete under the configuration active when they were admitted; there is no service restart. Because indirect injection can arrive through any RAG source or any reachable MCP server, a project-wide baseline is the safest default. Pairing that baseline with tight MCP access limits both the chance of an injection and the damage a successful one could do. ## Step 5: validate with sample injections in monitor mode Detection that has never been exercised against a real injection is an assumption, not a defence. Validation is monitor mode against representative traffic, and the procedure is in [Configure guardrails](/agent-router-enterprise/guides/operate-and-govern/safety-and-data-protection/configure-custom-guardrails/#step-6-validate-in-monitor-mode-before-enforcing). Three samples are specific to injection: - **A direct jailbreak** in prompt text, for example an instruction to ignore previous instructions. This exercises the `jailbreak` rule against content type `text`. - **An indirect injection**, shaped like a retrieved passage or a tool response with an embedded instruction directing the model to act. Submit it through the path that actually produces that content, rather than pasting it into a prompt, so the evaluation is recorded against `context` or `tool` and the indirect surface is genuinely exercised. - **Benign content that resembles an attack**, such as a document legitimately discussing prompt injection. This is the sample that catches an over-eager rule, and injection detection is unusually prone to it: security documentation, incident write-ups, and this guide itself all contain text that looks like the thing being detected. Resolving false detections and missed detections at this stage is far cheaper than discovering them once the rules are acting on real requests. Monitor mode produces the same trigger records that drive alerting in Step 6. ## Step 6: alert on the trigger record Every detection writes a **guardrail-trigger** record, whether the mode is enforce or monitor and whether the action is block or redact. That record is what makes injection detection observable and alertable rather than merely active. A monitor-mode rule no one is watching provides no protection. - One record is written per rule evaluation, carrying the rule and guardrail identifiers, the score, the action and mode, the stage, and the content type. The content type is what identifies the surface: a record with content type `tool` is an indirect attack arriving through a backend rather than a user, which is the case that warrants the promptest alert. - **The record can carry the offending text, and the data plane's own copy always does.** Where that content goes is controlled in two independent places: **Settings → Guardrails** governs what reaches the management plane (Full, Metadata only, or Off), while a `guardrailtrigger` subscription on a custom observability backend receives full content regardless of that setting. Both are documented in [Custom observability backends](/agent-router-enterprise/guides/observability-and-analytics/configure-custom-observability-backends/#guardrail-trigger-destinations). For injection detection this is usually welcome, because the payload is the evidence, but it should be a decision rather than a surprise. - Trigger records are a different record type from Audit Logs, which capture administrative mutations such as a change *to* this guardrail rather than the guardrail firing. To investigate one specific block, start from the correlation identifier returned with the `403`; that workflow is in [Guardrails best practices and FAQ](/reference/best-practices/guardrails-best-practices-and-faq/). The read-only `guardrail-reader` role exists so a security team can review triggers without being able to change them. - For alerting, subscribe a backend the security team already watches to the `guardrailtrigger` signal and raise an alert on rate or pattern. The general telemetry path is described in [Export telemetry to an observability stack](/agent-router-enterprise/guides/observability-and-analytics/export-telemetry-to-an-observability-stack/). A detection that fires into a record no one watches is indistinguishable from no detection at all. An alert closes that gap. ## Step 7: compose detection with content guardrails and DLP Injection detection is one control among several, and it is most effective as part of a layered safety posture rather than on its own. Each layer addresses a concern the others cannot. - Injection detection asks whether incoming text, from a user, a retrieved source, or a tool, is trying to subvert the model. It is the subject of this guide. - Content guardrails ask whether text is unsafe or disallowed in itself: toxicity, hate speech, banned topics, and similar check types, scored by `regexp`, `builtin`, or `tetrate`. See [Configure guardrail providers](/agent-router-enterprise/guides/operate-and-govern/safety-and-data-protection/configure-vendor-guardrails/) and [Configure guardrails](/agent-router-enterprise/guides/operate-and-govern/safety-and-data-protection/configure-custom-guardrails/). - Data-loss prevention asks whether sensitive data is leaving the boundary. PII or secrets detection with a redact or block action is Agent Router's DLP mechanism and limits what a successful injection could exfiltrate even if the injection itself evades detection. These layers operate in the same inline filter path, and all must permit a request for it to proceed. They are complementary by design: injection detection stops the model being subverted, content guardrails stop unsafe content passing in either direction, and DLP stops sensitive data leaving. Deciding which layer owns a given concern, rather than duplicating intent across all three, keeps the overall policy coherent and auditable. Tightening MCP access so that a subverted model can reach only the tools it genuinely needs, covered in [Govern MCP server access](/agent-router-enterprise/guides/operate-and-govern/govern-mcp-server-access/), reduces the blast radius further still. ## What to do next - **Configure guardrails**: add PII redaction and other rules alongside injection detection, including the DLP controls referenced here. See [Configure guardrails](/agent-router-enterprise/guides/operate-and-govern/safety-and-data-protection/configure-custom-guardrails/). - **Configure guardrail providers**: choose `regexp`, `builtin`, or `tetrate` for each check type. See [Configure guardrail providers](/agent-router-enterprise/guides/operate-and-govern/safety-and-data-protection/configure-vendor-guardrails/). - **Govern MCP server access**: limit which MCP servers a profile can reach, reducing the indirect-injection surface and the blast radius of a successful attack. See [Govern MCP server access](/agent-router-enterprise/guides/operate-and-govern/govern-mcp-server-access/). - **Custom observability backends**: deliver the trigger records written here to the stack the security team watches, and decide where their content goes. See [Custom observability backends](/agent-router-enterprise/guides/observability-and-analytics/configure-custom-observability-backends/#guardrail-trigger-destinations). - **Protect requests with guardrails**: the developer-side view of how safety controls appear in application code. See [Protect requests with guardrails](/agent-router-enterprise/guides/operate-and-govern/safety-and-data-protection/protect-requests-with-guardrails/). Where to go next Detect and redact sensitive data Add the DLP controls that limit what a successful injection could exfiltrate. Govern MCP server access Limit which MCP servers a profile can reach, reducing the indirect-injection surface. --- ## Detect and redact sensitive data Enterprise Tier Sensitive data leaks at the gateway long before anyone notices. A customer record pasted into a prompt, an API key embedded in a code snippet, a payment-card number copied into a support question: each of these can travel to a third-party model provider, and from there into a provider's own logs, the moment a request is forwarded unchecked. Data-loss prevention (DLP) is the discipline of stopping that traffic at the boundary: inspecting prompts on their way out and responses on their way back, recognising sensitive content, and removing, rejecting, or recording it before it crosses a line it should not cross. Tetrate Agent Router enforces DLP through the same rule engine that backs its content guardrails. A guardrail is a content-filtering rule enforced inline by the gateway; DLP is the sensitive-data-focused application of that engine, configured in the Admin Dashboard and tuned for the specific categories a security or compliance team cares about: personally identifiable information (PII), source secrets, and API keys. This guide covers the work of standing up a DLP policy in Agent Router: choosing what to detect, deciding what happens on a match, scoping the policy to the right project, applying it on both stages, testing it before it enforces, and deciding where the resulting trigger records go. This guide does not re-explain the rule engine itself. The mechanics of creating a guardrail, check types, providers, actions, and monitor mode are covered once, in [Configure guardrails](/agent-router-enterprise/guides/operate-and-govern/safety-and-data-protection/configure-custom-guardrails/). What follows is the DLP-specific reading of that material: which detectors to reach for, and how to assemble them into a policy that a compliance stakeholder will accept. **Persona:** Platform operator working in the Admin Dashboard, typically alongside the security and compliance stakeholders who own the underlying data-handling policy. **Estimated time:** 30 to 45 minutes for a first DLP policy, including time spent testing against seeded samples. ## When this guide applies This guide is the right starting point in any of these situations: | Situation | What it covers | | :---- | :---- | | Stripping PII from prompts before they reach an external provider | A `pii` rule on the input stage with a redact action | | Preventing customer records from leaving Agent Router | A detector in the project that handles that data | | Stopping API keys and secrets from being sent to or returned by a model | A `secrets` rule applied on both stages with a block action | | Encoding an organisation-specific record format the built-in detectors do not know | A custom pattern added alongside the built-in detectors | | Demonstrating sensitive-data controls for a compliance review | A project policy with redact and block actions, evidenced by its guardrail-trigger records | For the developer-side view, how a request path opts into the guardrails an operator has configured, see [Protect requests with guardrails](/agent-router-enterprise/guides/operate-and-govern/safety-and-data-protection/protect-requests-with-guardrails/). ## Outcomes By the end of this guide: - At least one DLP policy exists, built from one or more built-in detectors and any custom patterns the policy requires. - Each category of sensitive data carries a deliberate action (`redact` or `block`) and mode (`enforce` or `monitor`). - The policy inspects the input stage, the output stage, or both, matching the direction in which the data can leak. - The policy sits in the project that handles the data rather than being applied indiscriminately. - The policy has been tested against seeded sample content before being allowed to enforce. - The relationship between DLP, custom guardrails, and the guardrail-trigger record is clear. ## Prerequisites - Administrator access to the Admin Dashboard with permission to manage guardrails for the project, typically the `super_admin` role. The read-only `guardrail-reader` role can review what a policy did but cannot create or change one. - A written data-handling policy that states which categories of data are sensitive and what must happen to each. The most defensible DLP policies start from a compliance requirement expressed in plain language, not from a pattern invented at configuration time. - For any organisation-specific format (an internal record or account number) the pattern that describes it, ideally reviewed with the stakeholder who owns the data. - Seeded sample content for the testing step: text that should match each detector, and realistic text that should not. - Familiarity with [Configure guardrails](/agent-router-enterprise/guides/operate-and-govern/safety-and-data-protection/configure-custom-guardrails/), which describes the rule engine this guide applies. ## Step 1: decide what counts as sensitive DLP begins with a list, not a configuration screen. Before any rule is created, the categories of data the policy must catch should be named and agreed with the stakeholder who owns them. Each category then maps onto one of three check types. | Category | Check type | What it recognises | | :---- | :---- | :---- | | Personal data | `pii` | Names, email addresses, phone numbers, national identifiers, payment cards, IBANs, and similar well-known classes | | Credentials | `secrets` | AWS, GitHub, and Slack credentials, generic API keys, and other credential-bearing tokens | | Organisation-specific formats | `regexp` | Whatever the named patterns say: an internal record number, an account format, a document identifier | The `pii` and `secrets` check types recognise the common shapes of their categories without any pattern being written by hand. Where a category includes something organisation-specific, a named pattern in the Patterns library fills the gap: `regexp` rules take pattern names directly, and `builtin` detection patterns can each reference a catalog pattern instead of an inline expression. See [Configure guardrails](/agent-router-enterprise/guides/operate-and-govern/safety-and-data-protection/configure-custom-guardrails/) and [Configure guardrail providers](/agent-router-enterprise/guides/operate-and-govern/safety-and-data-protection/configure-vendor-guardrails/). A policy is usually assembled from several rules, each doing one job, rather than one rule stretched to cover every category. A `pii` rule for the well-known classes, a `secrets` rule for credentials, and a `regexp` rule naming the organisation's own patterns is a typical starting shape. The full check-type vocabulary is in the [guardrails reference](/reference/guardrails/#check-types). ## Step 2: choose the action for each category The action is what the gateway does when content matches a detector. Two enforcement actions are available (`redact` and `block`), and whether they apply is controlled by the rule's **mode** (`enforce` or `monitor`). A single DLP policy commonly uses more than one action: redact for data that should be stripped, block for data that must never pass, and monitor mode where visibility is the goal before enforcement. | Action / mode | What happens on a match | Where it fits in a DLP policy | | :---- | :---- | :---- | | Redact (enforce) | The matching span is masked or removed, and the request continues with the sanitised content | Personal data and customer records that should be stripped without stopping the interaction | | Block (enforce) | The request is rejected with HTTP `403`, or the response is withheld from the caller | Secrets and API keys, where a match means the interaction itself is not permitted | | Monitor mode | The content passes unchanged, but the would-be action is recorded | Observing how often a category would fire before enforcement, without disrupting traffic | Redact is the workhorse of a PII policy: the sensitive span is removed so the model never sees it, while the rest of the prompt proceeds and the interaction is not interrupted. For structured identifiers, the `regexp` provider's Patterns library and redaction format are a common choice; for semantic PII the `tetrate` provider's PII model can return entities and a masked copy. Block is the right action where the presence of the data is itself the problem: a credential or API key has no legitimate reason to travel to a model, so a match should stop the request outright. Monitor mode is the safest way to start any new rule: it produces the same trigger record without changing what callers experience, which makes it the natural pilot for Step 5. Action and mode are described in [Configure guardrails](/agent-router-enterprise/guides/operate-and-govern/safety-and-data-protection/configure-custom-guardrails/). ## Step 3: apply detection on both stages Sensitive data leaks in two directions, and a DLP policy has to account for both. A rule's **Run on** setting selects the input stage, the output stage, or both. - The input stage examines the prompt on its way to the model. This is where most PII redaction belongs: the sensitive content is removed before it leaves the organisation's boundary and before any provider can log it. - The output stage examines the model's response before it returns to the caller. A model can reproduce sensitive data it was given earlier in a conversation, or surface a secret it inferred from context; output inspection catches data on its way back out. - Setting **Run on** to both applies the same check each way. A credential should neither be sent to a model nor returned in a response, which makes `secrets` the clearest case for symmetric inspection. One restriction applies to that symmetry. On the ML providers, `secrets` is an input-stage check, so a rule that must inspect both stages for credentials belongs on `regexp` or `builtin`, where every check type runs on either stage. The stage restrictions are listed in the [guardrails reference](/reference/guardrails/#provider-availability-and-evaluation-order). The stage is the only placement decision the rule form offers. Which kind of content an evaluation ran against (text, image, document, tool call, thinking, or context) is recorded as the evaluation's content type rather than chosen on the rule, so a rule set to run on a stage inspects every kind of content arriving on it. Those values matter when reading trigger records, covered under [How DLP matches are recorded](#how-dlp-matches-are-recorded), not when creating the rule; they are listed in the [guardrails reference](/reference/guardrails/#rule-level-settings). ## Step 4: scope the policy to a project A DLP policy rarely applies uniformly to all traffic. The same prompt that is unremarkable from an internal analytics tool may be a violation from a public-facing assistant. The **project** is the only scope a guardrail has. Rules apply to the whole project, so guardrails in one project never touch another, and there is no per-app or per-team rule target. Separation therefore comes from which project a workload's API keys belong to: a project whose keys serve a customer-facing assistant can carry a strict policy while a project used for internal experiments carries none. Naming keys by purpose, as in [Onboard developers and issue keys](/agent-router-enterprise/guides/operate-and-govern/onboard-developers-and-issue-keys/), is what makes that boundary legible later. The practical consequence is that a category admitting no exception (credentials typically) belongs in every project that handles model traffic, while a category that is sensitive only in one context belongs in the project that context routes through. Where several rules apply to the same request, each is evaluated independently, and a block from any one of them stops the request. ## Step 5: validate in monitor mode against representative traffic A DLP policy that has never been validated against representative content will eventually redact something it should leave alone, or pass something it should have caught. There is no separate test-content surface that dry-runs a rule against pasted samples: validation is monitor mode against live or staged traffic, and the procedure is in [Configure guardrails](/agent-router-enterprise/guides/operate-and-govern/safety-and-data-protection/configure-custom-guardrails/#step-6-validate-in-monitor-mode-before-enforcing). Two things are specific to DLP validation: - **Sample content must be fabricated, never drawn from production.** A synthetic record, a key in the expected format, and a realistically shaped but invalid credential exercise the same patterns without creating the exposure the policy exists to prevent. - **Negative samples matter as much as positive ones.** A pattern for an internal record number will often match something innocuous, and an over-broad `pii` threshold will redact ordinary prose. Feed the rule realistic traffic that should *not* match and confirm it stays quiet. Resolving every false match and every missed match at this stage costs far less than discovering them once the policy is rejecting real requests. ## How DLP relates to guardrails DLP is not a separate subsystem. It is the same rule engine, on the same request path, aimed at one class of content: a DLP policy is the subset of guardrail rules that target personal data, credentials, and organisation-specific record formats. Everything in [Configure guardrails](/agent-router-enterprise/guides/operate-and-govern/safety-and-data-protection/configure-custom-guardrails/) therefore applies unchanged, and this guide adds only the DLP reading of it. ## How DLP matches are recorded Every match produces a **guardrail-trigger** record, and that record is what makes the policy defensible to a compliance stakeholder rather than merely active. One record is written per rule evaluation, carrying the rule and guardrail identifiers, the score, the action and mode, the stage and content type, the execution duration, and the project and consumer context. Guardrail triggers are a different record type from Audit Logs. Audit Logs capture administrative mutations, including changes *to* a guardrail; triggers capture the policy *firing*. An operator investigating a specific block starts from the correlation identifier returned with the `403`, which resolves to the rules that triggered, their scores, and the content that caused it. That workflow is in [Guardrails best practices and FAQ](/reference/best-practices/guardrails-best-practices-and-faq/). The read-only `guardrail-reader` role exists for exactly this review. **Trigger records can contain the matched content, and by default the data plane's own copy always does.** The record carries the matched message and, for a redaction, the sanitised message. Two independent controls govern where that content goes: - **Settings → Guardrails** in the Admin Dashboard sets what reaches the management plane: **Full** (matched and sanitised text stored), **Metadata only** (scores, check types, actions, and identifiers, without message content), or **Off** (no trigger records, and empty dashboard trigger views). - A `guardrailtrigger` subscription on a custom observability backend receives **full content regardless of that setting**, because the fan-out exists so an organisation can hold complete records in its own systems. A DLP policy therefore needs both settings decided deliberately: a control designed to keep sensitive data out of a provider's logs should not put it somewhere unintended instead. Both are documented in [Custom observability backends](/agent-router-enterprise/guides/observability-and-analytics/configure-custom-observability-backends/#guardrail-trigger-destinations). Retention of the management-plane copy is governed separately; see [Manage log retention and purge](/agent-router-enterprise/guides/operate-and-govern/compliance-and-governance/manage-log-retention-and-purge/). For streaming decisions into a central security system, see [Export audit and policy decisions to a SIEM](/agent-router-enterprise/guides/operate-and-govern/compliance-and-governance/export-audit-and-policy-decisions-to-a-siem/). ## What to do next - **Configure guardrails**: the full mechanics of the rule engine this guide applies, including check types, providers, and monitor mode. See [Configure guardrails](/agent-router-enterprise/guides/operate-and-govern/safety-and-data-protection/configure-custom-guardrails/). - **Configure guardrail providers**: how `regexp`, `builtin`, and `tetrate` score PII and secrets checks. See [Configure guardrail providers](/agent-router-enterprise/guides/operate-and-govern/safety-and-data-protection/configure-vendor-guardrails/). - **Protect requests with guardrails**: the developer-side view of how a request path sees the detectors configured here. See [Protect requests with guardrails](/agent-router-enterprise/guides/operate-and-govern/safety-and-data-protection/protect-requests-with-guardrails/). - **Custom observability backends**: decide where trigger records and their matched content are delivered. See [Custom observability backends](/agent-router-enterprise/guides/observability-and-analytics/configure-custom-observability-backends/#guardrail-trigger-destinations). - **Guardrails best practices and FAQ**: investigate a specific block from its correlation identifier. See [Guardrails best practices and FAQ](/reference/best-practices/guardrails-best-practices-and-faq/). - **Manage log retention and purge**: govern how long the management-plane copy of trigger records is kept. See [Manage log retention and purge](/agent-router-enterprise/guides/operate-and-govern/compliance-and-governance/manage-log-retention-and-purge/). - **Export audit and policy decisions to a SIEM**: stream DLP decisions to a central security system. See [Export audit and policy decisions to a SIEM](/agent-router-enterprise/guides/operate-and-govern/compliance-and-governance/export-audit-and-policy-decisions-to-a-siem/). - **Reference**: the definitions behind the terms used in this guide are in the [glossary](/reference/glossary/). Where to go next Configure guardrails The full mechanics of the rule engine this guide applies, including check types and monitor mode. Export audit and policy decisions to a SIEM Stream DLP decisions to a central security system. --- ## Guardrails on shared data planes Enterprise Tier A data plane is shared capacity: it can host gateways for many projects at once, and each of those projects can carry its own guardrails. This page explains how guardrail enforcement works on that shared capacity — how enforcement is scoped to each project, how gateway attachment determines which guardrails a data plane enforces, and which signals to wire into alerting. **Persona:** Platform operator or security stakeholder running data planes that serve more than one project. ## What you will learn - Each project's guardrails apply only to that project's traffic. - Gateway attachment health is enforcement health: while a project's gateway attachment is failed, that project's guardrails are not enforced. - Requests without project identity are blocked on a guardrail-carrying data plane, and the data plane counts them in metrics you should alert on. ## Each project's guardrails apply only to that project's traffic Guardrails are [project policies](/agent-router-enterprise/guides/operate-and-govern/safety-and-data-protection/about-guardrails/): a guardrail authored under one project belongs to that project alone. Every gateway serves exactly one project, and the platform stamps each request with the project it belongs to before guardrails are evaluated, so the gateway evaluates only the guardrails owned by the request's project. A project that has configured no guardrails is unaffected by the policies of the other projects sharing the data plane. Selection is per request. All served projects' guardrails coexist on the same data plane; on each request, the gateway selects the policy set that belongs to the project identity stamped on that request. ## Gateway attachment health decides what is enforced The set of projects whose guardrails a data plane enforces is derived from the project gateways attached to it — [provisioning a gateway](/agent-router-enterprise/guides/operate-and-govern/project-and-data-plane-management/provision-a-gateway/) for a project on a data plane is what enrols that project's guardrails there. That makes gateway attachment health part of enforcement health. While a project's gateway attachment is in a failed state, none of that project's guardrails are enforced on that data plane; enforcement resumes when the attachment is repaired. Traffic that cannot be attributed to a served project is blocked (see [the identity section below](#alert-on-requests-missing-project-identity)). When a gateway attachment fails: 1. Review the per-gateway health indicators on the data plane, following [Manage multiple gateways on a data plane](/agent-router-enterprise/guides/operate-and-govern/project-and-data-plane-management/manage-gateways-on-a-data-plane/). 1. Repair or re-provision the failed attachment to restore enforcement for that project. ## Alert on requests missing project identity The platform stamps identity onto every request as it crosses the gateway — the same identity surfaced on telemetry as the `x-tars-customer` and `x-router-project-id` header-mapped attributes in the [OpenTelemetry reference](/reference/otel-metrics/#metric-header-mapped-attributes). Applications do not set these; the data plane does, and guardrail selection depends on them. On a data plane with **any** guardrails configured, a request that reaches guardrail evaluation without this identity is **blocked**, with no guardrail evaluated: an unattributable request cannot be matched to its project's policy. On a data plane with no guardrails configured at all, such requests are allowed, since there is nothing to enforce. In both cases the data plane emits metrics counting requests that arrive without identity. These counters are the canary for a misconfigured or degraded request path, and they deserve a standing alert: - **Wire an alert on the missing-identity counters** in the metrics stream each data plane exposes (see [OpenTelemetry traces and metrics](/reference/otel-metrics/) for the delivery paths). Under normal operation the rate is zero. - **Treat a sustained non-zero rate as an incident.** On a guardrail-carrying data plane it means live traffic is being blocked; on a guardrail-free data plane it means requests are flowing without project attribution. Either way the request path needs repair, starting with the health of the data plane's gateways. Where to go next About guardrails The policy model: guardrails, rules, providers, patterns, and how a rule is evaluated. Manage multiple gateways on a data plane Review the gateways a data plane manages, their owning projects, and their health. Guardrails best practices & FAQ Roll out, tune, and operate guardrails in production without self-inflicted outages. OpenTelemetry traces and metrics The metric families and identity attributes the data plane exposes for alerting. --- ## Safety and Data Protection Guardrails are project policies the gateway enforces inline: on the way in (prompts) and on the way out (responses). Each policy is a collection of rules; each rule is a check type scored by a provider. [About guardrails](/agent-router-enterprise/guides/operate-and-govern/safety-and-data-protection/about-guardrails/) explains that model and how evaluation works; the guides that follow cover configuring guardrails and providers, redacting sensitive data, blocking prompt injection, applying stricter policy bundles to coding-assistant traffic, and the developer-side view of blocked and redacted requests. About guardrails The model behind the policies: rules, providers, evaluation order, and failure modes. Configure guardrails Create project policies and rules; validate in monitor mode before enforce. Configure guardrail providers Choose regexp, builtin, or tetrate for each check type. Detect and redact sensitive data Keep PII and credentials out of prompts and responses. Detect and block prompt injection Stop jailbreak patterns before they reach a model. Protect coding-assistant traffic Apply stricter policy bundles to agent and editor traffic without slowing normal use. Guardrails on shared data planes Per-project enforcement on shared capacity, gateway attachment health, and identity alerts. Protect requests with guardrails The developer-side view: recognising a block or a redaction and requesting coverage. --- ## Protect coding-assistant traffic Enterprise Tier Coding assistants are now a routine part of how software is written, and almost all of them reach a model through the same OpenAI-compatible interface that any other application uses. That convenience is also the risk. When a developer works inside Cursor, Continue, Cline, or a similar assistant, the prompts sent to the model are not casual questions; they are spans of the organisation's own source code, the surrounding files an assistant pulls in for context, configuration that frequently contains credentials, and whatever customer data happens to sit in the buffer being edited. Each of those requests is an opportunity for proprietary source, secrets, or regulated data to leave the organisation through a model call, or to be retained by a provider in a way the organisation never intended. Tetrate Agent Router sits in the path of those requests, which makes it the right place to treat coding-assistant traffic as its own class and hold it to a stricter standard than general application traffic. This guide covers that work end-to-end: why coding assistants are a distinct risk, how Agent Router distinguishes their traffic from everything else, how a stricter policy bundle, comprising intellectual property (IP) and secret detection with a redact or block action, is attached to that traffic, and how the result is verified by sending a sample request that carries a seeded secret and confirming the gateway blocks or redacts it. Because the developers who configure these assistants control how that traffic is labelled, a short section covers coordinating with them so the distinction actually holds in practice. **Persona:** Platform operator working in the Admin Dashboard, in coordination with the security or compliance owner of the underlying policy and the developers who configure the assistants. **Estimated time:** 30 to 60 minutes, including the verification step and the conversation with the development team that the labelling depends on. ## When this guide applies This guide is the right starting point in any of these situations: | Situation | What it covers | | :---- | :---- | | Coding assistants are in use and source code or secrets may be flowing through model calls | Distinguishing assistant traffic and attaching a stricter policy bundle to it | | A policy requires that proprietary code never reach an external model unredacted | IP and secret detection applied to the assistant's traffic with a redact or block action | | A record is required specifically for AI traffic originating in developer tooling | Where the guardrail-trigger records for assistant traffic are delivered | | The stricter controls need to be demonstrated before the evaluation is signed off | Verifying with a seeded-secret sample request | For the general mechanics of detecting and redacting sensitive content (the check types, providers, and match actions that this guide reuses) see [Detect and redact sensitive data](/agent-router-enterprise/guides/operate-and-govern/safety-and-data-protection/detect-and-redact-sensitive-data/) and [Configure guardrails](/agent-router-enterprise/guides/operate-and-govern/safety-and-data-protection/configure-custom-guardrails/). For how the assistants themselves are pointed at the gateway in the first place, see [Integrate the gateway with an app](/agent-router-service/guides/integrate-the-gateway-with-an-app/). ## Outcomes By the end of this guide: - Traffic originating from coding assistants can be distinguished from general application traffic at the gateway. - A stricter policy bundle, comprising IP and secret detection with a redact or block action, is attached to that traffic. - Where the guardrail-trigger records for that traffic are delivered, and how much of the matched content they carry, is understood. - A sample request carrying a seeded secret has been sent through an assistant's path and confirmed to be blocked or redacted. - The development team understands how its assistant configuration determines whether the stricter controls apply. ## Prerequisites - Administrator access to the Admin Dashboard with permission to manage guardrails and API keys for the project, typically the `super_admin` role. - A written statement of the policy the stricter bundle is meant to enforce: which categories of content must not leave the organisation through an assistant, and whether a match should be redacted or block the request outright. - An inventory of which coding assistants are in use (Cursor, Continue, Cline, and similar) and who configures them. The distinction this guide relies on cannot be enforced for an assistant nobody knows is running. - A non-production secret to seed for the verification step. A real credential must never be used as test content; a clearly fake but realistically shaped token serves the purpose without putting anything at risk. - Familiarity with project guardrails, covered in [Configure guardrails](/agent-router-enterprise/guides/operate-and-govern/safety-and-data-protection/configure-custom-guardrails/). ## Step 1: understand why coding-assistant traffic is a distinct risk A general application sends a model the content it was designed to send, and that content has usually been through some deliberate handling before it leaves the codebase. A coding assistant is different in three ways that together justify treating its traffic separately. - **The payload is proprietary source by default.** The assistant's purpose is to reason about the code being written, so source (often including files the developer did not consciously decide to share) is the normal content of every request, not an exception. - **Secrets travel with the code.** Configuration files, environment definitions, and inline constants frequently contain API keys, tokens, and connection strings. An assistant that reads the surrounding files for context reads those too, and forwards them to the model unless something stops it. - **Customer data is often in the buffer.** Test fixtures, sample records, and data pasted in while debugging routinely contain real or realistic customer information that the developer is not thinking of as an outbound disclosure. None of this is malicious; it is the ordinary behaviour of a useful tool. The point is that the volume and sensitivity of what flows through an assistant is high enough, and unconsidered enough, that the controls applied to general traffic are not sufficient. The remainder of this guide raises the floor for this one class of traffic. ## Step 2: distinguish coding-assistant traffic A stricter policy can only be attached to assistant traffic once that traffic can be told apart from everything else. Agent Router offers three ways to draw the distinction, in roughly increasing order of how much developer cooperation each requires. | Method | How the distinction is drawn | Trade-off | | :---- | :---- | :---- | | **Dedicated API key** | Each coding-assistant integration is issued its own API key, and the stricter bundle is scoped to that key | The cleanest signal; depends on developers using the issued key rather than a shared one | | **Request attributes or headers** | A request attribute or header set by the assistant's configuration marks the traffic as assistant-originated | Flexible, but only as reliable as the configuration that sets it | | **Routing path** | Assistant traffic is sent through a dedicated routing policy or path, distinct from general application traffic | Keeps the distinction in Agent Router rather than the client, at the cost of a separate path to maintain | The dedicated-key approach is the most robust and the one to prefer where it is practical, because it ties the distinction to a credential Agent Router already issues, tracks, and can revoke. The same pattern of one key per integration is recommended for observability in [Integrate the gateway with an app](/agent-router-service/guides/integrate-the-gateway-with-an-app/), so coding assistants frequently already have their own keys for that reason; this guide reuses that boundary for policy. Whichever method is chosen, it is only as good as the configuration on the developer side. Step 6 covers the coordination that keeps the distinction honest. ## Step 3: assemble the stricter policy bundle The "bundle" is the set of controls applied together to the distinguished traffic. It is built from the same guardrail mechanism used elsewhere in Agent Router, scoped to the assistant traffic identified in Step 2, with the action set tighter than it would be for general traffic. - **IP and secret detection.** Add rules that detect the categories the policy names: secrets such as API keys, tokens, and connection strings (`secrets` check type); and the markers of proprietary source, such as internal codenames, licence headers, or repository identifiers. Use `regexp` with the Patterns library for structured secrets, `builtin` for keyword and subject-plus-term lists, and `tetrate` where semantic detection is needed. Where generated code must be rejected outside coding assistants, prefer `ban_code` or `code` check types and scope them carefully. See [Configure guardrail providers](/agent-router-enterprise/guides/operate-and-govern/safety-and-data-protection/configure-vendor-guardrails/) and [Configure guardrails](/agent-router-enterprise/guides/operate-and-govern/safety-and-data-protection/configure-custom-guardrails/). - **Redact or block.** For an assistant, redaction lets the interaction continue with the sensitive span removed, useful where stripping a single secret should not derail an otherwise legitimate request. A block stops the request entirely and is the right action where a match means the content should never have been sent at all. The policy statement from the prerequisites decides which applies to each category; many bundles redact incidental secrets and block on the categories that admit no exception. - **A record of what the rules did.** Each rule evaluation writes a guardrail-trigger record carrying the rule and guardrail identifiers, the score, the action and mode, the stage, and the content type. No extra step produces it; Step 7 confirms it. Note that the record can also carry the matched text, which for assistant traffic means spans of source code: **Settings → Guardrails** governs what reaches the management plane (Full, Metadata only, or Off), and any custom observability backend subscribed to `guardrailtrigger` receives full content regardless of that setting. Both controls are org-wide rather than per-key, so a stricter bundle cannot be given its own retention or verbosity; see [Custom observability backends](/agent-router-enterprise/guides/observability-and-analytics/configure-custom-observability-backends/#guardrail-trigger-destinations). Keep the bundle in the **project** that owns the assistant traffic, and distinguish that traffic as in Step 2 (dedicated key, marked attribute, or path) so general traffic in the same organisation is not subjected to assistant-only controls. Where several rules apply to the same request, each is evaluated and a block from any one of them stops the request. ## Step 4: start in monitor mode A bundle that has never run against real assistant traffic will eventually redact something it should not or block a legitimate request. Before redact and block are allowed to affect callers, run every rule in the bundle in **monitor** mode against live assistant traffic for a period. The procedure is the same for any guardrail and is described once in [Validate in monitor mode before enforcing](/agent-router-enterprise/guides/operate-and-govern/safety-and-data-protection/configure-custom-guardrails/#step-6-validate-in-monitor-mode-before-enforcing). Two things are specific to assistant traffic: - Ordinary source code resembles the patterns a secret rule looks for. Placeholder tokens in test fixtures, example connection strings in documentation blocks, and high-entropy strings such as commit hashes all produce matches that must be resolved before enforcement, not after. - The pilot has to run long enough to cover the shape of real work, which for an assistant means more than one language, more than one repository, and the debugging sessions where data gets pasted into a buffer. A regular expression that matches more than intended is far cheaper to fix in monitor mode than once it is rejecting a developer's requests. ## Step 5: promote the bundle to enforcement With the bundle validated, switch mode from monitor to enforce for the redact or block actions decided in Step 3, and confirm the rules still apply only to the distinguished assistant traffic in the project. From this point the gateway enforces the stricter controls inline on every assistant request: a detected secret is redacted or the request is blocked (HTTP `403`), and each rule evaluation writes its guardrail-trigger record. It is worth re-checking the distinction explicitly at this step. A bundle that was validated against assistant keys in monitor mode but then applied to unrelated project traffic would begin blocking general traffic, which is exactly the surprise the monitor-mode pilot was meant to prevent. ## Step 6: coordinate with the developers who configure the assistants The distinction drawn in Step 2 lives partly on the developer side: in which API key an assistant is configured with, which header its configuration sets, or which endpoint it points at. If that configuration is wrong or absent, assistant traffic flows as general traffic and the stricter bundle never sees it. - Tell the development team which key, attribute, or path designates coding-assistant traffic, and confirm their assistant configuration uses it. The integration steps the developers follow are in [Integrate the gateway with an app](/agent-router-service/guides/integrate-the-gateway-with-an-app/). - Make clear what the stricter bundle does, so a redaction or block during normal work is understood as policy rather than a malfunction. A developer who knows that a blocked request means a secret was about to leave the organisation responds very differently from one who reads it as a broken tool. - Establish how a new assistant or a new developer is brought under the same controls, so the distinction does not erode as the team and its tooling change. The developer-facing view of attaching guardrails to a request path is covered in [Protect requests with guardrails](/agent-router-enterprise/guides/operate-and-govern/safety-and-data-protection/protect-requests-with-guardrails/); pointing the development team at it gives them the other half of the picture. ## Step 7: verify with a seeded secret The bundle is only proven once a request carrying a known secret has been sent through an assistant's path and observed to be blocked or redacted. This is the demonstration that the evaluation criterion asks for. 1. Take the non-production secret prepared in the prerequisites: a clearly fake but realistically shaped token, never a real credential. 1. Send it through the distinguished path, ideally from the configured assistant itself so the full path is exercised, embedding the seeded secret in a prompt that resembles real assistant content, such as a snippet of code containing the token. 1. Confirm the outcome the policy intends: a redaction action removes or masks the token before the request reaches the model, while a block action rejects the request so the token never leaves the gateway. 1. Confirm the guardrail-trigger record for that evaluation: which rule matched, the score, the action and mode, the stage, and the content type. Whether the record also carries the matched span depends on the controls described in Step 3, so this is the point at which to check that what reaches each destination matches what the policy allows for source code. 1. As a control, send a comparable request through a general (non-assistant) path and confirm the stricter bundle does not fire there, which proves the distinction from Step 2 is doing its job rather than the bundle catching everything indiscriminately. A pass on both halves (the secret is stopped on the assistant path and ordinary traffic on the general path is untouched) is the evidence that the controls distinguish assistant traffic and enforce the stricter policy against it. Note that a blocked response carries a correlation identifier, which an operator can later resolve to the rules that triggered and the content that caused the block; see [Guardrails best practices and FAQ](/reference/best-practices/guardrails-best-practices-and-faq/). ## What to do next - **Detect and redact sensitive data**: the broader treatment of sensitive-content detection that this bundle draws on. See [Detect and redact sensitive data](/agent-router-enterprise/guides/operate-and-govern/safety-and-data-protection/detect-and-redact-sensitive-data/). - **Configure guardrails**: the check types, providers, and match actions used to build the bundle. See [Configure guardrails](/agent-router-enterprise/guides/operate-and-govern/safety-and-data-protection/configure-custom-guardrails/). - **Integrate the gateway with an app**: how the assistants are pointed at the gateway, including the per-integration key boundary this guide reuses. See [Integrate the gateway with an app](/agent-router-service/guides/integrate-the-gateway-with-an-app/). - **Protect requests with guardrails**: the developer-side view to share with the team that configures the assistants. See [Protect requests with guardrails](/agent-router-enterprise/guides/operate-and-govern/safety-and-data-protection/protect-requests-with-guardrails/). - **Custom observability backends**: where guardrail-trigger records are delivered, and how much of the matched content each destination receives. See [Configure custom observability backends](/agent-router-enterprise/guides/observability-and-analytics/configure-custom-observability-backends/#guardrail-trigger-destinations). Where to go next Detect and redact sensitive data The broader treatment of sensitive-content detection that this bundle draws on. Configure guardrails The rule types and match actions used to build the bundle. --- ## Protect requests with guardrails Enterprise Tier A guardrail moves content inspection into the gateway, where project rules run inline on model traffic before a request reaches a provider and again before the response returns. Every integration whose key belongs to a project with active enforce-mode rules inherits that protection without code changes. This is the developer-facing view: which traffic is protected, how to recognise that a guardrail acted, and how to request coverage where it is missing. **Persona:** Developer working in the Developer Console. **Estimated time:** 10 to 15 minutes to review how guardrails apply to existing traffic and to confirm their effect in Request Logs. ## When this guide applies Guardrails are relevant whenever request or response content carries risk that should be handled before it crosses an application or provider boundary. The guide is especially useful in these situations: | Situation | What guardrails address | | :---- | :---- | | User-supplied prompts may contain PII | Redaction or blocking before content reaches an external provider | | Responses are shown directly to end users | Filtering of unsafe or disallowed content before it returns to the application | | A workload must meet a data-handling or compliance requirement | A consistent, centrally enforced control rather than per-service logic | | An integration must behave identically across multiple providers | Enforcement on the gateway applies regardless of which backend serves the request | Where the concern is which backend serves a request rather than what the request contains, [Route requests across providers](/agent-router-service/guides/route-requests-across-providers/) is the relevant guide instead. ## Outcomes By the end of this guide: - The role of a guardrail as an inline, gateway-enforced control is understood, along with the boundary between developer and operator responsibilities. - The way guardrails attach to traffic at the project is understood well enough to reason about which requests are protected. - A guardrail action can be recognised from the response signals, and the limits of what the Console shows about it are clear. - A request for guardrail coverage can be raised with the operator team with enough detail to act on. ## Prerequisites - A working API key with recent traffic against it, as set up in [Route requests across providers](/agent-router-service/guides/route-requests-across-providers/). - Familiarity with Request Logs, as covered in [Monitor traffic and usage](/agent-router-service/guides/monitor-traffic-and-usage/). Request Logs is the Console surface on which a refused request is located. - For requesting new coverage: a point of contact on the platform operator team, since guardrails are defined for a project in the Admin Dashboard rather than the Console. ## Step 1: understand how a guardrail acts on a request A guardrail is evaluated inline by the gateway on the model request path. Two points on that path can be inspected: - **On the request:** before the prompt is forwarded to a provider. A guardrail can redact matched content (for example, masking an email address or an account number) so that the sanitised prompt is what the provider receives, or it can block the request outright so that nothing is forwarded at all. - **On the response:** before the model output returns to the calling application. A guardrail can redact matched content from the response or block the response so that the disallowed content is never delivered. Standalone MCP profile traffic (`/mcp/*`) is not inspected by content guardrails. That path is governed by identity, access control, and audit instead. Rules use check types such as `pii`, `secrets`, `jailbreak`, and `toxicity`, each scored by a provider (`regexp`, `builtin`, `tetrate`, or another configured engine). The action on a match is redact or block; whether that action is applied is controlled by the rule's mode (enforce or monitor). Thresholds and patterns are operator-configured. From a developer's perspective, the contract is straightforward: a protected request is inspected on the way in and on the way out, and the gateway either passes it through, passes through a redacted version, or refuses it with HTTP `403`. ## Step 2: know which traffic is protected Guardrails are defined by the operator for a **project**, not chosen per request by the developer. A request is protected because it routes through a project that has active guardrails, not because the application opted in on the call itself. Protection is invisible in the request shape: the `model` field, the message content, and the headers are unaffected. This has two practical consequences: - **Protection follows the project.** Every integration whose API key belongs to a project with active enforce-mode rules is subject to those rules. Keys in a different project, or in the same project before any rules are enforced, are not. - **Coverage can differ between projects.** One project may carry a strict PII policy while another, used for an internal experiment, carries none. Naming keys by purpose (as in [Monitor traffic and usage](/agent-router-service/guides/monitor-traffic-and-usage/)) still helps when asking operators which project a key belongs to. Because the Console does not expose guardrail configuration to developers directly, the reliable way to confirm whether traffic is protected, and by what, is to ask the operator team. The response signals described in Step 3 confirm only that a rule acted, not what rules exist. ## Step 3: tell when a guardrail acted on a request A guardrail leaves an immediate signal in the response. The durable record of what a rule did is a guardrail-trigger record, which is an operator-side surface rather than a Console one. ### Response signals - **A redaction** is visible in the content itself. Where a guardrail masked matched text, the prompt the provider received, or the response the application received, contains the masked form rather than the original. A request that succeeds with altered content is the normal signature of a redaction guardrail. - **A block** is surfaced as a refusal rather than a model completion. The gateway returns HTTP `403`, carrying the guardrail's configured blocked-response message where one is set, and a correlation identifier. Application code that already handles non-success responses from the gateway will surface this in the same path it uses for other errors. The status code alone separates a guardrail block from the failures it is most often mistaken for: a rate limit returns `429` and an upstream failure returns `5xx`, so a `403` on a request that previously succeeded points at content rather than availability, and retrying it changes nothing. Step 5 covers why failover is not the remedy either. ### What the Console shows, and what it does not Request Logs records the request as the gateway processed it, so it confirms that a request was refused and with what status and message. The columns and detail panel are described in [Monitor traffic and usage](/agent-router-service/guides/monitor-traffic-and-usage/). For a request suspected of triggering a guardrail: 1. Open **Monitoring → Request Logs** in the Console. 1. Filter by the API key the integration uses and a time range covering the request. 1. Locate the request and open its detail panel. 1. Read the status and the error message in the response body. What Request Logs does not show is which rule fired, the score it returned, or whether the rule was in enforce or monitor mode. That detail lives in the guardrail-trigger record, which the operator team reads. The correlation identifier on the `403` is the value to quote when asking them, because an operator can resolve it to the rules that triggered and the content that caused the block; see [Guardrails best practices and FAQ](/reference/best-practices/guardrails-best-practices-and-faq/). A rule in monitor mode produces a trigger record and no response signal at all, so a monitored rule's matches are invisible from the Console by design. ## Step 4: request guardrail coverage for an application Because guardrails are defined and attached in the Admin Dashboard, adding or changing coverage for an application is a request to the operator team rather than a self-service action in the Console. A request that the operator can act on without a round trip includes: - **The project and API key or keys** the application uses, identified by name (for example, project `checkout` and key `checkout-service-prod`). Naming both ties the request to the policy boundary and the traffic path. - **The content of concern:** the categories of data or content that must be controlled, such as customer PII in prompts or disallowed categories in responses. - **The required action:** whether matched content should be redacted so the request still completes, or blocked so it does not. Redaction preserves functionality at the cost of altered content; blocking prioritises safety at the cost of failed requests. The right choice depends on the workload. - **The direction:** whether the concern is on the request (outbound to the model provider), the response (inbound to the application), or both. - **The expected traffic shape:** a representative example of a prompt and response helps the operator choose a check type and provider and tune the threshold without guesswork. The operator translates this into a concrete rule on the project's guardrail and validates it in monitor mode before enforcement. The mechanics of that work are covered in the operator guides linked under *What to do next*. Because monitor mode produces no response signal, the first observable effect on the application side is the one that appears when the rule is promoted to enforce. ## Step 5: reason about guardrails alongside routing and fallbacks Guardrails and routing operate on the same request but answer different questions. Routing decides *which backend* serves a request; a guardrail decides *whether and in what form* the content is allowed to pass. Because the guardrail is enforced on the gateway rather than per provider, a few properties hold regardless of routing configuration: - **Guardrails apply across every backend.** A request protected by a PII guardrail is inspected the same way whether routing sends it to the primary model or, after a failure, to a fallback. The protection does not have to be reconfigured per provider, and switching providers does not silently drop it. See [Improve resilience with fallbacks](/agent-router-service/guides/improve-resilience-with-fallbacks/) for the failover behaviour itself. - **A block is not a failure that fallback should retry.** Walking a fallback chain is the gateway's response to a backend that *failed* to serve a request. A request blocked by a guardrail was refused on content grounds, not failed on availability grounds, so it is not a candidate for failover to another backend; another provider would refuse the same content for the same reason. - **Redaction is consistent across a traffic split.** Where traffic is distributed by weight across two backends, a redaction guardrail sanitises content identically on whichever backend a given request lands on, so the split does not produce two different levels of protection. See [Reduce cost with traffic splitting](/agent-router-service/guides/reduce-cost-with-traffic-splitting/). The practical takeaway is that guardrails compose cleanly with routing: protection is a property of the project, applied uniformly across backends, rather than something that has to be re-established for each model provider a request might reach. ## What to do next - **[Configure guardrails](/agent-router-enterprise/guides/operate-and-govern/safety-and-data-protection/configure-custom-guardrails/)** (operator side): how the operator team creates project policies and rules. - **[Configure guardrail providers](/agent-router-enterprise/guides/operate-and-govern/safety-and-data-protection/configure-vendor-guardrails/)** (operator side): how the scoring engines (`regexp`, `builtin`, `tetrate`) differ. - **[Custom observability backends](/agent-router-enterprise/guides/observability-and-analytics/configure-custom-observability-backends/#guardrail-trigger-destinations)** (operator side): ship every guardrail-trigger record, full matched content included, from the data plane to an organisation's own SIEM or logging stack. - **[Monitor traffic and usage](/agent-router-service/guides/monitor-traffic-and-usage/):** the full Request Logs and Usage Analytics workflow used to locate and read a refused request. - **[Route requests across providers](/agent-router-service/guides/route-requests-across-providers/):** the routing setup that guardrails are layered on top of. - **[Glossary](/reference/glossary/):** definitions for guardrail, data plane, fallback policy, and the other terms used in this guide. Where to go next Monitor traffic and usage The Request Logs workflow used to locate a refused request. Route requests across providers The routing setup that guardrails are layered on top of. --- ## Secure MCP secrets and identity Enterprise Tier Once a Model Context Protocol (MCP) catalogue exists, the next governance problem is how the credentials behind that catalogue are held and how the gateway proves who is making each tool call. A server entry that carries a static bearer token in plain configuration is convenient and quietly dangerous: the token is readable by anyone with access to the configuration surface, it survives unchanged through every rotation cycle that nobody remembers to run, and when it leaks the blast radius is every request that server has ever served. Worse, when the downstream tool logs that activity, it attributes everything to a single shared identity rather than to the person who actually invoked it, which is exactly the attribution an auditor asks for and exactly what a shared token cannot provide. This guide covers the two practices that close those gaps. The first is sourcing MCP server secrets from an approved secret store (HashiCorp Vault or a cloud key management service (KMS)) so that the gateway fetches a credential at the moment it is needed rather than reading it from a configuration field. The second is identity pass-through: configuring OAuth and API-key delegation so the downstream tool receives a credential scoped to the real end user, and propagating that end-user identity into the tool call so the downstream tool's own logs name the person who acted. Together they turn the MCP catalogue from a set of shared secrets into a set of governed, auditable, per-user delegations. **Persona:** Platform operator working in the Admin Dashboard, in coordination with the security team that owns the secret store and the corporate identity provider. **Estimated time:** 45 to 90 minutes for a first integration, depending on how quickly access to the secret store and the downstream tool's OAuth registration can be arranged. ## When this guide applies This guide is the right starting point in any of these situations: | Situation | What it covers | | :---- | :---- | | Removing plaintext MCP server secrets from configuration | Pointing server entries at an approved secret store instead of an inline token | | Integrating HashiCorp Vault or a cloud KMS with Agent Router | Establishing the trust relationship and reference syntax used to resolve secrets at use time | | Delegating credentials to a downstream tool | Configuring OAuth or API-key pass-through so the tool receives a credential scoped to the caller | | Making downstream tool logs name the real user | Propagating end-user identity into the tool call so attribution survives the hop through the gateway | | Preparing for a security or compliance review of MCP access | Demonstrating that secrets resolve from the store and that user identity reaches downstream logs | This guide extends [Govern MCP server access](/agent-router-enterprise/guides/operate-and-govern/govern-mcp-server-access/), which establishes the MCP server catalogue and OAuth client configurations this guide secures. Read that guide first; the catalogue it builds is assumed throughout. The developer-side flow of assembling servers into a profile is covered in [Aggregate MCP servers into a profile](/agent-router-service/guides/aggregate-mcp-servers-into-a-profile/). ## Outcomes By the end of this guide: - At least one MCP server resolves its credential from an approved secret store rather than from a value stored in configuration. - At least one OAuth pass-through flow is configured and demonstrated, with the downstream tool receiving a credential scoped to the end user. - End-user identity is propagated to the tool call and is visible in the downstream tool's own logs. - The relationship between secret-store integration, identity pass-through, and Agent Router's single sign-on (SSO) and group mapping is clear. ## Prerequisites - Administrator access to the Admin Dashboard, typically the `super_admin` or `mcp_admin` role. - A populated MCP catalogue from [Govern MCP server access](/agent-router-enterprise/guides/operate-and-govern/govern-mcp-server-access/), including any OAuth client configurations the relevant servers require. - An approved secret store Agent Router is permitted to read from (a HashiCorp Vault instance or a cloud KMS) and the credentials or workload identity the gateway uses to authenticate to it. The security team usually owns this provisioning. - For OAuth pass-through: an application registered with the downstream tool's OAuth provider, plus the corresponding OAuth client configured in Agent Router as described in the governance guide. - For identity propagation: SSO configured so that signed-in users carry a verified identity. See [Configure SSO](/agent-router-enterprise/guides/identity-and-access-management/single-sign-on/sso-configuration-guide/). ## Step 1: decide which credentials belong in the secret store Not every value needs the same treatment, and being explicit about which do avoids both over-engineering and gaps. The distinction that matters is whether a value is a standing secret that grants access on its own. | Value | Treatment | | :---- | :---- | | Static bearer tokens for MCP servers | Move to the secret store; these grant access on their own and are the highest-value target | | OAuth client secrets | Move to the secret store; a leaked client secret undermines every delegated flow that uses it | | OAuth client IDs, authorization URLs, token URLs | May remain in configuration; these are identifiers, not secrets | | Per-user OAuth access tokens | Not stored by the operator at all; these are obtained per user at call time and never live in configuration | The guiding principle is that a credential should be fetched from the store at the moment it is used, not copied into a configuration field where it then has to be rotated separately and audited separately. Anything that grants access on its own belongs in the store. ## Step 2: connect Agent Router to an approved secret store Before a server entry can reference a stored secret, Agent Router has to be able to authenticate to the store and read the specific paths it is permitted to read. This is a one-time integration, performed with the security team. 1. Confirm with the security team which secret store backs this deployment (HashiCorp Vault or a cloud KMS) and which paths or key names the gateway is permitted to read. 1. Provision the identity the gateway uses to authenticate to the store. For Vault this is typically a role bound to the gateway's workload identity; for a cloud KMS it is typically the deployment's service account granted a decrypt or secret-access permission. The data plane runs in customer-managed infrastructure, so this binding is established on the customer side. 1. In the Admin Dashboard, open the secret-store integration settings and register the store: its address, the authentication method the gateway uses, and the scope of paths it may resolve. 1. Save the integration and confirm Agent Router reports a successful connection to the store. Scope the gateway's access to the store as narrowly as the store allows. A read-only binding limited to the specific MCP secret paths means that a compromise of the gateway cannot be used to read unrelated secrets, which is the entire point of putting the secrets in the store in the first place. ## Step 3: point an MCP server at a stored secret With the integration in place, a server's credential can be expressed as a reference into the store rather than as a literal value. 1. Open **MCP Servers** in the Admin Dashboard and open the server whose secret is to be externalised. 1. Locate the field that currently holds the bearer token or other inline secret. 1. Replace the literal value with a reference into the configured secret store. The exact syntax depends on the store; a Vault reference resolves a path and key, while a cloud KMS reference resolves a named secret. A representative Vault-style reference: ```text vault:secret/data/mcp/github-issues#token ``` 1. Save the server entry. At use time, the gateway resolves the reference against the store, retrieves the current value, and applies it to the upstream request. Because the value is fetched on each use rather than copied, a rotation performed in the store takes effect without any change to Agent Router configuration; the next tool call simply receives the new value. Removing the literal secret from configuration is the change a security review measures. The configuration now records *where* the secret lives, not the secret itself. ## Step 4: configure OAuth pass-through to the downstream tool API-key pass-through and OAuth pass-through solve the same problem, delivering the right credential to the downstream tool, but they differ in whose credential is delivered. Pass-through is what lets a tool act with the correct identity rather than with a single shared account. | Pass-through type | What reaches the downstream tool | When it applies | | :---- | :---- | :---- | | API-key pass-through | A key resolved from the secret store and attached to the upstream request | Tools that authenticate with a service or shared key held by Agent Router | | OAuth pass-through | A per-user access token obtained through the OAuth flow and attached on that user's behalf | Tools that grant access on behalf of a specific authenticated user | For API-key pass-through, the work is already done: a server configured with a bearer token resolved from the store (Step 3) attaches that key to every upstream request. The remaining steps below configure OAuth pass-through, where the credential delivered to the tool is scoped to the individual end user. 1. Confirm the downstream tool's MCP server entry has its auth type set to **OAuth** and is mapped to the correct OAuth client, as established in [Govern MCP server access](/agent-router-enterprise/guides/operate-and-govern/govern-mcp-server-access/). Confirm the client's secret resolves from the secret store rather than from an inline value. 1. Confirm the OAuth client requests the scopes the tool requires for the actions end users will perform. A token issued without a needed scope produces a downstream authorization failure that is easy to mistake for a gateway error. 1. Have a Console user reach the profile URL that includes this server. The OAuth flow runs: the user is redirected to the provider, authenticates, grants consent, and Agent Router exchanges the resulting code for a per-user access token. 1. The gateway attaches that per-user token to the tool call. The downstream tool now sees a credential that belongs to the consenting user, not a shared one. The result is delegation rather than impersonation. Each user's tool calls carry that user's own access token, so the downstream tool enforces that user's own permissions: a request a user is not entitled to make fails at the tool, not silently succeeds under a shared account's broader rights. ## Step 5: propagate end-user identity into the tool call OAuth pass-through gives the downstream tool a per-user token, which already carries identity in many cases. Where the tool keeps its own activity log keyed on a header or claim rather than solely on the token, the gateway can additionally propagate the verified end-user identity so that the tool's log names the real person. 1. Confirm SSO is configured and enabled so that every signed-in user carries a verified identity. The identity propagated downstream is the one the identity provider asserted, not a value the user can set. See [Configure SSO](/agent-router-enterprise/guides/identity-and-access-management/single-sign-on/sso-configuration-guide/). 1. In the server's configuration, enable identity propagation and select which attribute is forwarded, typically the user's email or a stable subject identifier. Avoid forwarding more than the downstream tool needs to attribute the action. 1. Where the downstream tool reads identity from a specific request header, configure the gateway to place the chosen attribute in that header on the upstream tool call. 1. Where group-based attribution matters (for example, distinguishing actions by business function in the downstream log) the same group mapping that drives Agent Router roles can inform what is propagated. See [Map Entra ID groups to business functions](/agent-router-enterprise/guides/identity-and-access-management/single-sign-on/map-entra-id-groups-to-business-functions/). Identity propagation depends on SSO supplying a trustworthy identity in the first place. A locally created account with a self-asserted email propagates a self-asserted email; the value of the audit trail is only as strong as the identity provider behind it. ## Step 6: verify secrets resolve and identity appears downstream The two claims this guide makes, that secrets come from the store and that identity reaches the downstream log, are both verifiable, and both should be confirmed before the configuration is treated as done. ### Verify the secret resolves from the store 1. Confirm the server's secret field holds a store reference, not a literal value, by reopening the server entry in **MCP Servers**. 1. Exercise the server through a profile and confirm the tool call succeeds, which establishes that the gateway resolved the reference to a working credential. 1. Rotate the secret in the store and exercise the server again without changing any Agent Router configuration. A successful call after rotation confirms the value is being fetched at use time rather than cached from configuration. ### Verify end-user identity in downstream logs 1. Have two distinct users each invoke a tool on the same server through their own profile sessions. 1. With the security team, inspect the downstream tool's own activity log. 1. Confirm each action is attributed to the correct individual user, with distinct identities for the two users, rather than to a single shared service account. A failure on either check points to a specific cause: an unresolved secret reference indicates the store binding or the reference syntax is wrong, while a single shared identity in the downstream log indicates either OAuth pass-through is not active for that server or identity propagation is not configured. The Agent Router's own [audit log](/agent-router-enterprise/guides/operate-and-govern/audit-platform-activity/) records the configuration changes made in this guide and complements the downstream tool's log on the access side. ## What to do next - **Govern MCP server access**: revisit the catalogue this guide secures to add, retire, or re-map servers and OAuth clients. See [Govern MCP server access](/agent-router-enterprise/guides/operate-and-govern/govern-mcp-server-access/). - **Aggregate MCP servers into a profile**: the developer-side flow that consumes the secured servers configured here. See [Aggregate MCP servers into a profile](/agent-router-service/guides/aggregate-mcp-servers-into-a-profile/). - **Configure SSO**: the SSO configuration that supplies the verified identity propagated to downstream tools. See [Configure SSO](/agent-router-enterprise/guides/identity-and-access-management/single-sign-on/sso-configuration-guide/). - **Map Entra ID groups to business functions**: drive group-based attribution from the corporate directory. See [Map Entra ID groups to business functions](/agent-router-enterprise/guides/identity-and-access-management/single-sign-on/map-entra-id-groups-to-business-functions/). - **Audit Agent Router activity**: review the events generated by the configuration changes made here. See [Audit Agent Router activity](/agent-router-enterprise/guides/operate-and-govern/audit-platform-activity/). Where to go next Govern MCP server access Revisit the catalogue this guide secures to add, retire, or re-map servers and OAuth clients. Configure SSO The SSO configuration that supplies the verified identity propagated to downstream tools. --- ## Admin Dashboard quickstart Enterprise Tier This 10-minute walkthrough covers the essential Admin dashboard tasks for a Fully Managed deployment: confirming the default project endpoint, reviewing model and provider access, managing user accounts, and monitoring organization-wide usage and audit history. The Admin dashboard is the operator-facing application. Administrators control which AI models are available to developers, manage the credentials the gateway uses to reach upstream providers, configure user access and roles, and maintain a tamper-resistant audit trail of all Agent Router changes. :::info The screenshots in this guide show a redacted URL. In a production environment the URL takes the form `dashboard..tetrate.ai`, and it differs between production and non-production deployments. ::: ## Step 1: sign in 1. Navigate to the Admin dashboard URL 1. Sign in with administrator credentials or SSO ![Sign in page](/img/screenshots/admin/quickstart-sign-in.jpeg) 1. If the organization uses single sign-on, click **Sign in with SSO** to authenticate through the corporate identity provider ![Sign in with SSO](/img/screenshots/admin/quickstart-sign-in-sso.jpeg) 1. The landing page is **Home**, showing an Agent Router-level overview: aggregate request volume, active users, model usage distribution, and any recent administrative events ![Admin Home](/img/screenshots/admin/quickstart-home.jpeg) ## Step 2: confirm the default project is ready Fully Managed deployments include a Tetrate-operated data plane. Onboarding sets up the default project automatically, so the gateway endpoint, the provider(s) connected, and the models selected are already in place. Project membership is the one exception: it is set by the onboarding wizard's Admin email field, not by anything else. A quick look confirms all of it before developers create keys or send traffic: 1. Open the project selector and choose the default project. 1. Open the project's **Gateway** tab and confirm the default gateway's **endpoint** matches the Gateway URL Tetrate assigned for the deployment, for example `https://proxy..tetrate.ai`. The value is already set, so there is nothing to configure. A data plane can host several gateways; this confirms the one the default project uses. 1. Confirm the administrator account is listed as a project member. If it is missing (the Admin email field was left blank during onboarding), add it with **Grant access**. 1. Confirm at least one provider is connected and enabled for the project. 1. Open the project-scoped **Models** page and confirm the models developers should call are linked and enabled. :::important Data-plane credentials are managed by Tetrate Do not generate or regenerate a data-plane service-account credential from the Admin Dashboard for a Fully Managed deployment. That action rotates the credential used by the managed data plane and can break it. If it is regenerated by mistake, contact Tetrate to restore service. ::: ## Step 3: review available models The model catalog controls which AI models are visible and accessible to Console users. Administrators can enable or disable models globally, and configure per-model settings such as context window limits or cost caps. Developers never interact with provider credentials; they select models by name, and the gateway routes to the appropriate provider automatically. 1. Go to **Models** in the sidebar 1. Browse the list of AI models in Agent Router. Each entry shows the model name, the provider it is sourced from, and its current availability status. ![Models page](/img/screenshots/admin/quickstart-models.jpeg) 1. Use the actions menu to enable or disable a model, or to update its configuration. Disabling a model removes it from the Console model catalog immediately; requests that reference it by name will receive a 404 error until the model is re-enabled or the developer updates their routing configuration. ![Models actions](/img/screenshots/admin/quickstart-models-actions.jpeg) ## Step 4: review providers Providers are the upstream AI services that Agent Router routes requests to, for example, OpenAI, Anthropic, Azure OpenAI, or Google Vertex AI. Each provider entry in the Admin dashboard holds the credentials and endpoint configuration the gateway uses when forwarding requests. Developers never see or use these credentials directly. 1. Go to **Providers** in the sidebar 1. View the list of configured AI provider connections. Each entry shows the provider name, connection status (healthy or unhealthy), and the number of models sourced from that provider. ![Providers page](/img/screenshots/admin/quickstart-providers.jpeg) 1. Use the actions menu to view connection details, update API credentials, or configure a custom endpoint URL (for example, a private endpoint for Azure OpenAI via Azure Private Link). Connection health is tested automatically; an unhealthy status indicates a credential or network issue that will prevent the gateway from routing to that provider. ![Providers actions](/img/screenshots/admin/quickstart-providers-actions.jpeg) ## Step 5: view users User management controls who can access Agent Router and with what level of permission. Each user is assigned one or more roles that determine which applications they can access (Console, Admin dashboard) and what operations they can perform within each. 1. Go to **Users** in the sidebar 1. Review the list of registered users, their assigned roles, and their last sign-in timestamps. This view is useful for identifying inactive accounts, confirming role assignments after onboarding, and auditing who has elevated access. ![Users page](/img/screenshots/admin/quickstart-users.jpeg) 1. Use the actions menu to modify a user's role, reset their credentials, or deactivate an account. Role changes take effect immediately; the user's next request is evaluated against their updated permissions. ![Users actions](/img/screenshots/admin/quickstart-users-actions.jpeg) ## Step 6: check usage analytics **Usage → Analytics** provides an organization-wide view of AI consumption across all users and API keys. Use it to understand cost trends, identify the heaviest consumers, compare model popularity, and inform decisions about model availability. 1. Go to **Usage → Analytics** in the sidebar 1. View aggregate usage across Agent Router. The default view shows total requests, token consumption, and cost estimates across all users and API keys. 1. Drill down to identify the top consumers by API key or user, the most frequently requested models, and cost trends over configurable time periods. To compare several people or teams on the same chart, see [Compare user and team spend in Analytics](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/compare-user-and-team-spend-in-analytics/). ![Usage Analytics](/img/screenshots/admin/quickstart-usage.jpeg) ## Step 7: view model metrics Model Metrics provides performance telemetry broken down by individual model. Use it to evaluate model reliability and latency before expanding availability, and to detect provider degradation early. 1. Go to **Model Metrics** in the sidebar 1. Review per-model performance data, including: - **Latency**: median and p95 response times, which surface models with inconsistent performance - **Throughput**: requests per minute, a measure of relative demand across the model catalog - **Error rates**: the proportion of requests that resulted in provider errors, which surface degraded provider health before it reaches developers ![Model Metrics](/img/screenshots/admin/quickstart-model-metrics.jpeg) ## Step 8: review audit logs Audit Logs record every administrative action taken in the Admin dashboard: model changes, provider credential updates, user modifications, SSO configuration edits, and more. The log is tamper-resistant and provides a complete accountability trail suitable for compliance reporting and incident investigation. 1. Go to **Audit Logs** in the sidebar 1. View a chronological history of administrative actions. Each entry records the actor, the action taken, the affected resource, and a precise timestamp. ![Audit Logs](/img/screenshots/admin/quickstart-audit-logs.jpeg) 1. Click **Refresh** to load the latest entries ![Audit Logs refresh](/img/screenshots/admin/quickstart-audit-logs-refresh.jpeg) 1. Click on any log entry to expand its full details, including the before and after state of the modified resource. This is particularly useful when investigating unexpected configuration changes or rolling back to a known-good state. ![Audit Log detail](/img/screenshots/admin/quickstart-audit-log-detail.jpeg) ## Evaluation checkpoint - [ ] Signed in to the Admin dashboard - [ ] Confirmed the default project has the correct gateway endpoint, project membership, provider connection, and enabled project models - [ ] Reviewed the model catalog and understand how to enable and disable models - [ ] Confirmed that all provider connections are healthy - [ ] Reviewed the user list and verified role assignments are correct - [ ] Reviewed organization-wide usage analytics and identified top consumers - [ ] Reviewed per-model latency and error rate metrics - [ ] Reviewed audit logs and understand what actions are captured Where to go next Console quickstart See the developer experience end to end. Managing models and providers Complete reference for the operator workflows. --- ## Data plane installation The data plane is a Kubernetes deployment containing the Controller and the Agent Router. This step installs those components in the target cluster and connects them to the management plane. Select the guide that matches the target cloud platform and preferred installation tooling. Select a guide Azure · AKS · tare CLI Azure using tare CLI Install on Azure Kubernetes Service using the tare CLI. The recommended path for AKS deployments when the tare CLI is available in the environment. Azure installation guide → Azure · AKS · Helm Azure using Helm Install on Azure Kubernetes Service using Helm and a private container registry such as Azure Container Registry. Use this guide when a private registry is required by the environment. Azure Helm guide → AWS · EKS · tare CLI AWS using tare CLI Install on Amazon Elastic Kubernetes Service using the tare CLI. AWS installation guide → GCP · GKE · tare CLI GCP using tare CLI Install on Google Kubernetes Engine using the tare CLI. GCP installation guide → Any cloud · Helm Helm (cloud-agnostic) Helm-based installation for platform teams that own the Helm release lifecycle and maintain a private container registry such as Harbor, JFrog Artifactory, Nexus, or Google Artifact Registry. Helm installation guide → Before you begin All data plane guides assume the following steps are already complete: - [Prerequisites](/agent-router-enterprise/self-hosted-data-plane/prerequisites/) confirmed: access, accounts, cloud quotas, and network rules. - [Management plane onboarding](/agent-router-enterprise/self-hosted-data-plane/onboarding/) complete through Step 5: organization details, providers, models, and the data plane credential (`service-account.json`). - Target Kubernetes cluster provisioned and meeting the [Kubernetes version requirement](/agent-router-enterprise/self-hosted-data-plane/prerequisites/#kubernetes-requirements). Cluster provisioning is not covered by Tetrate support. SSO is **not** required before data plane installation. Configure SSO after the gateway is running and before opening the Console to developers — see [SSO configuration](/agent-router-enterprise/guides/identity-and-access-management/single-sign-on/sso-configuration-guide/). After installation Gateway installation Next step: install the data plane gateway components that manage inbound access and configure request routing. SSO configuration After the gateway is running, connect an OIDC identity provider before developer handoff. Console quickstart Once SSO is working and the gateway is reachable, issue an API key and make a first routed request from the Console. --- ## Installation guide for AWS # Agent Router data plane installation for AWS This guide installs the Agent Router data plane on AWS Elastic Kubernetes Service (EKS). ## Overview Agent Router uses a split-plane model. A **Management Plane** hosted by **Tetrate** holds configuration and exposes the web applications. A **Data Plane** runs in your AWS account and handles all AI traffic. The two planes communicate over one outbound HTTPS connection initiated by the data plane; no inbound internet connections are required. ```mermaid architecture-beta group aws(cloud)[AWS] group tetrate(cloud)[Tetrate hosted] service mp(server)[Management Plane] in tetrate service dp(server)[Data Plane] in aws mp:R -- L:dp ``` By the end of this guide you will have: - An EKS cluster with the EBS CSI driver installed - An ECR registry mirroring the Agent Router images - The Agent Router data plane running in the `tars-system` and `tars-dataplane` namespaces - An Application Load Balancer fronting the data plane at a DNS name you own - An OpenTelemetry pipeline forwarding metrics to your observability stack Plan for 30–45 minutes of installation time, plus DNS propagation. ## Table of contents - **Prepare for the installation**: obtain the data plane credential and install the CLI * [Prerequisites](#prerequisites) * [Step 1: Obtain your data plane credential](#step-1-obtain-your-data-plane-credential) * [Step 2: Install the tare CLI](#step-2-install-the-tare-cli) - **Cluster setup**: create or reuse a Kubernetes cluster * [Step 3: Provision the EKS cluster](#step-3-provision-the-eks-cluster) * [Step 4: Install the EBS CSI driver addon](#step-4-install-the-ebs-csi-driver-addon) - **Registry setup**: mirror Agent Router images so the cluster can pull them * [Step 5: Create ECR repositories](#step-5-create-ecr-repositories) * [Step 6: Authenticate Docker to ECR](#step-6-authenticate-docker-to-ecr) * [Step 7: Sync Agent Router images to your ECR](#step-7-sync-agent-router-images-to-your-ecr) * [Step 8: Create the image-pull secret](#step-8-create-the-image-pull-secret) - **Data Plane installation**: deploy the data plane * [Step 9: Install the data plane](#step-9-install-the-data-plane) - **Ingress setup**: expose the data plane externally * [Step 10: Install the AWS Load Balancer Controller](#step-10-install-the-aws-load-balancer-controller) * [Step 11: Create the ALB Ingress](#step-11-create-the-alb-ingress) - **DNS configuration**: wire the hostname to the ingress and register the URL on the management plane * [Step 12: Wire DNS and register the URL](#step-12-wire-dns-and-register-the-url) - **Testing the installation**: verify the install works end-to-end * [Step 13: Verify the install](#step-13-verify-the-install) * [Step 14: Smoke tests](#step-14-smoke-tests) - **Operations** * [Upgrading](#upgrading) * [Cleanup](#cleanup) * [Troubleshooting](#troubleshooting) - **Appendices** * [Appendix A: TLS via ACM \+ HTTPS listener](#appendix-a-tls-via-acm--https-listener) * [Appendix B: Observability with OpenTelemetry](#appendix-b-observability-with-opentelemetry) ## Prerequisites ### Dashboard and Router app access Agent Router has two web surfaces. Both URLs are provided during onboarding: * **Dashboard** (admin): `https://dashboard..tetrate.ai`. Used in [Step 1](#step-1-obtain-your-data-plane-credential) and [Step 12](#step-12-wire-dns-and-register-the-url). * **Router app** (end-user): `https://router..tetrate.ai`. Used in [Step 14](#step-14-smoke-tests) to create API keys for callers. ### Tools Install the following on the host that will run the guide: | Tool | Install | | :- | :- | | aws CLI v2 | [https://docs.aws.amazon.com/cli/latest/userguide/install.html](https://docs.aws.amazon.com/cli/latest/userguide/install.html) | | eksctl ≥ 0.220 | [https://eksctl.io/installation/](https://eksctl.io/installation/) (or brew install eksctl). Older versions do not recognise current EKS K8s versions. | | kubectl | [https://kubernetes.io/docs/tasks/tools/](https://kubernetes.io/docs/tasks/tools/) | | helm (3+) | [https://helm.sh/docs/intro/install/](https://helm.sh/docs/intro/install/) | | docker | [https://docs.docker.com/get-docker/](https://docs.docker.com/get-docker/) | | curl | preinstalled on macOS and most Linux distros | | tare CLI | covered in [Step 2](#step-2-install-the-tare-cli) | A `data-plane-credentials.json` file is also required; see [Step 1](#step-1-obtain-your-data-plane-credential). ### Infrastructure A dedicated workload cluster must be provisioned before starting the installation. The cluster should consist of at least three (3) nodes. See [Cluster sizing](#cluster-sizing) for more details. :::warning Tetrate support does not cover client-side infrastructure provisioning or Kubernetes issues. The instructions for creating clusters and related infrastructure components are provided as a courtesy and should be carefully evaluated before executing them. ::: ### AWS permissions The operator needs permission to manage these AWS services on the target account: | Service | Used for | | :- | :- | | EKS (AmazonEKSClusterPolicy, AmazonEKSWorkerNodePolicy) | Cluster creation and node groups | | EC2 \+ VPC | Cluster networking, EBS volumes, ELB provisioning | | IAM | OIDC provider, IRSA roles for EBS CSI driver and AWS Load Balancer Controller | | CloudFormation | eksctl deploys everything via CloudFormation stacks | | ECR | Create repositories, push/pull images | | ELB (Elastic Load Balancing) | ALB created by the AWS Load Balancer Controller | The simplest setup is an admin role on a sandbox account. For a constrained role, AWS publishes the minimum policy `eksctl` needs at [https://eksctl.io/usage/minimum-iam-policies/](https://eksctl.io/usage/minimum-iam-policies/). ### Cluster sizing The chart's default install runs multiple always-on components, including the **Agent Router egress proxy** (Envoy-based; referred to as *egress envoy* in manifests and logs) with a minimum of 2 replicas, AI gateway controller and ext\_proc, controller / worker, Redis / rate-limit. This is not a single-node footprint. The **egress envoy** is the dominant resource consumer. Both CPU and memory usage grow with the **configuration size** the proxy holds in memory: the number of AIGatewayRoute and AIServiceBackend resources, header-mutation rules, and other per-route features. The AI gateway team's [control-plane scaling benchmark](https://aigateway.envoyproxy.io/blog/benchmarking-control-plane-scaling) shows roughly linear CPU and memory growth from adding routes, with memory staying elevated to keep the xDS state available to serve traffic. Plan for routes to scale up as the data plane adds providers, models, and projects. General-purpose EC2 families (m5.\* or m6i.\*, balanced CPU/RAM) are the right default. | Size | Use case | Recommended node pool | Approx allocatable target | | :- | :- | :- | :- | | Small | dev / test / low traffic | 3 × m5.large | ≥ 6 vCPU, ≥ 20 GiB RAM | | Medium | staging / light production | 3 × m5.xlarge | ≥ 12 vCPU, ≥ 40 GiB RAM | | High | production with burst headroom | 3 × m5.2xlarge (or split into system \+ data plane node groups) | ≥ 24 vCPU, ≥ 80 GiB RAM | Practical floor: 3 nodes minimum to survive a node drain or upgrade. Demo installs can start at Small; production should start at Medium. ## Conventions The steps below export environment variables (`AWS_REGION`, `EKS_CLUSTER_NAME`, `AWS_ACCOUNT`, `ECR_HOST`, `CREDENTIAL_FILE`, `SERVE_URL`, and others) as they become needed. Each later step assumes the variables defined earlier are still exported in the current shell. If you start a new shell mid-install, re-export them before continuing. ## Step 1: obtain your data plane credential In the dashboard, navigate to **System → Settings → Data plane credentials → \+ Generate Data plane credential**. Save the downloaded file as `data-plane-credentials.json` on the host where the install runs. This file is the long-lived identity the data plane uses to authenticate to the management plane. :::note Some parts of the product still use older "service account" naming for this file. CLI output and in-cluster paths may reference it as a service account; it is the same file. The dashboard is in the process of standardizing on "data plane credential". Each data plane uses its own credential. Revoke a credential from the dashboard or generate additional ones (for example, one per environment) at any time. ::: ## Step 2: install the tare CLI ```shell curl -sSL https://tare.tetrate.ai/tools/install.sh | bash ``` Output: ```none ==> tare installer ==> channel: stable ==> Detected platform: darwin-arm64 ==> Installing tare for darwin-arm64... ==> Downloading from: https://tare.tetrate.ai/tools/tags/v0.1.0-beta.2/tare-darwin-arm64.tar.gz ok Installed tare to /Users/johndoe/.tare/bin/tare ==> tare version: tare version v0.1.0-beta.2 ok Installation directory is already in your PATH ==> Get started: tare install identity.json --serve-url https://proxy.acme.com tare install --help ``` The installer prints the install path (typically `~/.tare/bin/tare`). Add it to `PATH` and verify the version: ```shell export PATH="$PATH:$HOME/.tare/bin" echo 'export PATH="$PATH:$HOME/.tare/bin"' >> ~/.zshrc # or ~/.bashrc ``` ```shell $ tare --version tare version v0.1.0-beta.2 ``` ## Step 3: provision the eks cluster ### Step 3.1: set environment variables ```shell AWS_REGION= # e.g. us-east-1 EKS_CLUSTER_NAME= AWS_ACCOUNT=$(aws sts get-caller-identity --query Account --output text) # Pick the regional default Kubernetes version so the version doesn't # go out of support unexpectedly. K8S_VERSION=$(aws eks describe-cluster-versions \ --region "${AWS_REGION}" \ --query 'clusterVersions[?defaultVersion==`true`].clusterVersion' \ --output text) # Optional resource tags — adapt or drop these for your organization. TAGS="Owner=,Team=,Purpose=development" ``` :::tip Confirm `eksctl version` reports ≥ 0.220 before continuing. Older binaries reject the K8s version EKS now defaults to and the cluster create call fails with `invalid version, supported values: 1.23, …, 1.31`. ::: ### Step 3.2: log in to AWS ```shell aws sts get-caller-identity ``` ### Step 3.3: create the eks cluster ```shell eksctl create cluster \ --name "${EKS_CLUSTER_NAME}" \ --region "${AWS_REGION}" \ --version "${K8S_VERSION}" \ --nodes 3 \ --nodes-min 3 \ --nodes-max 3 \ --node-type m5.xlarge \ --with-oidc \ --tags "${TAGS}" ``` Provisioning takes \~15 minutes. `--with-oidc` enables IAM Roles for Service Accounts (IRSA), which the EBS CSI driver and AWS Load Balancer Controller depend on. The `--nodes 3` and `--node-type m5.xlarge` values above match the **Medium** tier in [Cluster sizing](#cluster-sizing). Adjust for your environment. ### Step 3.4: verify ```shell kubectl get nodes ``` Expected: ```none NAME STATUS ROLES AGE VERSION ip-192-168-14-164.ec2.internal Ready 2m v1.35.x-eks-... ip-192-168-25-245.ec2.internal Ready 2m v1.35.x-eks-... ip-192-168-40-57.ec2.internal Ready 2m v1.35.x-eks-... ``` :::tip **Already have an EKS cluster?** Reuse it after confirming that: * OIDC provider is associated: `aws eks describe-cluster --name --region --query "cluster.identity.oidc.issuer"` * EBS CSI driver is installed: `aws eks list-addons --cluster-name --region --query 'addons' | grep aws-ebs-csi-driver` * K8s version is supported (`aws eks describe-cluster-versions --region ` shows the cluster's version is not deprecated) ::: ## Step 4: install the ebs csi driver addon The Agent Router egress proxy and Redis use persistent volumes, and EKS does not install a CSI driver by default. ```shell # IAM role for the addon (one-shot — keeps the role even after cluster recreation) eksctl create iamserviceaccount \ --name ebs-csi-controller-sa \ --namespace kube-system \ --cluster "${EKS_CLUSTER_NAME}" \ --region "${AWS_REGION}" \ --attach-policy-arn arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy \ --approve --role-only \ --role-name "${EKS_CLUSTER_NAME}-AmazonEKS_EBS_CSI_DriverRole" ``` ```shell # Install the addon itself eksctl create addon \ --cluster "${EKS_CLUSTER_NAME}" \ --name aws-ebs-csi-driver \ --region "${AWS_REGION}" \ --service-account-role-arn "arn:aws:iam::${AWS_ACCOUNT}:role/${EKS_CLUSTER_NAME}-AmazonEKS_EBS_CSI_DriverRole" \ --force ``` Wait for it to become `ACTIVE` (\~1 min): ```shell aws eks describe-addon --cluster-name "${EKS_CLUSTER_NAME}" \ --addon-name aws-ebs-csi-driver --region "${AWS_REGION}" \ --query 'addon.status' --output text ``` ## Step 5: create ecr repositories Agent Router images must live in your registry. Unlike most cloud registries, ECR requires each repository to be pre-created; `tare install --image-sync` does not auto-create them. The set of repos changes across tare releases. Rather than hand-maintaining a list, this guide uses a helper that loops `tare install --sync-only`, parses any `NAME_UNKNOWN` errors, creates the missing repo, and retries. ```shell ECR_HOST="${AWS_ACCOUNT}.dkr.ecr.${AWS_REGION}.amazonaws.com" ECR_PREFIX="${EKS_CLUSTER_NAME}" # repos will be created as ${EKS_CLUSTER_NAME}/ ``` The helper is in this repo at `tests/aws-dp-install/sync-images.sh`. Run it after the next step. ## Step 6: authenticate Docker to ecr ```shell aws ecr get-login-password --region "${AWS_REGION}" | docker login --username AWS --password-stdin "${ECR_HOST}" ``` ECR tokens expire after 12 hours. Re-run this command if subsequent steps fail with `unauthorized: authentication required`. ## Step 7: sync Agent Router images to your ecr Save `data-plane-credentials.json` to a known path, then run: ```shell CREDENTIAL_FILE=/path/to/data-plane-credentials.json SERVE_URL="http://" # e.g. http://proxy.example.com bash tests/aws-dp-install/sync-images.sh ``` The helper creates any missing ECR repos as `tare` reports them. Expect \~3–5 minutes on the first run. On success it prints: ```none ✓ Image sync done in 2m22s === sync complete === ``` :::note `--sync-only` still requires a `--serve-url` value (chicken-and-egg with the management plane URL registration). Use the hostname already pre-cleared with the DNS owner; the URL is registered in the next install step. ::: ## Step 8: create the image-pull secret ```shell kubectl create ns tars-system --dry-run=client -o yaml | kubectl apply -f - kubectl create ns tars-dataplane --dry-run=client -o yaml | kubectl apply -f - ECR_TOKEN=$(aws ecr get-login-password --region "${AWS_REGION}") for NS in tars-system tars-dataplane; do kubectl create secret docker-registry registry-secret \ --docker-server="${ECR_HOST}" \ --docker-username=AWS \ --docker-password="${ECR_TOKEN}" \ --docker-email=noreply@tetrate.io \ -n "${NS}" \ --dry-run=client -o yaml | kubectl apply -f - done ``` ECR tokens expire every 12 hours, so this secret stops working after that window. For production deployments, use IAM Roles for Service Accounts (IRSA) or EKS Pod Identity to mint tokens on demand instead of a static secret; see [AWS docs](https://docs.aws.amazon.com/AmazonECR/latest/userguide/pull-eks-from-ecr.html). ## Step 9: install the data plane ```shell tare install "${CREDENTIAL_FILE}" \ --image-sync "${ECR_HOST}/${ECR_PREFIX}" \ --image-pull-secret-name registry-secret \ --serve-url "${SERVE_URL}" ``` The command: * Re-syncs any images (idempotent after the previous step). * Installs the helm chart into `tars-system` and `tars-dataplane`. * Registers the `--serve-url` value with the management plane. Watch the pods come up: ```shell kubectl get pods -n tars-system kubectl get pods -n tars-dataplane ``` Expected output (all Running, 1/1 or 3/3): ```none NAME READY STATUS RESTARTS AGE ai-gateway-controller-... 1/1 Running 0 2m controller-... 1/1 Running 0 2m controller-worker-... 1/1 Running 0 2m envoy-gateway-... 1/1 Running 0 2m envoy-ratelimit-... 1/1 Running 0 1m tars-redis-master-... 1/1 Running 0 2m tars-tare-doctor-... 1/1 Running 0 1m NAME READY STATUS RESTARTS AGE egress-... 3/3 Running 0 1m ``` The install command also prints a `Dataplane unreachable` warning at this point because DNS does not resolve the data-plane hostname yet. This is expected; DNS is wired in [Step 12](#step-12-wire-dns-and-register-the-url) below. ## Step 10: install the AWS load balancer controller The data plane needs an L7 load balancer to terminate inbound HTTP and forward to the `egress` Service in `tars-dataplane`. The **AWS Load Balancer Controller** provisions an ALB from Kubernetes Ingress resources. ### Step 10.1: iam policy The latest controller version requires permissions beyond what the v2.7.1 reference policy covered. Fetch the current policy from `main`: ```shell curl -sSL -o /tmp/iam-policy.json https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/main/docs/install/iam_policy.json aws iam create-policy \ --policy-name AWSLoadBalancerControllerIAMPolicy \ --policy-document file:///tmp/iam-policy.json ``` If the policy already exists from a prior run, update it instead: ```shell POLICY_ARN=arn:aws:iam::${AWS_ACCOUNT}:policy/AWSLoadBalancerControllerIAMPolicy aws iam create-policy-version \ --policy-arn "${POLICY_ARN}" \ --policy-document file:///tmp/iam-policy.json \ --set-as-default ``` ### Step 10.2: iam service account ```shell eksctl create iamserviceaccount \ --cluster="${EKS_CLUSTER_NAME}" \ --region="${AWS_REGION}" \ --namespace=kube-system \ --name=aws-load-balancer-controller \ --attach-policy-arn="arn:aws:iam::${AWS_ACCOUNT}:policy/AWSLoadBalancerControllerIAMPolicy" \ --approve --override-existing-serviceaccounts ``` ### Step 10.3: Helm install ```shell helm repo add eks https://aws.github.io/eks-charts helm repo update VPC_ID=$(aws eks describe-cluster --name "${EKS_CLUSTER_NAME}" \ --region "${AWS_REGION}" \ --query "cluster.resourcesVpcConfig.vpcId" --output text) helm install aws-load-balancer-controller eks/aws-load-balancer-controller \ -n kube-system \ --set clusterName="${EKS_CLUSTER_NAME}" \ --set serviceAccount.create=false \ --set serviceAccount.name=aws-load-balancer-controller \ --set region="${AWS_REGION}" \ --set vpcId="${VPC_ID}" ``` Wait for the controller pods to roll out: ```shell kubectl rollout status -n kube-system deployment/aws-load-balancer-controller --timeout=180s ``` ## Step 11: create the alb ingress All data-plane traffic (`/v1/*`, `/mcp/*`, `/.well-known/*`) is served by the `egress` Service in `tars-dataplane` on port 10080. A single Ingress is enough. ```shell cat <<'EOF' | kubectl apply -f - apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: tars-ingress namespace: tars-dataplane annotations: alb.ingress.kubernetes.io/scheme: internet-facing alb.ingress.kubernetes.io/target-type: ip alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}]' spec: ingressClassName: alb rules: - http: paths: - path: / pathType: Prefix backend: service: name: egress port: number: 10080 EOF ``` Wait for the ALB to provision (\~1–2 min) and capture its hostname: ```shell kubectl get ingress tars-ingress -n tars-dataplane ``` The `ADDRESS` column populates with something like `k8s-tarsdata-tarsingr-xxxxxxxxxx-yyyyyyyyy.us-east-1.elb.amazonaws.com`. If `ADDRESS` stays empty for more than a few minutes, check the controller for `FailedDeployModel` events: ```shell kubectl get events -n tars-dataplane --sort-by=.lastTimestamp | tail -5 kubectl logs -n kube-system deployment/aws-load-balancer-controller --tail=30 | grep -iE "error|fail" ``` The most common cause is an outdated IAM policy missing a permission such as `elasticloadbalancing:DescribeListenerAttributes`. Re-run the IAM policy update above and restart the controller. The Ingress persists across `tare install` reinstalls; re-applying it during upgrades is not required. ## Step 12: wire DNS and register the URL ### Step 12.1: DNS Point the data-plane hostname at the ALB via a CNAME record at your DNS provider: ```none proxy.example.com CNAME k8s-tarsdata-tarsingr-xxxxxxxxxx-yyyyyyyyy.us-east-1.elb.amazonaws.com ``` Verify it resolves (propagation may take a couple of minutes): ```shell dig +short proxy.example.com ``` ### Step 12.2: register the URL on the management plane If the URL passed to `tare install --serve-url` matches the hostname just wired in DNS, registration is already done; `tare install` registered the gateway URL when it ran. Otherwise, in the dashboard: **System → Settings → Data planes → edit the data plane → URL**. ## Step 13: verify the install Run `tare doctor` to check pod, CRD, and policy state: ```shell tare doctor "${CREDENTIAL_FILE}" ``` A healthy install reports all components in `Accepted: True` state. If `tare doctor` flags any errors, contact Tetrate Support. ## Step 14: smoke tests Once `tare doctor` reports clean, validate end-to-end traffic. Create an API key from the router app (**Router → API Keys → \+ Create**): ```shell export DP_HOST=proxy.example.com export TARS_API_KEY= ``` Run the suite (sends each endpoint with and without auth; expects 200 vs 401): ```shell bash tests/aws-dp-install/smoke.sh all ``` Expected output: ```none ==== list models (GET /v1/models) ==== ok models auth http=200 expected=200 ok models NO-auth http=401 expected=401 ==== chat completions (POST /v1/chat/completions) ==== ok openai gpt-5-mini auth http=200 expected=200 ok openai gpt-5-mini NO-auth http=401 expected=401 ==== responses API (POST /v1/responses) ==== ok responses auth http=200 expected=200 ok responses NO-auth http=401 expected=401 ==== anthropic messages (POST /v1/messages) ==== ok anthropic auth http=200 expected=200 ok anthropic NO-auth http=401 expected=401 ==== embeddings (POST /v1/embeddings) ==== ok embeddings auth http=200 expected=200 ok embeddings NO-auth http=401 expected=401 ==== image generation (POST /v1/images/generations) ==== ok images auth http=200 expected=200 ok images NO-auth http=401 expected=401 - pass: 12 fail: 0 auth-bypass: 0 ``` If any row fails (in particular, if a NO-auth row returns 200 instead of 401), contact Tetrate Support. ## Upgrading An upgrade re-runs the install command. The helm chart, CRDs, and config in `tars-config` are upgraded; persistent state (Redis-backed rate-limit counters, accumulated audit data) survives. The ALB Ingress persists, the IAM policy does not change, and the EBS CSI addon remains installed. ```shell tare install "${CREDENTIAL_FILE}" \ --image-sync "${ECR_HOST}/${ECR_PREFIX}" \ --image-pull-secret-name registry-secret \ --serve-url "${SERVE_URL}" ``` :::note Some runtime patches (notably observability env vars on the egress deployment) are reset on every install. Re-apply the patches from [Appendix B](#appendix-b-observability-with-opentelemetry) after each upgrade. ::: ## Cleanup To remove everything created by this guide: ```shell # 1. Ingress (releases the ALB) kubectl delete ingress tars-ingress -n tars-dataplane --ignore-not-found ``` ```shell # 2. AWS Load Balancer Controller helm uninstall aws-load-balancer-controller -n kube-system 2>/dev/null || true eksctl delete iamserviceaccount \ --name aws-load-balancer-controller \ --namespace kube-system \ --cluster "${EKS_CLUSTER_NAME}" \ --region "${AWS_REGION}" 2>/dev/null || true ``` ```shell ALB_POLICY_ARN=$(aws iam list-policies \ --query "Policies[?PolicyName=='AWSLoadBalancerControllerIAMPolicy'].Arn" \ --output text 2>/dev/null) if [ -n "${ALB_POLICY_ARN}" ] && [ "${ALB_POLICY_ARN}" != "None" ]; then # Delete non-default versions first (IAM rule) for V in $(aws iam list-policy-versions --policy-arn "${ALB_POLICY_ARN}" \ --query 'Versions[?IsDefaultVersion==`false`].VersionId' --output text); do aws iam delete-policy-version --policy-arn "${ALB_POLICY_ARN}" --version-id "${V}" done aws iam delete-policy --policy-arn "${ALB_POLICY_ARN}" fi ``` ```shell # 3. ECR repositories (the cluster prefix matches what was used at create time) for REPO in $(aws ecr describe-repositories --region "${AWS_REGION}" \ --query "repositories[?starts_with(repositoryName, '${EKS_CLUSTER_NAME}/')].repositoryName" \ --output text); do aws ecr delete-repository --repository-name "${REPO}" \ --region "${AWS_REGION}" --force done ``` ```shell # 4. EKS cluster (also tears down the EBS CSI addon and its IAM role via # CloudFormation — takes ~10 min) eksctl delete cluster --name "${EKS_CLUSTER_NAME}" --region "${AWS_REGION}" ``` ```shell # 5. Local kubeconfig file rm -f "${HOME}/kubeconfig-${EKS_CLUSTER_NAME}" ``` :::note DNS records (CNAMEs at your DNS provider) must be removed manually. ::: ## Troubleshooting | Symptom | Likely cause | Fix | | :- | :- | :- | | eksctl create cluster errors with invalid version, supported values: 1.23, …, 1.31 | eksctl is older than the K8s version EKS currently defaults to | brew upgrade eksctl (or re-download), then retry | | tare install \--sync-only errors with NAME\_UNKNOWN: The repository with name '...' does not exist | ECR repo for that image was never created | The sync-images.sh helper handles this; re-run it | | Ingress ADDRESS stays empty for more than 2 min | LB Controller IAM policy is missing newer permissions | Update the IAM policy from main, restart the controller | | Pods stuck ImagePullBackOff with no basic auth credentials | ECR token in registry-secret expired (12h lifetime) | Re-run the image-pull-secret creation step ([Step 8](#step-8-create-the-image-pull-secret)) | | egress pods restart loop with connection refused to redis | EBS CSI driver missing, so Redis PV never binds | Install the EBS CSI addon ([Step 4](#step-4-install-the-ebs-csi-driver-addon)) | ## Appendix a: TLS via acm \+ HTTPS listener The default flow uses HTTP on port 80. For production, terminate TLS at the ALB using an AWS Certificate Manager (ACM) certificate. 1. **Request the certificate** in ACM for the data-plane hostname. Validate via DNS (CNAME) or email per the domain control method. If this gateway serves a multi-gateway project fronted by DNS failover, the certificate must **also** carry the project's client-facing name as a SAN — a name missing here stays invisible until the first failover. See the [multi-SAN worked example](/agent-router-enterprise/guides/operate-and-govern/project-and-data-plane-management/gateway-sets-and-dns-failover/#worked-example-acm-multi-san-certificates-on-albs) in the failover guide. 2. **Get the certificate ARN**: ```shell CERT_ARN=$(aws acm list-certificates --region "${AWS_REGION}" \ --query "CertificateSummaryList[?DomainName=='proxy.example.com'].CertificateArn" \ --output text) ``` 3. **Update the Ingress** to listen on HTTPS:443 and redirect HTTP → HTTPS: ```none metadata: annotations: alb.ingress.kubernetes.io/scheme: internet-facing alb.ingress.kubernetes.io/target-type: ip alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS": 443}]' alb.ingress.kubernetes.io/ssl-redirect: '443' alb.ingress.kubernetes.io/certificate-arn: alb.ingress.kubernetes.io/ssl-policy: ELBSecurityPolicy-TLS13-1-2-2021-06 ``` 4. **Update the gateway URL** on the management plane to `https://...`. Re-run `tare install` to re-register the `--serve-url` value automatically. The ALB picks up the annotation change without redeploying. Verify with: ```shell curl -I https://proxy.example.com/v1/models -H "Authorization: Bearer ${TARS_API_KEY}" ``` ## Appendix b: observability with OpenTelemetry The data plane emits OTLP metrics (`router_*` family). To forward them to your observability backend, deploy an OpenTelemetry Collector in-cluster. ```shell cat <<'EOF' | kubectl apply -f - apiVersion: v1 kind: Namespace metadata: name: otel-system apiVersion: v1 kind: ConfigMap metadata: name: otel-collector-config namespace: otel-system data: config.yaml: | receivers: otlp: protocols: http: endpoint: 0.0.0.0:4318 grpc: endpoint: 0.0.0.0:4317 processors: batch: timeout: 5s transform/strip_scope: metric_statements: - context: metric statements: - replace_pattern(name, "^dynamicmodulescustom\\.", "router_") exporters: debug: verbosity: detailed # Replace this with your real backend exporter (Datadog, Grafana Cloud, # SigNoz, CloudWatch via the awsemf exporter, etc.). service: pipelines: metrics: receivers: [otlp] processors: [batch, transform/strip_scope] exporters: [debug] apiVersion: apps/v1 kind: Deployment metadata: name: otel-collector namespace: otel-system spec: replicas: 1 selector: matchLabels: app: otel-collector template: metadata: labels: app: otel-collector spec: containers: - name: collector image: otel/opentelemetry-collector-contrib:0.98.0 ports: - containerPort: 4317 - containerPort: 4318 volumeMounts: - name: config mountPath: /etc/otelcol-contrib volumes: - name: config configMap: name: otel-collector-config apiVersion: v1 kind: Service metadata: name: otel-collector namespace: otel-system spec: selector: app: otel-collector ports: - name: otlp-http port: 4318 targetPort: 4318 - name: otlp-grpc port: 4317 targetPort: 4317 EOF ``` Point the Agent Router egress proxy at the collector: ```shell kubectl set env deploy/egress -n tars-dataplane \ OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector.otel-system.svc.cluster.local:4318 \ OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf ``` Generate traffic via the smoke tests, then verify metrics are flowing: ```shell kubectl logs -n otel-system deployment/otel-collector --tail=50 | grep "Name:" ``` Expected metrics include `router_requests_total`, `router_auth_attempts_total`, and similar. :::note The `kubectl set env` patch is reset by `tare install` re-runs. Re-apply after every upgrade until the chart accepts these settings via Helm values. ::: Where to go next Gateway installation Install the data plane gateway components that manage inbound access and configure request routing. Self-Hosted Data Plane See the full install sequence: gateway, SSO, then developer handoff. --- ## Installation Guide for Azure using Helm This guide installs the Agent Router data plane on Azure Kubernetes Service (AKS) using Helm and a private container registry. **Important** This guide assumes platform teams own the Helm release lifecycle and operate a private container registry (Azure Container Registry, Harbor, JFrog Artifactory, Nexus, or Google Artifact Registry). If not, refer to the [Installation guide for Azure](/agent-router-enterprise/self-hosted-data-plane/data-plane-installation/installation-guide-az/). :::warning About Helm-based installation The recommended way to installing a data plane is by using the `TARE` command line utility. Using Helm is only for advanced or custom cases, and should be carefully considered. ::: ## Architecture Agent Router Enterprise uses a split-plane model. A **management plane** hosted by **Tetrate** holds configuration, license data, and the web dashboard. A **data plane** runs in a customer-managed Kubernetes cluster and handles all AI traffic. The two planes communicate over a single outbound HTTPS connection initiated by the data plane. No inbound internet connections are required. The data plane requires a stable public hostname so applications can reach it; TLS certificates are issued for hostnames, not IP addresses. Gateway exposure, DNS, and TLS are out of scope for this guide. ```mermaid architecture-beta group azure(cloud)[Azure] group tetrate(cloud)[Tetrate hosted] service mp(server)[Management Plane] in tetrate service dp(server)[Data Plane] in azure mp:R -- L:dp ``` The workflow is: - The `tare` CLI syncs Agent Router release artifacts into the private registry. - `tare` applies CRDs (Custom Resource Definitions) outside Helm. - `tare` renders Helm values. - `helm` installs or upgrades the data plane workload from the private registry. This guide covers data plane installation only. It does not create a public Gateway, ingress, DNS record, or TLS certificate. For gateway exposure, follow the published cloud install guide for the target platform, or use the organization's preferred ingress or Gateway stack. ## Outcomes By the end of this guide: - Agent Router release artifacts are mirrored into a private container registry. - Agent Router CRDs (Custom Resource Definitions) are applied to the target Kubernetes cluster. - The data plane is deployed via Helm into the `tars-system` and `tars-dataplane` namespaces. - The in-cluster `egress` service is verified and reachable via port-forward for smoke tests. Plan for 30 to 60 minutes, plus separate time for Agent Router team to provision the gateway, DNS, and TLS. ## In this guide - **Prepare for the installation**: obtain the data plane credential and install the CLI - [Prerequisites](#prerequisites) - [Step 1: Obtain the data plane credential](#step-1-obtain-the-data-plane-credential) - [Step 2: Install the `tare` CLI](#step-2-install-the-tare-cli) - [Step 3: Set environment variables](#step-3-set-variables) - **Registry setup** - mirror Agent Router artifacts into the private registry - [Step 4: Sync release artifacts](#step-4-sync-release-artifacts) - **Data Plane installation** - apply CRDs and deploy with Helm - [Step 5: Apply CRDs](#step-5-apply-crds) - [Step 6: Render Helm values](#step-6-render-helm-values) - [Step 7: Create namespaces and pull secrets](#step-7-create-namespaces-and-pull-secrets) - [Step 8: Install or upgrade with Helm](#step-8-install-or-upgrade-with-helm) - **Verify** - confirm the installation works end-to-end - [Step 9: Verify](#step-9-verify) - [Step 10: Expose the data plane](#step-10-expose-the-data-plane) - [Step 11: Run validation and smoke tests](#step-11-run-validation-and-smoke-tests) - **Operations** - manage and troubleshoot the installation - [Upgrade and rollback](#upgrade-and-rollback) - [If artifacts are already mirrored](#if-artifacts-are-already-mirrored) - [Troubleshooting](#troubleshooting) ## Prerequisites ### Dashboard and Router app access Agent Router exposes two web surfaces. Both URLs are provided during onboarding: - **Dashboard** (admin): `https://dashboard..tetrate.ai`. - **Router app** (end-user): `https://router..tetrate.ai`. ### Required tools Install the following on the workstation used for this guide: | Tool | Install | | :--- | :--- | | `az` CLI | `https://learn.microsoft.com/cli/azure/install-azure-cli` | | `kubectl` | `https://kubernetes.io/docs/tasks/tools` | | `helm` (3+) | `https://helm.sh/docs/intro/install` | | `docker` | `https://docs.docker.com/get-docker` | | `curl` | Preinstalled on macOS and most Linux distributions | | `tare` CLI | Covered in [Step 2](#step-2-install-the-tare-cli) | A `data-plane-credentials.json` file is also required. See [Step 1](#step-1-obtain-the-data-plane-credential). ### Infrastructure A dedicated workload cluster must be provisioned before starting the installation. The cluster requires at least three (3) nodes. See [Cluster sizing](#cluster-sizing) for more detail. :::warning Tetrate support does not cover client-side infrastructure provisioning or Kubernetes issues. The instructions for creating clusters and related infrastructure components are provided as a courtesy and should be carefully evaluated before executing them. ::: Furtermore, make sure you have sufficient systems/infrastructure permissions to: - Access the target Kubernetes cluster. - Push images to the private registry mirror. - Pull images from the target cluster to the same private registry. ### Cluster sizing The default chart installs multiple always-on components, including the **Agent Router egress proxy** (Envoy-based; referred to as *egress envoy* in manifests and logs) with a minimum of 2 replicas, AI gateway controller and ext_proc, controller and worker, Redis, and rate-limit services. A single-node footprint is not sufficient. The **egress envoy** is the dominant resource consumer. Its CPU and memory usage scale with the configuration size held in memory: the count of `AIGatewayRoute` and `AIServiceBackend` resources, header-mutation rules, and per-route features. The AI gateway team's [control-plane scaling benchmark](https://aigateway.envoyproxy.io/blog/benchmarking-control-plane-scaling) shows roughly linear CPU and memory growth as routes are added. Plan capacity for route counts that grow as providers, models, and projects are added. General-purpose VM sizes from the `Standard_D*s_v5` family provide a balanced default. | Size | Use case | Recommended node pool | Approximate allocatable target | | :--- | :--- | :--- | :--- | | Small | Dev / test / low traffic | 3 × `Standard_B2s` | ≥ 6 vCPU, ≥ 20 GiB RAM | | Medium | Staging / light production | 3 × `Standard_D4s_v5` | ≥ 12 vCPU, ≥ 40 GiB RAM | | High | Production with burst headroom | 3 × `Standard_D8s_v5` (or split into system + data plane pools) | ≥ 24 vCPU, ≥ 80 GiB RAM | Maintain a minimum of three nodes to tolerate node upgrades and evictions. Demo installs may start at Small; production installations should start at Medium. :::note AKS Automatic is not supported. ::: ## Conventions All commands assume the environment variables defined in [Step 3](#step-3-set-variables) are exported in the current shell. Re-export them when opening a new terminal. Replace every `` value with a site-specific one. ## Step 1: obtain the data plane credential In the dashboard, go to **Settings > Data planes**. If the data plane is not registered yet: 1. Click **Register Data plane**. 1. Set **Data plane ID** to a stable identifier such as `acme-prod`. 1. Optionally set **Name**, **Description**, and **Labels**. 1. Leave **URL** blank for now unless the final gateway hostname is already known. 1. Click **Register Data plane**. Then generate the credential from the data plane row: 1. Click the row's edit action to open the **Data plane** drawer. 1. In the **Credential** section, click **Generate credential**. 1. Save the downloaded file as `data-plane-credentials.json` on the machine where the install will run. Each data plane uses its own credential. Credentials can be rotated or revoked from the same **Credential** section in the data plane edit drawer. ## Step 2: install the `tare` CLI Run the installer script: ```shell curl -sSL https://tare.tetrate.ai/tools/install.sh | bash ``` Output: ```none ==> tare installer ==> channel: stable ==> Detected platform: darwin-arm64 ==> Installing tare for darwin-arm64... ==> Downloading from: https://tare.tetrate.ai/tools/tags/v0.1.0-beta.4/tare-darwin-arm64.tar.gz ok Installed tare to /Users/johndoe/.tare/bin/tare ==> tare version: tare version v0.1.0-beta.4 ok Installation directory is already in your PATH ==> Get started: tare install identity.json --serve-url https://proxy.acme.com tare install --help ``` The installer prints the install path (typically `~/.tare/bin/tare`). Add it to `PATH` and verify the version: ```shell export PATH="$PATH:$HOME/.tare/bin" echo 'export PATH="$PATH:$HOME/.tare/bin"' >> ~/.zshrc # or ~/.bashrc ``` ```shell $ tare --version tare version v0.1.0-beta.4 ``` ## Step 3: set variables For Azure Container Registry: ```bash DP_CREDENTIAL=./data-plane-credentials.json # Obtained in Step 1 PRIVATE_REGISTRY_HOST=${ACR_NAME}.azurecr.io PRIVATE_IMAGE_REGISTRY=${ACR_NAME}.azurecr.io/tare PULL_SECRET=tare-registry-pull # this value will be used later on, leave it as-is ``` For another private registry: ```bash DP_CREDENTIAL=./data-plane-credentials.json # Obtained in Step 1 PRIVATE_REGISTRY_HOST=registry.acme.example.com # Replace with your own registry host PRIVATE_IMAGE_REGISTRY=registry.acme.example.com/tare # Replace with your own registry full URL PULL_SECRET=tare-registry-pull # this value will be used later on, leave it as-is REGISTRY_USERNAME= REGISTRY_PASSWORD= ``` ## Step 4: sync release artifacts Log in locally with credentials that can push to the destination registry. For ACR: ```bash az acr login --name "${ACR_NAME}" ``` For a generic private registry: ```bash printf '%s' "${REGISTRY_PASSWORD}" | \ docker login "${PRIVATE_REGISTRY_HOST}" \ --username "${REGISTRY_USERNAME}" \ --password-stdin ``` Copy the pinned Agent Router images and the `serve-helm` OCI chart: ```bash tare install "${DP_CREDENTIAL}" \ --image-sync "${PRIVATE_IMAGE_REGISTRY}" \ --sync-only \ --parallel 2 \ --stall-threshold 5m ``` `tare` uses the data plane credential to pull from Tetrate's source registry. The local registry login is used only to push to the private registry. ## Step 5: apply CRDs Apply CRDs outside Helm: ```bash tare install "${DP_CREDENTIAL}" \ --crds-only \ --image-registry "${PRIVATE_IMAGE_REGISTRY}" ``` CRDs are managed outside the Helm release so that `helm uninstall` does not delete cluster-scoped APIs and cascade-delete custom resources. ## Step 6: render Helm values Render values for the Helm release: ```bash tare install "${DP_CREDENTIAL}" \ --image-registry "${PRIVATE_IMAGE_REGISTRY}" \ --print-helm-values > values.yaml ``` Treat `values.yaml` as sensitive: it contains the data plane identity secret. Do not commit it in plaintext. Use SOPS, Sealed Secrets, External Secrets, or the organization's approved secret workflow. If the public gateway URL is not known yet, confirm the rendered values do not set it to the management-plane API URL. The following fields must remain empty: ```yaml global: serveUrl: "" controller: config: proxyUrl: "" ``` Override them during Helm install if needed: ```bash --set-string global.serveUrl= \ --set-string controller.config.proxyUrl= ``` ## Step 7: create namespaces and pull secrets Create the data plane namespaces: ```bash kubectl create namespace tars-system --dry-run=client -o yaml | kubectl apply -f - kubectl create namespace tars-dataplane --dry-run=client -o yaml | kubectl apply -f - ``` For AKS attached to ACR, no image pull secret is usually required after: ```bash az aks update \ --resource-group "${RESOURCE_GROUP}" \ --name "${AKS_CLUSTER_NAME}" \ --attach-acr "${ACR_NAME}" ``` For a private registry that requires Kubernetes pull credentials, create the same secret in both namespaces: ```bash kubectl create secret docker-registry "${PULL_SECRET}" \ --docker-server="${PRIVATE_REGISTRY_HOST}" \ --docker-username="${REGISTRY_USERNAME}" \ --docker-password="${REGISTRY_PASSWORD}" \ --namespace tars-system \ --dry-run=client -o yaml | kubectl apply -f - kubectl create secret docker-registry "${PULL_SECRET}" \ --docker-server="${PRIVATE_REGISTRY_HOST}" \ --docker-username="${REGISTRY_USERNAME}" \ --docker-password="${REGISTRY_PASSWORD}" \ --namespace tars-dataplane \ --dry-run=client -o yaml | kubectl apply -f - ``` When using a pull secret, render values with: ```bash tare install "${DP_CREDENTIAL}" \ --image-registry "${PRIVATE_IMAGE_REGISTRY}" \ --image-pull-secret-name "${PULL_SECRET}" \ --print-helm-values > values.yaml ``` ## Step 8: install or upgrade with Helm Set the chart version for the target Agent Router release: ```bash CHART_VERSION="0.1.0-alpha.1+17f076b" ``` Install or upgrade the release: ```bash helm upgrade --install tars "oci://${PRIVATE_IMAGE_REGISTRY}/serve-helm" \ --version "${CHART_VERSION}" \ -f values.yaml \ -n tars-system \ --skip-crds \ --set ai-gateway-crds.enabled=false \ --set-string global.serveUrl= \ --set-string controller.config.proxyUrl= ``` Use `--skip-crds` and `--set ai-gateway-crds.enabled=false` together. The first skips chart `crds/` entries; the second disables AI Gateway CRDs rendered as normal Helm templates. Helm expects the SemVer build-metadata form with `+`, such as `0.1.0-alpha.1+17f076b`. OCI stores the tag internally with `_`, but `helm --version` must use the `+` form. ## Step 9: verify Check the Helm release: ```bash helm list -n tars-system helm status tars -n tars-system ``` Wait for workloads: ```bash kubectl rollout status deployment/controller -n tars-system --timeout=300s kubectl rollout status deployment/egress -n tars-dataplane --timeout=300s ``` Confirm images come from the private registry: ```bash kubectl get deploy egress -n tars-dataplane \ -o jsonpath='{range .spec.template.spec.containers[*]}{.name}={.image}{"\n"}{end}' ``` Run doctor: ```bash tare doctor "${DP_CREDENTIAL}" --verbose ``` A fresh data plane with no routes can report warnings for missing `RouteDeployments`, `AIGatewayRoutes`, `HTTPRoutes`, or `EnvoyPatchPolicies`. The core install is healthy when pods roll out and images pull from the private registry. ## Step 10: expose the data plane This Helm workflow installs the data plane but does not create a public Gateway, ingress, DNS record, or TLS certificate. For gateway exposure, follow the published cloud install guide for the target platform, or use the organization's preferred ingress or Gateway stack. The data plane serves traffic through the in-cluster `egress` service: - **Service:** `egress` - **Namespace:** `tars-dataplane` - **Port:** `10080` - **Paths:** `/v1/*`, `/mcp/*`, and `/.well-known/*` Once the gateway and DNS are ready, set the gateway URL in the dashboard: **Settings > Data planes > edit row > URL**. ## Step 11: run validation and smoke tests Validate the Helm install first: ```bash helm status tars -n tars-system kubectl rollout status deployment/controller -n tars-system --timeout=300s kubectl rollout status deployment/egress -n tars-dataplane --timeout=300s kubectl get deploy egress -n tars-dataplane \ -o jsonpath='{range .spec.template.spec.containers[*]}{.name}={.image}{"\n"}{end}' tare doctor "${DP_CREDENTIAL}" --verbose ``` Run traffic smoke tests after onboarding has configured provider keys and routes. This Helm workflow does not create a public Gateway or DNS record. Until the gateway is exposed, use a local port-forward to the in-cluster `egress` service: ```bash kubectl port-forward -n tars-dataplane svc/egress 18080:10080 ``` In another terminal, set the local endpoint and an API key from the router app: ```bash export DP_SCHEME=http export DP_HOST=127.0.0.1:18080 export TARS_API_KEY= ``` At minimum, verify model listing with and without authentication: ```bash curl -s -o /tmp/tare-models.json -w "%{http_code}\n" \ "${DP_SCHEME}://${DP_HOST}/v1/models" \ -H "Authorization: Bearer ${TARS_API_KEY}" curl -s -o /tmp/tare-models-no-auth.json -w "%{http_code}\n" \ "${DP_SCHEME}://${DP_HOST}/v1/models" ``` Expected result: the authenticated request returns `200`; the unauthenticated request returns `401`. Also smoke test every API shape enabled during onboarding, such as `/v1/chat/completions`, `/v1/responses`, `/v1/messages`, `/v1/embeddings`, `/v1/images/generations`, and MCP profile paths. ## Upgrade and rollback Use the same split workflow for upgrades: update CRDs first, then upgrade the Helm workload release. If the new Agent Router release includes CRD changes, re-run CRD apply: ```bash tare install "${DP_CREDENTIAL}" \ --crds-only \ --image-registry "${PRIVATE_IMAGE_REGISTRY}" ``` Render values for the new release: ```bash tare install "${DP_CREDENTIAL}" \ --image-registry "${PRIVATE_IMAGE_REGISTRY}" \ --print-helm-values > values.yaml ``` When using a pull secret, include it when rendering values: ```bash tare install "${DP_CREDENTIAL}" \ --image-registry "${PRIVATE_IMAGE_REGISTRY}" \ --image-pull-secret-name "${PULL_SECRET}" \ --print-helm-values > values.yaml ``` Upgrade the workload release: ```bash CHART_VERSION="" helm upgrade tars "oci://${PRIVATE_IMAGE_REGISTRY}/serve-helm" \ --version "${CHART_VERSION}" \ -f values.yaml \ -n tars-system \ --skip-crds \ --set ai-gateway-crds.enabled=false \ --set-string global.serveUrl= \ --set-string controller.config.proxyUrl= ``` Rollback uses Helm's normal rollback flow: ```bash helm rollback tars -n tars-system ``` CRDs are not affected by rollback because they are not part of the Helm release manifest. Existing custom resources remain in the cluster. ## If artifacts are already mirrored If Agent Router team has already mirrored the Agent Router images and chart, skip the `--image-sync` step and start at CRD apply: ```bash tare install "${DP_CREDENTIAL}" \ --crds-only \ --image-registry "${PRIVATE_IMAGE_REGISTRY}" ``` Then render values and install with Helm as described above. ## Troubleshooting ### Destination registry reports repository not found Create the destination repository before syncing. For example, with Google Artifact Registry: ```bash gcloud artifacts repositories create tare \ --repository-format docker \ --location ``` ACR and many enterprise registries create repositories on first push, but some registries require pre-creation. ### Helm reports invalid ownership metadata This usually means Helm is trying to adopt CRDs that were already applied by `tare --crds-only`. Example: ```text CustomResourceDefinition "aigatewayroutes.aigateway.envoyproxy.io" exists and cannot be imported into the current release: invalid ownership metadata ``` Install or upgrade with both flags: ```bash --skip-crds \ --set ai-gateway-crds.enabled=false ``` ### Helm reports improper constraint `helm --version` expects a SemVer constraint, not a bare commit SHA. Use the chart's full SemVer, such as `0.1.0-alpha.1+17f076b`, or a release-build Agent Router version such as `v0.1.0-beta.3`. Bare commit SHAs are rejected before Helm contacts the registry. The registry stores SemVer build metadata with `_`, but `helm --version` must use the SemVer form with `+`. ### Pods are stuck in imagepullbackoff Check that: - the private registry contains every image tag for the Agent Router release - the cluster identity can pull from the private registry - pull secrets exist in both `tars-system` and `tars-dataplane` when required - the pull secret is referenced from `values.yaml`; if not, re-render values with `--image-pull-secret-name` - `values.yaml` points `global.imageRegistry` at the intended registry ### Helm cannot pull the chart from the mirror If `helm install` or `helm upgrade` fails locally with `pull access denied`, log in to the OCI registry from the operator machine: ```bash helm registry login "${PRIVATE_REGISTRY_HOST}" ``` This is separate from Kubernetes image pull access. Helm pulls the chart artifact from the local machine; the cluster pulls workload images using node identity or image pull secrets. Where to go next Gateway installation Install the data plane gateway components that manage inbound access and configure request routing. Self-Hosted Data Plane See the full install sequence: gateway, SSO, then developer handoff. --- ## Installation guide for Azure This guide installs the Agent Router data plane on Azure Kubernetes Service (AKS). ## Architecture Agent Router Enterprise uses a split-plane model. A **Management Plane** hosted by **Tetrate** holds configuration, license data, and the web dashboard. A **Data Plane** runs in a customer-managed Kubernetes cluster and handles all AI traffic. The two planes communicate over a single outbound HTTPS connection initiated by the data plane. No inbound connections from the internet are required. The data plane needs a stable public hostname so applications can reach it; TLS certificates are issued for hostnames, not IP addresses. DNS and TLS are configured in [Step 8](#step-8-wire-dns-and-register-the-url) and [Appendix B](#appendix-b-enable-tls). ```mermaid architecture-beta group azure(cloud)[Azure] group tetrate(cloud)[Tetrate hosted] service mp(server)[Management Plane] in tetrate service dp(server)[Data Plane] in azure mp:R -- L:dp ``` The procedure produces: - An Azure Container Registry mirroring the Agent Router images - An AKS cluster running the data plane in the `tars-system` and `tars-dataplane` namespaces - An Azure Application Gateway fronting the data plane at a customer-owned DNS name - A gateway URL registered on the management plane Plan for 30 to 60 minutes of installation time, plus DNS propagation. ## Table of contents - **Prepare for the installation**: obtain the data plane credential and install the CLI - [Prerequisites](#prerequisites) - [Step 1: Obtain the data plane credential](#step-1-obtain-the-data-plane-credential) - [Step 2: Install the `tare` CLI](#step-2-install-the-tare-cli) - **Cluster setup**: create or reuse a Kubernetes cluster - [Step 3: Provision the AKS cluster](#step-3-provision-the-aks-cluster) - **Registry setup**: mirror Agent Router images so the cluster can pull them - [Step 4: Create an Azure Container Registry](#step-4-create-an-azure-container-registry) - [Step 5: Sync Agent Router images to ACR](#step-5-sync-agent-router-images-to-acr) - **Data Plane installation**: deploy the data plane - [Step 6: Install the Agent Router data plane](#step-6-install-the-agent-router-data-plane) - **Ingress setup**: expose the data plane externally - [Step 7: Expose the data plane via AGIC](#step-7-expose-the-data-plane-via-agic) - **DNS configuration**: wire the hostname to the ingress and register the URL on the management plane - [Step 8: Wire DNS and register the URL](#step-8-wire-dns-and-register-the-url) - **Testing the installation**: verify the install works end-to-end - [Step 9: Verify provider routes](#step-9-verify-provider-routes) - [Step 10: Verify the install](#step-10-verify-the-install) - [Step 11: Smoke tests](#step-11-smoke-tests) - **Appendices** - [Appendix A: Alternative ingress (AGC) for Cilium / Overlay clusters](#appendix-a-alternative-ingress-agc) - [Appendix B: Enable TLS](#appendix-b-enable-tls) - [Appendix C: Forward observability data to an OpenTelemetry Collector](#appendix-c-forward-observability-data-to-an-opentelemetry-collector) - [Appendix D: Use an existing private registry](#appendix-d-use-an-existing-private-registry) ## Prerequisites ### Dashboard and Router app access Agent Router exposes two web surfaces. Both URLs are provided during onboarding: - **Dashboard** (admin): `https://dashboard..tetrate.ai`. Used in [Step 1](#step-1-obtain-the-data-plane-credential) and [Step 8](#step-8-wire-dns-and-register-the-url). - **Router app** (end-user): `https://router..tetrate.ai`. Used in [Step 11](#step-11-smoke-tests) for creating API keys and MCP profiles. ### Required tools Install the following on the workstation used for this guide: | Tool | Install | | :--- | :--- | | `az` CLI | `https://learn.microsoft.com/cli/azure/install-azure-cli` | | `kubectl` | `https://kubernetes.io/docs/tasks/tools` | | `helm` (3+) | `https://helm.sh/docs/intro/install` | | `docker` | `https://docs.docker.com/get-docker` | | `curl` | Preinstalled on macOS and most Linux distributions | | `tare` CLI | Covered in [Step 2](#step-2-install-the-tare-cli) | A `data-plane-credentials.json` file is also required. See [Step 1](#step-1-obtain-the-data-plane-credential). ### Infrastructure A dedicated workload cluster must be provisioned before starting the installation. The cluster requires at least three nodes. See [Cluster sizing](#cluster-sizing) for more detail. :::warning Tetrate support does not cover client-side infrastructure provisioning or Kubernetes issues. The instructions for creating clusters and related infrastructure components are provided as a courtesy and should be carefully evaluated before executing them. ::: ### Azure permissions The following role assignments are required on the subscription used for deployment: | Role | Scope | Required for | | :--- | :--- | :--- | | Contributor | The resource group | AKS, ACR, App Gateway creation | | Azure Kubernetes Service Contributor Role | The AKS cluster | Enabling the AGIC addon | | AcrPush | The container registry | Pushing synced images | | User Access Administrator | The resource group | Attaching ACR to AKS | | Network Contributor | The AKS managed RG (`MC_*`) | Letting AGIC manage App Gateway state | | Log Analytics Contributor | The linked Log Analytics RG | Required only when Container Insights is enabled | ### Cluster sizing The default chart installs multiple always-on components, including the **Agent Router egress proxy** (Envoy-based; referred to as *egress envoy* in manifests and logs) with a minimum of 2 replicas, AI gateway controller and ext_proc, controller and worker, Redis, and rate-limit services. A single-node footprint is not sufficient. The **egress envoy** is the dominant resource consumer. Its CPU and memory usage scale with the configuration size held in memory: the count of `AIGatewayRoute` and `AIServiceBackend` resources, header-mutation rules, and per-route features. The AI gateway team's [control-plane scaling benchmark](https://aigateway.envoyproxy.io/blog/benchmarking-control-plane-scaling) shows roughly linear CPU and memory growth as routes are added. Plan capacity for route counts that grow as providers, models, and projects are added. General-purpose VM sizes from the `Standard_D*s_v5` family provide a balanced default. | Size | Use case | Recommended node pool | Approximate allocatable target | | :--- | :--- | :--- | :--- | | Small | Dev / test / low traffic | 3 × `Standard_B2s` | ≥ 6 vCPU, ≥ 20 GiB RAM | | Medium | Staging / light production | 3 × `Standard_D4s_v5` | ≥ 12 vCPU, ≥ 40 GiB RAM | | High | Production with burst headroom | 3 × `Standard_D8s_v5` (or split into system + data plane pools) | ≥ 24 vCPU, ≥ 80 GiB RAM | Maintain a minimum of three nodes to tolerate node upgrades and evictions. Demo installs may start at Small; production installations should start at Medium. :::note AKS Automatic is not supported. ::: ## Conventions All commands assume the environment variables defined in [Step 3](#step-3-provision-the-aks-cluster) are exported in the current shell. Re-export them when opening a new terminal. Replace every `` value with a site-specific one. ## Step 1: obtain the data plane credential In the dashboard, navigate to **System → Settings → Data plane** and click **Generate Data plane credential**. Save the downloaded file as `data-plane-credentials.json` on the workstation used for installation. This file is the long-lived identity the data plane uses to authenticate to the management plane. :::note Some parts of the product still use the older "service account" naming for this file. The dashboard is standardizing on "data plane credential"; the file is the same. Each data plane uses its own credential. Credentials can be revoked from the dashboard, and additional credentials can be generated (for example, one per environment) at any time. ::: ## Step 2: install the `tare` CLI Run the installer script: ```shell curl -sSL https://tare.tetrate.ai/tools/install.sh | bash ``` Output: ```none ==> tare installer ==> channel: stable ==> Detected platform: darwin-arm64 ==> Installing tare for darwin-arm64... ==> Downloading from: https://tare.tetrate.ai/tools/tags/v0.1.0-beta.4/tare-darwin-arm64.tar.gz ok Installed tare to /Users/johndoe/.tare/bin/tare ==> tare version: tare version v0.1.0-beta.4 ok Installation directory is already in your PATH ==> Get started: tare install identity.json --serve-url https://proxy.acme.com tare install --help ``` The installer prints the install path (typically `~/.tare/bin/tare`). Add it to `PATH` and verify the version: ```shell export PATH="$PATH:$HOME/.tare/bin" echo 'export PATH="$PATH:$HOME/.tare/bin"' >> ~/.zshrc # or ~/.bashrc ``` ```shell $ tare --version tare version v0.1.0-beta.4 ``` ## Step 3: provision the aks cluster [Step 7](#step-7-expose-the-data-plane-via-agic) uses the AGIC addon (Application Gateway Ingress Controller). The `az aks create` flags below configure the cluster networking for AGIC compatibility. :::tip **Reusing an existing AKS cluster.** Check AGIC compatibility before continuing: ```shell az aks show -n -g \ --query 'networkProfile.{plugin: networkPlugin, mode: networkPluginMode, dataplane: networkDataplane}' ``` - If the output is `{plugin: "azure", mode: null, dataplane: "azure"}`, AGIC is supported. Skip to [Step 4](#step-4-create-an-azure-container-registry). - If `mode` is `overlay` or `dataplane` is `cilium`, AGIC is not supported. Use [Appendix A: AGC](#appendix-a-alternative-ingress-agc) for ingress. ::: ### Step 3.1: set environment variables ```shell RESOURCE_GROUP= LOCATION= AKS_CLUSTER_NAME= # Use the regional default Kubernetes version to avoid an unexpected end-of-support situation. K8S_VERSION=$(az aks get-versions --location "${LOCATION}" --query "values[?isDefault].version | [0]" -o tsv) # Optional resource tags; adapt or remove as needed. TAGS=( owner="" team="" purpose=development ) ``` ### Step 3.2: sign in to Azure ```shell az login az account set --subscription ``` ### Step 3.3: create the resource group ```shell az group create \ --name "${RESOURCE_GROUP}" \ --location "${LOCATION}" \ --tags "${TAGS[@]}" ``` ### Step 3.4: create the aks cluster ```shell az aks create \ --resource-group "${RESOURCE_GROUP}" \ --name "${AKS_CLUSTER_NAME}" \ --location "${LOCATION}" \ --kubernetes-version "${K8S_VERSION}" \ --node-count 3 \ --node-vm-size Standard_D4s_v5 \ --network-plugin azure \ --network-dataplane azure \ --enable-managed-identity \ --generate-ssh-keys \ --tags "${TAGS[@]}" ``` Provisioning takes approximately five minutes. AKS includes a default CSI driver, so no additional configuration is required for the persistent volumes used by the data plane's Redis state. The `--node-count` and `--node-vm-size` values above correspond to the **Medium** tier in [Cluster sizing](#cluster-sizing). Adjust as needed. ### Step 3.5: fetch the kubeconfig ```shell az aks get-credentials \ --resource-group "${RESOURCE_GROUP}" \ --name "${AKS_CLUSTER_NAME}" \ --file ~/kubeconfig-${AKS_CLUSTER_NAME} export KUBECONFIG=~/kubeconfig-${AKS_CLUSTER_NAME} kubectl get nodes ``` Expected output: ```none NAME STATUS ROLES AGE VERSION aks-nodepool1-xxxxxxxx-vmss000000 Ready 2m v1.34.x aks-nodepool1-xxxxxxxx-vmss000001 Ready 2m v1.34.x ``` ## Step 4: create an Azure container registry Create the registry that the next step syncs Agent Router images into: ```shell ACR_NAME= # 5–50 lowercase alphanumeric chars az acr create \ --resource-group "${RESOURCE_GROUP}" \ --name "${ACR_NAME}" \ --sku Standard \ --tags "${TAGS[@]}" ``` ## Step 5: sync Agent Router images to acr ### Step 5.1: authenticate Docker to the acr ```shell az acr login --name "${ACR_NAME}" ``` This refreshes local Docker credentials for `${ACR_NAME}.azurecr.io` for roughly three hours. Re-run this command if `unauthorized: authentication required` errors occur during image sync. ### Step 5.2: sync images Copy the container images from Tetrate's registry into the registry created above. The `tare` CLI authenticates to the source registry automatically. Only the destination ACR requires a local login. ```shell tare install /path/to/data-plane-credentials.json \ --image-sync ${ACR_NAME}.azurecr.io/tare \ --sync-only ``` The sync produces no progress output and takes several minutes. After it completes, verify the images: ```shell az acr repository list --name "${ACR_NAME}" -o tsv ``` The output should list ten repositories under the `tare/` prefix, including `ai-gateway-controller`, `envoy-tars`, `gateway`, `liaison`, `ratelimit`, `redis`, `tare-doctor`, and `valet`. ### Step 5.3: grant aks pull access to acr Attach the ACR to AKS so the cluster's managed identity can pull images: ```shell az aks update \ --resource-group "${RESOURCE_GROUP}" \ --name "${AKS_CLUSTER_NAME}" \ --attach-acr "${ACR_NAME}" ``` No image-pull secret is required; AKS handles authentication via its managed identity. :::tip If the account lacks `User Access Administrator` on the resource group, the command above fails with `Could not create a role assignment for ACR`. Fall back to the admin-user flow: ```shell az acr update --name "${ACR_NAME}" --admin-enabled true ACR_USERNAME=$(az acr credential show --name "${ACR_NAME}" --query "username" -o tsv) ACR_PASSWORD=$(az acr credential show --name "${ACR_NAME}" --query "passwords[0].value" -o tsv) ``` Pipe these credentials to `tare install` in [Step 6](#step-6-install-the-agent-router-data-plane) using `--image-pull-secret-stdin`. ::: ## Step 6: install the Agent Router data plane ```shell tare install /path/to/data-plane-credentials.json \ --image-sync ${ACR_NAME}.azurecr.io/tare ``` The `tare install` command performs the following actions: - Creates the `tars-system` and `tars-dataplane` namespaces. - Installs the Agent Router data plane via Helm. :::tip If the admin-user fallback from [Step 5.3](#step-53-grant-aks-pull-access-to-acr) was used, pipe the credentials so `tare install` creates the image-pull secret: ```shell echo "${ACR_USERNAME}:${ACR_PASSWORD}" | \ tare install /path/to/data-plane-credentials.json \ --image-sync ${ACR_NAME}.azurecr.io/tare \ --image-pull-secret-stdin ``` ::: ## Step 7: expose the data plane via agic The data plane terminates external traffic on a single in-cluster service: `egress` in `tars-dataplane` on port `10080`. It serves both LLM API requests (`/v1/*`) and MCP traffic (`/mcp/*`, `/.well-known/*`); no separate routes are needed. ### Step 7.1: enable agic on the cluster This provisions an Azure Application Gateway (Standard_v2) and wires it to AKS. Provisioning takes approximately five minutes. ```shell az aks enable-addons \ --resource-group "${RESOURCE_GROUP}" \ --name "${AKS_CLUSTER_NAME}" \ --addons ingress-appgw \ --appgw-name "${AKS_CLUSTER_NAME}-appgw" \ --appgw-subnet-cidr 10.225.0.0/24 ``` :::note The `--appgw-subnet-cidr` must not overlap any existing subnet in the VNet (the default AKS subnet is `10.224.0.0/16`). Keep the size at `/24`; this works for any cluster networking and is required for clusters that ever used Overlay. ::: Verify the controller is running: ```shell kubectl get pods -n kube-system -l app=ingress-appgw ``` A single `ingress-appgw-deployment-*` pod should be `Running`. A small number of restarts during the first few minutes is normal while the controller reconciles against the in-progress ARM provisioning. ### Step 7.2: create the ingress ```shell cat <<'EOF' | kubectl apply -f - apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: tars-ingress namespace: tars-dataplane annotations: # AGIC's default health probe is GET / on the backend, but the Agent Router egress proxy # only serves /v1/* and /mcp/* paths and returns 404 for /. Without these # two annotations, AGIC marks the backend unhealthy and every request # returns 502 Bad Gateway. appgw.ingress.kubernetes.io/health-probe-path: / appgw.ingress.kubernetes.io/health-probe-status-codes: "200-499" spec: ingressClassName: azure-application-gateway rules: - http: paths: - path: / pathType: Prefix backend: service: name: egress port: number: 10080 EOF ``` ### Step 7.3: retrieve the public ip ```shell kubectl get ingress tars-ingress -n tars-dataplane ``` Expected output: ```none NAME CLASS HOSTS ADDRESS PORTS AGE tars-ingress azure-application-gateway * 40.x.x.x 80 30s ``` Record the `ADDRESS` value; it is used in [Step 8](#step-8-wire-dns-and-register-the-url). :::note The Ingress resource persists across `tare install` re-runs and does not need to be re-applied. ::: :::warning **TLS is required for production.** The Ingress above listens on HTTP/80 only, which is acceptable for local testing but not for any customer-facing deployment. Configure TLS on the Ingress before going to production. [Appendix B](#appendix-b-enable-tls) describes two example mechanisms: bring-your-own certificate and cert-manager. ::: ## Step 8: wire DNS and register the URL ### Step 8.1: add the DNS a record Create an A record pointing the data plane hostname to the App Gateway address from [Step 7.3](#step-73-retrieve-the-public-ip): ```none . A ``` The App Gateway listens on port 80 by default; no port suffix is required. DNS propagation typically takes one to two minutes. Verify with: ```shell dig +short # Should return the App Gateway IP ``` ### Step 8.2: register the URL on the management plane In the dashboard, navigate to **System → Settings → Profile → Proxy URL** and set it to `https://` (or `http://...` if TLS was skipped for local testing). Changes propagate to the data plane within approximately 30 seconds. :::tip For demo installations without a domain, `.nip.io` resolves automatically. Use it as a placeholder, then switch to a real hostname before going to production. ::: ## Step 9: verify provider routes Providers (OpenAI, Anthropic, and others) and their upstream API keys are configured during Agent Router onboarding, not during this install. The data plane retrieves that configuration automatically once it is connected to the management plane. Verify the data plane received the provider routes: ```shell kubectl get aigatewayroutes -A kubectl get aiservicebackends -A ``` Both should show `Accepted` resources within a minute of the data plane coming up. If the lists are empty, consult the Agent Router onboarding guide to confirm providers are configured. ## Step 10: verify the install Run `tare doctor`: ```shell tare doctor /path/to/data-plane-credentials.json --verbose ``` Pass criteria: all in-cluster checks report `Status: OK` (or `Healthy`) with `0 errors, 0 warnings`, and the final line confirms the health-report bundle was accepted (`Sending health report ... OK (bundle )`). Expected output (abridged): ```none CHECKS PERFORMED: - Namespace existence (system, dataplane) - CRD presence (Gateway API, AI Gateway, RouteDeployment) - Controller deployments ready (AI Gateway etc) - Proxy deployment ready - GatewayClass and Gateway accepted/programmed - EnvoyPatchPolicy acceptance (per instance) - EnvoyProxy acceptance (per instance) - Egress EnvoyProxy image uses envoy-tars - Identity Secret and ConfigMap present - AIServiceBackend acceptance - Envoy Gateway Backend acceptance - BackendTrafficPolicy acceptance - BackendSecurityPolicy acceptance - BackendTLSPolicy acceptance - ClientTrafficPolicy acceptance - HTTPRouteFilter presence/acceptance - ReferenceGrant presence - RouteDeployment status conditions - AIGatewayRoute acceptance/resolution - HTTPRoute parent acceptance/resolution - MCPRoute parent acceptance/resolution - Proxy admin and forward endpoints - Pod CrashLoopBackOff (excluding tars-config-monitor) Sending health report to https://api..tetrate.ai/v1/dataplane-status... OK (bundle ) ``` Send a request with an invalid token to confirm auth is enforced: ```shell curl -sS -o /dev/null -w "HTTP %{http_code}\n" \ "http:///v1/chat/completions" \ -X POST -H "Content-Type: application/json" \ -H "Authorization: Bearer NotREAL" \ -d '{"model":"gpt-5-mini","messages":[{"role":"user","content":"hi"}]}' # Expected: HTTP 401 ``` If the response is `HTTP 200`, contact support. Auth is enforced automatically on every install. If `tare doctor` reports `Status: Broken`, see [Troubleshooting](#troubleshooting). ## Step 11: smoke tests In the router app (`https://router..tetrate.ai`), select **API Keys** in the sidebar and create a key. Applications (and the tests below) use this key as `Authorization: Bearer ...`. Set the host and key once: ```shell export DP_HOST= export TARS_API_KEY="" ``` ### Chat Completions (OpenAI shape) ```shell curl -s "http://${DP_HOST}/v1/chat/completions" \ -X POST -H "Content-Type: application/json" \ -H "Authorization: Bearer ${TARS_API_KEY}" \ -d '{ "model": "gpt-5-mini", "messages": [{"role": "user", "content": "hello, what are you?"}] }' ``` ### Anthropic Messages (native shape) ```shell curl -s "http://${DP_HOST}/v1/messages" \ -X POST -H "Content-Type: application/json" \ -H "anthropic-version: 2023-06-01" \ -H "Authorization: Bearer ${TARS_API_KEY}" \ -d '{ "model": "claude-haiku-4-5", "max_tokens": 64, "messages": [{"role": "user", "content": "hello"}] }' ``` ### List available models ```shell curl -s "http://${DP_HOST}/v1/models" \ -H "Authorization: Bearer ${TARS_API_KEY}" | jq '.data[].id' | sort -u ``` ### Streaming ```shell curl -s "http://${DP_HOST}/v1/chat/completions" \ -X POST -H "Content-Type: application/json" \ -H "Authorization: Bearer ${TARS_API_KEY}" \ -d '{ "model": "gpt-5-mini", "stream": true, "messages": [{"role": "user", "content": "count from 1 to 5"}] }' ``` ### MCP Create an MCP profile in the router app: **MCP Profiles** (sidebar) → **Create profile**. The profile ID is used in the URL below. ```shell curl -s "http://${DP_HOST}/mcp/" \ -X POST -H "Content-Type: application/json" \ -H "Authorization: Bearer ${TARS_API_KEY}" \ -d '{"jsonrpc":"2.0","method":"tools/list","id":1}' ``` To register the MCP profile with Claude Code: ```shell claude mcp add --transport http \ http://${DP_HOST}/mcp/ \ --header "Authorization: Bearer ${TARS_API_KEY}" ``` ## Upgrading To upgrade the data plane to a newer Agent Router release: 1. Install the new `tare` CLI version (re-run [Step 2](#step-2-install-the-tare-cli)). 1. Re-run image sync ([Step 5.2](#step-52-sync-images)) and `tare install` ([Step 6](#step-6-install-the-agent-router-data-plane)). Cluster state (namespaces, ingress, DNS, and dashboard configuration) persists across upgrades. ## Cleanup To remove the deployment, delete resources in dependency order: ```shell # 1. Ingress + AGIC addon (also releases the App Gateway) kubectl delete ingress tars-ingress -n tars-dataplane --ignore-not-found az aks disable-addons \ --resource-group "${RESOURCE_GROUP}" \ --name "${AKS_CLUSTER_NAME}" \ --addons ingress-appgw 2>/dev/null || true # 2. Helm release helm uninstall tars -n tars-system 2>/dev/null || true # 3. ACR az acr delete --resource-group "${RESOURCE_GROUP}" --name "${ACR_NAME}" --yes # 4. AKS cluster az aks delete \ --resource-group "${RESOURCE_GROUP}" \ --name "${AKS_CLUSTER_NAME}" \ --yes --no-wait # 5. Resource group (catches anything left behind) az group delete --name "${RESOURCE_GROUP}" --yes --no-wait # 6. Kubeconfig rm -f ~/kubeconfig-${AKS_CLUSTER_NAME} ``` :::note If `helm uninstall tars` hangs on finalizers (for example, `GatewayClass//tars-egress still exists`), see the finalizer cleanup procedure in [Troubleshooting](#cleanup-helm-uninstall-hangs). ::: ## Troubleshooting Issues are grouped by the step where they are most likely to occur. ### Image synchronization issues | Symptom | Cause | Fix | | :--- | :--- | :--- | | `401 Unauthorized` on `HEAD https://registry.tetrate.ai/v2/...` | The credential is not authorized to pull from `registry.tetrate.ai`. | Regenerate the credential from **Dashboard → System → Settings → Data plane → Credentials** and retry. | | `unauthorized: authentication required` from the destination registry | The `az acr login` token has expired (~3h default). | Re-run `az acr login --name "${ACR_NAME}"` and retry the sync. | ### Acr pull access | Symptom | Cause | Fix | | :--- | :--- | :--- | | Pods in `ImagePullBackOff` with `403 Forbidden` from ACR | The image-pull secret was not created or is missing from the pod's namespace. | Confirm in both namespaces: `kubectl get secret tars-image-pull-secret -n tars-dataplane` and `kubectl get secret tars-image-pull-secret -n tars-system`. If missing, re-run `tare install` with the `--image-pull-secret-stdin` flow from the [Step 6](#step-6-install-the-agent-router-data-plane) tip. | ### Agic | Symptom | Cause | Fix | | :--- | :--- | :--- | | `az aks enable-addons` fails with `AuthorizationFailed: ... managedClusters/write` | Caller lacks AKS write permission. | Grant `Azure Kubernetes Service Contributor Role` on the AKS resource. | | `LinkedAuthorizationFailed: ... Microsoft.OperationalInsights/workspaces/sharedkeys/read` | Container Insights is enabled and AGIC requires read access on the linked Log Analytics workspace. | Grant `Log Analytics Contributor` on the linked workspace RG, or disable Container Insights: `az aks disable-addons -n -g --addons monitoring`. | | Ingress has no `ADDRESS` after five minutes; AGIC log reports `App Gateway in stopped state` | AGIC reconciled too early. | Restart the AGIC controller: `kubectl delete pod -n kube-system -l app=ingress-appgw`. The replacement pod re-reads state and programs the gateway. | | AGIC log loops on `Waiting for overlay extension config to be ready` | Cluster uses Cilium dataplane or Azure CNI Overlay; AGIC does not support either. | Switch to [Appendix A: AGC](#appendix-a-alternative-ingress-agc), or recreate the cluster with traditional Azure CNI. | | Ingress has an `ADDRESS` but `curl` returns `502 Bad Gateway` | AGIC's default health probe is `GET /` and the Agent Router egress proxy returns 404 there. | The Ingress YAML in [Step 7.2](#step-72-create-the-ingress) sets the `health-probe-path` and `health-probe-status-codes: "200-499"` annotations. Add them if missing; AGIC reconciles within ~30 seconds. | ### Testing | Symptom | Cause | Fix | | :--- | :--- | :--- | | HTTP 404 with body `No matching route found. It is likely because the model specified in your request is not configured in the Gateway.` | The requested model name is not configured, or no providers are configured. | Verify `kubectl get aigatewayroutes -A` shows `Accepted` rows. If empty, consult the Agent Router onboarding guide. Changes propagate to the data plane within ~30 seconds. | | HTTP 404 with empty body | No `AIGatewayRoute` resources exist in the cluster. | Check the data plane is connected to the management plane: `kubectl logs -n tars-system deployment/controller-worker --tail=50`. The log entry `No secret found for provider` indicates the provider key did not reach the data plane; contact the Agent Router onboarding team. | | HTTP 401 with a valid bearer | The API key was issued against a different management plane than this data plane is registered to. | Issue a new key from the router app for this tenant (`https://router..tetrate.ai` → API Keys → Create). | | HTTP 502 from the dashboard playground (but not from direct `curl`) | The URL registered on the management plane does not match what the App Gateway serves. Most common cause: registered `https://` but the App Gateway only listens on HTTP/80. | Either enable TLS on the App Gateway (see [Appendix B](#appendix-b-enable-tls)) and keep the `https://` URL, or set the registered URL in **Dashboard → System → Settings → Data planes** to `http://` to match. | ### Cleanup: Helm uninstall hangs | Symptom | Cause | Fix | | :--- | :--- | :--- | | `helm uninstall tars` times out with `resource GatewayClass//tars-egress still exists. status: Terminating` | Custom resource finalizers block namespace deletion when controllers exit before the finalizer drains. | Force-clear finalizers in two passes. (1) Clear gateway-related CRs: `kubectl patch gatewayclass tars-egress --type=merge -p '{"metadata":{"finalizers":[]}}'` and repeat for `aiservicebackends`, `backendsecuritypolicies`, `mcproutes`, `tarsroutedeployments`. (2) Once namespaces start terminating, do the same for `aigatewayroutes`. Then `kubectl delete ns tars-system tars-dataplane`. | ## Appendix a: alternative ingress (agc) Use AGC when the AKS cluster runs the **Cilium dataplane** or **Azure CNI Overlay**, neither of which is supported by AGIC, and recreating the cluster is impractical. AGC works on any CNI. :::note AGC planning is required at cluster creation time: the AKS cluster needs `--enable-oidc-issuer` and `--enable-workload-identity`. Existing clusters without these flags can be updated using `az aks update`. No rebuild is required. ::: High-level steps: 1. Register the resource provider: `az provider register --namespace Microsoft.ServiceNetworking` 1. Create a user-assigned managed identity for the ALB controller. 1. Grant the identity `AppGw for Containers Configuration Manager` on the cluster's node resource group and `Network Contributor` on the cluster's VNet. 1. Federate the identity with the AKS OIDC issuer. 1. Install the ALB controller via Helm (`oci://mcr.microsoft.com/application-lb/charts/alb-controller`). 1. Create a delegated subnet for AGC and an `ApplicationLoadBalancer` CR. 1. Create a `Gateway` (Gateway API) and `HTTPRoute` instead of an `Ingress`. For the full walkthrough, see the [Microsoft documentation](https://learn.microsoft.com/azure/application-gateway/for-containers/). Replace [Step 7](#step-7-expose-the-data-plane-via-agic) with the AGC setup. [Step 8](#step-8-wire-dns-and-register-the-url) and all subsequent steps are unchanged; only the ingress provisioning differs. ## Appendix b: enable TLS The main flow uses HTTP-only on port 80 so the install can complete without a certificate. Production deployments require TLS on the App Gateway. AGIC supports any certificate delivery mechanism that produces a `kubernetes.io/tls` Secret in the cluster. Two example flows are described below. Existing TLS provisioning workflows (corporate CA, Azure Key Vault, internal PKI) can be used by delivering the resulting certificate and key as a `tls` Secret named in the Ingress. ### B.1: bring your own certificate Create the secret from an existing fullchain and key: ```shell kubectl create secret tls tars-ingress-tls \ --cert=path/to/fullchain.pem \ --key=path/to/privkey.pem \ -n tars-dataplane ``` Update the Ingress to use the secret, with host scoping and HTTP-to-HTTPS redirect: ```yaml metadata: annotations: appgw.ingress.kubernetes.io/health-probe-path: / appgw.ingress.kubernetes.io/health-probe-status-codes: "200-499" appgw.ingress.kubernetes.io/ssl-redirect: "true" spec: ingressClassName: azure-application-gateway tls: - hosts: [proxy.example.com] secretName: tars-ingress-tls rules: - host: proxy.example.com http: paths: - path: / pathType: Prefix backend: service: name: egress port: number: 10080 ``` ### B.2: cert-manager and let's encrypt cert-manager auto-issues and auto-renews certificates from Let's Encrypt. The HTTP-01 challenge runs through the same AGIC ingress configured in [Step 7](#step-7-expose-the-data-plane-via-agic), so no additional infrastructure is required. This option suits sites without an existing certificate workflow. **Prerequisites:** - The DNS A record from [Step 8.1](#step-81-add-the-dns-a-record) must be live (`dig +short ` returns the App Gateway IP). Let's Encrypt validates over DNS and HTTP. - The App Gateway must listen on HTTP/80 (default from [Step 7](#step-7-expose-the-data-plane-via-agic)). - An email address for Let's Encrypt expiry notices. #### Step b.2.1: install cert-manager One-time, cluster-wide: ```shell helm repo add jetstack https://charts.jetstack.io helm repo update helm install cert-manager jetstack/cert-manager \ --namespace cert-manager --create-namespace \ --version v1.18.2 \ --set crds.enabled=true ``` Verify the install: ```shell kubectl get pods -n cert-manager # Expect 3 pods Running: cert-manager-*, cert-manager-cainjector-*, cert-manager-webhook-* ``` #### Step b.2.2: create the clusterissuer ```shell cat <<'EOF' | kubectl apply -f - apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: letsencrypt-prod spec: acme: email: ops@example.com # Replace with a real address server: https://acme-v02.api.letsencrypt.org/directory privateKeySecretRef: name: letsencrypt-prod-account-key solvers: - http01: ingress: ingressClassName: azure-application-gateway EOF ``` Confirm the issuer reached the `Ready` state: ```shell kubectl get clusterissuer letsencrypt-prod # NAME READY AGE # letsencrypt-prod True 20s ``` :::tip For testing, point `server:` at `https://acme-staging-v02.api.letsencrypt.org/directory`. The staging issuer has higher rate limits and a separate root, preserving the production quota. Switch to production once the certificate issues cleanly on staging. ::: #### Step b.2.3: update the ingress with TLS and cert-manager annotations Replace the Ingress from [Step 7.2](#step-72-create-the-ingress) with the version below. Two additions: a `tls:` block referencing a Secret cert-manager will create, and the `cert-manager.io/cluster-issuer` annotation that triggers issuance. ```shell cat <<'EOF' | kubectl apply -f - apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: tars-ingress namespace: tars-dataplane annotations: cert-manager.io/cluster-issuer: letsencrypt-prod appgw.ingress.kubernetes.io/ssl-redirect: "true" appgw.ingress.kubernetes.io/health-probe-path: / appgw.ingress.kubernetes.io/health-probe-status-codes: "200-499" spec: ingressClassName: azure-application-gateway tls: - hosts: [proxy.example.com] # Replace with the data plane hostname secretName: tars-ingress-tls rules: - host: proxy.example.com # Must match tls.hosts http: paths: - path: / pathType: Prefix backend: service: name: egress port: number: 10080 EOF ``` :::important The `host:` on the rule must match the hostname in `tls.hosts` and the hostname registered for the data plane on the management plane. A mismatch causes either AGIC to reject the rule or Let's Encrypt to fail the HTTP-01 challenge. ::: #### Step b.2.4: wait for cert-manager to issue the certificate ```shell kubectl get certificate -n tars-dataplane -w # Initial: READY=False (cert-manager solves the HTTP-01 challenge) # After ~1m: READY=True ``` If the certificate remains `READY=False` for more than a couple of minutes, inspect the order and challenge status: ```shell kubectl describe certificate tars-ingress-tls -n tars-dataplane kubectl get challenge -n tars-dataplane kubectl describe challenge -n tars-dataplane # Reports the precise solver error ``` Common challenge failures and fixes: | Failure message | Fix | | :--- | :--- | | `Self-check failed: ... acme: server returned a non-2xx HTTP status` (404) | AGIC has not yet programmed the solver path. Wait ~30 seconds; cert-manager creates a solver Ingress and AGIC reconciles. | | `dns: NXDOMAIN` or `no IP for hostname` | DNS A record has not propagated. Confirm with `dig +short `. | | `urn:ietf:params:acme:error:rateLimited` | Let's Encrypt quota exceeded. Switch to the staging issuer (see Step B.2.2 tip) and retry. | #### Step b.2.5: verify HTTPS end-to-end ```shell curl -v https:///v1/models \ -H "Authorization: Bearer ${TARS_API_KEY}" \ 2>&1 | grep -E '^[<>] (HTTP|x-amz|Authorization|expire date|issuer)' | head -10 ``` Expected: a clean TLS handshake (no certificate errors) and `HTTP/2 200`. Verify the certificate chain: ```shell echo | openssl s_client -servername -connect :443 2>/dev/null \ | openssl x509 -noout -subject -issuer -dates # subject= CN = proxy.example.com # issuer= C = US, O = Let's Encrypt, CN = R10 # notAfter=... (~90 days from issue) ``` #### Step b.2.6: update the registered URL If an `http://...` URL was registered in [Step 8.2](#step-82-register-the-url-on-the-management-plane), update it to `https://...`. The gateway URL registered on the management plane must match the protocol the App Gateway serves. **Renewal:** cert-manager renews automatically at two-thirds of the certificate's lifetime (approximately 60 days for Let's Encrypt's 90-day certificates). No manual action is required. ## Appendix c: forward observability data to an OpenTelemetry collector The data plane can stream envoy HTTP access logs and `router_*` application metrics to a customer-managed OpenTelemetry Collector, which forwards to any backend (Azure Monitor, Datadog, Grafana Cloud, SigNoz). The two streams are configured in separate fields on the `EnvoyProxy` resource: | Stream | EnvoyProxy field | Contents | | :--- | :--- | :--- | | Access logs (per-request HTTP metadata) | `accessLog.sinks[]` | Method, status, path, latency, MCP headers, downstream/upstream addresses | | Metrics (`router_*` and envoy native stats) | `metrics.sinks[]` | `router_requests_total`, `router_model_requests_total`, plus envoy cluster/listener counters | Either stream can be configured independently; the instructions below cover both in order. ### Available metrics | Name | Type | Labels | | :--- | :--- | :--- | | `router_requests_total` | Counter | method, endpoint, status_code | | `router_request_duration_ms` | Histogram | method, endpoint, status_code | | `router_errors_total` | Counter | type, endpoint, status, model, provider | | `router_streaming_requests_total` | Counter | model, provider, endpoint | | `router_model_requests_total` | Counter | model, provider, endpoint, byok | | `router_auth_attempts_total` | Counter | result, auth_mode | | `router_balance_checks_total` | Counter | result | | `router_overrun_protections_total` | Counter | reason | Two equivalent scrape paths are available: 1. **Direct envoy admin** (`:19001/stats/prometheus`): metric names appear as listed above. This is the lightest setup for Prometheus-only consumers that do not need access logs. 1. **OpenTelemetry Collector** (configured below): exposes the same `router_*` names on `otel-collector.tars-dataplane.svc:9464/metrics`, plus an OTLP-gRPC receiver for the access-log stream. Recommended when a single collection point fans out to multiple backends. :::note If `router_*` metrics do not appear after sending traffic, ask the MP operator to verify `PROXY_CONFIG` is set on the management plane; these metrics require it. ::: ### Filter probe and scanner noise before building dashboards When the data plane is exposed on a public address, two sources contribute noise to `router_errors_total` and `router_auth_attempts_total`: - **Health probes** (AGIC, AGC, or any load balancer): the probe pings the backend every few seconds. The Agent Router egress proxy applies its auth filter before path matching, so unauthenticated probes register as authentication failures on the probe path (default `/`). - **Internet bot and scanner traffic**: any public IP attracts opportunistic scans targeting paths such as `/wiki`, `/favicon.ico`, `/SDK/webLanguage`, and `/invoker/EJBInvokerServlet`. Each scan increments the auth-failure counter. These counters reach the thousands within a few hours. Unfiltered charts make a healthy service appear to be failing. Filter to the data plane's real endpoints (`/v1/*` and `/mcp/*`): ```promql # Prometheus: keep only real customer traffic router_requests_total{endpoint=~"^/v1/.*|^/mcp/.*"} router_errors_total{endpoint=~"^/v1/.*|^/mcp/.*"} ``` ```kql // Azure Log Analytics / Application Insights equivalent | extend endpoint = tostring(customDimensions.endpoint) | where endpoint startswith "/v1/" or endpoint startswith "/mcp/" ``` ### Step c.1: deploy the OpenTelemetry collector Deploy into the `tars-dataplane` namespace; any other namespace fails with `unknown namespace for the cache`. The ConfigMap below wires both logs and metrics pipelines, exposes `router_*` on a Prometheus scrape endpoint (`:9464`), and is ready to fan out to additional backends. See [Send to a real observability backend](#send-to-a-real-observability-backend). ```shell cat <<'EOF' | kubectl apply -f - apiVersion: v1 kind: ConfigMap metadata: name: otel-collector-config namespace: tars-dataplane data: config.yaml: | receivers: otlp: protocols: http: { endpoint: 0.0.0.0:4318 } grpc: { endpoint: 0.0.0.0:4317 } processors: batch: { timeout: 5s } # Strip envoy's internal dynamic-modules scope prefix so router_* metrics # ship with their canonical names (router_requests_total, etc.) instead # of dynamicmodulescustom.router_requests_total. transform/strip_scope: metric_statements: - context: metric statements: - replace_pattern(name, "^dynamicmodulescustom\\.", "") exporters: # In-cluster Prometheus scrape target. Names land clean as router_*. prometheus: endpoint: 0.0.0.0:9464 namespace: "" send_timestamps: true metric_expiration: 30m resource_to_telemetry_conversion: { enabled: true } service: pipelines: metrics: receivers: [otlp] processors: [transform/strip_scope, batch] exporters: [prometheus] # Add a 'logs' pipeline when forwarding envoy HTTP access logs # (per-request method, status, path, MCP headers) to a backend # such as Azure Monitor or Datadog. See "Send to a real # observability backend" below for an example. --- apiVersion: apps/v1 kind: Deployment metadata: name: otel-collector namespace: tars-dataplane spec: replicas: 1 selector: { matchLabels: { app: otel-collector } } template: metadata: { labels: { app: otel-collector } } spec: containers: - name: collector image: otel/opentelemetry-collector-contrib:0.98.0 ports: - { containerPort: 4317, name: otlp-grpc } - { containerPort: 4318, name: otlp-http } volumeMounts: - { name: config, mountPath: /etc/otelcol-contrib } resources: requests: { cpu: 100m, memory: 128Mi } limits: { cpu: 250m, memory: 256Mi } volumes: - name: config configMap: { name: otel-collector-config } --- apiVersion: v1 kind: Service metadata: name: otel-collector namespace: tars-dataplane spec: selector: { app: otel-collector } ports: - { name: otlp-grpc, port: 4317, targetPort: 4317 } - { name: otlp-http, port: 4318, targetPort: 4318 } - { name: prometheus, port: 9464, targetPort: 9464 } EOF ``` Once the manifest is applied, an in-cluster Prometheus can scrape `http://otel-collector.tars-dataplane.svc:9464/metrics` and find clean `router_requests_total`, `router_model_requests_total`, and similar names. The `transform/strip_scope` processor removes envoy's internal scope prefix before export, so dashboards and alerts work without dealing with the OTel encoding. ### Step c.2: add the metrics sink to the envoyproxy Push `router_*` and envoy native stats from the Agent Router egress proxy into the collector. The access-log sink is a separate, opt-in step described in [Forwarding access logs](#forwarding-access-logs-optional). ```shell kubectl patch envoyproxy tars-egress-proxy -n tars-system --type=merge -p '{ "spec":{"telemetry":{"metrics":{"sinks":[ {"type":"OpenTelemetry","openTelemetry":{"backendRefs":[ {"group":"","kind":"Service","name":"otel-collector","namespace":"tars-dataplane","port":4317,"weight":1} ]}} ]}}} }' ``` ### Step c.3: restart the egress proxy deployment ```shell kubectl rollout restart -n tars-dataplane deployment/egress kubectl rollout status -n tars-dataplane deployment/egress --timeout=120s ``` ### Step c.4: verify After running the smoke tests, scrape the collector's Prometheus endpoint to confirm `router_*` metrics are flowing with clean names: ```shell POD=$(kubectl get pods -n tars-dataplane -l app=otel-collector -o jsonpath='{.items[0].metadata.name}') kubectl port-forward -n tars-dataplane pod/$POD 9464:9464 & sleep 2 curl -s http://localhost:9464/metrics | grep '^router_' | head ``` Expected: `router_requests_total`, `router_model_requests_total`, `router_auth_attempts_total`, `router_request_duration_ms_bucket`, and similar names with non-zero counts matching the traffic sent. If `router_*` metrics do not appear after traffic, ask the MP operator to check `PROXY_CONFIG` on the management plane. For deeper diagnostics, the collector's self-metrics report pipeline throughput: ```shell kubectl port-forward -n tars-dataplane pod/$POD 8888:8888 & sleep 2 curl -s http://localhost:8888/metrics | grep otelcol_exporter_sent ``` Non-zero `otelcol_exporter_sent_metric_points` confirms metrics are leaving the collector toward each configured exporter. The `_log_records` counter appears only after a logs pipeline is added (see [Forwarding access logs](#forwarding-access-logs-optional)). :::note The EnvoyProxy metrics-sink patch resets on every `tare install` re-run. Re-apply after each reinstall, or script it as a post-install hook. ::: ### Send to a real observability backend The starter configuration exports `router_*` metrics to an in-cluster Prometheus endpoint only. Two common extensions: 1. **Fan out metrics to a managed backend** (Azure Monitor, Datadog, Grafana Cloud): add a backend exporter alongside `prometheus`. 1. **Forward HTTP per-request access logs**: opt-in. Requires both an EnvoyProxy patch (to make egress emit access logs) and a logs pipeline in the collector. The Azure Monitor walkthrough below shows both. #### Example: Azure monitor / application insights Create the Application Insights resource (workspace-based, reusing the AKS Log Analytics workspace): ```shell az monitor app-insights component create \ --app tars-dp-insights \ --location "${LOCATION}" \ --kind web \ --resource-group "${RESOURCE_GROUP}" \ --workspace "/subscriptions//resourceGroups//providers/Microsoft.OperationalInsights/workspaces/" CONN_STR=$(az monitor app-insights component show \ --app tars-dp-insights -g "${RESOURCE_GROUP}" \ --query connectionString -o tsv) ``` Store the connection string in a secret and inject it as an environment variable into the collector pod: ```shell kubectl create secret generic otel-azure-creds -n tars-dataplane \ --from-literal=APP_INSIGHTS_CONN_STR="${CONN_STR}" kubectl set env deploy/otel-collector -n tars-dataplane --from secret/otel-azure-creds ``` Add the `azuremonitor` exporter (ships with `otel/opentelemetry-collector-contrib`) alongside the default `prometheus`. Keep the `transform/strip_scope` processor in the metrics pipeline so names appear in Azure Monitor as clean `router_*`. The logs pipeline carries envoy HTTP per-request access logs and does not include `prometheus`, since that exporter handles only metrics: ```yaml exporters: prometheus: endpoint: 0.0.0.0:9464 namespace: "" send_timestamps: true metric_expiration: 30m resource_to_telemetry_conversion: { enabled: true } azuremonitor: connection_string: ${env:APP_INSIGHTS_CONN_STR} service: pipelines: metrics: receivers: [otlp] processors: [transform/strip_scope, batch] exporters: [prometheus, azuremonitor] logs: receivers: [otlp] processors: [batch] exporters: [azuremonitor] ``` :::tip **Environment variable syntax.** The collector requires `${env:VAR_NAME}` (with the `env:` prefix). Plain `${VAR_NAME}` silently fails to substitute and the exporter does not load. The collector log shows no error, so check the `otelcol_exporter_sent_log_records` and `_metric_points` self-metrics on `:8888` to confirm. ::: Restart and verify both exporters are shipping: ```shell kubectl rollout restart deploy/otel-collector -n tars-dataplane # After a curl to /v1/chat/completions: POD=$(kubectl get pods -n tars-dataplane -l app=otel-collector -o jsonpath='{.items[0].metadata.name}') kubectl port-forward -n tars-dataplane pod/$POD 8888:8888 & sleep 2 curl -s http://localhost:8888/metrics | grep otelcol_exporter_sent ``` Both `prometheus` and `azuremonitor` exporters should show non-zero `otelcol_exporter_sent_log_records` and `_metric_points`. **Where to view data:** Azure Portal → Application Insights `tars-dp-insights` → Logs (for KQL) or Workbooks (for custom dashboards). :::note Default Application Insights panes will not populate. The Overview, Performance, Failures, and Application Map panes require AI-native event types (requests, dependencies, exceptions). The `azuremonitor` OTel exporter does not translate envoy access logs into those types; the data resides in `customMetrics` (envoy and `router_*`) and `traces` (envoy access logs). Build a Workbook with the queries below for a usable dashboard. ::: Useful queries (paste into the Logs pane of `tars-dp-insights`): ```kql // Recent envoy access logs traces | where timestamp > ago(15m) | extend method = tostring(customDimensions["method"]), status = tostring(customDimensions["response_code"]), route = tostring(customDimensions["route_name"]) | project timestamp, method, status, route, duration=customDimensions["duration"] | order by timestamp desc // router_* metrics: latest cumulative values per metric customMetrics | where timestamp > ago(1h) | where name startswith "router_" | extend endpoint = tostring(customDimensions.endpoint) | where endpoint startswith "/v1/" or endpoint startswith "/mcp/" // Exclude probe and scanner noise | summarize total = max(valueSum) by name | order by name asc // Per-minute request rate by status code customMetrics | where timestamp > ago(1h) | where name == "router_requests_total" | extend status = tostring(customDimensions.status_code) | summarize cum = max(valueSum) by bin(timestamp, 1m), status | order by status, timestamp asc | serialize | extend per_min = cum - prev(cum, 1, 0) | where per_min >= 0 | render timechart ``` :::important **Counter aggregation: use `max`, not `sum`.** Every `router_*_total` is a cumulative counter. OTel re-ships the current value on every flush (~5s default), so `customMetrics` rows accumulate by hundreds per hour. `sum(valueSum)` inflates the result by orders of magnitude (for example, 1.8M when the real cumulative count is ~2,300). - For cumulative totals: `max(valueSum)` (latest snapshot). - For rates over time: compute deltas with `serialize | extend ... = cum - prev(cum, 1, 0)`. ::: #### Forwarding access logs (optional) Adds envoy HTTP per-request access logs (method, status, path, latency, MCP headers, downstream and upstream addresses) on top of the metrics. Two changes are required; both can be applied incrementally without re-running `tare install`. **1. Patch the EnvoyProxy** to emit access logs to the collector. The default `accessLog` block varies between tare builds; check the existing shape before patching: ```shell kubectl get envoyproxy tars-egress-proxy -n tars-system \ -o jsonpath='{.spec.telemetry.accessLog}' # Non-empty: use Path A. Empty: use Path B. ``` **Path A: JSON-patch** (default `accessLog` already present, append a sink): ```shell kubectl patch envoyproxy tars-egress-proxy -n tars-system --type=json -p '[ { "op": "add", "path": "/spec/telemetry/accessLog/settings/0/sinks/-", "value": { "type": "OpenTelemetry", "openTelemetry": { "backendRefs": [ {"group":"","kind":"Service","name":"otel-collector","namespace":"tars-dataplane","port":4317,"weight":1} ] } } } ]' ``` **Path B: merge-patch** (no default `accessLog`, create the entire block): ```shell kubectl patch envoyproxy tars-egress-proxy -n tars-system --type=merge -p '{ "spec":{"telemetry":{"accessLog":{"settings":[ {"sinks":[ {"type":"OpenTelemetry","openTelemetry":{"backendRefs":[ {"group":"","kind":"Service","name":"otel-collector","namespace":"tars-dataplane","port":4317,"weight":1} ]}} ]} ]}}} }' ``` **2. Add a logs pipeline** to the collector ConfigMap, pointing at the chosen backend exporter (for example, `azuremonitor`). The Azure Monitor walkthrough above shows the full ConfigMap diff; the relevant addition is: ```yaml service: pipelines: logs: receivers: [otlp] processors: [batch] exporters: [azuremonitor] # Or datadog, otlphttp, and similar ``` **3. Restart egress** so the new accessLog sink loads: ```shell kubectl rollout restart -n tars-dataplane deployment/egress ``` Verify access logs are flowing: ```shell # Azure Monitor: open Application Insights → Logs and run # traces | where timestamp > ago(15m) | take 10 # # Datadog / Grafana Cloud / others: check the corresponding Logs explorer. # # Real-time verification on the collector side: temporarily add 'debug' to # the logs pipeline exporters list and grep: # kubectl logs -n tars-dataplane deployment/otel-collector --tail=100 \ # | grep otel_envoy_accesslog ``` :::note The EnvoyProxy `accessLog` patch resets on every `tare install` re-run. Re-apply after each reinstall. ::: #### Other backends | Platform | Exporter | Reference | | :--- | :--- | :--- | | Datadog | `datadog` | `https://docs.datadoghq.com/opentelemetry/otel_collector_datadog_exporter/` | | Grafana Cloud | `otlphttp` to a grafana.net endpoint | `https://grafana.com/docs/grafana-cloud/send-data/otlp/` | | SigNoz Cloud | `otlphttp` with the `signoz-access-token` header | `https://signoz.io/docs/instrumentation/opentelemetry-collector/` | | Splunk Observability | `signalfx` | `https://docs.splunk.com/observability/en/gdi/opentelemetry/exporters/signalfx-exporter.html` | | In-cluster SigNoz, Jaeger, or Grafana | `otlp` or `otlphttp` to the local service | Agent Router–specific | The pattern is consistent across backends: define the exporter in the collector's ConfigMap, add it to the relevant pipelines, then restart the collector. The EnvoyProxy patches applied above remain unchanged. :::note Both EnvoyProxy patches (metrics and accessLog) reset on every `tare install` re-run. Re-apply after each reinstall. ::: ## Appendix d: use an existing private registry Use this path when an organization-wide private container registry (Nexus, Harbor, JFrog Artifactory, or another enterprise registry) is already in place. This appendix replaces [Step 4](#step-4-create-an-azure-container-registry) and [Step 5](#step-5-sync-agent-router-images-to-acr) in the main flow. Two separate registry credentials are involved: - **Operator credentials**: used by the workstation running `tare install --image-sync` to push images into the private registry. - **Kubernetes pull credentials**: stored as an image-pull secret so AKS nodes can pull images from the private registry. The data plane credential is still required. `tare` uses it to authenticate to Tetrate's source registry while syncing images. The private registry username and password are used only for the destination registry and Kubernetes image pulls. ### Step d.1: set variables for the private registry ```shell DP_CREDENTIAL=./data-plane-credentials.json PRIVATE_REGISTRY_HOST=registry.acme.example.com PRIVATE_IMAGE_REGISTRY=registry.acme.example.com/tare PULL_SECRET=acme-registry-pull REGISTRY_USERNAME= REGISTRY_PASSWORD= ``` ### Step d.2: sync images into the private registry Log in locally with credentials that can push to the private registry: ```shell printf '%s' "${REGISTRY_PASSWORD}" | \ docker login "${PRIVATE_REGISTRY_HOST}" \ --username "${REGISTRY_USERNAME}" \ --password-stdin ``` Copy the pinned Agent Router images into the registry: ```shell tare install "${DP_CREDENTIAL}" \ --image-sync "${PRIVATE_IMAGE_REGISTRY}" \ --sync-only ``` `tare` authenticates to the Tetrate source registry using the data plane credential. The local Docker login authenticates to the destination registry. ### Step d.3: create Kubernetes pull secrets Create the same pull secret in both namespaces: ```shell kubectl create namespace tars-system --dry-run=client -o yaml | kubectl apply -f - kubectl create namespace tars-dataplane --dry-run=client -o yaml | kubectl apply -f - kubectl create secret docker-registry "${PULL_SECRET}" \ --docker-server="${PRIVATE_REGISTRY_HOST}" \ --docker-username="${REGISTRY_USERNAME}" \ --docker-password="${REGISTRY_PASSWORD}" \ --namespace tars-system \ --dry-run=client -o yaml | kubectl apply -f - kubectl create secret docker-registry "${PULL_SECRET}" \ --docker-server="${PRIVATE_REGISTRY_HOST}" \ --docker-username="${REGISTRY_USERNAME}" \ --docker-password="${REGISTRY_PASSWORD}" \ --namespace tars-dataplane \ --dry-run=client -o yaml | kubectl apply -f - ``` ### Step d.4: install from the private registry Install Agent Router with `--image-registry` pointing at the private registry and `--image-pull-secret-name` referencing the existing secret: ```shell tare install "${DP_CREDENTIAL}" \ --image-registry "${PRIVATE_IMAGE_REGISTRY}" \ --image-pull-secret-name "${PULL_SECRET}" \ --wait ``` This does not create the secret. It tells the Helm install to use the secret that already exists in `tars-system` and `tars-dataplane`. :::note If a platform team mirrors Agent Router images into the private registry before the install, skip the `--image-sync` step from [Step D.2](#step-d2-sync-images-into-the-private-registry) and run only the `tare install` command above. ::: After the install, continue with [Step 7](#step-7-expose-the-data-plane-via-agic) or the organization's preferred ingress path. Where to go next Gateway installation Install the data plane gateway components that manage inbound access and configure request routing. Self-Hosted Data Plane See the full install sequence: gateway, SSO, then developer handoff. --- ## Installation guide for GCP # Agent Router data plane installation for Google Cloud Platform This guide installs the Agent Router data plane on Google Cloud Platform (GCP) using Google Kubernetes Engine (GKE). ## Architecture overview Agent Router Enterprise uses a split-plane model. A **Management Plane** hosted by **Tetrate** holds configuration and serves the web UI. A **Data Plane** runs in a customer-managed GKE cluster and handles all AI traffic. The two planes communicate over a single outbound HTTPS connection initiated by the data plane; no inbound connections from the internet reach the management plane. The data plane requires a stable public hostname so applications can call it; TLS certificates are issued for hostnames, not IPs. The gateway-install step provisions a Google-managed certificate against that hostname automatically. ```mermaid architecture-beta group gcp(cloud)[GCP] group tetrate(cloud)[Tetrate hosted] service mp(server)[Management Plane] in tetrate service dp(server)[Data Plane] in gcp mp:R -- L:dp ``` By the end of this guide you will have: - An Artifact Registry repository mirroring the Agent Router images - A GKE cluster running the data plane in the `tars-system` and `tars-dataplane` namespaces - A Google Cloud Application Load Balancer fronting the data plane at a DNS name you own - A gateway URL registered on the management plane Plan for 30–45 minutes of installation time, plus DNS propagation. ## Table of contents - **Prepare for the installation**: obtain the data plane credential and install the CLI - [Prerequisites](#prerequisites) - [Step 1: Obtain your data plane credential](#step-1-obtain-your-data-plane-credential) - [Step 2: Install the `tare` CLI](#step-2-install-the-tare-cli) - **Cluster setup**: create or reuse a Kubernetes cluster - [Step 3: Provision the GKE cluster](#step-3-provision-the-gke-cluster) - **Registry setup**: mirror Agent Router images so the cluster can pull them - [Step 4: Create an Artifact Registry repository](#step-4-create-an-artifact-registry-repository) - [Step 5: Sync Agent Router images to the registry](#step-5-sync-agent-router-images-to-the-registry) - **Data Plane Installation**: deploy the data plane - [Step 6: Install the Agent Router data plane](#step-6-install-the-agent-router-data-plane) - **Gateway setup**: provision the public gateway, certificate, and DNS authorization - [Step 7: Install the gateway](#step-7-install-the-gateway) - **DNS configuration**: point a domain at the gateway and register the URL - [Step 8: Wire DNS and register the URL](#step-8-wire-dns-and-register-the-url) - **Testing the installation**: verify the install works end-to-end - [Step 9: Verify provider routes](#step-9-verify-provider-routes) - [Step 10: Verify the install](#step-10-verify-the-install) - [Step 11: Smoke tests](#step-11-smoke-tests) * **Appendices** - [Appendix A: Cross-project and external-registry pull access](#appendix-a-cross-project-and-external-registry-pull-access) - [Appendix B: Forward observability data to an OpenTelemetry Collector](#appendix-b-forward-observability-data-to-an-opentelemetry-collector) ## Prerequisites ### Dashboard and Router app access Agent Router exposes two web surfaces. Both URLs are provided during onboarding: - **Dashboard** (admin): `https://dashboard..tetrate.ai`. Used in [Step 1](#step-1-obtain-your-data-plane-credential) and [Step 8](#step-8-wire-dns-and-register-the-url). - **Router app** (end-user): `https://router..tetrate.ai`. Used in [Step 11](#step-11-smoke-tests) for creating API keys and MCP profiles. ### Required tools Install the following on the workstation used to run this guide: | Tool | Install | | :--- | :--- | | `gcloud` | [cloud.google.com/sdk/docs/install](https://cloud.google.com/sdk/docs/install) | | `kubectl` | `gcloud components install kubectl` (or [kubernetes.io/docs/tasks/tools](https://kubernetes.io/docs/tasks/tools/)) | | `curl` | Preinstalled on macOS and most Linux distributions | | `jq` | [jqlang.github.io/jq/download](https://jqlang.github.io/jq/download/) | | `tare` CLI | Installed in [Step 2](#step-2-install-the-tare-cli) | A `data-plane-credentials.json` file is also required. See [Step 1](#step-1-obtain-your-data-plane-credential). ### Infrastructure A dedicated workload cluster must be provisioned before starting the installation. The cluster should consist of at least three (3) nodes. See [Cluster sizing](#cluster-sizing) for more details. :::warning Tetrate support does not cover client-side infrastructure provisioning or Kubernetes issues. The instructions for creating clusters and related infrastructure components are provided as a courtesy and should be carefully evaluated before executing them. ::: ### GCP iam roles The following roles are required on the project that will host the data plane: | Role | Scope | Required for | | :--- | :--- | :--- | | `roles/container.admin` | The GCP project | Creating and managing the GKE cluster | | `roles/artifactregistry.admin` | The GCP project (or specific repo) | Creating Artifact Registry and pushing images | | `roles/iam.serviceAccountUser` | Attached service accounts | Cluster node SA, gateway provisioner SA | | `roles/compute.networkAdmin` | The GCP project | Reserving static IPs for the gateway | | `roles/certificatemanager.editor` | The GCP project | Provisioning Google-managed certificates | | `roles/dns.admin` | The DNS-managing project (often separate) | Adding A and CNAME records when the IAM principal manages DNS directly; not required when records are handed off to a DNS team | ### Cluster type: standard or autopilot GKE **Standard** is the recommended default: full control over node pools, predictable scheduling, and easier cost and performance tuning. This guide assumes Standard. GKE **Autopilot** is supported, but its placement and security constraints may reject the chart's resource and security assumptions on the first run. For Autopilot, validate the install against the cluster's policy before applying: ```shell # Dry-run validation against Autopilot policy tare install /path/to/data-plane-credentials.json --print-resources | \ kubectl apply --dry-run=server -f - ``` Size pod requests and limits to match the policy, and add headroom since Autopilot scaling is request-driven. ### Cluster sizing The default chart installs multiple always-on components, including the **Agent Router egress proxy** (Envoy-based; referred to as *egress envoy* in manifests and logs) with a minimum of 2 replicas, AI gateway controller and ext_proc, controller and worker, Redis, and rate-limit services. This is not a single-node footprint. The **egress envoy** is the dominant resource consumer. Its CPU and memory usage scale with the configuration size held in memory: the number of `AIGatewayRoute` and `AIServiceBackend` resources, header-mutation rules, and per-route features. Plan capacity for route counts that grow as providers, models, and projects are added. General-purpose machine types (`n2-*` family, balanced CPU and RAM) provide a balanced default. | Size | Use case | Recommended node pool | Approximate allocatable target | | :--- | :--- | :--- | :--- | | Small | Dev / test / low traffic | 3 × `e2-standard-2` | ≥ 6 vCPU, ≥ 20 GiB RAM | | Medium | Staging / light production | 3 × `n2-standard-4` | ≥ 12 vCPU, ≥ 40 GiB RAM | | High | Production with burst headroom | 3 × `n2-standard-8` (or split into system + data plane pools) | ≥ 24 vCPU, ≥ 80 GiB RAM | Maintain a minimum of three nodes to tolerate node upgrades and evictions. Demo installs may start at Small; production installations should start at Medium. ## Conventions All commands assume the environment variables defined in [Step 3](#step-3-provision-the-gke-cluster) and [Step 4](#step-4-create-an-artifact-registry-repository) are exported in the current shell. Re-export them when opening a new terminal. Replace any `` value with one of your own. ## Step 1: obtain your data plane credential In the dashboard, navigate to **System → Settings → Data plane credentials** and click **+ Generate Data plane credential**. Save the downloaded file as `data-plane-credentials.json` on the workstation used for installation. This is the long-lived identity the data plane uses to authenticate to the management plane. :::note Some parts of the product still use the older "service account" naming for this file. The dashboard is standardizing on "data plane credential"; the file is the same. Each data plane uses its own credential. Credentials can be revoked from the dashboard, and additional credentials can be generated (for example, one per environment) at any time. ::: ## Step 2: install the `tare` CLI Run the installer script: ```shell curl -sSL https://tare.tetrate.ai/tools/install.sh | bash ``` Output: ```none ==> tare installer ==> channel: stable ==> Detected platform: darwin-arm64 ==> Installing tare for darwin-arm64... ==> Downloading from: https://tare.tetrate.ai/tools/tags/v0.1.0-beta.2/tare-darwin-arm64.tar.gz ok Installed tare to /Users/jonhdoe/.tare/bin/tare ==> tare version: tare version v0.1.0-beta.2 ok Installation directory is already in your PATH ==> Get started: tare install identity.json --serve-url https://proxy.acme.com tare install --help ``` The installer prints the install path (typically `~/.tare/bin/tare`). Add it to `PATH` and verify the version: ```shell export PATH="$PATH:$HOME/.tare/bin" echo 'export PATH="$PATH:$HOME/.tare/bin"' >> ~/.zshrc # or ~/.bashrc ``` ```shell $ tare --version tare version v0.1.0-beta.2 ``` ## Step 3: provision the gke cluster ### Step 3.1: set environment variables ```shell PROJECT_ID= REGION=us-central1 ZONE=us-central1-a CLUSTER_NAME=tare-dp ``` ### Step 3.2: select the project ```shell gcloud config set project "${PROJECT_ID}" ``` ### Step 3.3: create the cluster ```shell gcloud container clusters create "${CLUSTER_NAME}" \ --zone "${ZONE}" \ --num-nodes 3 \ --machine-type n2-standard-4 \ --release-channel regular \ --enable-ip-alias \ --workload-pool "${PROJECT_ID}.svc.id.goog" \ --gateway-api=standard ``` `--gateway-api=standard` enables the GKE Gateway controller and installs the `GCPBackendPolicy` and `HealthCheckPolicy` CRDs that the gateway Helm chart depends on. Without it, [Step 7](#step-7-install-the-gateway) fails with `resource mapping not found for ... GCPBackendPolicy`. The `--num-nodes` and `--machine-type` values above correspond to the **Medium** tier in [Cluster sizing](#cluster-sizing). Adjust as needed. Provisioning takes approximately 5–10 minutes. :::tip **Already have a GKE cluster?** Reuse it after confirming two things: - Workload Identity is enabled: ```shell gcloud container clusters describe --zone \ --format='value(workloadIdentityConfig.workloadPool)' ``` - Gateway API is enabled: ```shell kubectl get crd gateways.gateway.networking.k8s.io kubectl get crd gcpbackendpolicies.networking.gke.io ``` If either is missing, enable Gateway API: ```shell gcloud container clusters update --zone --gateway-api=standard ``` ::: ### Step 3.4: fetch the kubeconfig ```shell gcloud container clusters get-credentials "${CLUSTER_NAME}" --zone "${ZONE}" kubectl get nodes ``` Expected output: ```none NAME STATUS ROLES AGE VERSION gke-tare-dp-default-pool-xxxxxxxx-xxxx Ready 5m v1.34.x gke-tare-dp-default-pool-xxxxxxxx-yyyy Ready 5m v1.34.x gke-tare-dp-default-pool-xxxxxxxx-zzzz Ready 5m v1.34.x ``` ## Step 4: create an artifact registry repository ```shell AR_REGION=us-central1 AR_REPO=tare ``` Create the repository: ```shell gcloud artifacts repositories create "${AR_REPO}" \ --repository-format=docker \ --location="${AR_REGION}" \ --description="Agent Router data plane images" ``` The repository is addressable as `${AR_REGION}-docker.pkg.dev/${PROJECT_ID}/${AR_REPO}`. ## Step 5: sync Agent Router images to the registry ### Step 5.1: authenticate Docker to artifact registry ```shell gcloud auth configure-docker "${AR_REGION}-docker.pkg.dev" ``` ### Step 5.2: sync images Copy the container images from Tetrate's registry into the registry created above. The `tare` CLI authenticates to the source registry automatically. ```shell tare install /path/to/data-plane-credentials.json \ --image-sync "${AR_REGION}-docker.pkg.dev/${PROJECT_ID}/${AR_REPO}" \ --sync-only ``` The sync produces no progress output and takes several minutes. After it completes, verify the images: ```shell gcloud artifacts docker images list \ "${AR_REGION}-docker.pkg.dev/${PROJECT_ID}/${AR_REPO}" \ --include-tags --limit 20 ``` The output should list ten repositories under the `tare/` prefix, including `ai-gateway-controller`, `envoy-tars`, `gateway`, `liaison`, `ratelimit`, `redis`, `tare-doctor`, and `valet`. :::tip To preview which images would sync without pulling them, replace `--sync-only` with `--print-images`. ::: ## Step 6: install the Agent Router data plane Select the hostname used to expose the data plane externally and pass it as `--serve-url`. DNS for this hostname is configured in [Step 8](#step-8-wire-dns-and-register-the-url) after the gateway is up. ```shell tare install /path/to/data-plane-credentials.json \ --image-sync "${AR_REGION}-docker.pkg.dev/${PROJECT_ID}/${AR_REPO}" \ --serve-url https:// ``` `tare install` performs the following actions: - Creates the `tars-system` and `tars-dataplane` namespaces. - Installs the Agent Router data plane via Helm. :::note The `--serve-url` flag is currently required by the CLI (legacy behavior). The URL does not need to resolve at install time; `tare install` records it on the management plane. The URL can be changed later through **Dashboard → System → Settings → Data planes**. ::: :::note When the cluster and Artifact Registry reside in the same project, GKE nodes pull images using the node service account automatically; no image-pull secret is required. For cross-project or external-registry setups, see [Appendix A](#appendix-a-cross-project-and-external-registry-pull-access). ::: ## Step 7: install the gateway `tare gateway install` reads a JSON config and creates the global static IP, the Google-managed certificate (via Certificate Manager), the DNS authorization, and the Kubernetes Gateway and HTTPRoute that wire it all together. :::tip **Already have GCP infrastructure?** `tare gateway install` is opinionated. It assumes a public global L7 load balancer, a Google-managed certificate, and DNS records published by the operator. Two escape hatches when this does not fit the environment: - **Use `--ack-prereqs` instead of `--apply-prereqs`** when the static IP, certificate, DNS authorization, and certificate map have already been provisioned by an infrastructure-as-code tool. Reference the existing names in `gcp-gateway.json`; `tare` will use them without creating or modifying anything. - **Skip `tare gateway install` entirely** and author a `Gateway` and `HTTPRoute` against a preferred `GatewayClass` (for example, `gke-l7-rilb` for an internal load balancer, or cert-manager-managed certs through any `GatewayClass`). The data plane only requires an Ingress or Gateway that routes to the `egress` service in `tars-dataplane` on port `10080`. Everything else in this section is a convenience wrapper around the GCP-native flow. ::: ### Step 7.1: configure the gateway Create a file named `gcp-gateway.json` in the same directory as the credential file: ```json { "projectId": "", "serveDomain": "", "serveUrl": "", "customer": "", "environment": "production", "certificateMap": { "name": "tare-cert-map" }, "certificate": { "name": "tare-gateway-cert" }, "dnsAuthorization": { "name": "tare-dns-auth" }, "namespaces": { "gateway": "tars-gateway", "system": "tars-system", "dataplane": "tars-dataplane" }, "gcloud": { "skipExisting": true } } ``` Field reference: - `projectId`: the project hosting the gateway infrastructure. - `serveDomain` / `serveUrl`: the fully-qualified hostname clients will use. - `customer`: the customer identifier (visible in the credential or on the dashboard). - `environment`: a free-form label used as a Helm value. - `certificateMap`, `certificate`, `dnsAuthorization`: resource names. Prefix as needed. - `gcloud.skipExisting: true`: makes the command idempotent across retries. :::tip **Reserve a static IP up front** to allow DNS coordination to start before the gateway finishes provisioning: ```shell gcloud compute addresses create tare-gateway-ip --global --project "${PROJECT_ID}" gcloud compute addresses describe tare-gateway-ip --global --project "${PROJECT_ID}" \ --format='value(address)' ``` Then add `"gateway": { "staticIpName": "tare-gateway-ip" }` to `gcp-gateway.json`. Without a reserved IP, the gateway receives an ephemeral address that can change during maintenance. ::: ### Step 7.2: validate the config ```shell tare gateway config lint --config gcp-gateway.json ``` Errors block the install; warnings include remediation guidance and can be configured to fail CI runs. ### Step 7.3: preview the plan ```shell tare gateway install /path/to/data-plane-credentials.json \ --type gcp \ --config gcp-gateway.json \ --plan-only ``` The output is a deployment plan: resource diffs against the current GCP and Kubernetes state, with each value traced back to its source flag or config field. Review before applying. ### Step 7.4: apply ```shell tare gateway install /path/to/data-plane-credentials.json \ --type gcp \ --config gcp-gateway.json \ --apply-prereqs \ --wait ``` `--apply-prereqs` instructs `tare` to create the cloud resources directly. When these resources have already been provisioned by an infrastructure-as-code tool, use `--ack-prereqs` instead; `tare` will reference but not modify the assets. On success, the command prints the static IP and the CNAME record required for certificate validation. Record both. :::note Certificate activation takes approximately 5–20 minutes after DNS propagates (see [Step 8](#step-8-wire-dns-and-register-the-url)). Track activation with: ```shell gcloud certificate-manager certificates describe \ --format='yaml(managed.state,managed.domainStatus)' ``` ::: ## Step 8: wire DNS and register the URL After the gateway is up, two DNS records are required: an A record for traffic and a CNAME for certificate validation. The exact values are printed by the gateway-install output and can be fetched at any time. ### Step 8.1: retrieve the DNS values ```shell # Static IP for the A record gcloud compute addresses describe tare-gateway-ip --global --format='value(address)' ``` ```shell # DNS authorization values for the CNAME record gcloud certificate-manager dns-authorizations describe tare-dns-auth \ --project "${PROJECT_ID}" \ --format='value(dnsResourceRecord.name,dnsResourceRecord.type,dnsResourceRecord.data)' ``` ### Step 8.2: add the DNS records Add the following records in the DNS provider (Cloud DNS, Route 53, Cloudflare, registrar, and similar): ```none # Traffic . A TTL 300 # Certificate authorization _acme-challenge.. CNAME TTL 300 ``` Verify propagation: ```shell dig +short # Should return the static IP dig +short _acme-challenge. # Should return the Google-managed CNAME target ``` Once DNS resolves, the certificate's `managed.state` flips to `ACTIVE` and the gateway begins serving HTTPS. ### Step 8.3: register the URL on the management plane In the dashboard, navigate to **System → Settings → Profile → Proxy URL** and set it to `https://`. Changes propagate to the data plane within approximately 30 seconds. ## Step 9: verify provider routes Providers (OpenAI, Anthropic, and others) and their upstream API keys are configured during Agent Router onboarding, not during this install. The data plane retrieves that configuration automatically once it is connected to the management plane. Verify the data plane received the provider routes: ```shell kubectl get aigatewayroutes -A kubectl get aiservicebackends -A ``` Both should show `Accepted` resources within a minute of the data plane coming up. If the lists are empty, consult the Agent Router onboarding guide to confirm providers are configured. ## Step 10: verify the install Run `tare doctor`: ```shell tare doctor /path/to/data-plane-credentials.json --verbose ``` Pass criteria: all in-cluster checks report `Status: OK` (or `Healthy`) with `0 errors, 0 warnings`, and the final line confirms the health-report bundle was accepted (`Sending health report ... OK (bundle )`). Expected output (abridged): ```none CHECKS PERFORMED: - Namespace existence (system, dataplane) - CRD presence (Gateway API, AI Gateway, RouteDeployment) - Controller deployments ready (such as AI Gateway, controller components) - Proxy deployment ready - GatewayClass and Gateway accepted/programmed - EnvoyPatchPolicy acceptance (per instance) - EnvoyProxy acceptance (per instance) - Egress EnvoyProxy image uses envoy-tars - Identity Secret and ConfigMap present - AIServiceBackend acceptance - Envoy Gateway Backend acceptance - BackendTrafficPolicy acceptance - BackendSecurityPolicy acceptance - BackendTLSPolicy acceptance - ClientTrafficPolicy acceptance - HTTPRouteFilter presence/acceptance - ReferenceGrant presence - RouteDeployment status conditions - AIGatewayRoute acceptance/resolution - HTTPRoute parent acceptance/resolution - MCPRoute parent acceptance/resolution - Proxy admin and forward endpoints - Pod CrashLoopBackOff (excluding tars-config-monitor) Sending health report to https://api..tetrate.ai/v1/dataplane-status... OK (bundle ) ``` Then send a request with an invalid token to confirm auth is enforced: ```shell curl -sS -o /dev/null -w "HTTP %{http_code}\n" \ "https:///v1/chat/completions" \ -X POST -H "Content-Type: application/json" \ -H "Authorization: Bearer NotREAL" \ -d '{"model":"gpt-5-mini","messages":[{"role":"user","content":"hi"}]}' # Expected: HTTP 401 ``` If the response is `HTTP 200`, contact Tetrate Support; auth is enforced automatically on every install. ## Step 11: smoke tests In the router app (`https://router..tetrate.ai`), select **API Keys** in the sidebar and create a key. This is the key applications (and the tests below) use as `Authorization: Bearer ...`. Set the host and key once: ```shell export DP_HOST= export TARS_API_KEY="" ``` ### Chat Completions (OpenAI shape) ```shell curl -s "https://${DP_HOST}/v1/chat/completions" \ -X POST -H "Content-Type: application/json" \ -H "Authorization: Bearer ${TARS_API_KEY}" \ -d '{ "model": "gpt-5-mini", "messages": [{"role": "user", "content": "hello, what are you?"}] }' ``` ### Anthropic Messages (native shape) ```shell curl -s "https://${DP_HOST}/v1/messages" \ -X POST -H "Content-Type: application/json" \ -H "anthropic-version: 2023-06-01" \ -H "Authorization: Bearer ${TARS_API_KEY}" \ -d '{ "model": "claude-haiku-4-5", "max_tokens": 64, "messages": [{"role": "user", "content": "hello"}] }' ``` ### List available models ```shell curl -s "https://${DP_HOST}/v1/models" \ -H "Authorization: Bearer ${TARS_API_KEY}" | jq '.data[].id' | sort -u ``` ### Streaming ```shell curl -s "https://${DP_HOST}/v1/chat/completions" \ -X POST -H "Content-Type: application/json" \ -H "Authorization: Bearer ${TARS_API_KEY}" \ -d '{ "model": "gpt-5-mini", "stream": true, "messages": [{"role": "user", "content": "count from 1 to 5"}] }' ``` ### MCP :::note MPC is licensed separately from inference and guardrails. Contact your Tetrate sales contact for more information. ::: Create an MCP profile in the router app: **MCP Profiles** (sidebar) → **Create profile**. The profile ID is used in the URL below. ```shell curl -s "https://${DP_HOST}/mcp/" \ -X POST -H "Content-Type: application/json" \ -H "Authorization: Bearer ${TARS_API_KEY}" \ -d '{"jsonrpc":"2.0","method":"tools/list","id":1}' ``` To register the MCP profile with Claude Code: ```shell claude mcp add --transport http \ https://${DP_HOST}/mcp/ \ --header "Authorization: Bearer ${TARS_API_KEY}" ``` ## Upgrading To upgrade the data plane to a newer Agent Router release: 1. Install the new `tare` CLI version (re-run [Step 2](#step-2-install-the-tare-cli)). 1. Re-run [Step 5.2](#step-52-sync-images) and [Step 6](#step-6-install-the-agent-router-data-plane). 1. The gateway, certificate, and DNS authorizations persist; `tare gateway install` does not need to be re-run unless its config has changed. :::note Cluster state (namespaces, gateway, DNS authorization, certificate, and registered URL) persists across upgrades. ::: ## Cleanup To remove the deployment, delete resources in dependency order: ```shell # 1. Helm release helm uninstall tars -n tars-system 2>/dev/null || true ``` ```shell # 2. Gateway resources (cert, DNS auth, static IP, namespaces) tare gateway uninstall /path/to/data-plane-credentials.json \ --type gcp \ --config gcp-gateway.json \ --wait ``` ```shell # 3. Remove any GCP assets that survived the uninstall (idempotent) gcloud certificate-manager certificates delete tare-gateway-cert --quiet || true gcloud certificate-manager certificate-maps delete tare-cert-map --quiet || true gcloud certificate-manager dns-authorizations delete tare-dns-auth --quiet || true gcloud compute addresses delete tare-gateway-ip --global --quiet || true ``` ```shell # 4. Artifact Registry repository gcloud artifacts repositories delete "${AR_REPO}" --location="${AR_REGION}" --quiet ``` ```shell # 5. GKE cluster (only when not required for other workloads) gcloud container clusters delete "${CLUSTER_NAME}" --zone "${ZONE}" --quiet ``` ```shell # 6. Local kubeconfig kubectl config delete-context "gke_${PROJECT_ID}_${ZONE}_${CLUSTER_NAME}" || true ``` :::note DNS records (the A record and the `_acme-challenge` CNAME) live in the DNS provider and must be removed manually. ::: ## Troubleshooting Issues are grouped by the step where they are most likely to occur. ### Image synchronization issues | Symptom | Cause | Fix | | :--- | :--- | :--- | | `401 Unauthorized` on `HEAD https://registry.tetrate.ai/v2/...` | The credential is not authorized to pull from `registry.tetrate.ai`. | Regenerate the credential from **Dashboard → System → Settings → Data plane credentials** and retry. | | `unauthenticated` / `permission denied` from the destination registry | The `gcloud auth configure-docker` token is invalid for this registry. | Re-run `gcloud auth configure-docker ${AR_REGION}-docker.pkg.dev` and retry the sync. | ### Image pull from the cluster fails | Symptom | Cause | Fix | | :--- | :--- | :--- | | Pods in `ImagePullBackOff` with `permission denied` from Artifact Registry | The cluster and registry reside in different projects, or the node service account lacks `roles/artifactregistry.reader`. | Grant the node SA reader access on the cross-project repo, or use the image-pull-secret approach in [Appendix A](#appendix-a-cross-project-and-external-registry-pull-access). | ### Gateway installation issues | Symptom | Cause | Fix | | :--- | :--- | :--- | | Wait timeout while gateway provisions | Google Cloud Load Balancer provisioning is in flight; first-run provisioning takes 5–15 minutes. | Watch `kubectl get gateway -n tars-gateway -o wide` and look for an `ADDRESS`. If still empty after 20 minutes, inspect Gateway events for the underlying error. | | `Failed to load dynamic module: composer` in Agent Router egress proxy logs | The auth filter wiring did not program. On current builds this indicates the management plane is missing the proxy-settings config. | Contact support; this should not appear on a correctly configured tenant. | | Certificate stays `PROVISIONING` indefinitely | The DNS authorization CNAME does not resolve, or the A record points at the wrong IP. | Verify with `dig +short _acme-challenge.` and `dig +short `. Both must return the values shown by `tare gateway install`. | ### Failed tests | Symptom | Cause | Fix | | :--- | :--- | :--- | | HTTP 404 with body `No matching route found.` | The requested model is not configured for any provider, or no providers are configured. | Verify `kubectl get aigatewayroutes -A` shows `Accepted` rows. If empty, consult the Agent Router onboarding guide. | | HTTP 401 with a valid bearer | The API key was issued against a different management plane than this data plane is registered to. | Issue a new key from the router app for this tenant. | | HTTPS connect failure (`SSL_ERROR`, `unable to verify the first certificate`) | The certificate is not yet `ACTIVE`, or DNS has not propagated. | Check `gcloud certificate-manager certificates describe --format='value(managed.state)'` and wait until `ACTIVE`. | ## Appendix a: cross-project and external-registry pull access The main flow assumes the GKE cluster and the Artifact Registry reside in the **same** GCP project. In that setup, GKE uses the node service account automatically; no image-pull secret is required. For other arrangements: ### A.1: cross-project artifact registry (no pull secret) Grant the GKE cluster's node service account `roles/artifactregistry.reader` on the destination project or repository: ```shell NODE_SA=$(gcloud container clusters describe "${CLUSTER_NAME}" \ --zone "${ZONE}" --format='value(nodeConfig.serviceAccount)') gcloud projects add-iam-policy-binding "${PROJECT_ID}" \ --member="serviceAccount:${NODE_SA}" \ --role="roles/artifactregistry.reader" ``` After the binding, `tare install` works exactly as in the main flow; no additional flags are required. ### A.2: cross-project via impersonated service account (pull secret) When the node service account cannot be added to the destination project, pipe a short-lived OAuth token to `tare install`: ```shell echo "oauth2accesstoken:$(gcloud auth print-access-token --impersonate-service-account=)" | \ tare install /path/to/data-plane-credentials.json \ --image-registry "${AR_REGION}-docker.pkg.dev/${PROJECT_ID}/${AR_REPO}" \ --image-pull-secret-stdin \ --serve-url https:// \ --wait ``` :::note `--image-pull-secret-stdin` creates the `tars-image-pull-secret` Secret in both Agent Router namespaces. ::: ### A.3: external registry (not artifact registry) Pipe a registry username and password directly: ```shell echo ":" | \ tare install /path/to/data-plane-credentials.json \ --serve-url https:// \ --image-sync / \ --image-pull-secret-stdin \ --image-pull-secret-name tars-image-pull-secret \ --wait ``` ## Appendix b: forward observability data to an OpenTelemetry collector The data plane can stream envoy HTTP access logs and `router_*` application metrics to a customer-managed OpenTelemetry Collector, which can then forward to any backend (Google Cloud Monitoring via the `googlecloud` exporter, Datadog, Grafana Cloud, SigNoz, and others). The two streams are configured in separate fields on the `EnvoyProxy` resource: | Stream | EnvoyProxy field | Contents | | :--- | :--- | :--- | | Access logs (per-request HTTP metadata) | `accessLog.sinks[]` | Method, status, path, latency, MCP headers, downstream and upstream addresses | | Metrics (`router_*` and envoy native stats) | `metrics.sinks[]` | `router_requests_total`, `router_model_requests_total`, plus envoy cluster and listener counters | Either stream can be configured independently. ### Inline at install time (recommended) `tare install` accepts `--otel-collector-endpoint` and `--otel-exporter-auth-headers` so the data plane ships telemetry from the first request: ```shell tare install /path/to/data-plane-credentials.json \ --image-sync "${AR_REGION}-docker.pkg.dev/${PROJECT_ID}/${AR_REPO}" \ --serve-url https:// \ --otel-collector-endpoint 'https://' \ --otel-exporter-auth-headers 'Bearer ' ``` ### Alternative: deploy a collector in-cluster The collector ConfigMap and `EnvoyProxy` patch shapes are documented in [Appendix C of the Azure installation guide](/agent-router-enterprise/self-hosted-data-plane/data-plane-installation/installation-guide-az/#appendix-c-forward-observability-data-to-an-opentelemetry-collector). The data plane resources are Kubernetes-native, so the manifests apply identically on GCP. For GCP-specific backends: | Backend | Exporter | Reference | | :--- | :--- | :--- | | Google Cloud Monitoring (Stackdriver) | `googlecloud` | [GoogleCloudPlatform/opentelemetry-operations-collector](https://github.com/GoogleCloudPlatform/opentelemetry-operations-collector) | | Google Cloud Trace / Logging | `googlecloud` | Same as above | | Datadog | `datadog` | [docs.datadoghq.com/opentelemetry/otel_collector_datadog_exporter](https://docs.datadoghq.com/opentelemetry/otel_collector_datadog_exporter/) | | Grafana Cloud | `otlphttp` to a grafana.net endpoint | [grafana.com/docs/grafana-cloud/send-data/otlp](https://grafana.com/docs/grafana-cloud/send-data/otlp/) | :::note The EnvoyProxy patches reset on every `tare install` re-run. Re-apply after each reinstall, or use the inline `--otel-collector-endpoint` flags so the install owns the configuration. ::: Where to go next Gateway installation Install the data plane gateway components that manage inbound access and configure request routing. Self-Hosted Data Plane See the full install sequence: gateway, SSO, then developer handoff. --- ## Installation guide for Helm # Agent Router data plane installation using Helm This guide is for platform teams that own the Helm release lifecycle and already have a private container registry, such as Azure Container Registry, Harbor, JFrog Artifactory, Nexus, or Google Artifact Registry. In this guide: - `tare` syncs TARE release artifacts into the private registry. - `tare` applies CRDs outside Helm. - `tare` renders Helm values. - Helm installs or upgrades the data plane workload from the private registry. Plan for 30 to 60 minutes, plus the time to mirror artifacts into the private registry. :::info This guide covers data plane installation only. It does not create a public Gateway, ingress, DNS record, or TLS certificate. For gateway exposure, follow the published cloud install guide for the target platform, or use the organization's preferred ingress or Gateway stack. ::: ## Architecture overview Agent Router Enterprise uses a split-plane model. A **Management Plane** hosted by **Tetrate** holds configuration and serves the web UI. A **Data Plane** runs in customer-managed cluster infrastructure and handles all AI traffic. The two planes communicate over a single outbound HTTPS connection initiated by the data plane; no inbound connections from the internet reach the management plane. ```mermaid architecture-beta group customer(cloud)[Customer hosted] group tetrate(cloud)[Tetrate hosted] service mp(server)[Management Plane] in tetrate service dp(server)[Data Plane] in customer mp:R -- L:dp ``` ## Overview - **Prepare**: obtain the credential and set install variables - [Prerequisites](#prerequisites) - [Step 1 - Obtain the data plane credential](#step-1---obtain-the-data-plane-credential) - [Step 2 - Set variables](#step-2---set-variables) - **Mirror**: copy TARE artifacts to the private registry - [Step 3 - Sync release artifacts](#step-3---sync-release-artifacts) - **Install**: apply CRDs and deploy with Helm - [Step 4 - Apply CRDs](#step-4---apply-crds) - [Step 5 - Render Helm values](#step-5---render-helm-values) - [Step 6 - Create namespaces and pull secrets](#step-6---create-namespaces-and-pull-secrets) - [Step 7 - Install or upgrade with Helm](#step-7---install-or-upgrade-with-helm) - **Verify and expose**: confirm health, expose the service, and smoke test - [Step 8 - Verify](#step-8---verify) - [Step 9 - Expose the data plane](#step-9---expose-the-data-plane) - [Step 10 - Validation and smoke tests](#step-10---validation-and-smoke-tests) - **Operate** - [Upgrade and rollback](#upgrade-and-rollback) - [If artifacts are already mirrored](#if-artifacts-are-already-mirrored) - [Troubleshooting](#troubleshooting) --- ## Prerequisites - A TARE data plane credential from the dashboard, saved as `data-plane-credentials.json`. - `tare`, `helm`, and `kubectl` installed locally. - Access to the target Kubernetes cluster. - Push access to the private registry mirror. - Pull access from the target cluster to the same private registry. ## Conventions Later steps reuse the shell variables defined in [Step 2 - Set variables](#step-2---set-variables), such as `DP_CREDENTIAL` and `PRIVATE_IMAGE_REGISTRY`. The steps assume a single shell session. If the session changes, re-export the variables before continuing. ## Step 1 - Obtain the data plane credential In the dashboard, go to **Settings -> Data planes**. If this data plane is not registered yet: 1. Click **Register Data plane**. 1. Set **Data plane ID** to a stable ID such as `acme-prod`. 1. Optionally set **Name**, **Description**, and **Labels**. 1. Leave **URL** blank for now unless the final gateway hostname is already known. 1. Click **Register Data plane**. Then generate the credential from the data plane row: 1. Click the row's edit action to open the **Data plane** drawer. 1. In the **Credential** section, click **Generate credential**. 1. Save the downloaded file as `data-plane-credentials.json` on the machine the install runs from. Each data plane uses its own credential. A credential can be rotated or revoked from the same **Credential** section in the data plane edit drawer. ## Step 2 - Set variables For Azure Container Registry: ```bash DP_CREDENTIAL=./data-plane-credentials.json PRIVATE_REGISTRY_HOST=${ACR_NAME}.azurecr.io PRIVATE_IMAGE_REGISTRY=${ACR_NAME}.azurecr.io/tare PULL_SECRET=tare-registry-pull ``` For another private registry: ```bash DP_CREDENTIAL=./data-plane-credentials.json PRIVATE_REGISTRY_HOST=registry.acme.example.com PRIVATE_IMAGE_REGISTRY=registry.acme.example.com/tare PULL_SECRET=tare-registry-pull REGISTRY_USERNAME= REGISTRY_PASSWORD= ``` ## Step 3 - Sync release artifacts Log in locally with credentials that can push to the destination registry. For ACR: ```bash az acr login --name "${ACR_NAME}" ``` For a generic private registry: ```bash printf '%s' "${REGISTRY_PASSWORD}" | \ docker login "${PRIVATE_REGISTRY_HOST}" \ --username "${REGISTRY_USERNAME}" \ --password-stdin ``` Copy the pinned TARE images and the serve-helm OCI chart: ```bash tare install "${DP_CREDENTIAL}" \ --image-sync "${PRIVATE_IMAGE_REGISTRY}" \ --sync-only \ --parallel 2 \ --stall-threshold 5m ``` `tare` uses the data plane credential to pull from Tetrate's source registry. The local registry login is used only to push to the private registry. ## Step 4 - Apply CRDs Apply CRDs outside Helm: ```bash tare install "${DP_CREDENTIAL}" \ --crds-only \ --image-registry "${PRIVATE_IMAGE_REGISTRY}" ``` CRDs are intentionally managed outside the Helm release so `helm uninstall` does not delete cluster-scoped APIs and cascade-delete custom resources. ## Step 5 - Render Helm values Render values for the Helm release: ```bash tare install "${DP_CREDENTIAL}" \ --image-registry "${PRIVATE_IMAGE_REGISTRY}" \ --print-helm-values > values.yaml ``` Treat `values.yaml` as sensitive because it contains the data plane identity secret. Do not commit it as plaintext. Use SOPS, Sealed Secrets, External Secrets, or the organization's approved secret workflow. If the public gateway URL is not known yet, make sure the rendered values do not set it to the management-plane API URL. Keep these fields empty: ```yaml global: serveUrl: "" controller: config: proxyUrl: "" ``` If needed, override them during Helm install: ```bash --set-string global.serveUrl= \ --set-string controller.config.proxyUrl= ``` ## Step 6 - Create namespaces and pull secrets Create the data plane namespaces: ```bash kubectl create namespace tars-system --dry-run=client -o yaml | kubectl apply -f - kubectl create namespace tars-dataplane --dry-run=client -o yaml | kubectl apply -f - ``` For AKS attached to ACR, no image pull secret is usually needed after: ```bash az aks update \ --resource-group "${RESOURCE_GROUP}" \ --name "${AKS_CLUSTER_NAME}" \ --attach-acr "${ACR_NAME}" ``` For a private registry that requires Kubernetes pull credentials, create the same secret in both namespaces: ```bash kubectl create secret docker-registry "${PULL_SECRET}" \ --docker-server="${PRIVATE_REGISTRY_HOST}" \ --docker-username="${REGISTRY_USERNAME}" \ --docker-password="${REGISTRY_PASSWORD}" \ --namespace tars-system \ --dry-run=client -o yaml | kubectl apply -f - kubectl create secret docker-registry "${PULL_SECRET}" \ --docker-server="${PRIVATE_REGISTRY_HOST}" \ --docker-username="${REGISTRY_USERNAME}" \ --docker-password="${REGISTRY_PASSWORD}" \ --namespace tars-dataplane \ --dry-run=client -o yaml | kubectl apply -f - ``` When using a pull secret, render values with: ```bash tare install "${DP_CREDENTIAL}" \ --image-registry "${PRIVATE_IMAGE_REGISTRY}" \ --image-pull-secret-name "${PULL_SECRET}" \ --print-helm-values > values.yaml ``` ## Step 7 - Install or upgrade with Helm Use the chart version for the TARE release: ```bash CHART_VERSION="0.1.0-alpha.1+17f076b" ``` Install or upgrade the release: ```bash helm upgrade --install tars "oci://${PRIVATE_IMAGE_REGISTRY}/serve-helm" \ --version "${CHART_VERSION}" \ -f values.yaml \ -n tars-system \ --skip-crds \ --set ai-gateway-crds.enabled=false \ --set-string global.serveUrl= \ --set-string controller.config.proxyUrl= ``` Use `--skip-crds` and `--set ai-gateway-crds.enabled=false` together. The first skips chart `crds/` entries; the second disables AI Gateway CRDs rendered as normal Helm templates. Helm expects the SemVer build-metadata form with `+`, such as `0.1.0-alpha.1+17f076b`. OCI stores the tag internally with `_`, but `helm --version` should still use `+`. ## Step 8 - Verify Check the Helm release: ```bash helm list -n tars-system helm status tars -n tars-system ``` Wait for workloads: ```bash kubectl rollout status deployment/controller -n tars-system --timeout=300s kubectl rollout status deployment/egress -n tars-dataplane --timeout=300s ``` Confirm images come from the private registry: ```bash kubectl get deploy egress -n tars-dataplane \ -o jsonpath='{range .spec.template.spec.containers[*]}{.name}={.image}{"\n"}{end}' ``` Run doctor: ```bash tare doctor "${DP_CREDENTIAL}" --verbose ``` A fresh data plane with no routes can report warnings for missing RouteDeployments, AIGatewayRoutes, HTTPRoutes, or EnvoyPatchPolicies. The core install is healthy when pods roll out and images pull from the private registry. ## Step 9 - Expose the data plane This Helm workflow installs the data plane but does not create a public Gateway, ingress, DNS record, or TLS certificate. For gateway exposure, follow the published cloud install guide for the target platform, or use the organization's preferred ingress or Gateway stack. The data plane serves traffic through the in-cluster egress service: - Service: egress - Namespace: tars-dataplane - Port: 10080 - Paths: `/v1/*`, `/mcp/*`, and `/.well-known/*` After the gateway and DNS are ready, set the gateway URL in the dashboard: **Settings -> Data planes -> edit row -> URL**. ## Step 10 - Validation and smoke tests Validate the Helm install first: ```bash helm status tars -n tars-system kubectl rollout status deployment/controller -n tars-system --timeout=300s kubectl rollout status deployment/egress -n tars-dataplane --timeout=300s kubectl get deploy egress -n tars-dataplane \ -o jsonpath='{range .spec.template.spec.containers[*]}{.name}={.image}{"\n"}{end}' tare doctor "${DP_CREDENTIAL}" --verbose ``` Then run traffic smoke tests after onboarding has configured provider keys and routes. This Helm workflow does not create a public Gateway or DNS record. Until the gateway is exposed, use a local port-forward to the in-cluster egress service: ```bash kubectl port-forward -n tars-dataplane svc/egress 18080:10080 ``` In another terminal, set the local endpoint and an API key from the router app: ```bash export DP_SCHEME=http export DP_HOST=127.0.0.1:18080 export TARS_API_KEY= ``` At minimum, verify model listing with and without authentication: ```bash curl -s -o /tmp/tare-models.json -w "%{http_code}\n" \ "${DP_SCHEME}://${DP_HOST}/v1/models" \ -H "Authorization: Bearer ${TARS_API_KEY}" curl -s -o /tmp/tare-models-no-auth.json -w "%{http_code}\n" \ "${DP_SCHEME}://${DP_HOST}/v1/models" ``` Expected result: the authenticated request returns 200; the unauthenticated request returns 401. Also smoke test every API shape enabled during onboarding, such as `/v1/chat/completions`, `/v1/responses`, `/v1/messages`, `/v1/embeddings`, `/v1/images/generations`, and MCP profile paths. ## Upgrade and rollback Use the same split workflow for upgrades: update CRDs first, then upgrade the Helm workload release. If the new TARE release includes CRD changes, re-run CRD apply: ```bash tare install "${DP_CREDENTIAL}" \ --crds-only \ --image-registry "${PRIVATE_IMAGE_REGISTRY}" ``` Render values for the new release: ```bash tare install "${DP_CREDENTIAL}" \ --image-registry "${PRIVATE_IMAGE_REGISTRY}" \ --print-helm-values > values.yaml ``` If a pull secret is used, include it when rendering values: ```bash tare install "${DP_CREDENTIAL}" \ --image-registry "${PRIVATE_IMAGE_REGISTRY}" \ --image-pull-secret-name "${PULL_SECRET}" \ --print-helm-values > values.yaml ``` Upgrade the workload release: ```bash CHART_VERSION="" helm upgrade tars "oci://${PRIVATE_IMAGE_REGISTRY}/serve-helm" \ --version "${CHART_VERSION}" \ -f values.yaml \ -n tars-system \ --skip-crds \ --set ai-gateway-crds.enabled=false \ --set-string global.serveUrl= \ --set-string controller.config.proxyUrl= ``` Rollback uses Helm's normal rollback flow: ```bash helm rollback tars -n tars-system ``` CRDs are not affected by rollback because they are not part of the Helm release manifest. Existing custom resources remain in the cluster. ## If artifacts are already mirrored If Agent Router team has already mirrored the TARE images and chart, skip the `--image-sync` step and start at CRD apply: ```bash tare install "${DP_CREDENTIAL}" \ --crds-only \ --image-registry "${PRIVATE_IMAGE_REGISTRY}" ``` Then render values and install with Helm as above. ## Troubleshooting ### Destination registry reports repository not found Create the destination repository before syncing. For example, with Google Artifact Registry: ```bash gcloud artifacts repositories create tare \ --repository-format docker \ --location ``` ACR and many enterprise registries create repositories on first push, but some registries require pre-creation. ### Helm reports invalid ownership metadata This usually means Helm is trying to adopt CRDs already applied by `tare --crds-only`. Example: ```text CustomResourceDefinition "aigatewayroutes.aigateway.envoyproxy.io" exists and cannot be imported into the current release: invalid ownership metadata ``` Install or upgrade with both: ```bash --skip-crds \ --set ai-gateway-crds.enabled=false ``` ### Helm reports improper constraint `helm --version` expects a SemVer constraint, not a bare commit SHA. Use the chart's full SemVer, such as `0.1.0-alpha.1+17f076b`, or a release-build TARE version such as `v0.1.0-beta.3`. Bare commit SHAs are rejected before Helm contacts the registry. The registry stores SemVer build metadata with `_`, but `helm --version` should use the SemVer form with `+`. ### Pods are stuck in ImagePullBackOff Check that: - the private registry contains every image tag for the TARE release - the cluster identity can pull from the private registry - pull secrets exist in both tars-system and tars-dataplane when required - the pull secret is referenced from `values.yaml`; if not, re-render values with `--image-pull-secret-name` - `values.yaml` points `global.imageRegistry` at the intended registry ### Helm cannot pull the chart from the mirror If `helm install` or `helm upgrade` fails locally with pull access denied, log in to the OCI registry from the operator machine: ```bash helm registry login "${PRIVATE_REGISTRY_HOST}" ``` This is separate from Kubernetes image pull access. Helm pulls the chart artifact from the local machine; the cluster pulls workload images using node identity or image pull secrets. Where to go next Gateway installation Install the data plane gateway components that manage inbound access and configure request routing. Self-Hosted Data Plane See the full install sequence: gateway, SSO, then developer handoff. --- ## Retrieve data plane resources # Retrieve Agent Router data plane Kubernetes resources for review or GitOps This guide renders the Kubernetes resources that tare install would deploy, without applying them to a cluster. Use this workflow to review the manifest, run a server-side dry run, or commit the rendered YAML to a GitOps repository before deployment. ## Prerequisites - A data plane credential file from the dashboard, saved locally as `data-plane-credentials.json`. - The `tare` CLI installed and available on `PATH`. - `helm` and `kubectl` available locally. Recent `tare` releases can download their own tool bundle, but most enterprise workstations manage these tools directly. Install the CLI if needed: ```shell curl -sSL https://tare.tetrate.ai/tools/install.sh | bash export PATH="$PATH:$HOME/.tare/bin" tare --version ``` The sections below describe two different options for retrieving the resource definitions: * Option 1 - Rendering the manifest as a **single file** (YAML format) * Option 2 - Rendering the manifest using an **Artifact Registry** ## Option 1: render the manifest Run `tare install` with `--print-resources`: ```shell tare install ./data-plane-credentials.json --print-resources > tare-data-plane.yaml ``` The command writes the rendered Kubernetes YAML to stdout. Progress and preflight messages go to stderr, so redirecting stdout produces a clean manifest file. The rendered file contains the same Kubernetes resources `tare install` would deploy through Helm: namespaces, CRDs, service accounts, RBAC, secrets, config maps, services, deployments, autoscaling resources, gateway resources, and install jobs. :::tip The gateway URL can be set later from the dashboard after the gateway and DNS are ready: * Log in the the Admin Dashboard. * Navigate to **System** → **Settings** * Select tab **Data planes**. * Click the **Edit** action icon. * Locate item **URL**. * Update the URL. * Click button **Save changes**. ::: ## Option 2: render with an artifact registry When the cluster pulls from a mirrored registry, set the image registry in the rendered Helm values: ```shell tare install ./data-plane-credentials.json \ --image-registry us-central1-docker.pkg.dev/acme-prod/tare \ --print-resources > tare-data-plane.yaml ``` When the registry requires an image pull secret that already exists in the target namespaces, reference it: ```shell tare install ./data-plane-credentials.json \ --image-registry us-central1-docker.pkg.dev/acme-prod/tare \ --image-pull-secret-name tare-registry-pull \ --print-resources > tare-data-plane.yaml ``` To render the pull-secret object from credentials on stdin: ```shell printf '%s\n' 'user:password' | tare install ./data-plane-credentials.json \ --image-registry us-central1-docker.pkg.dev/acme-prod/tare \ --image-pull-secret-stdin \ --print-resources > tare-data-plane.yaml ``` ## Validate before applying Run a server-side dry run against the target cluster: ```shell kubectl apply --dry-run=server -f tare-data-plane.yaml ``` Apply once the manifest has been reviewed: ```shell kubectl apply -f tare-data-plane.yaml ``` For GitOps, commit `tare-data-plane.yaml` to the repository managed by Argo CD, Flux, or the internal deployment system instead of applying it directly. ## Common issues ### Error message: `tare: command not found` The installer places the binary in `~/.tare/bin` by default. Add it to `PATH`: ```shell export PATH="$PATH:$HOME/.tare/bin" ``` ### Error message: `unknown command "cli" for "tare"` Use `tare install`, not `tare cli`: ```shell tare install ./data-plane-credentials.json --print-resources ``` If an older CLI requires a URL, upgrade `tare` first. When an immediate upgrade is not possible, include `--serve-url` as a compatibility workaround: ```shell tare install ./data-plane-credentials.json \ --serve-url https://proxy.acme.example.com \ --print-resources ``` ### Error message: `Missing serve URL` Upgrade `tare` and retry. Current releases allow `--print-resources` without a data plane Proxy URL: ```shell curl -sSL https://tare.tetrate.ai/tools/install.sh | bash tare install ./data-plane-credentials.json --print-resources ``` When an immediate upgrade is not possible, pass a valid planned public hostname: ```shell tare install ./data-plane-credentials.json \ --serve-url https://proxy.acme.example.com \ --print-resources ``` The hostname does not need to resolve while rendering. After the gateway and DNS are live, set or update the gateway URL in the dashboard. Where to go next Data plane installation Apply the rendered resources by running a full data plane installation on AWS, Azure, or GCP. Gateway installation Install the gateway components that manage inbound access to the data plane. --- ## Gateway installation guide for EKS This guide exposes the Agent Router data plane in an Amazon Elastic Kubernetes Service (EKS) cluster using the tare command-line utility. The install drives a Network Load Balancer through the egress EnvoyProxy, so the endpoint answers on a customer-facing hostname without an AWS Load Balancer Controller and without the aws CLI. By the end of this guide the cluster has: - The egress EnvoyProxy (`tars-egress-proxy` in `tars-system`) patched to expose its service as a LoadBalancer, annotated for a Network Load Balancer. - The `egress` Service in `tars-dataplane` reporting an NLB hostname. - A customer-facing DNS `CNAME` record pointing at that NLB hostname. Plan for **10 to 20 minutes** end-to-end. The NLB itself usually publishes a hostname within a minute of the patch. ## Architecture ```text ┌───────────────────────────────────────────┐ │ proxy. ← customer DNS │ │ │ (CNAME) │ │ ▼ │ │ ┌────────────────┐ │ │ │ Network Load │ ← provisioned by the │ │ │ Balancer │ in-tree EKS cloud │ │ └────────┬───────┘ provider │ │ │ │ │ ┌────────▼───────┐ │ │ │ egress Service │ ← reconciled by │ │ │ (tars-dataplane)│ Envoy Gateway from │ │ └────────┬───────┘ the EnvoyProxy │ │ │ │ │ ┌────────▼───────┐ │ │ │ Agent Router │ │ │ │ Dataplane │ ← installed earlier │ │ └────────────────┘ via tare CLI │ └───────────────────────────────────────────┘ ``` This is a layer 4 path. The `tare` utility patches one resource: the egress EnvoyProxy. It sets `spec.provider.kubernetes.envoyService.type` to `LoadBalancer` and adds `service.beta.kubernetes.io/aws-load-balancer-type: nlb`. Envoy Gateway then reconciles the `egress` Service, and the in-tree EKS cloud provider provisions the NLB from the annotation. The operator provides one customer-facing DNS record at the top. Driving the load balancer through the EnvoyProxy is what makes the exposure durable. The `egress` Service is owned by Envoy Gateway, so a direct Service patch reverts to `ClusterIP` on the next reconcile; the EnvoyProxy setting is re-asserted across pod restarts, Envoy Gateway restarts, and `tare upgrade`. :::info Choose an exposure path: 1. **NLB via `tare` (this guide's default).** Use when an EKS cluster already has the data plane installed, a layer 4 endpoint is enough, and TLS terminates upstream of the load balancer or at the client. Smoke-tests use plain HTTP on port `10080`. This path is **not** sufficient for membership-probe DNS failover, which requires HTTPS on port 443. 1. **NLB with ACM TLS on the load balancer.** Keep the NLB shape and terminate TLS with an ACM certificate via the AWS Load Balancer Controller — see [Optional: terminate TLS on the NLB](#optional-terminate-tls-on-the-nlb). Use this when you need HTTPS at the edge on layer 4, including any project fronted by health-checked DNS failover ([Gateway sets and DNS-level failover](/agent-router-enterprise/guides/operate-and-govern/project-and-data-plane-management/gateway-sets-and-dns-failover/)). 1. **ALB with ACM TLS.** Use the [AWS data plane installation guide](/agent-router-enterprise/self-hosted-data-plane/data-plane-installation/installation-guide-aws/) when layer 7 ingress is required, or when you prefer an ALB Ingress with ACM (steps 10 and 11 install the AWS Load Balancer Controller). Also use that guide when the data plane is not yet installed — gateway install needs the egress EnvoyProxy to exist before it can patch it. These are alternative exposure shapes, not a required sequence. For high-availability gateway sets, prefer option 2 (or an equivalent HTTPS-on-443 front for each member). ::: ## Prerequisites - `tare` CLI `v0.1.0-beta.2` or later. Install or upgrade with `curl -fsSL https://tare.tetrate.ai/tools/install.sh | bash`. The binary lands at `~/.tare/bin/tare`; add it to `PATH`. - `kubectl` pointed at the EKS cluster (`aws eks update-kubeconfig --region $REGION --name $CLUSTER`); confirm with `kubectl config current-context`. - Data plane already installed, so the egress EnvoyProxy exists: `kubectl get envoyproxy tars-egress-proxy -n tars-system` returns a resource. - `identity.json` (the data plane credential and service-account file). The same file used for `tare install`. - Kubernetes RBAC allowing `patch` on `envoyproxies.gateway.envoyproxy.io` in `tars-system`. The pre-flight runs `kubectl auth can-i` and fails with a pointer to the required permission when it is denied. No `aws` CLI, no AWS Load Balancer Controller, and no IAM policy work are required for this path. The `--apply-prereqs` and `--ack-prereqs` flags belong to the GCP and Azure paths and do not apply here. :::note Credential-less upstream authentication to Bedrock is a separate concern from ingress. See [Bedrock workload identity](/agent-router-enterprise/guides/operate-and-govern/provision-models-and-providers/bedrock-workload-identity/) for IRSA and EKS Pod Identity. ::: ## Conventions The commands below assume the following environment variables are exported in the current shell: ```shell export SERVE_DOMAIN="proxy.acme.example.com" export REGION="" export CLUSTER="" ``` Each step assumes these variables remain set; export them again after opening a new shell. ## Step 1: preview the patch ```shell tare gateway install identity.json --type aws --plan-only ``` The Plan view lists the EnvoyProxy that will be patched, the annotations that will be applied, and the Service that Envoy Gateway will reconcile into an NLB. Neither `--plan-only` nor `--print-resources` touches the cluster, so both are safe to run first. To see the exact merge patch instead of the summary: ```shell tare gateway install identity.json --type aws --print-resources ``` Unlike the GCP and Azure paths, `--type aws` takes no config file. It is configured entirely by flags. ## Step 2: expose the egress as an NLB ```shell tare gateway install identity.json --type aws --wait ``` `--wait` polls the `egress` Service until the cloud provider publishes a hostname; NLBs surface a hostname rather than an IP address. Add `--yes` to skip the `Proceed?` prompt, which is required in non-TTY contexts such as CI. The wait timeout defaults to `10m` and is set with `--timeout`. For an internal NLB that is reachable only from inside the VPC: ```shell tare gateway install identity.json --type aws --aws-internal --wait ``` This adds `service.beta.kubernetes.io/aws-load-balancer-internal: true` to the same patch. ### Install sequence The install proceeds through the following stages: 1. **Plan and confirm** - the plan prints before any cluster access, then the CLI prompts unless `--yes` is passed. 1. **Pre-flight `kubectl` and cluster** - verifies `kubectl` is available and the cluster is reachable. 1. **Pre-flight EnvoyProxy** - confirms `tars-egress-proxy` exists in `tars-system` and that the current context can patch it. A missing resource fails with an "is the data plane installed?" hint. 1. **Patch the EnvoyProxy** - a JSON merge patch sets `envoyService.type=LoadBalancer` and the NLB annotations, preserving the rest of `provider.kubernetes`. 1. **`--wait`** - polls `status.loadBalancer.ingress[0].hostname` on the `egress` Service until the NLB hostname appears. 1. **Completion output** - prints the DNS CNAME action and a smoke-test command. ### Flags for the AWS path | Flag | Default | Description | | :---- | :---- | :---- | | `--aws-internal` | `false` | Provision an internal NLB (adds the `aws-load-balancer-internal` annotation) | | `--aws-service` | `egress` | Egress Service that Envoy Gateway exposes as an NLB | | `--aws-envoyproxy` | `tars-egress-proxy` | Egress EnvoyProxy resource to drive the NLB through | ## Step 3: wire DNS and verify Read the NLB hostname from the Service: ```shell kubectl get service egress -n tars-dataplane \ -o jsonpath='{.status.loadBalancer.ingress[0].hostname}' # k8s-tarsdata-egress-xxxxxxxxxx-yyyyyyyyy.elb.us-east-1.amazonaws.com ``` Point the serve hostname at that value with a `CNAME` record at your DNS provider: ```text proxy.acme.example.com CNAME k8s-tarsdata-egress-xxxxxxxxxx-yyyyyyyyy.elb.us-east-1.amazonaws.com ``` Smoke-test the endpoint on the egress port: ```shell curl -sk -o /dev/null -w 'HTTP %{http_code}\n' \ http://:10080/v1/chat/completions # expected: HTTP 401 ``` `401` is the success signal: the request reached the `tars-proxy` filter, which is asking for an API key. `200` means the filter is not loaded, and `502` means the load balancer health check is failing. ## Optional: terminate TLS on the NLB The layer 4 path above leaves TLS to whatever sits in front of the load balancer, and the [ALB path](/agent-router-enterprise/self-hosted-data-plane/data-plane-installation/installation-guide-aws/#appendix-a-tls-via-acm--https-listener) terminates it at a layer 7 ingress. There is a third option: keep the NLB and have **it** terminate TLS with an ACM certificate. This suits deployments that want a layer 4 endpoint but still need HTTPS at the edge — including any project fronted by health-checked DNS failover, since the membership probe those health checks target is an HTTPS-on-443 contract. It requires the **AWS Load Balancer Controller**, which reads these annotations; the in-tree provider the `tare` path uses does not claim Services annotated `external`. Set them on the egress `EnvoyProxy` in Helm values, never on the Service directly — Envoy Gateway owns the Service and reverts direct edits: ```yaml envoyProxy: service: type: LoadBalancer annotations: service.beta.kubernetes.io/aws-load-balancer-type: external service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing service.beta.kubernetes.io/aws-load-balancer-ssl-cert: service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "" service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp ``` The `annotations` map is passed through verbatim, so any annotation the controller supports can be set the same way. :::warning `ssl-ports` must name a port the Service exposes Envoy Gateway derives the Service's ports from the **Gateway's listeners**, and the egress Gateway ships exactly one — HTTP on `10080`. So out of the box the Service exposes only `10080`, and `ssl-ports: "443"` matches nothing: the load balancer comes up with no TLS listener at all. Check what the Service actually exposes before setting the annotation: ```shell kubectl get service egress -n tars-dataplane \ -o jsonpath='{range .spec.ports[*]}{.name} {.port}->{.targetPort}{"\n"}{end}' ``` Publishing on 443 — which the membership probe requires, since its `Host` match is portless — therefore needs the Gateway to expose a 443 listener so the Service carries that port. That is a change to the egress Gateway's listener set, and its shape depends on your release: confirm it with your Tetrate field engineer rather than deriving it. ::: Certificates are regional, and a project served by **several** gateways needs each member's certificate to cover both the member's own hostname and the shared client-facing name. Plan that before the second gateway exists — see [Gateway sets and DNS-level failover](/agent-router-enterprise/guides/operate-and-govern/project-and-data-plane-management/gateway-sets-and-dns-failover/). ## Troubleshooting ### `EnvoyProxy "tars-egress-proxy" not found` The data plane is not installed in the current context, or the EnvoyProxy is named differently. Run `tare install` first, or pass the actual name: ```shell tare gateway install identity.json --type aws --aws-envoyproxy --wait ``` ### Egress service stays `ClusterIP` after patching Confirm the EnvoyProxy carries the setting: ```shell kubectl get envoyproxy tars-egress-proxy -n tars-system \ -o jsonpath='{.spec.provider.kubernetes.envoyService.type}' # expected: LoadBalancer ``` When the EnvoyProxy reads `LoadBalancer` but the Service does not, Envoy Gateway has not reconciled it. Check the `envoy-gateway` controller logs. ### NLB hostname never appears Inspect the Service events for provisioning errors: ```shell kubectl describe service egress -n tars-dataplane ``` The in-tree cloud provider discovers subnets by tag. Missing or incorrect `kubernetes.io/role/elb` subnet tags are the usual cause. ### Kubernetes RBAC denies the patch The pre-flight reports that the current context cannot patch `envoyproxies.gateway.envoyproxy.io`. Grant a role with `patch` on that resource in `tars-system` and rerun. ## Notes - The exposure is durable. Because Envoy Gateway owns the `egress` Service through the EnvoyProxy, the NLB is re-asserted across pod restarts, Envoy Gateway restarts, and `tare upgrade`. - The install is idempotent. Re-running applies the same merge patch, so it is safe to repeat after changing `--aws-internal`. - This path handles ingress only. Upstream provider credentials, including credential-less Bedrock access, are configured separately. Where to go next Bedrock workload identity Give the data plane credential-less access to Bedrock with IRSA or EKS Pod Identity. Console quickstart Issue an API key and confirm a first routed request through the new endpoint. --- ## Gateway installation guide for AKS This guide installs the Agent Router gateway in an Azure Kubernetes Service (AKS) environment using the tare command-line utility. The resulting ingress terminates TLS for the data plane and routes traffic to it from a customer-facing hostname. By the end of this guide the cluster has: - AGIC (Application Gateway Ingress Controller) enabled on the AKS cluster. - A new Application Gateway (Standard_v2) provisioned in the `MC_` resource group. - An Ingress resource in the `tars-dataplane` namespace, annotated for AGIC health probes. - A customer-facing DNS `A` record pointing at the App Gateway frontend. Plan for **45 to 90 minutes** end-to-end: - 10 to 15 minutes of CLI work. - 15 to 60 minutes of background certificate provisioning. ## Architecture ```text ┌───────────────────────────────────────────┐ │ proxy. ← customer DNS │ │ │ │ │ ▼ │ │ ┌────────────────┐ │ │ │ Static IP │ ← Global address │ │ └────────┬───────┘ │ │ │ │ │ ┌────────▼───────┐ │ │ │ Cert Map + │ ← TLS termination │ │ │ Managed Cert │ │ │ └────────┬───────┘ │ │ │ │ │ ┌────────▼───────┐ │ │ │ Agent Router | ← installed in this │ │ │ Gateway(k8s) │ guide │ │ └────────┬───────┘ │ │ │ │ │ ┌────────▼───────┐ │ │ │ Agent Router | | | | Dataplane │ ← installed earlier │ │ └────────────────┘ via tare CLI │ └───────────────────────────────────────────┘ ``` The `tare` utility provisions everything from the static IP down to the gateway routes. The operator provides one customer-facing DNS record at the top. :::info This guide replaces the manual AGIC enable plus Ingress YAML in the main Azure runbook (Step 9a) with a single command. The command runs the same pre-flights as `tare doctor`, enables the AGIC addon, applies the dataplane Ingress with the correct health-probe annotations, and waits for the App Gateway to come up. Use this guide when: - An AKS cluster already exists and the dataplane is installed (`tare install ...` has already run). - The cluster is AGIC-compatible: `networkProfile.networkPlugin=azure`, `networkPluginMode=null`, `networkDataplane=azure`. Cilium and Overlay clusters must use AGC (a separate path; support is planned). - A reproducible, scripted install is preferred over running `az aks enable-addons` plus `kubectl apply -f ingress.yaml` by hand. Do not use this guide when: - The cluster runs Cilium or Overlay. The pre-flight refuses; use the AGC path in the main runbook. - The dataplane is not yet installed. Gateway install needs the `tars-dataplane` Service to point its Ingress at. - Only an inspection of intended changes is required. Use `--plan-only` or `--dry-run-prereqs` first. ::: ## Prerequisites - `tare` CLI `v0.1.0-beta.2` or later. Install or upgrade with `curl -fsSL https://tare.tetrate.ai/tools/install.sh | bash`. The binary lands at `~/.tare/bin/tare`; add it to `PATH`. - `az` CLI signed in to the subscription that owns the AKS cluster. - `kubectl` pointed at the AKS cluster (`az aks get-credentials -g $RG -n $CLUSTER`); confirm with `kubectl config current-context`. - Dataplane already installed; `kubectl get deploy -n tars-dataplane tars-dataplane` returns a deployment. - `identity.json` (the dataplane credential and service-account file). The same file used for `tare install`. - Azure RBAC on the signed-in identity: - `Azure Kubernetes Service Contributor Role` on the AKS resource group (for `az aks enable-addons`). - `Network Contributor` on the `MC_` resource group (`MC___`). AGIC requires this to manage the App Gateway lifecycle. The grant must be scoped to `MC_RG`; the parent-RG scope does not propagate. - The pre-flight prints the exact `az role assignment create` commands if either role is missing. It degrades gracefully when role-list read is denied, printing a warning rather than failing fast. ## Conventions The commands below assume the following environment variables are exported in the current shell: ```shell export CUSTOMER="acme" export SERVE_DOMAIN="proxy.acme.example.com" export SUBSCRIPTION_ID="" export AKS_RG="" export AKS_CLUSTER_NAME="" export REGION="" ``` Each step assumes these variables remain set; export them again after opening a new shell. ## Step 1: generate the config ```shell tare gateway config init --type azure \ --customer "${CUSTOMER}" \ --serve-domain "${SERVE_DOMAIN}" \ --azure-subscription-id "${SUBSCRIPTION_ID}" \ --azure-resource-group "${AKS_RG}" \ --aks-cluster-name "${AKS_CLUSTER_NAME}" ``` The command writes `azure-gateway.json` to the current directory. The App Gateway name (`-appgw`) and subnet CIDR (`10.225.0.0/24`) are filled in by the resolver at install time using documented defaults. Edit `azure-gateway.json` to override either. The interactive wizard runs when the flags are omitted; it prompts in order for customer ID, serve domain, subscription, resource group, and cluster name. ## Step 2: lint the config ```shell tare gateway config lint --config azure-gateway.json --type azure ``` The lint command is hermetic and makes no cloud calls. GCP-only rules self-gate, so they do not produce phantom warnings against an Azure config. Exit code 0 with warnings is acceptable for a fresh wizard output; exit code 1 means a required field is missing or malformed. ## Step 3: preview the install ```shell tare gateway install identity.json --type azure --config azure-gateway.json --plan-only ``` The Plan view shows: - The pre-flights that will run and the checks they perform. - The `az aks enable-addons` command, or a skip notice if AGIC is already enabled. - The Ingress manifest that will be applied to `tars-dataplane`. Review the Plan before applying. To change any value, edit `azure-gateway.json` and rerun. ## Step 4: apply ### Step 4.1: dry-run the cloud commands (optional) ```shell tare gateway install identity.json --type azure --config azure-gateway.json --dry-run-prereqs ``` The dry-run prints the `az` commands without executing them. The output is suitable for capture into a change ticket or for execution from a workstation with different credentials. ### Step 4.2: run the install ```shell tare gateway install identity.json --type azure --config azure-gateway.json --apply-prereqs --wait --yes ``` `--yes` skips the `Proceed?` prompt and is required in non-TTY contexts such as CI and scripts. Omit it in an interactive terminal to keep a final confirmation prompt. ### Step 4.3: install sequence The install proceeds through the following stages: 1. **Pre-flight `networkProfile`** - fails fast if the cluster is AGIC-incompatible (Cilium or Overlay). 1. **Pre-flight Azure RBAC** - checks `AKS Service Contributor` on the AKS RG and `Network Contributor` on `MC_RG` against the signed-in identity. Prints exact `az role assignment create` commands and exits when a role is missing. When role-list read is denied, the step warns and continues. 1. **Pre-flight `kubectl auth can-i`** - verifies the current kubeconfig can `create ingress` in `tars-dataplane`. 1. **`az aks enable-addons ingress-appgw`** - provisions a new App Gateway Standard_v2 (approximately 5 minutes). Idempotent: when the addon is already enabled, the step is skipped. 1. **Wait for `operationalState=Running`** - polls the App Gateway resource. A known race exists where AGIC enable returns success while the App Gateway is briefly `Stopped`; the wait loop auto-recovers by issuing a `start` on the gateway. 1. **Apply Ingress** to `tars-dataplane` with the two AGIC health-probe annotations (`appgw.ingress.kubernetes.io/health-probe-path: /` and `health-probe-status-codes: "200-499"`). Without these annotations, AGIC marks the Agent Router data plane backend (egress proxy) unhealthy and every request returns 502. 1. **`--wait`** - polls until the Ingress reports an `address`. If the address stays empty for 60 seconds or more after the App Gateway is `Running`, the install issues a one-shot AGIC re-nudge (cycling the addon). The AGIC controller sometimes misses the first reconcile. ## Step 5: wire DNS and verify ```shell kubectl get ingress -n tars-dataplane # NAME CLASS HOSTS ADDRESS PORTS AGE # tars-dataplane azure-application-gateway proxy.acme.example.com 20.62.x.y 80 2m ``` Point the wildcard or serve-domain `A` record at the `ADDRESS` value. The App Gateway listens on port 80. Smoke-test the endpoint: ```shell curl -sS -o /dev/null -w '%{http_code}\n' "https://${SERVE_DOMAIN}/healthz" # expected: 200 (after DNS propagates) ``` ## Step 6: confirm via tare doctor ```shell tare doctor identity.json ``` The doctor should report Layer 9 (cloud ingress / Application Gateway) as healthy. If the smoke test fails, run `/agentrouter dp diagnose` for the layer-by-layer walk. ## Troubleshooting ### Appgw `operationalState=Stopped` after enable Known race: AGIC sometimes returns `addon enabled` before the gateway finishes starting. The wait loop in `tare gateway install` auto-recovers by issuing `az network application-gateway start`. For manual recovery: ```shell az network application-gateway start \ --resource-group "MC_${AKS_RG}_${AKS_CLUSTER_NAME}_${REGION}" \ --name "${AKS_CLUSTER_NAME}-appgw" ``` ### Ingress address stays empty after appgw is running AGIC occasionally misses the first reconcile. The `--wait` flow handles this by cycling the addon after 60 seconds. For manual recovery: ```shell az aks disable-addons --addons ingress-appgw -g "${AKS_RG}" -n "${AKS_CLUSTER_NAME}" az aks enable-addons --addons ingress-appgw -g "${AKS_RG}" -n "${AKS_CLUSTER_NAME}" \ --appgw-name "${AKS_CLUSTER_NAME}-appgw" ``` The recovery re-uses the existing App Gateway and is safe to repeat. ### `AuthorizationFailed` despite freshly granted rbac The Azure CLI caches tokens for approximately one hour. Mint a fresh token: ```shell az logout az login ``` Then rerun `tare gateway install`. ### Pre-flight fails with `networkProfile incompatible` The cluster runs Cilium or Overlay. AGIC will never converge - the App Gateway provisions but stays `Stopped`, and AGIC logs `failed to reconcile overlay CNI`. Two options apply: 1. **Switch to AGC** (Application Gateway for Containers). See Path 9b in the main Azure runbook. AGC works on any AKS dataplane and is Microsoft's recommended replacement. 1. **Rebuild the cluster** with traditional Azure CNI: `az aks create ... --network-plugin azure --network-dataplane azure` (omit `--network-plugin-mode overlay`). The `az` CLI 2.60 and later defaults silently to Cilium and Overlay; pass `--network-dataplane azure` explicitly. ### Backend health unhealthy, every request returns 502 When the Ingress is applied manually without the AGIC health-probe annotations, AGIC defaults to `GET /` expecting `200`, but the Agent Router egress proxy returns `404` on `/`. Re-apply the Ingress through `tare gateway install` (the install always writes the two annotations), or patch the Ingress directly: ```shell kubectl annotate ingress -n tars-dataplane tars-dataplane \ appgw.ingress.kubernetes.io/health-probe-path=/ \ appgw.ingress.kubernetes.io/health-probe-status-codes='200-499' --overwrite ``` ## Notes - AGC (the Cilium and Overlay path) is **not** implemented in `tare gateway install --type azure` yet — support is planned. Until then, AGC clusters follow Step 9b of the main Azure runbook manually. - AWS (`--type aws`) exposes the data plane's egress as a Network Load Balancer instead of an Ingress. See [Gateway installation guide for EKS](/agent-router-enterprise/self-hosted-data-plane/gateway-installation/gateway-installation-guide-aws/). - The install is idempotent at the addon level (re-enabling is a no-op). The Ingress apply uses `kubectl apply`, so re-running picks up annotation changes made by hand in `azure-gateway.json`. Where to go next SSO configuration Connect an OIDC identity provider before developer handoff. Console quickstart After SSO is working, issue an API key and confirm a first routed request. --- ## Gateway installation guide for GCP This guide installs the Agent Router gateway in a Google Cloud project using the tare command-line utility. The resulting ingress terminates TLS for the data plane and routes traffic to it from a customer-facing hostname. Plan for **45 to 90 minutes** end-to-end: * 10–15 minutes of CLI work. * 15–60 minutes of background certificate provisioning. ## Architecture ```text ┌───────────────────────────────────────────┐ │ proxy. ← customer DNS │ │ │ │ │ ▼ │ │ ┌────────────────┐ │ │ │ Static IP │ ← GCP global address │ │ └────────┬───────┘ │ │ │ │ │ ┌────────▼───────┐ │ │ │ Cert Map + │ ← TLS termination │ │ │ Managed Cert │ │ │ └────────┬───────┘ │ │ │ │ │ ┌────────▼───────┐ │ │ │ Agent Router | ← installed in this │ │ │ Gateway(k8s) │ guide │ │ └────────┬───────┘ │ │ │ │ │ ┌────────▼───────┐ │ │ │ Agent Router | | | | Dataplane │ ← installed earlier │ │ └────────────────┘ via tare CLI │ └───────────────────────────────────────────┘ ``` The `tare` utility provisions everything from the static IP down to the gateway routes. The operator provides one customer-facing DNS record at the top. ## Prerequisites The following are required before starting: * **Access to a GCP project** with at least the following roles: * `roles/compute.networkAdmin` (static IP). * `roles/certificatemanager.editor` (cert map, cert, DNS auth). * **Access to the Kubernetes cluster** where the Agent Router dataplane runs. `kubectl cluster-info` must succeed against the target cluster. * **Control over the DNS provider** for the domain in use (for example Cloudflare, Route 53, or an internal DNS team). Two records are created during the install. * **The Agent Router dataplane already installed.** This guide assumes `tare install` has already run successfully against the target cluster. * **`gcloud` installed and authenticated** when using `--apply-prereqs` to let the CLI provision the GCP resources. When those resources are managed externally by an infrastructure-as-code tool, skip `gcloud` and use `--ack-prereqs` instead. * **Two files from Tetrate:** * The identity file (`identity.json`), issued by the management plane during onboarding. * A starter `gcp-gateway.json`. A Tetrate field engineer typically provides a template tuned to the target environment. The `tare` binary ships `helm` and `kubectl` embedded; no separate install is required. ## Step 1: prepare the config The gateway install is driven by a single JSON contract: `gcp-gateway.json`. Every value can also be passed as a CLI flag, but the config file is the recommended unit for version control and review. A complete example: ```json { "projectId": "withfraser", "serveDomain": "proxy.acme.tetrate.ai", "customer": "acme", "environment": "production", "serveUrl": "proxy.acme.tetrate.ai", "certificateMap": { "name": "acme-tetrate-ai" }, "certificate": { "name": "acme-serve-cert" }, "dnsAuthorization": { "name": "proxy-acme-tetrate-ai-dns-auth" }, "namespaces": { "gateway": "tars-gateway", "system": "tars-system", "dataplane": "tars-dataplane" }, "gcloud": { "skipExisting": true } } ``` Commonly edited fields: | Field | Description | | -- | -- | | `projectId` | The GCP project that hosts the static IP, cert, and cert map. | | `serveDomain` | The fully-qualified hostname customers will use. Single host only; wildcards are not yet supported. | | `customer` | The customer identifier, matching the value in the identity file. | | `environment` | Free-form label propagated to Helm values. Typical values: `production`, `staging`. | | `certificateMap.name` | Name for the GCP certificate map. Convention: `-`. | | `certificate.name` | Name for the managed certificate. Convention: `-serve-cert`. | | `gcloud.skipExisting` | When `true`, re-runs are idempotent; re-installs do not fail when a resource already exists. Recommended `true`. | For the full schema, including optional fields such as `gateway.staticIpName` and `securityPolicy`, see the `gcp-gateway.json` schema reference. :::tip Set `gateway.staticIpName` to pin the gateway to a reserved static IP. Without it, the gateway receives an ephemeral GCP address that may change on reschedule or upgrade, silently breaking the DNS A record from Step 5. ::: ## Step 2: validate the config Lint the config before any cloud or cluster operation: ```shell tare gateway config lint --config gcp-gateway.json ``` The check is hermetic (no GCP or Kubernetes calls) and safe to run in CI on every config change. The lint command produces one of three outcomes. **All clear:** ```text Summary: 0 error, 0 warn, 0 info ``` **Warnings to review:** ```text ⚠ [TAREL010] gateway.staticIpName — gateway will receive an ephemeral GCP forwarding-rule address; customer DNS A records may break on reschedule. Remediate: set gateway.staticIpName in gcp-gateway.json or pass --static-ip-name on install. Summary: 0 error, 1 warn, 0 info ``` Warnings are advisory; the install proceeds, but each finding should be reviewed. Findings include a remediation hint and a docs link (`https://docs.tetrate.ai/tare/lint/`). **Errors that block install:** ```text ✗ [TAREL001] customer — required value cannot be resolved tried: --customer (empty), gcp-gateway.json:customer (empty), identity.json:customerId (empty) Remediate: add "customer" to gcp-gateway.json, or pass --customer. Summary: 1 error, 0 warn, 0 info ``` Errors must be fixed before the install runs. For the full rule list and how to suppress accepted findings (for example via a `lint.ignore` block in the config), see `tare gateway config lint`. **CI usage:** `tare gateway config lint --config gcp-gateway.json --fail-on-warn` exits non-zero on any warning or error, making it the recommended pre-merge gate for a config repository. ## Step 3: preview the install (plan) Render a Plan before any side effects: ```shell tare gateway install identity.json \ --type gcp \ --config gcp-gateway.json \ --plan-only ``` Sample output: ```text Plan: tare gateway install (provider=gcp) Identity: identity.json (client_email: ops@acme.com) Project: withfraser (from --config) Customer: acme (from --config) Environment: production (from --config) Serve host: proxy.acme.tetrate.ai (from --config:serveDomain) serveUrl: proxy.acme.tetrate.ai (same as serveDomain) GCP delta: + static IP acme-gateway-ip [will create] + cert map acme-tetrate-ai [will create] + dns auth proxy-acme-…-dns-auth [will create] + certificate acme-serve-cert [will create, single host] + map entry serve-entry [will create] Kubernetes delta: + namespace tars-gateway [will create] ~ namespace tars-system [exists] ~ namespace tars-dataplane [exists] + helm release tars-gateway [will install, chart v1.2.5] After install, customer must: Create A record: proxy.acme.tetrate.ai →
Create CNAME: (provided after install) ``` Verify the following before proceeding: 1. **Project** is correct. Misclicking between `acme-prod` and `acme-staging` is the most common deploy mistake. 1. **Customer** matches the identity file's owner. 1. **Serve host** matches the customer-facing hostname. 1. **`+` rows** are resources that will be created. **`~` rows** already exist and are left alone (or modified in place). 1. **No error findings** appear in the inline lint section. The Plan output is also available as JSON (`--output json`) for use by other tooling. The JSON shape is a stable contract (`apiVersion: tare.tetrate.io/v1alpha1`). ## Step 4: apply Two routes are supported. ### Option a: CLI provisions GCP resources When `tare` is permitted to run `gcloud` directly: ```shell tare gateway install identity.json \ --type gcp \ --config gcp-gateway.json \ --apply-prereqs \ --wait ``` `--apply-prereqs` creates the static IP, DNS authorization, certificate map, certificate, and map entry via `gcloud`. `--wait` blocks the command until the gateway has a routable address. ### Option b: GCP resources already provisioned When the GCP resources are owned externally by an infrastructure-as-code tool, skip the prereq automation: ```shell tare gateway install identity.json \ --type gcp \ --config gcp-gateway.json \ --ack-prereqs \ --wait ``` `--ack-prereqs` acknowledges that the static IP, certificate map, certificate, and DNS authorization are already in place. The CLI refuses to start the install without one of `--apply-prereqs` or `--ack-prereqs`. ### Install sequence The CLI prints the Plan, prompts `Proceed? [y/N]:` in a TTY (pass `--yes` in CI), then runs through the following stages: 1. Preflight: verify Helm, kubectl, and cluster reachability. 1. Cloud prereqs: runs only with `--apply-prereqs`. 1. Namespace creation: idempotent. 1. Helm install: applies the gateway chart. 1. Wait for address: runs only with `--wait`; reports the gateway address once Kubernetes assigns one (typically under a minute). The final output looks like: ```text Gateway installation complete. Customer action required (DNS): Create DNS A record: Host: proxy.acme.tetrate.ai Type: A Value: 34.110.x.x TTL: 300 Create DNS authorization CNAME record: Host: _abc123.proxy.acme.tetrate.ai. Type: CNAME Value: abc123..googleusercontent.com. Certificate provisioning note: Certificate may still be PROVISIONING after gateway install. HTTPS will be fully ready once certificate state is ACTIVE. ``` Copy these values for use in the next step. ## Step 5: configure DNS Two DNS records are required in the DNS provider. Both must exist; HTTPS does not work without either. ### Record 1: a record for customer traffic | | | | -- | -- | | **Host** | `proxy.acme.tetrate.ai` (the `serveDomain` value) | | **Type** | `A` | | **Value** | The address printed in Step 4 (for example `34.110.x.x`) | | **TTL** | `300` (5 minutes) recommended | Customer clients resolve this record to reach the gateway. ### Record 2: DNS authorization cname | | | | -- | -- | | **Host** | The CNAME host printed in Step 4 (for example `_abc123.proxy.acme.tetrate.ai.`) | | **Type** | `CNAME` | | **Value** | The `googleusercontent.com` target printed in Step 4 | | **TTL** | `300` recommended | Google Certificate Manager uses this record to verify **domain ownership** before issuing a TLS certificate. Without it, the certificate stays in `PROVISIONING` indefinitely. To fetch the CNAME details at any time after the install: ```shell gcloud certificate-manager dns-authorizations describe \ proxy-acme-tetrate-ai-dns-auth \ --project withfraser \ --format='value(dnsResourceRecord.name,dnsResourceRecord.type,dnsResourceRecord.data)' ``` ## Step 6: wait for the certificate to activate Once both DNS records are live, the managed certificate moves from `PROVISIONING` to `ACTIVE` automatically. Typical timing: * **First 5 minutes**: DNS propagation; certificate stays `PROVISIONING`. * **5–20 minutes**: Google validates the DNS authorization, issues the certificate, and the certificate flips to `ACTIVE`. * **Up to a few hours (rare)**: when DNS propagation is slow at the registrar. Check the certificate state: ```shell gcloud certificate-manager certificates describe acme-serve-cert \ --project withfraser \ --format="yaml(name,managed.state,managed.domainStatus)" ``` Expected output: ```yaml name: projects/withfraser/locations/global/certificates/acme-serve-cert managed: state: ACTIVE domainStatus: proxy.acme.tetrate.ai: ACTIVE ``` If `state: FAILED_NOT_VISIBLE` appears, the DNS authorization CNAME from Step 5 is missing or incorrect. Verify that it resolves: ```shell dig +short CNAME _abc123.proxy.acme.tetrate.ai ``` The result must match the value returned by `dns-authorizations describe`. ## Step 7: verify Once the certificate is `ACTIVE`, verify end-to-end. **Cluster resources are healthy:** ```shell kubectl get gateway -n tars-gateway kubectl get httproute -n tars-system kubectl get httproute -n tars-dataplane ``` All gateways and routes should report `True` for the `Accepted` and `Programmed` conditions. **Hostname resolution:** ```shell dig +short A proxy.acme.tetrate.ai # Expected: 34.110.x.x (the address from Step 4) ``` **TLS termination and gateway response:** ```shell curl -v https://proxy.acme.tetrate.ai/healthz # Expected: HTTP/2 200, valid certificate for proxy.acme.tetrate.ai ``` When the `healthz` endpoint is not exposed, any request that reaches the gateway and returns an Agent Router-shaped response (even `404` or `401`) confirms that the gateway and TLS are working. ## Troubleshooting | Symptom | Likely cause | Resolution | | -- | -- | -- | | `customer not resolved, tried: --customer (empty), gcp-gateway.json:customer (empty), identity.json:customerId (empty)` | No customer set anywhere. | Add `"customer": ""` to `gcp-gateway.json` or pass `--customer`. | | GCP flags such as `--certificate-map-name` are ignored | `--type aws` passed. | The AWS path is ingress-only and takes no config file. Follow [Gateway installation guide for EKS](/agent-router-enterprise/self-hosted-data-plane/gateway-installation/gateway-installation-guide-aws/), or pass `--type gcp` for this guide. | | Plan shows a `?` row for a resource | The CLI could not probe live state (credentials, IAM, or network issue). | The install treats the resource as `will-create`. Check the `probeError` field in `--output json` for details. | | `helm deploy failed` | Cluster not reachable, RBAC issue, or a chart-level error. | Run `kubectl cluster-info`; when that fails, fix the kubeconfig context first. | | Gateway never reports an address (`--wait` times out) | Cloud load-balancer provisioning is slow, or the static IP is not attached. | Run `kubectl describe gateway -n tars-gateway` and check Kubernetes events for the gateway resource. | | Certificate stuck `PROVISIONING` for \>30 min | DNS authorization CNAME is missing, wrong, or not yet propagated. | Re-check Step 5. Use `dig` to verify the CNAME resolves. | | Certificate state `FAILED_NOT_VISIBLE` | DNS authorization record cannot be resolved from Google's side. | Recreate the CNAME exactly as printed by `dns-authorizations describe`. | | `curl` returns a TLS error after cert is `ACTIVE` | A record points to the wrong IP, or local DNS cache. | Verify with `dig`; flush the local resolver. | | Re-running install fails with `already exists` from gcloud | `gcloud.skipExisting` is `false`. | Set `"gcloud": { "skipExisting": true }` in `gcp-gateway.json`. | For issues not covered above, re-run the install with the Plan visible (`--plan-only` is sufficient) and share the output with the Tetrate contact. The Plan captures every resolved value and live state probe, which is usually enough to diagnose. ## Need help? When blocked, capture: 1. The output of `tare gateway install … --plan-only`. 1. The output of `tare gateway config lint --format json`. 1. The output of `gcloud certificate-manager certificates describe …`. Share these with the Tetrate contact. Together they describe the config, the live cloud state, and the certificate state, which is usually enough to unblock without further back-and-forth. Where to go next SSO configuration Connect an OIDC identity provider before developer handoff. Console quickstart After SSO is working, issue an API key and confirm a first routed request. --- ## Gateway installation overview Gateway resources expose and manage AI traffic to the data plane. This page describes the tare install-gateway command and its workflow; Agent Router–specific runbooks follow. ## Synopsis ```shell tare install-gateway [flags] ``` ## Description The `tare install-gateway` command deploys the gateway Helm chart required to expose Agent Router serve traffic. It manages the lifecycle of the gateway resources, from prerequisite validation to final DNS configuration guidance. ## High-level workflow 1. **Identity loading:** Validates and loads the provided ``. 1. **Configuration merging:** Loads the optional `gcp-gateway.json` via `--config` and resolves values across flags, config, and identity defaults. 1. **Prerequisite handling:** Runs optional prerequisite workflows with `--dry-run-prereqs` or `--apply-prereqs`. 1. **Preflight validation:** Verifies environment readiness. 1. **Installation:** Deploys the Agent Router gateway chart through the embedded Helm engine. 1. **Finalization:** Optionally waits for the Gateway address (`--wait`) and outputs the required DNS A-record actions. ## Provider support | Provider | Status | | :---- | :---- | | Google Cloud Platform (GCP) | Fully supported | | Azure Kubernetes Service (AKS) | Fully supported | | Amazon Web Services (AWS) | Fully supported | :::note The AWS path is ingress-only and takes no config file. Instead of deploying the gateway chart, it exposes the data plane's egress as a Network Load Balancer by patching the egress EnvoyProxy, so no AWS Load Balancer Controller and no `aws` CLI are required. ::: Where to go next Gateway installation on GCP Install the Agent Router gateway in a Google Cloud project using the tare CLI. Gateway installation on AKS Install the Agent Router gateway on Azure Kubernetes Service using the tare CLI. Gateway installation on EKS Expose the data plane on Amazon EKS as a Network Load Balancer using the tare CLI. --- ## Gateway Installation With the data plane running, the gateway is the component that answers on the Proxy URL chosen during onboarding. Installation wires the gateway into the cluster, then connects DNS and TLS so the endpoint answers over HTTPS. A generic guide covers the common path; cloud-specific variants cover GCP, Azure, and AWS particulars. On AWS the certificate is yours rather than installer-provisioned. Gateway installation overview The common installation path: deploy the gateway and wire DNS and TLS. Gateway installation on GCP GCP-specific steps for load balancing, DNS, and certificates. Gateway installation on Azure Azure-specific steps for load balancing, DNS, and certificates. Gateway installation on EKS Expose the data plane on EKS as a Network Load Balancer, optionally terminating TLS with your ACM certificate. --- ## Agent Router Enterprise Self-Hosted Data Plane Agent Router Enterprise · Self-Hosted Data Plane Install the data plane in your Kubernetes cluster; Tetrate hosts the management plane. Step 1 Prerequisites Cluster, network, accounts, and quotas before install. Verify prerequisites → Step 2 Management plane onboarding Complete through Step 5 (data plane credential). SSO can wait. Management plane onboarding → Step 3 Data plane installation Controller and Agent Router using the credential from onboarding. Data plane guides → Step 4 Gateway installation Inbound access and request routing for the data plane. Gateway installation → Step 5 SSO configuration OIDC sign-in before developer handoff. Configure SSO → Step 6 Console quickstart API key and a first routed request once the gateway is reachable. Console quickstart → --- ## Self-Hosted onboarding Onboard a Self-Hosted Data Plane deployment through a one-time Admin Dashboard wizard. Steps 1 to 5 collect configuration, Step 5 downloads the credential used to install the data plane, and Step 6 confirms completion. Two values can be deferred: SSO (Step 2), until after the gateway is running but before the Console is opened to developers, and the gateway URL (Step 1), until after the gateway and DNS are live. ## The onboarding wizard The wizard launches automatically on the first sign-in to the Admin Dashboard; there is no separate onboarding URL. Each step has **Next** and **Previous** controls. **Next** is enabled once the current step's required fields are filled; **Previous** returns to any earlier step to review or change a value. Step 1 Organization Name Identify the tenant, and optionally register the gateway URL. Step 2 Corporate Login Configure SSO over OpenID Connect (OIDC). Can be deferred until after the gateway is running. Step 3 LLM Providers Connect providers such as OpenAI and Anthropic. Step 4 LLM Models Choose which models are exposed. Step 5 Configure AI gateway Download the `service-account.json` credential the data plane needs. Step 6 Setup Complete The Admin Dashboard can now be accessed by signing in with SSO. Onboarding produces a configured management plane and a `service-account.json` file used as input to the data plane installation. Plan for 15 to 30 minutes, plus any back-and-forth with the identity provider (IdP) team if SSO is configured now. ## Prerequisites Have the following on hand before launching the wizard: - Access to the Admin Dashboard. Tetrate provides the URL during provisioning. - Initial admin credentials for the first sign-in. SSO is configured in the wizard; the first login uses these credentials. - Your API credentials for at least one LLM provider, for example an OpenAI API key. **Important: make sure you have at least one (1) API key available during onboarding.** - OIDC IdP details (issuer URL, client ID, client secret, scopes) if SSO is configured now. Agent Router supports **OIDC only**. SSO can also be deferred; see [SSO configuration](/agent-router-enterprise/guides/identity-and-access-management/single-sign-on/sso-configuration-guide/). - An IdP administrator reachable during Step 2. The wizard shows a redirect URI that must be registered on the IdP side before SSO works. - Optionally, a planned hostname for the gateway, for example `https://ar-proxy.my-company.com`. This value can be entered as the **Data plane URL** in Step 1, or passed later as `tare install --serve-url`. Neither entry point is mandatory: the URL can also be registered from the Admin Dashboard once the gateway and DNS are live. ## Get started The wizard appears automatically when the Admin Dashboard URL is first visited. Click **Start setup** to begin. ### Step 1: configure organization name Fill in: | Field | Required | Description | Sample value | | :---- | :---- | :---- | :---- | | Organization Name | Yes | The organization's display name | `Acme Corp.` | | Data plane URL | No | The public URL where the gateway (proxy) is reachable. Always include the scheme (`https://`). The same value can be passed to `tare install --serve-url` later. | `https://proxy.tare-pn-12345a678.tetrate.ai` | :::info The Data plan URL can be deferred The gateway does not need to exist yet, and this field can be left empty. **Next** is enabled once **Organization Name** is filled in. Onboarding only stores the value; the cluster, ingress, and DNS come later during install. Leaving the field empty means the default project is not serving traffic when the wizard finishes, which Step 6 reports. The URL can be registered at any time afterwards under **System → Settings → Data planes**, or automatically by passing `--serve-url` to `tare install`. See [Data plane installation](/agent-router-enterprise/self-hosted-data-plane/data-plane-installation/). ::: :::note Field label The wizard labels this field **Data plane URL**, but the value is a gateway endpoint: a data plane hosts gateways, and each gateway owns the URL that clients call. See [Manage multiple gateways on a data plane](/agent-router-enterprise/guides/operate-and-govern/project-and-data-plane-management/manage-gateways-on-a-data-plane/). ::: ### Step 2: configure corporate login (single sign-on) :::info SSO can wait For a Self-Hosted Data Plane install, configure SSO here or defer it until after the data plane and gateway are running. Developers need SSO before using the Console, so complete it before developer handoff. See the [SSO configuration guide](/agent-router-enterprise/guides/identity-and-access-management/single-sign-on/sso-configuration-guide/). ::: :::warning Agent Router supports **OIDC only**. SAML, LDAP, and other protocols are not supported. ::: If configuring SSO now, the wizard displays the **OIDC redirect URI** for the deployment, following the pattern: ```text https://dashboard./login ``` Register Agent Router as an OIDC application in the IdP using that exact URI, then fill in: | Field | Required | Description | Sample value | | :---- | :---- | :---- | :---- | | Issuer (IdP URL) | Yes | OIDC issuer URL exposed by the IdP | `https://login.example.com/` | | Client ID | Yes | OAuth client ID created for Agent Router | `agent-router` | | Client Secret | Yes | OAuth client secret paired with the Client ID | `••••••••` | | Discovery URL | No | OIDC discovery document URL. If omitted, Agent Router constructs it as `/.well-known/openid-configuration`. | `https://login.example.com/.well-known/openid-configuration` | | Scopes | Yes | OAuth scopes to request | `openid profile email` | | Admin email | Yes | At least one user granted admin rights after the first SSO login | `admin@example.com` | | Attribute mapping | Yes | Maps an IdP claim to the Agent Router user's email. The typical value is `email`. | `email` | ### Step 3: configure LLM provider(s) Connect Agent Router to one or more LLM providers. At least one provider is required to route traffic. The wizard shows a radio button for each supported provider, including OpenAI, Anthropic, Google, Azure OpenAI, Mistral, Cohere, Together, and Perplexity. See [Provisioning models and providers](/agent-router-enterprise/guides/operate-and-govern/provision-models-and-providers/) for the authoritative list. Select a provider, click **Configure**, then fill in: | Field | Required | Description | Sample value | | :---- | :---- | :---- | :---- | | Display Name | No | Optional label shown in tables and filters | `OpenAI` | | API Base URL | Yes | Base URL for the provider API. Pre-filled for supported providers. | `https://api.openai.com/v1` | | Auth method | Yes | Authentication method. Pre-filled per provider. | `API Key` | | Secret Type | Yes | `Password` for single-string API keys; `Service Account JSON` for providers that authenticate with a JSON credential file. | `Password` | | Provider Secret | Yes | API key, token, or service-account JSON. Use **Rotate** in the UI to replace a stored secret without clearing the rest of the form. | `sk-proj-...` | | Enable provider | Yes | Toggle on to expose the provider with this configuration. | toggle | Repeat for each provider to expose. ### Step 4: configure LLM models The wizard shows a single list of all models Agent Router supports, maintained by Tetrate. By default, a selective set of common models is enabled. Deselect models to control cost, meet data-handling requirements, or scope to the intended use case. Models remain configurable later from the Admin Dashboard. If a required model is missing, contact Tetrate support. :::note Custom model entries are not configurable in onboarding. ::: ### Step 5: Configure AI gateway This final step generates the credentials the data plane uses to authenticate to the management plane. Click **Create Service Account**. A `service-account.json` file downloads. The credential is valid immediately and can be used to run `tare install` as soon as it lands. :::warning The download is **one-shot**. If the browser blocks it or the file is lost, the same credential cannot be downloaded again; regenerate it instead (see below). ::: :::warning Treat `service-account.json` as a long-lived secret. Store it in a secret manager; do not commit it to source control. ::: #### Regenerate the service account file If the file is lost, regenerate it from the Admin Dashboard: 1. Sign in to the Admin Dashboard. 1. Go to **System | Settings**. 1. Open the **Licenses** tab. 1. Click **+ Generate New License**. A new `service-account.json` file downloads. Older files remain valid until revoked. ### Complete the process Click **Complete setup** to finish onboarding. Where to go next Data plane installation Install the data plane with the `service-account.json` credential from Step 5. Configure SSO (if deferred) Complete OIDC setup if SSO was skipped in the wizard. Required before developer handoff. --- ## Prerequisites Before starting with Tetrate Agent Router, verify that you have the following access and environment requirements in place. This page covers what each role needs (developers and administrators) to use Agent Router successfully from day one. ## Management Plane and Data Plane The very first prerequisite for installing, configuring or using a self-hosted data plane, is a provisioned Management Plane. Both the provisioning and operational management is done by Tetrate, making the management plane a hosted components. At the moment, it is not possible to fully self-host Agent Router Enterprise, only the data plane. ![management-vs-data-plane](/img/screenshots/about/architecture/TARE_arch.png) For more information on hosting options and the underlying architecture, consult [Planes and core components](/product-architecture/planes-and-components/). ## Account access Tetrate Agent Router serves different web applications, each serving a distinct role. Depending on your function, you may need access to one or both. | Requirement | Details | | --- | --- | | **User account** | An email/password or SSO credential provided by your administrator. For Proof-of-Concept (PoC) environments, Tetrate provisions accounts with appropriate roles. | | **Console URL** | The developer-facing application for API keys, model routing, and usage monitoring (e.g., `https://router.tare-pn-12345678.tetrate.ai/`) | | **Admin Dashboard URL** | The operator-facing application for managing models, providers, and users (e.g., `https://dashboard.tare-pn-12345678.tetrate.ai/`) | Your administrator will provide URLs and initial credentials during onboarding. If your organization uses SSO, you will authenticate through your corporate identity provider; no separate password is required. Agent Router access is role-based: each user is granted access only to the applications appropriate to their function. ## Browser requirements Tetrate Agent Router is fully browser-based; no desktop software installation is required. Both applications support the following browsers: - Google Chrome 90+ - Mozilla Firefox 90+ - Microsoft Edge 90+ - Safari 15+ JavaScript must be enabled. A screen resolution of 1280×720 or higher is recommended for the best experience, particularly when viewing usage dashboards and request log details. ## For developers (Console users) Developers use the Console to create API keys, browse the model catalog, configure routing policies, and monitor usage. In addition to account access, you will need: | Requirement | Purpose | | --- | --- | | **API key** | Generated in the Console under **API Keys**. Authenticates all AI requests sent through the gateway and determines which routing policies apply. | | **Proxy endpoint URL** | The address your application sends requests to instead of calling AI providers directly. The gateway handles routing, fallback, guardrail enforcement, and provider translation transparently. Shown on your Dashboard (e.g., `https://my-first-proxy.acme.com`). | | **SDK or HTTP client** | Any OpenAI-compatible client library or HTTP tool. No gateway-specific SDK is required. | ### Supported SDKs and tools Tetrate Agent Router is compatible with any tool or library that speaks the OpenAI API format. To connect, replace the provider's base URL with your proxy endpoint URL and use your Console API key. No other changes are required. | Category | Examples | | --- | --- | | **Python libraries** | `openai`, `langchain`, `crewai`, `pydantic-ai`, `llama-index` | | **TypeScript/JavaScript** | `openai` SDK, Vercel AI SDK | | **Code assistants** | Cline, Aider, Continue, Roo Code, Goose | | **Chat interfaces** | Open WebUI | | **MCP clients** | Claude Code, Cursor, VS Code | For step-by-step setup instructions for each tool, see [Integrations](/agent-router-service/guides/integrate-the-gateway-with-an-app/). ## For administrators (Admin Dashboard users) Administrators use the Admin Dashboard to manage model availability, configure AI provider credentials, control user access, and maintain Agent Router audit trail. You need: | Requirement | Purpose | | --- | --- | | **Admin role** | Assigned by a Agent Router administrator. Grants access to the Admin Dashboard and all operator-level controls. | | **Provider credentials** | API keys for each AI provider you plan to configure (e.g., OpenAI, Anthropic, Azure OpenAI, Google Vertex AI). These are stored securely by Agent Router and are never exposed to Console users. | | **SSO metadata** | OIDC discovery URL, issuer, client ID, and client secret from your corporate identity provider. Agent Router supports OIDC only. | In PoC environments, provider credentials and initial Agent Router configuration are preconfigured by Tetrate. Administrators can review and modify these settings from first sign-in. ## Network requirements Where network firewalls, security groups, or outbound proxies are in use, the connections listed below must be permitted. ### Endpoints, direction, and ports Direction is stated from the customer's perspective: **Out** is traffic that leaves the customer domain, **In** is traffic that enters it. All ports are TCP, and every Tetrate-hosted endpoint is HTTPS on port 443. Tetrate-hosted endpoints follow the pattern `..tetrate.ai`, where `` is the identifier issued during onboarding (for example `tare-pn-12345678`). Which connections apply depends on the deployment model, and the difference is larger than it first appears. In a Self-Hosted Data Plane deployment the gateway runs in a customer-managed cluster, which is what makes the cluster-side rows and the single inbound path necessary. In an [Enterprise Fully Managed](/agent-router-enterprise/fully-managed/) deployment Tetrate runs both planes and nothing is installed in the customer environment, so every required connection is outbound from a browser or an application and no inbound rule exists at all. Both tables are given below; see [Deployment models](/product-architecture/deployment-models/) for the wider comparison. #### Self-Hosted Data Plane This is the model described by this section: Tetrate hosts the management plane, and the Controller and the gateway run in a customer-managed Kubernetes cluster. | Endpoint | Direction | Port | Opened by | Purpose | | --- | --- | --- | --- | --- | | `https://router..tetrate.ai` | Out | 443 | Developer browsers | Console access: API keys, model catalog, usage monitoring | | `https://dashboard..tetrate.ai` | Out | 443 | Administrator browsers | Admin Dashboard access: models, providers, users, settings | | `https://auth..tetrate.ai` | Out | 443 | All browsers, at sign-in | Authentication, including the OIDC callback used by SSO | | `https://api..tetrate.ai` | Out | 443 | Controller, from inside the cluster | Configuration polling and data plane health reports (`/v1/dataplane-status`) | | `https://registry.tetrate.ai` | Out | 443 | Cluster nodes and the install host | Container images and the Helm chart. Install and upgrade only, and avoidable by mirroring to a private registry | | `https://tare.tetrate.ai` | Out | 443 | Operator workstation | `tare` CLI download. Once per workstation, not needed by the cluster | | AI provider APIs, such as `https://api.openai.com` and `https://api.anthropic.com` | Out | 443 | Gateway | Model requests to each configured provider. Not required where every provider is self-hosted or reached over private connectivity | | Remote MCP servers referenced by MCP profiles | Out | 443 | Gateway | MCP tool calls to servers hosted outside the cluster | | External observability backend, such as `https://otlp.nr-data.net` or `https://trace.agent.datadoghq.com` | Out | 4317 (OTLP gRPC) or 443 | In-cluster OpenTelemetry collector | Telemetry export. Not required where the observability stack runs inside the customer environment | | `https://acme-v02.api.letsencrypt.org` | Out | 443 | cert-manager | Certificate issuance for the gateway hostname. Only where Let's Encrypt is used rather than an internal certificate authority | | Gateway endpoint on a customer-owned hostname, such as `https://ai-gateway.acme.com` | In | 443 | Client applications, agents, and coding assistants | AI request routing. This ingress is created and controlled by the customer and can be internal-only. It terminates on the in-cluster `egress` service on port `10080` | | Corporate OIDC identity provider: discovery, authorization, and token endpoints | In | 443 | Tetrate-hosted authentication service | Validation of the discovery document and exchange of the authorization code. Applies only where the identity provider is self-hosted and not otherwise reachable from outside the customer domain. No rule is needed for a SaaS identity provider such as Entra ID, Okta, or Google | | Let's Encrypt HTTP-01 challenge validation | In | 80 | Let's Encrypt validators | Certificate issuance only. Avoidable with the DNS-01 solver or a certificate supplied from an internal public key infrastructure | | Tetrate management plane into the cluster | None | None | Nothing | No connection is ever initiated from Tetrate into the customer environment, so no inbound rule is required on Tetrate's behalf | Only two of these entries are unavoidable: the outbound connection from the Controller to `api..tetrate.ai`, and the inbound path from client applications to the gateway, which the customer defines. Every other row is either conditional on a feature being used or removable, as described in the sections that follow. #### Enterprise Fully Managed, for comparison Nothing is installed in the customer environment, so there is no cluster, no ingress to create, and no cluster-side egress. The gateway is Tetrate-hosted and is called outbound like any other SaaS API. | Endpoint | Direction | Port | Opened by | Purpose | | --- | --- | --- | --- | --- | | `https://router..tetrate.ai` | Out | 443 | Developer browsers | Console access: API keys, Base URL, request logs, usage | | `https://dashboard..tetrate.ai` | Out | 443 | Administrator browsers | Admin Dashboard access: models, providers, users, settings | | `https://auth..tetrate.ai` | Out | 443 | All browsers, at sign-in | Authentication, including the OIDC callback used by SSO | | `https://proxy..tetrate.ai/v1` | Out | 443 | Applications, agents, and coding assistants | AI request routing. This is the Base URL shown in the Console, and it is the row that reverses direction in a Self-Hosted Data Plane deployment, where the same function is served by a customer-owned hostname | | Corporate OIDC identity provider: discovery, authorization, and token endpoints | In | 443 | Tetrate-hosted authentication service | As in the self-hosted case, and required only where the identity provider is self-hosted and not otherwise reachable from outside the customer domain | Three consequences of the model are worth noting before a firewall change request is raised: - Nothing that appears in the self-hosted table as cluster-side traffic applies. `registry.tetrate.ai` and `tare.tetrate.ai` are not contacted, no certificate is issued for a customer hostname, and no inbound rule is opened for the gateway. - Connections to AI providers and to remote MCP servers originate from Tetrate's environment rather than the customer's. Where a provider account restricts access by source address, the allowlist must therefore cover Tetrate's egress and not the customer's. - Telemetry export to a customer-hosted observability backend is the one case that reverses direction, since the exporter would run on Tetrate's side. Confirm the arrangement with Tetrate before assuming a self-hosted collector can be used. The remainder of this page describes the self-hosted case. ### Data plane connections In a [Self-Hosted Data Plane](/agent-router-enterprise/self-hosted-data-plane/) deployment the data plane (the Controller and the Agent Router gateway) runs in a customer-managed Kubernetes cluster, and Tetrate hosts the management plane. The connections the cluster needs are listed below, stated relative to the customer environment. | Direction | Connection | Port | Required | | --- | --- | --- | --- | | Outbound | Controller to the management plane, for configuration polling | 443 | Yes. This is the only connection the data plane needs in steady state. | | Outbound | Gateway to AI provider endpoints | 443 | Yes, unless every configured provider is self-hosted or reached over private connectivity. | | Outbound | Cluster to `registry.tetrate.ai`, for container images and the Helm chart | 443 | At install and upgrade only. Avoidable by mirroring the images and chart into a private registry. | | Inbound | Client applications to the gateway | 443 | Yes, but this ingress is created and controlled by the customer, and it can be internal-only. | | Inbound | Tetrate or the public internet into the cluster | None | No. Nothing outside the environment opens a connection into the data plane. | Mirroring images and the chart to a private registry is covered under [Data plane installation](/agent-router-enterprise/self-hosted-data-plane/data-plane-installation/); keeping provider traffic off the public internet is covered under [Private provider connectivity](/product-architecture/network-and-security/#private-provider-connectivity). ### Outbound-only applies to every self-hosted deployment The outbound-only design is a property of the architecture, not a separate deployment mode. [Network and Security](/product-architecture/network-and-security/) discusses it under the heading **Intranet deployment**, which describes the most restrictive network posture rather than a distinct product configuration: the whole page applies to Self-Hosted Data Plane deployments. A standard self-hosted install therefore has the same property as the intranet case. The management plane never opens a connection into the customer environment, and no inbound firewall rules are needed on Tetrate's behalf. What an intranet posture adds is a choice the customer makes about the gateway's own ingress. Where every application calling the gateway is itself internal, the gateway can be exposed on an internal address only and no public ingress is needed at all. ### Summary for a security review The points a security team most often asks about, collected for a review or a firewall change request: - The management plane never initiates a connection into the customer environment. There is no exposed listener for Tetrate to call and no inbound firewall rule to open. - Configuration reaches the data plane over a single outbound HTTPS connection on port 443, opened from inside the cluster by the Controller and used to poll for updates. - Prompts and responses travel only through the data plane in the customer's environment. The outbound configuration connection carries configuration, not request content. What request detail is sent to the management plane is itself configurable; see [Control what request data leaves the cluster](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/configuring-request-logs/). - Traffic to AI providers can be held on private connectivity (Azure Private Link, AWS PrivateLink, GCP Private Service Connect) so it never crosses the public internet, and a fallback chain built only from private endpoints cannot route to a public provider. - Gateway ingress is the customer's own. It can be restricted to internal networks, and mutual TLS can be required of callers. - Registry access is needed only at install and upgrade, and can be removed entirely by mirroring images and the Helm chart into a private registry. Full detail on each control, written for a reader without a networking background, is in [Network and Security](/product-architecture/network-and-security/). For topology, see [Deployment models](/product-architecture/deployment-models/) and [Architecture Overview](/product-architecture/architecture-overview/). ### The Data plane URL field The onboarding wizard's **Data plane URL** field is described in the interface as the URL the management plane uses to reach the AI Gateway. That label is misleading and has caused the field to be read as an inbound connection requirement. It is not one. The value is the customer-facing gateway URL, the address that **client applications** call. The management plane stores it against the workspace and pushes it into the data plane's own configuration, so that the Console can show developers the endpoint to use and the gateway knows the address it serves. Nothing in the management plane dials it. Three consequences follow, all useful to know before onboarding starts: - The URL does not need to be reachable from the management plane or from the internet. It needs to be reachable by whatever calls the gateway, so an internal-only hostname is correct when every caller is internal. - The URL does not need to resolve at install time. It is recorded as configuration, and DNS can be wired afterwards. - The field is optional. It can be left empty during onboarding and registered later under **System → Settings → Data planes**, or passed to `tare install --serve-url`. See [Management plane onboarding](/agent-router-enterprise/self-hosted-data-plane/onboarding/) for the field in context, and [Manage multiple gateways on a data plane](/agent-router-enterprise/guides/operate-and-govern/project-and-data-plane-management/manage-gateways-on-a-data-plane/) for why a data plane can own more than one such URL. ## Kubernetes requirements Self-hosted data plane installs require a Kubernetes cluster running **Kubernetes 1.32 or later**. ## Which version to install The registry publishes a long list of versions carrying pre-release identifiers such as `alpha` and `rc`, which makes the choice look more consequential than it is. Which version applies depends on the install path. **With `tare install`, no version is chosen.** The CLI carries an embedded Helm chart and deploys that, so the data plane version follows the version of the `tare` binary in use. Installing a current CLI is the whole of the decision, and `tare --version` reports what is installed. The `--chart-version` flag exists only for overriding the embedded chart with a remote or OCI one through `--chart-path`, which is not the usual path. **With a direct Helm install**, a version is required, because the chart is pulled from the registry as an OCI artifact: ```bash CHART_VERSION="0.1.0-alpha.1+17f076b" ``` Unless Tetrate has pinned a version for the deployment, use the most recently published chart version. The pre-release identifiers currently in use reflect a pre-1.0 release train that is moving to a plain SemVer scheme; they are not a reason to prefer an older tag, and there is no separate stable channel to select from. Where a version is being chosen for a production install, confirm the intended one with Tetrate rather than inferring it from the tag list. Two constraints on the version string itself cause most install failures: - The value must be a full SemVer version. A bare commit SHA is rejected before Helm contacts the registry. - Build metadata is joined with `+`, as in `0.1.0-alpha.1+17f076b`. The registry stores the same tag internally with `_`, but Helm's `--version` requires the `+` form. A release-build version such as `v0.1.0-beta.3` is also accepted. Record the version that was installed, since upgrades and support conversations both depend on knowing it. The mechanics of both paths, including mirroring the chart to a private registry, are in [Data plane installation](/agent-router-enterprise/self-hosted-data-plane/data-plane-installation/) and the [`tare install` reference](/reference/tare-cli/install/). ## SSO requirements Single sign-on (SSO) lets users access the Console and Admin Dashboard through your corporate identity provider. Any OIDC-compliant identity provider is supported. Before configuring SSO, collect the following details: - Issuer (Identity Provider URL) - Client ID - Client Secret - Discovery URL (optional) **When SSO is required depends on your deployment path.** Open your path from the Get started sidebar, or see Deployment models if you are unsure which applies. - **Agent Router Service:** SSO is handled by Tetrate; no customer SSO setup. - **Enterprise Fully Managed:** Configure SSO during onboarding (wizard Step 2) or confirm it is already set up in a PoC environment. - **Enterprise Self-Hosted Data Plane:** Gather SSO details early. SSO can be configured after the data plane and gateway are running, provided it is done before opening the Console to developers. In Tetrate-provisioned PoC environments, SSO may already be configured; confirm with your administrator before starting the onboarding wizard. Mapping IdP roles to Agent Router roles is optional and can be done after the data plane is running. ## Evaluation checkpoint Use this checklist to confirm your environment is ready before proceeding to the quickstarts: - [ ] you have completed Management plane onboarding (triggered when first opening the Admin Dashboard) - [ ] You can access the Console URL and sign in successfully - [ ] You can access the Admin Dashboard URL (if applicable to your role) - [ ] Your browser meets the minimum version requirements - [ ] For self-hosted data plane installs: your cluster meets the [Kubernetes version requirement](#kubernetes-requirements) - [ ] For self-hosted data plane installs: outbound HTTPS on port 443 from the cluster to the management plane is permitted, and any security review has the [connection summary](#summary-for-a-security-review) - [ ] For self-hosted data plane installs: the [version to install](#which-version-to-install) is known, whether that is a `tare` CLI version or an explicit chart version - [ ] You have an API key (or can create one in the Console) - [ ] You know your proxy endpoint URL Where to go next Management plane onboarding Continue self-hosted setup: sign in to the Admin Dashboard and complete organization details, providers, and models. Self-Hosted Data Plane See the full install sequence from management plane onboarding through developer handoff. --- ## AI capabilities Agent Router passes through AI capabilities (structured outputs, multimodal inputs, and function calling) to any provider that supports them. Requests are forwarded as-is when the input format already matches the target provider's native schema, and translated automatically when cross-provider mappings exist. ## Capabilities covered on this page Structured outputs Constrain model responses to a JSON schema. Supported in passthrough and (where available) translated modes across providers. Go to section → Multimodal Image generation and vision inputs. Formats are translated automatically when routing across provider boundaries. Go to section → Function calling Tool definitions and tool call responses forwarded unchanged. Full examples for OpenAI and Anthropic formats. Go to section → ## Structured outputs Structured outputs constrain model responses to a specific JSON schema, guaranteeing that the model's reply conforms to a defined data shape. This eliminates the need to parse free-form text, reduces the risk of malformed responses in production, and makes AI outputs directly usable in downstream systems without an intermediate validation step. Agent Router supports structured outputs in two modes: - **Passthrough**: the request is forwarded to the provider unchanged, because the request format already matches the provider's native structured output schema - **Translated**: Agent Router adapts the request to the target provider's structured output format when routing across provider boundaries | Input Format | Backend | Mode | Status | | --- | --- | --- | --- | | OpenAI `/v1/chat/completions` | OpenAI | Passthrough | Supported | | OpenAI `/v1/chat/completions` | GCP Anthropic (Vertex) | Translated | Coming in next EAG release | | Anthropic `/v1/messages` | Anthropic | Passthrough | Supported | | Anthropic `/v1/messages` | GCP Anthropic (Vertex) | Passthrough | Supported | ### Example: OpenAI structured output The following request asks the model to return a list of planets as a JSON object conforming to a strict schema. Setting `"strict": true` enables guaranteed schema adherence: the model will not return fields outside the defined schema. `PROXY_URL` stands for the proxy endpoint from the Console Dashboard, scheme included and ending in `/v1`. ```bash curl PROXY_URL/chat/completions \ -H "Authorization: Bearer YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "gpt-4o", "messages": [{"role": "user", "content": "List 3 planets"}], "response_format": { "type": "json_schema", "json_schema": { "name": "planets", "strict": true, "schema": { "type": "object", "properties": { "planets": { "type": "array", "items": { "type": "string" } } }, "required": ["planets"], "additionalProperties": false } } } }' ``` :::note Anthropic Structured Outputs When using the Anthropic Messages API (`/v1/messages`), use the `output_config.format` field for structured output configuration. The older `output_format` field is deprecated. ::: :::info Cross-Provider Translation Cross-provider structured output translation (e.g., sending an OpenAI-format structured output request that the gateway routes to an Anthropic backend) is being added in an upcoming **Tetrate Agent Router** release (upstream: [envoyproxy/ai-gateway#1846](https://github.com/envoyproxy/ai-gateway/pull/1846)). Until this lands, structured outputs work in passthrough mode only; the request format must match the target provider's native schema. ::: ## Multimodal Agent Router supports multimodal AI capabilities including image generation and vision (image understanding). Payloads are forwarded to the backend as-is, and Agent Router translates formats automatically when routing across providers, enabling, for example, an OpenAI-format vision request to be served by an Anthropic Claude backend without any changes to the client application. ### Image generation Use the `/v1/images/generations` endpoint to generate images from text prompts. Requests are routed to the configured image generation provider and model (e.g., OpenAI's `gpt-image-1`). The response contains the generated image as base64-encoded data. ```bash curl -H "Content-Type: application/json" \ -H "Authorization: Bearer $TARE_API_KEY" \ -d '{ "model": "gpt-image-1", "prompt": "a serene mountain landscape at sunrise in watercolor", "size": "1024x1024", "n": 1 }' \ -X POST $TARE_BASE_URL/images/generations | jq -r '.data[0].b64_json' | base64 -d > output.png ``` The example above decodes the base64 payload and saves it to `output.png`: ![Example generated image, watercolor mountain landscape at sunrise](/img/screenshots/ai-features/image-generation-example.png) ### Vision (image understanding) To send an image to a model for analysis, include it in the `messages` array using the OpenAI content parts format: a `content` array that combines text items and image items in a single message. This works with any vision-capable model regardless of provider. Images can be supplied as inline base64 data URIs (for local files) or as publicly accessible URLs. **Inline base64 (local file):** ```bash IMAGE_BASE64=$(base64 < image.png) curl -H "Content-Type: application/json" \ -H "Authorization: Bearer $TARE_API_KEY" \ -d '{ "model": "claude-sonnet-4-5", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "What is in this image?" }, { "type": "image_url", "image_url": { "url": "data:image/png;base64,'"$IMAGE_BASE64"'" } } ] } ] }' \ -X POST $TARE_BASE_URL/v1/chat/completions ``` **URL reference (publicly accessible image):** ```bash curl -H "Content-Type: application/json" \ -H "Authorization: Bearer $TARE_API_KEY" \ -d '{ "model": "gpt-4o", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in detail." }, { "type": "image_url", "image_url": { "url": "https://example.com/image.png" } } ] } ] }' \ -X POST $TARE_BASE_URL/v1/chat/completions ``` Agent Router translates the content parts format automatically when routing across providers; an OpenAI-format vision request can be routed to an Anthropic Claude backend transparently. For a full list of models that support vision inputs, see the [Model Catalog](/agent-router-service/guides/route-requests-across-providers/). ## Function calling Function calling (also called tool use) allows the model to request that your application execute a function and return the result. This enables the model to take actions and retrieve information it cannot access on its own, such as querying a database, calling an API, or reading a file. Agent Router forwards tool definitions and tool call responses unchanged to providers that support native function calling. The interaction follows a two-round pattern: 1. **Round 1**: Your application sends a request with a `tools` array defining the available functions. If the model decides it needs to call a tool to answer the question, it returns a response with `finish_reason: "tool_calls"` (OpenAI) or `stop_reason: "tool_use"` (Anthropic) instead of a final answer. 1. **Round 2**: Your application executes the function, then sends a follow-up request containing the original messages, the model's tool call, and the function result. The model uses this context to generate its final response. ### OpenAI format Use the `tools` array and optional `tool_choice` parameter with the Chat Completions API. **Round 1, request with tool definition:** ```json { "model": "gpt-4o", "messages": [{"role": "user", "content": "What's the weather in NYC?"}], "tools": [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a city", "parameters": { "type": "object", "properties": { "location": { "type": "string" } }, "required": ["location"] } } } ] } ``` **Round 1, model response requesting a tool call:** ```json { "choices": [{ "finish_reason": "tool_calls", "message": { "role": "assistant", "tool_calls": [{ "id": "call_abc123", "type": "function", "function": { "name": "get_weather", "arguments": "{\"location\": \"NYC\"}" } }] } }] } ``` **Round 2, follow-up request with the function result:** ```json { "model": "gpt-4o", "messages": [ {"role": "user", "content": "What's the weather in NYC?"}, { "role": "assistant", "tool_calls": [{"id": "call_abc123", "type": "function", "function": {"name": "get_weather", "arguments": "{\"location\": \"NYC\"}"}}] }, { "role": "tool", "tool_call_id": "call_abc123", "content": "{\"temperature\": 72, \"condition\": \"sunny\"}" } ], "tools": [...] } ``` ### Anthropic format Use the `tools` array in the Anthropic Messages API (`/v1/messages`). **Round 1, request with tool definition:** ```json { "model": "claude-sonnet-4-20250514", "max_tokens": 1024, "messages": [{"role": "user", "content": "What's the weather in NYC?"}], "tools": [ { "name": "get_weather", "description": "Get the current weather for a city", "input_schema": { "type": "object", "properties": { "location": { "type": "string" } }, "required": ["location"] } } ] } ``` **Round 1, model response requesting tool use:** ```json { "stop_reason": "tool_use", "content": [{ "type": "tool_use", "id": "toolu_01abc", "name": "get_weather", "input": {"location": "NYC"} }] } ``` **Round 2, follow-up request with the tool result:** ```json { "model": "claude-sonnet-4-20250514", "max_tokens": 1024, "messages": [ {"role": "user", "content": "What's the weather in NYC?"}, {"role": "assistant", "content": [{"type": "tool_use", "id": "toolu_01abc", "name": "get_weather", "input": {"location": "NYC"}}]}, {"role": "user", "content": [{"type": "tool_result", "tool_use_id": "toolu_01abc", "content": "{\"temperature\": 72, \"condition\": \"sunny\"}"}]} ], "tools": [...] } ``` :::tip Function Calling vs. MCP Native function calling (described here) is application-side: your code defines the tools, detects tool call responses, executes the function, and sends the result back in a second request. [MCP Profiles](/agent-router-service/guides/aggregate-mcp-servers-into-a-profile/) provide gateway-side tool execution: Agent Router connects models to external MCP servers directly, so your application sends a single request and receives a final answer without implementing the tool execution loop. ::: ## Known limitations Current constraints Cross-provider structured output translation is not yet supported. Structured outputs currently work in passthrough mode only; the request format must match the target provider's native schema. Translation is coming in an upcoming Tetrate Agent Router release (upstream: envoyproxy/ai-gateway#1846). Embeddings, images, and rerank endpoints accept requests in passthrough mode only. No provider translation is applied; the request must already match the target provider's native format. Function calling cross-provider translation follows the same support matrix as the base API formats. Routing an OpenAI-format tool call request to an Anthropic backend is supported for the standard message format; check Gateway APIs for current coverage. Gateway APIs Full reference for all three API formats with streaming examples. MCP Profiles Gateway-side tool execution via MCP servers. Model Catalog Browse model capabilities and multimodal support across providers. --- ## Architecture overview Tetrate Agent Router is an AI gateway that provides unified, policy-controlled access to large language models across multiple providers. All AI traffic flows through a data plane. Where that data plane runs — Tetrate-hosted or in the customer's own infrastructure — depends on the deployment model. This page builds the mental model of how Agent Router is put together, starting from first principles and defining each term as it appears. No prior familiarity with Tetrate Agent Router is assumed. The deeper topic pages linked at the end expand on each area once the overall shape is clear. The material here is conceptual: it explains what the pieces are and why they are arranged as they are, not how to install or configure them. Readers who want to start using Agent Router can move straight to the [Console quickstart](/agent-router-service/quickstarts/quickstart-console/); readers choosing a deployment path should see [Deployment models](/product-architecture/deployment-models/). New to the vocabulary? The Key concepts page introduces the platform's terms in narrative form, and the Glossary is a quick lookup for any single word. This page defines terms inline as they come up. ## What an AI gateway is A large language model (LLM) is an AI model that takes a piece of text (a *prompt*) and returns generated text (a *response*). Models are hosted by *providers*: companies and services such as OpenAI, Anthropic, Google, Azure, Amazon, and Perplexity, as well as models an organisation runs itself (*self-hosted* models). Each provider exposes its own API, its own credentials, and its own endpoints. Without a gateway, every application wires itself directly to each provider it uses. That means scattered API keys, per-provider integration code, no single place to see what is being spent, and no consistent way to apply security or routing rules. Swapping or adding a provider becomes an application change. An AI gateway is a single entry point that sits between the applications making requests and the providers serving them. Applications send every request to the gateway instead of to a provider directly. The gateway decides which provider and model should handle each request, attaches the right credentials, applies policy (rate limits, content rules, access control), records what happened, and forwards the request. To the application, it looks like one endpoint that speaks a familiar API; behind it, the gateway can reach 200+ models across every supported provider. Adding, swapping, or removing a provider becomes a configuration change rather than an application rewrite. The effect is easiest to see in a concrete case. Suppose an application asks for a general-purpose chat model. The gateway can send that request to one provider today and a cheaper or higher-quality provider tomorrow, retry against a second provider if the first is overloaded, redact sensitive text before it is forwarded, and reject the request outright when the calling key exceeds its rate limit. None of that logic lives in the application, and none of it requires an application change: the request keeps arriving at one endpoint, while the decisions move into configuration held centrally. The same central position is what makes consolidated spend reporting and a single audit trail possible, because every request is observed in one place. Tetrate Agent Router is that gateway, plus the applications and services used to configure and operate it. It ships in two forms that share the same core: Agent Router Service, a self-serve tier for individual developers, and Agent Router Enterprise, which adds team-level cost attribution, guardrails, single sign-on, and a choice of data-plane hosting (Tetrate-operated Agent Router Enterprise Fully Managed, or Agent Router Enterprise Self-Hosted Data Plane where the customer runs the data plane in its own infrastructure). This documentation describes the Enterprise architecture. ## Component overview The diagram below shows how the main applications sit above a shared backend layer, which in turn connects to the AI gateway routing traffic to upstream providers. Three layers appear in the diagram, from top to bottom: - **The two applications** are the human-facing surfaces. The **Developer Console** (the Console) is where developers issue credentials, define routing rules, and watch their own traffic. The **Admin Dashboard** is where operators provision models and providers, manage users, and set the policies developers work within. Both are web applications served from the same backend, and both are read-and-write surfaces onto the same configuration. What differs is scope: the Console shows a developer only their own keys, routes, and usage, while the Admin Dashboard sees every user and every setting across the deployment. - **The shared backend** stores all configuration (routing rules, policies, provider settings, user accounts) and serves it to the rest of the system. It is the single source of truth: a change saved in either application is recorded here first, and every other component reads from it rather than holding its own copy. The **Controller** shown here is the component that carries that configuration down to the gateway. - **The AI Gateway** is the component that every request actually passes through. It routes each request to a provider and applies policy inline, meaning as part of handling the request rather than in a separate step. Because policy runs inside the request path, a blocked or redacted prompt is stopped before it reaches a provider, not flagged after the fact. ## The split-plane model Those layers do not all run in the same place. Agent Router divides into two halves, called *planes*, that run in different environments and have different jobs. - The **management plane** is hosted and operated by Tetrate. It holds the configuration and exposes the two web applications. Think of it as the control room: it is where decisions are recorded, but no live AI traffic passes through it. Because it is fully managed, upgrades, availability, and scaling of the control surfaces are Tetrate's responsibility, not the customer's. - The **data plane** handles all AI traffic — every prompt and every response passes through it, and nowhere else. In fully hosted deployments ([Agent Router Service](/agent-router-service/quickstarts/) or [Agent Router Enterprise Fully Managed](/agent-router-enterprise/fully-managed/)), Tetrate operates the data plane. In [Agent Router Enterprise Self-Hosted Data Plane](/agent-router-enterprise/self-hosted-data-plane/), you deploy it inside your own Kubernetes cluster (Kubernetes is the industry-standard system for running containerised software). Where the data plane runs is what distinguishes the deployment models. In fully hosted options, Tetrate runs the data plane as well — the fastest way to start. In Agent Router Enterprise Self-Hosted Data Plane, you run the data plane in your cluster while Tetrate continues to host the management plane; that is what keeps traffic inside your boundary. All three paths are compared in [Deployment models](/product-architecture/deployment-models/). The diagram below shows the split for **Agent Router Enterprise Self-Hosted Data Plane** (data plane in the customer domain). In Agent Router Service and Agent Router Enterprise Fully Managed the same components run with the data plane in Tetrate's environment. ![data-paths](/img/screenshots/about/architecture/TARE_split-pane.png) The separation is the central idea of the architecture. In Agent Router Enterprise Self-Hosted Data Plane deployments, **prompt and response data never leaves your environment** because the data plane runs in infrastructure you control; the management plane only ever holds configuration, never traffic. In fully hosted deployments, Tetrate operates both planes. The two planes still have to stay in sync, and they do so over a single connection with a deliberate direction. The data plane reaches out to the management plane to pull down the latest configuration: this is *outbound* from the data plane. In Agent Router Enterprise Self-Hosted Data Plane that means outbound from the customer's environment; the management plane never initiates a connection *into* the customer's environment. This "outbound-only" design means the customer's firewall needs no inbound openings for Agent Router, which is simpler to approve and reduces the ways in which the environment can be attacked. Split-plane model, management plane, data plane, and the outbound-only connection are covered in depth in Planes and core components. ## Core components of the data plane The data plane is not a single program. It is a small set of components that run together wherever the data plane is hosted (in the customer's cluster for Agent Router Enterprise Self-Hosted Data Plane, or on Tetrate infrastructure for Agent Router Service and Agent Router Enterprise Fully Managed). | Component | What it does, in plain terms | | :---- | :---- | | **Agent Router (the AI Gateway)** | The proxy every request passes through. A *proxy* is a server that receives a request, does something with it, and forwards it on. This one chooses the provider, attaches credentials, applies policy, and forwards the request. It is built on Envoy, a widely used open-source proxy. | | **Controller** | The bridge between the two planes. It repeatedly asks the management plane for the latest configuration and translates it into settings the gateway understands, keeping routing rules, policies, and credentials current. | | **Dynamic modules** | Add-on logic that runs *inside* the gateway rather than as a separate service, so it adds no extra network hop. Modules are compiled shared libraries (written in Rust or Go) loaded into the gateway's filter chain, and they provide capabilities such as routing decisions, cost controls, and guardrails. | | **Redis** | A fast in-memory datastore the gateway uses to count requests, so that rate limits are enforced consistently even when several copies of the gateway run at once. | These components run in dedicated Kubernetes namespaces, keeping Agent Router's resources separate from other workloads on the same cluster. Because the gateway is horizontally scalable, more copies can be added to handle load; the shared Redis counter is what lets those copies enforce a single rate limit between them rather than each counting in isolation. An optional remote guardrail provider (for example the shipped `tetrate` ML provider, or an external guardrail service where one is configured) can be connected to the gateway for advanced content analysis; that callout lives outside the cluster and is only involved when a rule selects such a provider. ## How a request flows end to end Bringing the pieces together, here is the path of a single request once Agent Router is running: 1. An application sends a request to the gateway's endpoint, using an API shape it already supports (Agent Router is OpenAI-compatible, so existing code often works unchanged). The request carries an *API key*, a credential that identifies the caller. 1. The gateway authenticates the key and identifies which user or team the request belongs to. 1. Routing rules decide which provider and model should serve the request. If a *fallback policy* is configured (an ordered list of backends to try), the gateway knows what to try next should the first choice fail. 1. Guardrails inspect the prompt. A *guardrail* is a content rule, for example removing personally identifiable information or blocking a disallowed request, enforced inline before the prompt is forwarded. 1. Rate limits are checked, so that no key exceeds its allotted tokens or requests. A *token* is the unit of text LLMs measure work in; roughly a few characters each. 1. The gateway attaches the provider credential and forwards the request. If that provider returns an error or is unreachable, the gateway walks to the next backend in the fallback chain, within the same request, with no action needed from the application. 1. The provider's response returns through the gateway, where guardrails can inspect it again, and details of the exchange are recorded as *telemetry* (operational data such as latency, token counts, and cost) before the response is returned to the application. When the caller requests a streamed response, the gateway relays it token by token as the provider produces it, rather than waiting for the full text. Every step above happens inside the data plane. The management plane supplied the rules beforehand but sees none of this traffic. Where the data plane runs depends on the [deployment model](/product-architecture/deployment-models/). Two behaviours make this flow easier to operate. First, errors from different providers are normalized into a consistent shape, so an application handles one error format regardless of which backend was reached. Second, every request is tagged with a correlation identifier returned in the response, which ties an application-side failure to the exact exchange recorded in telemetry and makes a single request traceable end to end. Both are detailed in [Gateway behavior](/reference/gateway-behavior/). ## Key terms The vocabulary introduced above, collected for reference: | Term | Meaning | | :---- | :---- | | **Large language model (LLM)** | An AI model that turns a text prompt into a generated text response. | | **Provider** | A service that hosts models (OpenAI, Anthropic, Google, Azure, Amazon, Perplexity), or a model the organisation self-hosts. | | **Prompt / response** | The text sent to a model, and the text it returns. | | **AI gateway** | The single entry point requests pass through, which routes and applies policy before reaching a provider. | | **Management plane** | The Tetrate-hosted half that stores configuration and serves the web applications. Holds no traffic. | | **Data plane** | The half that all AI traffic flows through. Tetrate-operated in Agent Router Service and Agent Router Enterprise Fully Managed; customer-hosted in Kubernetes for Agent Router Enterprise Self-Hosted Data Plane. | | **Controller** | The data-plane component that pulls configuration from the management plane and applies it to the gateway. | | **Routing** | Choosing which provider and model serves a given request. | | **Fallback policy** | An ordered list of backends the gateway tries in turn when one fails. | | **Guardrail** | An inline content rule (for example PII redaction or blocking) applied to prompts and responses. | | **Rate limit** | A cap on the tokens, requests, or concurrency a key may consume. | | **Telemetry** | Operational data (latency, token counts, cost, logs) recorded about traffic and exported to monitoring tools. | | **Bring Your Own Key (BYOK)** | Using a caller's own provider credentials, so that spend is settled with the provider directly. See [Use your own provider credentials](/agent-router-service/guides/use-your-own-provider-credentials/). | | **Model Context Protocol (MCP)** | A standard for connecting AI agents to external tools, which the gateway can aggregate behind one endpoint. See [Aggregate MCP servers into a profile](/agent-router-service/guides/aggregate-mcp-servers-into-a-profile/). | ## Explore the architecture Planes and core components The split-plane model, the operating responsibilities of each plane, and the data-plane components that route traffic. Data flows The configuration, request, and observability paths, and how they stay isolated from one another. Guardrails How the Guardrails dynamic module inspects and enforces policy inline at the gateway. Deployment models Agent Router Service, Agent Router Enterprise Fully Managed, and Agent Router Enterprise Self-Hosted Data Plane compared. Network and security Private connectivity, network isolation, transport security, and WAF policies. Operations and reliability Control-plane resilience, multi-region deployment, scaling, and per-request fallback. What's next Deployment models Choose Agent Router Service, Agent Router Enterprise Fully Managed, or Agent Router Enterprise Self-Hosted Data Plane. Agent Router Enterprise Fully Managed Tetrate hosts both planes; nothing to install. Agent Router Enterprise Self-Hosted Data Plane prerequisites Verify access requirements before installing a customer-managed data plane. Console quickstart Create an API key and make your first routed AI request. Admin quickstart Review models, providers, and usage as an operator. Gateway behavior Error normalization, correlation IDs, and the request debugging workflow. --- ## Data flows Tetrate Agent Router moves three different kinds of data, and each kind travels its own route. The three routes are configuration polling (how the gateway learns its rules), the prompt and response request path (the live AI traffic itself), and the observability pipeline (the operational data recorded about that traffic). Keeping them separate means a configuration update, a live AI request, and a telemetry export never share a channel, which makes the system easier to secure, to reason about, and to operate. This page explains each of those three flows from first principles, defining every term as it appears. No prior familiarity with Tetrate Agent Router is assumed. A *data flow* is simply a category of information that moves between two points in a predictable direction: from where it is produced to where it is consumed. Understanding the three flows, and why they are deliberately kept apart, is the clearest way to understand how the platform protects sensitive content while still staying centrally configurable and fully observable. These flows cross the two halves of the platform (the data plane and the management plane) that are introduced in Planes and core components. That page is useful background, but the essentials are restated here so this page can be read on its own. ## The two places data moves between Before the flows themselves, two terms are worth fixing in place, because every flow is described in relation to them. - The **data plane** is the part of the platform that handles all live AI traffic — every prompt and every response passes through it, and nowhere else. Its central component is the **AI gateway**, the server that every request travels through on its way to a provider. Where the data plane runs depends on the deployment model: in [Self-Hosted Data Plane](/agent-router-enterprise/self-hosted-data-plane/) it sits inside the customer's own Kubernetes cluster (Kubernetes is the industry-standard system for running containerised software); in fully hosted paths (Service or Enterprise Fully Managed), Tetrate operates it. The three paths are compared in [Deployment models](/product-architecture/deployment-models/). - The **management plane** is hosted and operated by Tetrate in every deployment model. It stores all configuration (routing rules, policies, provider settings, user accounts) and exposes the two web applications used to manage the platform. No live AI traffic ever passes through the management plane. A *provider* is a service that hosts large language models, for example OpenAI, Anthropic, Google, Azure, Amazon, or Perplexity. A *large language model (LLM)* is an AI model that takes a piece of text (a *prompt*) and returns generated text (a *response*). The diagram below shows all three flows spanning both planes for **Enterprise Self-Hosted Data Plane**, where the data plane runs in the customer's environment (Customer Managed) and Tetrate hosts only the management plane. Fully hosted deployments keep the same three flows, but the data plane sits in Tetrate's environment instead. ![data-paths](/img/screenshots/admin/data-flow-01.png) ## Configuration flow: how the gateway learns its rules The configuration flow carries the rules the gateway needs in order to do its job: which providers exist, which models are available, how requests should be routed, what spending and content policies apply, and which credentials to use. Those rules are decided and stored in the management plane, but they have to reach the gateway in the data plane before they can take effect. This is handled by the **Controller**, a component that runs alongside the gateway in the data plane and acts as the bridge between the two planes. The Controller performs *configuration polling*: on a regular schedule it reaches out to the management plane, asks for the latest configuration, and receives whatever has changed. *Polling* means the data plane repeatedly asks for updates on its own timer, rather than waiting for the management plane to push them. Once it has the update, the Controller translates it into settings the gateway understands and applies them, keeping routing rules, policies, and credentials current. The direction of this connection is deliberate and important. It is *outbound only*, meaning the data plane always initiates the connection out to the management plane, and the management plane never initiates a connection the other way. In Self-Hosted Data Plane deployments that means the customer's firewall (the network barrier that controls what may enter and leave their environment) needs no inbound openings for Tetrate Agent Router at all. A firewall that only has to allow outbound connections is simpler to approve, and it reduces *attack surface*: the number of ways an outside party could try to reach into the environment. Because inbound access is never required, there is simply nothing there to attack. One consequence of this design is worth noting: the data plane keeps working even if the connection to the management plane is briefly interrupted. It continues routing requests using the configuration it already holds, and it picks up any changes on the next successful poll. The Controller and its role as the bridge between the planes are described further in Planes and core components. ## Request flow: the live prompt and response traffic The request flow is the live AI traffic itself, and it is the only one of the three flows that carries the actual content of prompts and responses. Requests originate from *client applications* (the programs an organisation runs that need to call an AI model), pass through the gateway, and are forwarded to the selected provider. Responses travel the same route in reverse. Along the way, the gateway does several things to each request in turn, all inside the data plane: 1. It authenticates the request and identifies which caller it belongs to, using the *API key* the request carries. An API key is a credential that names and authorises the caller. 1. It applies *routing decisions*: choosing which provider and model should serve the request, based on the configuration delivered by the configuration flow. 1. It enforces *rate limiting*. A *rate limit* is a cap on how much a given key may consume in a period, measured in requests or in tokens (a *token* is the unit of text that LLMs measure work in, roughly a few characters each). Rate limiting protects both the provider and the customer's budget from a single caller sending far more traffic than intended. 1. It applies *guardrails*. A *guardrail* is a content rule enforced inline on model traffic, for example removing personally identifiable information from a prompt or blocking a disallowed request. Guardrails can inspect the response on the way back as well. They do not evaluate standalone MCP profile traffic. 1. It attaches the correct provider credential and forwards the request to the chosen provider. If that provider fails or is unreachable, the gateway can walk to the next backend in a configured fallback order, within the same request, with no action needed from the application. The defining property of this flow is that it never passes through the management plane. In Self-Hosted Data Plane deployments, prompt and response content also never leaves the customer's environment except to reach the provider that was selected to serve the request: because the gateway runs in infrastructure the customer controls, sensitive content (which can include proprietary code, customer records, or internal documents) stays inside the customer's network boundary. In fully hosted deployments the same separation holds — traffic stays on the data plane and never reaches the management plane — but the data plane itself is operated by Tetrate. Keeping live content off the management plane is the central reason the platform separates the planes at all. What happens to a request inside the gateway is covered step by step in Guardrails, and the gateway's normalised errors and per-request identifiers are documented in Gateway behavior. ## Observability flow: the operational record of traffic The observability flow carries the operational record of what the platform is doing, so that it can be watched, measured, and alerted on. It carries no prompt or response content by default; it carries facts *about* the traffic. Three kinds of data travel this flow: - **Telemetry** is the general term for operational data recorded about traffic, such as latency (how long a request took), token counts, cost, and the outcome of each request. - **Metrics** are numeric measurements aggregated over time, such as total requests, error rates, and latency percentiles. Metrics are the raw material for dashboards and alerting rules. - **Structured logs** are individual event records in a consistent, machine-readable format, so that tools can search and correlate them rather than parsing free text. The flow is assembled from two stages. First, a **Grafana Alloy** agent running locally in the data plane collects the observability data. Grafana Alloy is a widely used open-source collector: an agent that *scrapes* (periodically pulls) data from the components that produce it, in this case the gateway, the Controller, and the Redis instance the gateway uses for rate-limit counting. Second, the collected data is forwarded to an **OpenTelemetry (OTEL) Collector**. OpenTelemetry is an open, vendor-neutral standard for describing and transporting telemetry; the OTEL Collector is a routing hub that receives telemetry in that standard format and forwards it onward. From the OTEL Collector, the telemetry is routed to whichever *monitoring backends* the organisation already runs. A monitoring backend is a platform that stores telemetry and turns it into dashboards, searches, and alerts. Common examples are **Grafana** (open-source dashboards and visualisation), **Datadog** (a hosted monitoring platform), and **New Relic** (a hosted observability platform). Because the platform emits telemetry in the open OpenTelemetry format rather than a proprietary one, it slots into the tooling an observability team already operates, instead of forcing a second, separate console. The point of this two-stage design (a local collector that scrapes, feeding a collector that routes) is that the platform does not decide where telemetry ultimately lives. It emits telemetry in an open format and lets the organisation's existing stack absorb it, which is why storage, retention, and alerting rules stay with the destination platform rather than inside Tetrate Agent Router. ![data-paths](/img/screenshots/admin/data-flow-02.png) The operational detail of reading this data lives in Monitor traffic and usage (the in-Console surfaces) and Export telemetry to an observability stack (sending it to an external backend). The exact metrics and span attributes are listed in OpenTelemetry metrics. ## The three flows at a glance The flows differ in direction, in what crosses them, and in who consumes the result. The request flow is the only one that carries prompt and response content, and it never touches the management plane. In Self-Hosted Data Plane deployments it also stays inside the customer environment except to reach the selected provider. Reading the table by column shows why the separation matters: no single channel mixes configuration, live content, and operational data, so each can be secured and scaled on its own terms. | Flow | Direction | Carries | Consumed by | Why it is separate | | :---- | :---- | :---- | :---- | :---- | | Configuration | Data plane to management plane, outbound only | Policy, routing, and credential updates | The Controller, which applies them to the gateway | Keeps the gateway in sync; in Self-Hosted, with no inbound connections into the customer environment | | Request | Client to gateway to provider, and back | Prompts and responses (sensitive content) | The client application, via the provider | Isolates live content on the data plane so it never touches the management plane | | Observability | Data plane to the organisation's monitoring backends | Telemetry, metrics, structured logs | Grafana, Datadog, New Relic, and similar | Lets existing monitoring tools absorb operational data without carrying live traffic | The configuration and observability flows are covered in operational detail in [Monitor traffic and usage](/agent-router-service/guides/monitor-traffic-and-usage/) and [Export telemetry to an observability stack](/agent-router-enterprise/guides/observability-and-analytics/export-telemetry-to-an-observability-stack/). ## Where to go next - **Planes and core components**: the planes these flows cross and the components that produce them. See [Planes and core components](/product-architecture/planes-and-components/). - **Guardrails**: what happens to a request inside the gateway on the request flow. See [Guardrails](/product-architecture/guardrails/). Where to go next Planes and core components The planes these flows cross and the components that produce them. Guardrails What happens to a request inside the gateway on the request flow. --- ## Deployment models Tetrate Agent Router can be run in three get-started paths. All offer the same gateway and feature set. What changes is where the data plane runs — the part that carries AI traffic — and who operates that infrastructure. Fully hosted paths are the fastest way to start; Self-Hosted Data Plane keeps all AI traffic inside your environment while Tetrate still hosts the management plane. This page explains deployment models from first principles, defining each term as it appears. No prior familiarity with Tetrate Agent Router or cloud infrastructure is assumed. Once the shape of each option is clear, the linked pages at the end go deeper on networking, security, and operations. In every Enterprise deployment, Tetrate hosts and operates the management plane. "Self-hosted" means only the data plane runs in your Kubernetes cluster. You do not install, patch, or scale the management plane — the Console, Admin Dashboard, and configuration backend stay with Tetrate. Deployment model is separate from product tier. Agent Router Service and Agent Router Enterprise describe which features an account has (see the welcome page). This page describes where the software runs and maps to the three get-started paths: Agent Router Service, Agent Router Enterprise Fully Managed, and Agent Router Enterprise Self-Hosted Data Plane. ## The one idea to hold on to first Tetrate Agent Router is built in two halves, called *planes*, that run in different places and do different jobs. - The **management plane** is the control room. It stores configuration (routing rules, policies, provider settings, user accounts) and serves the two web applications operators and developers use. No live AI traffic passes through it. **In every deployment model, the management plane is hosted and operated by Tetrate.** - The **data plane** is the road the traffic travels. Every prompt sent to a model and every response returned passes through it, and nowhere else. The data plane is where deployment models differ. *Prompt* here means the text an application sends to an AI model, and *response* means the text the model returns. Because everything sensitive rides on the data plane (which can include proprietary code, customer records, or internal documents), where the data plane runs is the question that matters most for security and compliance. The split-plane model is covered in full in [Planes and core components](/product-architecture/planes-and-components/). The rest of this page is, in effect, the answer to one question: *where should the data plane run?* ## Three paths and who runs each plane | Get-started path | Tier | Management plane | Data plane | Customer installs | | :---- | :---- | :---- | :---- | :---- | | [Agent Router Service](/agent-router-service/quickstarts/) | Service | Tetrate | Tetrate | Nothing | | [Agent Router Enterprise Fully Managed](/agent-router-enterprise/fully-managed/) | Enterprise | Tetrate | Tetrate | Nothing | | [Agent Router Enterprise Self-Hosted Data Plane](/agent-router-enterprise/self-hosted-data-plane/) | Enterprise | **Tetrate** | **Your Kubernetes cluster** | Data plane + gateway only | **Agent Router Enterprise Self-Hosted Data Plane** is a hybrid deployment: Tetrate hosts the management plane; you install only the data plane in your Kubernetes cluster. Service and Fully Managed are both fully hosted — Tetrate runs both planes. They differ by tier (features and scale), not by where software runs. Self-Hosted Data Plane is the only path where you operate infrastructure, and that infrastructure is the data plane alone. ## What you host vs what Tetrate hosts | Component | Fully Managed (Enterprise) | Self-Hosted Data Plane (Enterprise) | | :---- | :---- | :---- | | Management plane (Console, Admin Dashboard, config) | Tetrate | **Tetrate** | | Data plane (gateway, Controller) | Tetrate | **Your Kubernetes** | | Prompt and response traffic | Tetrate infrastructure | **Your environment** | | Customer installs | None | Data plane + gateway only | ## The two Enterprise deployment models at a glance Enterprise offers two deployment models. Both use the same Enterprise feature set; both keep the management plane with Tetrate. Agent Router Enterprise · Fully Managed Management plane: Tetrate · Data plane: Tetrate Tetrate runs both planes. There is nothing for you to install or operate. The gateway is reachable as soon as onboarding completes. Hosted by Tetrate end to end; nothing to provision or install Operational from the moment onboarding completes The model catalog is kept current by Tetrate as providers and models are added Suited to teams that want Enterprise features without running infrastructure Agent Router Enterprise · Self-Hosted Data Plane Management plane: Tetrate · Data plane: your cluster You install the data plane in your Kubernetes cluster. The management plane remains hosted and operated by Tetrate. This suits organisations with data-residency requirements, private network topologies, or a need for deeper control over where AI traffic runs. Data plane deployed in your Kubernetes cluster, in any cloud or on-premises Management plane hosted and operated by Tetrate — no management-plane infrastructure to maintain All prompts and responses stay within your environment and never transit Tetrate systems Full control over model availability, provider credentials, network policy, and data residency Supports private provider connectivity to keep model traffic off the public internet **Self-hosted data plane does not mean:** - You run the Admin Dashboard or Console on your infrastructure - You operate the management-plane backend - Prompts flow through Tetrate-hosted gateway infrastructure **Self-hosted data plane does mean:** - You run the gateway and Controller in your cluster - Tetrate still hosts SSO configuration, the model catalog, users, and audit UI - The data plane pulls configuration outbound from the management plane; Tetrate never connects into your environment ## Agent Router Service (fully hosted) [Agent Router Service](/agent-router-service/quickstarts/) is the Service-tier fully hosted option. Tetrate runs the entire platform — both planes. An account is created and the gateway is reachable straight away at a Tetrate-hosted address. There is no cluster to stand up, no software to install, and no infrastructure to keep patched or scaled. Because the data plane is operated by Tetrate, prompts and responses are processed on Tetrate-hosted infrastructure rather than inside your network boundary. For many teams that is the right trade: the gateway works immediately without the infrastructure Self-Hosted Data Plane requires. Service suits individual developers, small teams, evaluations, and workloads without strict requirements about where data is processed. ## Enterprise Fully Managed in depth In the Fully Managed model, Tetrate runs both planes. You configure the management plane through onboarding and operate from the Admin Dashboard or Console. Applications point at the Tetrate-hosted gateway endpoint and send requests using an API shape they already support (Agent Router is OpenAI-compatible, so existing code often works unchanged). Fully Managed suits Enterprise teams that want the full feature set without deploying or operating a data plane in their own cluster. ## Enterprise Self-Hosted Data Plane in depth In the Self-Hosted Data Plane model, only the data plane runs on infrastructure you control. The management plane stays with Tetrate. This is the option when the deciding question is not "does it work" but "where exactly does our traffic go, and who can touch it." A few terms make this model easier to follow: - **Kubernetes** is the industry-standard system for running containerised software (applications packaged so they run the same way anywhere). The data plane is deployed into a Kubernetes *cluster*, meaning a group of machines Kubernetes manages as one. That cluster can sit in any major cloud or in your own data centre. - **On-premises** (often shortened to on-prem) means running software on infrastructure the organisation owns and operates itself, typically in its own data centre, rather than on a cloud provider's infrastructure. Self-Hosted Data Plane supports on-premises deployment of the data plane. - **Data residency** is the requirement that data be processed and stored within a specific geographic region (for example, keeping European traffic inside the European Union). Running the data plane in a location you choose is how a residency requirement is satisfied at the infrastructure level. Residency can also be enforced at the routing level; see [Compliance](/reference/compliance/). Because the data plane sits inside your environment, every prompt and every response stays inside that boundary and never transits Tetrate systems. The management plane still supplies the configuration, but it never sees the traffic. The two planes stay in sync over a single connection that the data plane opens *outbound* to the management plane to pull down the latest configuration; the management plane never opens a connection *into* your environment. This outbound-only design means your firewall needs no inbound openings for Agent Router. The connection direction and network posture are covered in [Network and security](/product-architecture/network-and-security/). Self-Hosted Data Plane also unlocks capabilities that depend on running inside your network: - **Private connectivity** keeps traffic to AI providers off the public internet by routing it over the cloud provider's own private backbone instead. Each major cloud offers this under its own name: Azure Private Link, AWS PrivateLink, and Google Cloud Private Service Connect (GCP is Google Cloud Platform). When the data plane runs inside one of these clouds, a provider can be configured to be reached over the matching private-connectivity option. See [Network and security](/product-architecture/network-and-security/). - **Chargeback** is the practice of attributing AI spend back to the team, project, or cost centre that incurred it, so internal costs can be settled or reported. In Self-Hosted Data Plane there is no payment processing to run; usage is tracked for internal audit and chargeback rather than for billing by Tetrate. - **Guardrails** are inline content rules, for example redacting personally identifiable information or blocking a disallowed request, applied to prompts and responses as they pass through the gateway. Because the guardrails run inside your own data plane, sensitive content is inspected without leaving the environment. Guardrails are covered in [Guardrails](/product-architecture/guardrails/). - **Bring Your Own Key (BYOK)** means using the caller's own provider credentials, so that spend is settled directly with the provider rather than through the platform. This works transparently with the routing features in Self-Hosted Data Plane. Self-Hosted Data Plane suits organisations with data-residency or regulatory requirements, teams that require AI traffic on private networks, and deployments that need control over model availability, provider credentials, and network policy. ## Comparing deployment models The management plane is hosted by Tetrate in every model. Differences follow from where the data plane runs. | Dimension | Agent Router Service | Enterprise Fully Managed | Enterprise Self-Hosted Data Plane | | :---- | :---- | :---- | :---- | | Tier | Service | Enterprise | Enterprise | | Management plane operator | Tetrate | Tetrate | **Tetrate** | | Where the data plane runs | Tetrate-hosted infrastructure | Tetrate-hosted infrastructure | Your Kubernetes cluster (any cloud or on-premises) | | Who operates the data plane | Tetrate | Tetrate | You | | Where prompts and responses are processed | On Tetrate-hosted infrastructure | On Tetrate-hosted infrastructure | Inside your environment; never transits Tetrate systems | | Data residency control | Not customer-selected at the infrastructure level | Not customer-selected at the infrastructure level | Data plane runs in a location you choose | | Private connectivity to providers | Not applicable | Not applicable | Supported where the cloud offers it (Azure Private Link, AWS PrivateLink, GCP Private Service Connect) | | Setup effort | None; operational once the account is activated | Onboarding only; no data plane install | Onboarding plus data plane and gateway install in your cluster | | Best suited to | Individual developers, fast starts, evaluations | Enterprise features without running a data plane | Data residency, private networks, infrastructure control | Whichever model is chosen, the gateway, its APIs, and the way applications call it stay the same. Moving from one model to another is a change in where the data plane runs, not a change to the applications built on top of it. ## Which path should I choose? - **Need to start quickly with no infrastructure?** → [Agent Router Service](/agent-router-service/quickstarts/) or [Agent Router Enterprise Fully Managed](/agent-router-enterprise/fully-managed/) - **Need Enterprise features but don't want to run a cluster?** → [Agent Router Enterprise Fully Managed](/agent-router-enterprise/fully-managed/) - **Need prompts and responses to stay in your environment?** → [Agent Router Enterprise Self-Hosted Data Plane](/agent-router-enterprise/self-hosted-data-plane/) — management plane still Tetrate-hosted; you install the data plane only Where to go next Agent Router Enterprise Fully Managed Tetrate hosts both planes. Configure onboarding and operate from the Console. Agent Router Enterprise Self-Hosted Data Plane Tetrate hosts the management plane; you install the data plane in your cluster. Network and security Private connectivity, isolation, and transport security for a customer-managed data plane. Operations and reliability How a customer-managed data plane stays available and scales. Planes and core components The split-plane model and the data-plane components that route traffic. Compliance Data-residency options, certifications, and upstream-provider retention settings. --- ## Guardrails A guardrail is a named project policy that inspects model traffic as it passes through the gateway and can change or stop that traffic before it reaches a model, or before a model's reply reaches the application. In Tetrate Agent Router, guardrails run as a dynamic module inside the gateway rather than as safety code written into each application. Every model request that crosses the gateway passes through the module, which evaluates configured rules, and acts on the verdict before the request reaches a model provider. The same inspection applies to the response on the way back. Standalone Model Context Protocol (MCP) profile traffic (`/mcp/*`) does not pass through content guardrails; it is governed by identity, access control, and audit instead. This page explains how that inspection works from first principles, defining each term as it appears. No prior familiarity with Tetrate Agent Router, with proxies, or with the internals of an AI gateway is assumed. The operator and developer guides linked at the end cover how guardrails are configured and used once the underlying mechanism is clear. For the wider context this page sits in, the Architecture overview explains what the gateway is and where it runs, and Planes and core components describes the data-plane components (including the dynamic modules) in one place. The Glossary is a quick lookup for any single term. Three pages cover guardrails, and they answer different questions. This page explains *where the module sits and what that placement costs and buys*. [About guardrails](/agent-router-enterprise/guides/operate-and-govern/safety-and-data-protection/about-guardrails/) explains *how an operator reasons about guardrails, rules, patterns, and providers* as objects to configure. The [guardrails reference](/reference/guardrails/) is the lookup surface for the fixed vocabularies: the 23 check types, every rule-level setting and its permitted values, and the provider-by-check-type matrix. ## What a guardrail is, and why it runs at the gateway A large language model (LLM) takes a piece of text (a *prompt*) and returns generated text (a *response*). Prompts and responses are not always safe to pass along untouched. A prompt might contain Personally Identifiable Information (PII), meaning data that identifies a person, such as a name, an email address, a phone number, or a payment-card number. A prompt might also be an attack. A response might reproduce sensitive data, or return content the organisation does not permit. A *guardrail* is a named policy that addresses these risks. On its own it does nothing: its power lives in the **rules** it contains. Each rule names a **check type** (what to look for) and a **provider** (the engine that scores the content). Providers return a normalised score; the Guardrails module compares that score to a threshold and applies the rule's **action** (`block` or `redact`) when the rule's **mode** is enforce. The full vocabulary of check types, and every setting a rule carries, is tabulated in the [guardrails reference](/reference/guardrails/#rule-level-settings). Placing that policy at the gateway (on the model path every LLM request already passes through) rather than inside each application means one policy protects every application in the project at once, and applications need no safety code of their own. Guardrails are **bound to a project**, not to the organisation as a whole. The Admin Dashboard's project selector switches which set of guardrails, patterns, and guardrail providers is under management. A guardrail authored under one project does not apply to another. The rest of this page explains the machinery that makes that inspection possible, and then how a single request moves through it. ## Proxies, filters, and dynamic modules Understanding guardrails means understanding a little about how the gateway is built. A *proxy* is a server that sits between a client and a destination: it receives a request, does something with it, and forwards it on. The Agent Router gateway is a proxy. It receives each AI request, selects a model provider, applies policy, and forwards the request to that provider. It is built on Envoy, a widely used open-source proxy. A proxy like this handles a request through a *filter chain*: an ordered pipeline of small processing stages called *filters*. Each filter inspects or modifies the request in turn, and the request travels through the chain in sequence before being routed to its destination. A filter chain is how a proxy composes many independent behaviours (authentication, routing, rate limiting, safety checks) into one coherent path without any single filter needing to know about the others. Adding a new behaviour to that pipeline used to mean recompiling the proxy from C++, or accepting the slower execution of a WebAssembly (Wasm) or Lua extension. A *dynamic module* is the modern alternative: add-on logic written as a high-performance extension (often in the Rust or Go programming languages) and loaded into the proxy at runtime as a *shared library*, a compiled file (for example a `.so` file) that a program loads and runs as part of itself rather than as a separate process. Because the module runs inside the proxy, it adds no extra network hop and executes at close to native speed. Guardrails are one such dynamic module. Cost controls are another. The key property is *inline processing*: the module runs as part of handling the request, inside the same filter chain, rather than as a separate service the gateway has to call over the network and wait for. The Guardrails module is one of the data-plane components. Its place alongside the Controller and Redis is described in Planes and core components. Remote evaluation, when used, calls out to a configured guardrail provider (for example the shipped tetrate ML provider, or an external guardrail service where one is configured). ## How the Guardrails dynamic module works The module operates in five stages within the filter chain: interception, evaluation, enforcement, egress inspection, and recording. The first three apply to the prompt on its way to the model. The fourth applies the same logic to the response on its way back. The fifth is not sequential: a record is emitted for each rule evaluation, wherever in the request it occurred. ### Stage 1: interception (the prompt) When an application sends a prompt to an AI model, the request reaches the gateway first, not the model provider. The Guardrails module intercepts the HTTP *payload* (the payload is the actual content the request carries) before it is forwarded to the downstream model provider, whether that provider is an external service such as OpenAI or a model the organisation runs itself. A payload is not necessarily one block of prose. A rule declares which part of it to inspect, choosing between all content, plain text, an image, a document, tool output, the model's thinking, or retrieved context; the permitted values are listed in the [guardrails reference](/reference/guardrails/#rule-level-settings). The distinction matters because the surfaces differ in who controls them. Text is what the user typed, while tool output and retrieved context arrive from a backend and are read by the model with the same authority as the rest of the prompt, which is what makes them the vector for the indirect attacks described under evaluation below. Interception is what makes everything that follows possible: the module holds the prompt while it decides what to do, and nothing is forwarded until that decision is made. ### Stage 2: evaluation Once the module holds the payload, it runs the project's active guardrail rules. Each rule is scored by its **provider**: - Locally: engines such as `regexp` (named patterns from the Patterns catalog) and `builtin` (a multi-layer keyword and regex pipeline) run in process inside the gateway. Local checks add minimal latency because no network call is involved and the data never leaves the gateway. - Remotely, by callout: for analysis that local pattern matching cannot perform, a provider such as `tetrate` (or an external guardrail service, where one is configured) pauses the request, calls a remote endpoint, and resumes once the service replies. A callout is more capable but slower than a local check, because it involves a network round trip. **Providers score; the module decides.** A provider returns a normalised score in the range 0.0 to 1.0 (and, for redaction, a sanitised copy of the text). It never chooses to block or redact and never compares against a threshold. The module compares `score ≥ threshold` to decide whether the rule was violated. How each engine arrives at its score is covered in [About guardrails](/agent-router-enterprise/guides/operate-and-govern/safety-and-data-protection/about-guardrails/#guardrail-providers). Evaluation is bounded by the guardrail's **evaluation timeout**, and a provider can also simply fail. Either way the rule produces no score, and the guardrail's **failure mode** decides the outcome: fail-close, the default, treats the rule as violated and the content is blocked; fail-open skips the rule and the content passes. This is the most consequential piece of the design to hold in mind, because it means an unreachable remote provider stops traffic unless the guardrail was deliberately configured not to. Providers are not retried, so that deterministic behaviour is the gateway's alone. Two categories of threat are worth naming here, because they drive what evaluation looks for. - Prompt injection is text that tries to subvert the model rather than being unsafe in itself: instructions smuggled into the prompt (or into content the model is given to work with) that make the model ignore its own instructions, leak data, or act incorrectly. - A jailbreak is a direct form of prompt injection in the user's own prompt, the familiar "ignore previous instructions" family of attacks and role-play framings intended to escape the model's configured behaviour. ### Stage 3: enforcement (action and mode) When a rule is violated, two orthogonal fields decide what happens: - **Action:** `redact` alters the payload (for example replacing a detected credit-card number with a placeholder) and lets the request continue; `block` stops the request and returns HTTP `403` with an optional message and a correlation identifier, so the model provider never receives the request. - **Mode:** `enforce` applies the action; `monitor` records the would-be action and lets the content pass unchanged. Monitor mode is how a rule is validated against live traffic before enforcement, and a monitor rule is always effectively fail-open, because it never stops a request. A block ends evaluation immediately. A redaction does not: because the payload has changed, the module swaps in the sanitised text and re-runs any providers whose rules are still pending against the new content. One rule's redaction therefore determines what later rules see, which is why the order in which cheap deterministic engines and expensive remote ones are stacked has an effect beyond latency. If no rule is violated, the filter chain continues and the request is forwarded. ### Stage 4: egress inspection (the response) Guardrails are not limited to incoming prompts. *Egress* means traffic leaving toward its destination, and here it refers to the model's response on its way back to the application. When the model generates a response, that response flows back through the proxy, and the module inspects it using the same rule evaluation before it reaches the caller. Egress inspection is what catches a response that reproduces sensitive data or returns disallowed content, closing the gap that inspecting only the prompt would leave open. Egress inspection is not free, and its cost is paid in streaming. A model normally returns its answer token by token, and the gateway forwards each fragment as it arrives. A rule cannot judge content it has not seen in full, so if any rule on the output stage is in enforce mode, the response is buffered until it is complete, evaluated, and only then delivered. The caller receives one whole answer rather than a stream, and waits for the full generation before receiving anything. Where every output-stage rule is in monitor mode the trade does not arise: evaluation runs asynchronously and streaming is unaffected. Enforcing on the output stage is therefore a deliberate exchange of perceived latency for the ability to stop a response, and it is the one architectural decision on this page that end users notice directly. ### Stage 5: recording Every rule evaluation emits a **guardrail-trigger** record, whether the rule fired or not and whether its mode was enforce or monitor. The record carries the rule and guardrail identifiers, the score, the action and mode, the stage, the content type, the execution duration, and the project and consumer context. This is a distinct record type from the Audit Logs, which capture administrative mutations such as a change *to* a guardrail rather than the rule firing. Where those records go is a data-protection question in its own right, because a trigger record can contain the matched content, and the copy the data plane's own collector produces always does. The management-plane copy is the one that can be scoped, through the Admin Dashboard's **Settings → Guardrails** card, to full content, metadata only, or off. A backend subscribed to the `guardrailtrigger` signal receives full content regardless of that setting, by design, so that a strict management-plane setting cannot silently break an organisation's own security pipeline. [Custom observability backends](/agent-router-enterprise/guides/observability-and-analytics/configure-custom-observability-backends/#guardrail-trigger-destinations) covers both controls. ## The request path, end to end Putting the stages in order, a single guarded request moves like this: 1. An application sends a prompt to the gateway. The Guardrails module intercepts the payload before it is forwarded (interception). 1. The module evaluates active rules against the content types each one declares: providers score the content; the module compares scores to thresholds. A provider that times out or errors is resolved by the guardrail's failure mode (evaluation). 1. On a violation, the module applies the rule's action when mode is enforce, or records the would-be action when mode is monitor. A block ends the request; a redaction re-runs the providers with rules still pending against the sanitised content (enforcement). 1. If the request was allowed or redacted, the gateway forwards it to the model provider, and the response returns through the proxy. 1. The module inspects the response with the same logic before it reaches the application, buffering the response first where any output-stage rule enforces (egress inspection). 1. Each evaluation emits a guardrail-trigger record to the data plane's collector, and to the management plane subject to the configured content mode (recording). Every step happens inside the data plane, the environment all AI traffic flows through. Local checks keep the content inside it entirely. A remote provider call is the one point at which content is sent outside, and only when such a provider is configured and selected by a rule. Who operates that data plane depends on the deployment model, and this is the question to settle before treating the paragraph above as a residency guarantee. On Agent Router Service and Agent Router Enterprise Fully Managed, Tetrate runs the data plane, so prompts and responses are processed on Tetrate-hosted infrastructure. On Agent Router Enterprise Self-Hosted Data Plane, the data plane runs in the organisation's own Kubernetes cluster and the content never leaves it. See [Deployment models](/product-architecture/deployment-models/). ## Why guardrails run as a module, not per application Running guardrails inside a dynamic module rather than writing safety checks into each application has two consequences that motivate the design. - Centralised governance: every application that uses the project's gateway path is protected by the same rules automatically, with no per-application integration work. A rule changed once applies everywhere in that project the moment it takes effect. - Cost and safety at the edge: blocking a non-compliant or malicious prompt before it reaches the model means the model provider is never invoked for a request that would fail policy. The organisation does not pay for a call it never wanted to make, and unsafe content never reaches the model in the first place. ## Key terms The vocabulary introduced above, collected for reference. **Guardrail**, **Dynamic module**, and **Provider** are defined in the [Glossary](/reference/glossary/) alongside the rest of the Agent Router vocabulary, and the permitted values of every rule setting are in the [guardrails reference](/reference/guardrails/#rule-level-settings). | Term | Meaning | | :---- | :---- | | **Rule** | One check type plus action and mode, run by a chosen provider against input, output, or both. | | **Check type** | A name from a fixed vocabulary (`pii`, `toxicity`, `jailbreak`, and others): *what* a rule looks for. | | **Content type** | Which part of a payload a rule inspects: all, text, image, document, tool output, thinking, or context. | | **Pattern** | A named, reusable regex in a shared catalog that rules can reference by name. | | **Threshold** | Sensitivity in the range 0.0 to 1.0; the module flags a violation when the provider score is at or above it. | | **Action** | `block` or `redact`: what enforcement would do on a violation. | | **Mode** | `enforce` applies the action; `monitor` records the would-be action only. | | **Failure mode** | If a provider cannot evaluate a rule: fail-close (default) treats it as blocked; fail-open lets traffic pass. | | **Evaluation timeout** | The bound on how long evaluation may take before the failure mode decides the outcome. | | **Prompt / response** | The text sent to a model, and the text it returns. | | **Proxy** | A server that receives a request, processes it, and forwards it on. The gateway is a proxy, built on Envoy. | | **Filter / filter chain** | A single processing stage, and the ordered pipeline of such stages a request passes through in the proxy. | | **WebAssembly (Wasm)** | A portable sandboxed code format; an older, slower way to extend a proxy than a dynamic module. | | **Shared library** | A compiled file (for example a `.so`) that a program loads and runs as part of itself. | | **Inline processing** | Work performed as part of handling the request, rather than in a separate service called over the network. | | **Payload** | The content a request carries; here, the prompt or the response. | | **Personally Identifiable Information (PII)** | Data that identifies a person, such as a name, email address, phone number, or payment-card number. | | **Redaction** | Removing or masking matched content so the request can still proceed without the sensitive text. | | **Callout** | A pause in processing to call a remote guardrail provider and resume on its reply. | | **Prompt injection / jailbreak** | Text that tries to subvert the model; a jailbreak is a direct injection in the user's own prompt. | | **Egress** | Traffic leaving toward its destination; here, the model's response inspected on its way back to the application. | | **Guardrail-trigger record** | The record one rule evaluation emits, distinct from an Audit Log entry, which records a change to a guardrail. | ## What to do next The operational configuration and developer-side use of guardrails are covered in the guides below: - [About guardrails](/agent-router-enterprise/guides/operate-and-govern/safety-and-data-protection/about-guardrails/) is the operator's conceptual view of the same machinery: the objects, the rule form, and how each engine detects. - [Guardrails reference](/reference/guardrails/) is the lookup surface: the 23 check types, every rule-level setting, and the provider-by-check-type matrix. - [Configure guardrails](/agent-router-enterprise/guides/operate-and-govern/safety-and-data-protection/configure-custom-guardrails/) and [Configure guardrail providers](/agent-router-enterprise/guides/operate-and-govern/safety-and-data-protection/configure-vendor-guardrails/) set up the policies and engines the module enforces. - [Detect and redact sensitive data](/agent-router-enterprise/guides/operate-and-govern/safety-and-data-protection/detect-and-redact-sensitive-data/) and [Detect and block prompt injection](/agent-router-enterprise/guides/operate-and-govern/safety-and-data-protection/detect-and-block-prompt-injection/) cover common check workflows. - [Protect requests with guardrails](/agent-router-enterprise/guides/operate-and-govern/safety-and-data-protection/protect-requests-with-guardrails/) is the developer-side view. Where to go next Configure guardrails Set up the policies and rules the module enforces inline. Data flows Where guardrail evaluation sits on the request path through the gateway. --- ## Key concepts A handful of terms show up again and again across these docs. Some are common to any AI gateway; others are specific to how Tetrate Agent Router is built and run. Read it end to end the first time through. When you just need a quick definition later, the Glossary in the Reference section gives the one-line version. Structural Routing Credentials Agent infrastructure Observability ## Structural concepts These are the building blocks: the components of Agent Router itself and how they're deployed. ### Data plane and management plane Agent Router splits into two halves that talk over a small, well-defined interface. The **data plane** handles the request path. It contains the Controller and a proxy component (the AI gateway), and every AI request and response flows through it. The data plane receives application requests, evaluates routing policy, attaches the right credentials, calls the upstream AI provider, and returns the response. Where it runs depends on the deployment model: in [Agent Router Service](/agent-router-service/quickstarts/) and [Agent Router Enterprise Fully Managed](/agent-router-enterprise/fully-managed/), Tetrate operates it; in [Agent Router Enterprise Self-Hosted Data Plane](/agent-router-enterprise/self-hosted-data-plane/), you deploy it in a Kubernetes cluster you control, so traffic, payloads, prompts, completions, and credentials stay inside your network. The three paths are compared in [Deployment models](/product-architecture/deployment-models/). The **management plane** is the Tetrate-hosted control surface in every deployment model. It stores routing rules, policies, model and provider definitions, user records, API key metadata, audit history, and analytics rollups. The data plane pulls its configuration from here and pushes telemetry back, but no request payloads ever cross the boundary. You get a centrally managed control surface without operating that stack yourself. ### The Developer Console and the Admin Dashboard Two web applications front Agent Router, each scoped to a different audience. The **Developer Console** is the developer-facing application. It's where you issue API keys, author routing policies, assemble MCP profiles, configure integrations, and test prompts in the Playground. Developers spend most of their time here. The **Admin Dashboard** is the platform-operator application. It's where you provision models and providers, manage users, set budgets and rate limits, review audit logs, and configure SSO. Most of Agent Router's governance happens here. The two applications share the same underlying system. They're two views of the same data, each scoped to the responsibilities of its audience. ### Projects A **project** is the logical boundary that owns configuration and access in Agent Router. Everything a team works with day to day belongs to a project: the models and providers granted to it, its API keys, its MCP servers and profiles, its routing and policy, its budgets, and its members. It is also the unit of attribution, so usage and audit records are tied to the project the traffic ran under. If you have used the concept of a workspace or a tenant in other tools, a project plays a similar role here: it is the container that keeps one team's or one application's setup separate from the next. #### Projects, gateways, and data planes The project is one of three objects that work together, and the clearest way to understand a project is to see how it relates to the other two. They divide into a *logical* layer and a *physical* layer: - A **project** is *logical*. It is the configuration and access boundary described above. It decides what should be possible (which models, which members, which policy), but it runs no software of its own. - A **gateway** is *physical*. It is a single inference endpoint with one customer-facing URL: the thing an application actually calls. Each gateway serves exactly one project. - A **data plane** is *physical* too. It is the layer that provisions and manages gateways — Tetrate-operated in Agent Router Service and Agent Router Enterprise Fully Managed, or deployed in a [Kubernetes cluster you control](/product-architecture/planes-and-components/) in Agent Router Enterprise Self-Hosted Data Plane. It is the capacity a gateway runs on. These three connect through a small set of fixed relationships: | Objects | How they relate | | :---- | :---- | | Project and gateway | **Default one gateway; each gateway serves exactly one project.** Selecting a data plane auto-provisions the project's first gateway. On Agent Router Enterprise with multi-gateway projects, a project may optionally attach additional like-configured gateways for high availability (see [Gateway sets and DNS-level failover](/agent-router-enterprise/guides/operate-and-govern/project-and-data-plane-management/gateway-sets-and-dns-failover/)). A gateway never serves more than one project. | | Data plane and gateways | **One-to-many.** A single data plane provisions and manages many gateways. Typically that is one gateway per project placed on it; when a project uses optional HA members, those members usually run on separate data planes. | | Management plane and data planes | **One-to-many.** The single management plane manages many data planes, which can sit in different clouds and regions. | Read together, the picture is straightforward: the one management plane holds every project's configuration, each data plane is a pool of capacity in some location, and each gateway is a project's live endpoint running on one of those data planes. A project comes to life by requesting a gateway from a data plane. The data plane provisions the runtime, the management plane maps the project to that gateway and publishes its desired configuration, and once the gateway reports ready its URL appears on the project. Until then, a project is just configuration with nowhere to send traffic. Most deployments stop there; optional additional members are an Enterprise HA concern, not the default path. Keeping the logical project separate from the physical gateway and data plane is deliberate, and it is what makes the model flexible. Because a project's identity (its URL, its keys, its configuration) is decoupled from the capacity that serves it, a gateway's runtime can be upgraded in place or moved to a different data plane without the project's configuration or its URL changing, and without dropping in-flight requests. The same decoupling lets you place projects wherever they need to run: several projects can share one data plane, each with its own gateway, or be spread across data planes by region or cloud (for example, one project served from an AWS data plane and another from an Azure data plane). #### Why projects matter The project is the **isolation boundary**, and that isolation is enforced in two places. In the applications, every view is scoped to the selected project: it lists only the models, MCP servers, MCP profiles, and request logs that belong to that project, and switching projects with the **project switcher** re-scopes the whole app. At the gateway, cross-project requests are refused at request time. A key issued for one project does not work on another project's gateway (it is rejected with a `403`), and a model a project has not been granted cannot be called through it (a `404`). This runtime enforcement is what lets several teams or applications share one platform, or even one data plane, while staying fully separated from one another. That boundary is why projects are the organising idea for running Agent Router at scale. They give you a hard line for security and access, clean per-team attribution for cost and audit, and the freedom to move or grow the underlying infrastructure without disturbing the applications that depend on it. A default project is created for each customer during onboarding, and administrators can add more afterwards: one per team, application, or environment (for example, separate production and staging projects). The operator tasks for creating projects, provisioning their gateways, and placing them on data planes are collected under [Project and data plane management](/agent-router-enterprise/guides/operate-and-govern/project-and-data-plane-management/), and the fleet-level arrangement of data planes is described in [Operating multiple data planes](/product-architecture/planes-and-components/#operating-multiple-data-planes). ## Routing concepts Routing is the gateway's central job: for each incoming AI request, deciding which backend should serve it. A few concepts describe how that decision gets made. ### Backends, models, and providers A **provider** is an upstream AI service the gateway can talk to: OpenAI, Anthropic, Google, Azure OpenAI, Mistral, and many others, plus self-hosted endpoints and customer-specific deployments. A **model** is a specific offering from a provider, such as `gpt-4o` or `claude-sonnet-4-6`. A **backend**, in routing terms, is a model-on-a-provider combination paired with the credentials needed to reach it. Every routing decision ultimately resolves to a choice of backend. ### Routing chains and policies Each API key carries a **routing chain**: an ordered or weighted list of backends the gateway considers when a request arrives. The chain encodes the consumer's preferences, fallbacks, and constraints, and one or more **routing policies** attached to the key shape how it behaves. ### Fallback policy A **fallback policy** is an ordered list of backends. When a request arrives, the gateway tries the first backend in the chain; if that call fails with a recoverable error (a 5xx, a rate-limit response, or a connection timeout), it moves to the next backend and tries again, continuing until one succeeds or the chain runs out. The fallback decision happens inside the gateway, so your application sees a single request that either succeeds or fails. It never has to implement retry logic of its own. Fallback policies are how you manage provider risk: a primary commercial model with one or more secondary models behind it, or a commercial model with a self-hosted open-weights model as a last-resort backstop. ### Traffic splitting A **traffic splitting** policy distributes requests across two or more backends by percentage rather than by priority. A 70/30 split, for example, sends 70 % of requests to the first backend and 30 % to the second. Use it to evaluate one model against another under real production traffic, to manage cost by routing a portion of traffic to a cheaper backend, or to roll out a new provider gradually. Splitting and fallback combine: a split picks the primary backend for a given request, and a fallback chain takes over if that backend fails. ### Advanced routing rules When neither weighted splitting nor priority-ordered fallback is enough, **advanced routing rules** dispatch requests based on attributes of the request itself: the model the caller asked for, custom headers, request metadata, or other signals. Use them to encode policies such as "route requests from this tenant to this provider" or "send anything tagged `code-completion` to a model tuned for that task". ## Credentials and identity Identity in Agent Router is layered. Applications authenticate to the gateway with API keys; human operators authenticate to the Console and Admin Dashboard with single sign-on; and the gateway authenticates to upstream providers using credentials that are either centrally managed or supplied by the consumer. ### API keys An **API key** is the credential an application presents when it calls the gateway. Each key ties to a consumer (a person or a service), a routing chain, and any policies attached to that chain. Keys are issued from the Admin Dashboard or claimed by developers in the Console, depending on your onboarding model. Revocation, rotation, and per-key budgets are all managed centrally. ### Bring your own key (BYOK) **Bring Your Own Key**, or **BYOK**, is the model under which a consumer's own upstream provider credentials are used in place of centrally managed ones. A team that holds its own OpenAI account, for example, can register those credentials with Agent Router and have its requests routed through them rather than through the centrally pooled credentials. BYOK is a first-class part of the routing model: BYOK and centrally managed backends can sit in the same routing chain, and policy, not application code, decides between them. Reach for BYOK when billing, compliance, or contractual relationships require a specific team's traffic to land on a specific provider account. ### Single sign-on and role mapping Agent Router integrates with corporate identity providers through **single sign-on (SSO)** over OIDC. Microsoft Entra ID, Okta, Auth0, Keycloak, and any OIDC-compliant provider can handle authentication for the Console and the Admin Dashboard. Once SSO is configured, role assignment is driven from the identity provider itself: app role or group claims in the OIDC token map to Agent Router roles on every login, so directory changes propagate without manual work in Agent Router. The [Configuring SSO: Mapping Roles and Groups](/agent-router-enterprise/guides/identity-and-access-management/single-sign-on/sso-role-mapping-guide/) guide covers the mapping mechanics in detail. ## Agent infrastructure Beyond model routing, Agent Router gives you a control surface for the agent ecosystem, particularly the Model Context Protocol servers that AI clients consume. ### Model context protocol (MCP) The **Model Context Protocol (MCP)** is an emerging standard for exposing tools, data sources, and context to AI clients. MCP servers act as adapters: one wraps a file system, another a ticketing API, another a documentation source, and so on. AI clients such as Claude Code, Cursor, and VS Code consume those servers to extend their own capabilities. ### MCP profile An **MCP profile** is a single endpoint that aggregates multiple MCP servers behind one interface. Instead of configuring each AI client with a separate connection to every server it needs, you assemble a profile centrally, govern it with Agent Router's identity and access model, and expose it as one URL that clients connect to once. Profiles get rid of the per-agent fan-out that becomes unmanageable as the MCP ecosystem grows, and they bring MCP traffic under the same audit, access-control, and observability regime as standard model traffic. ## Extensibility and observability Two more concepts shape how Agent Router plugs into your surrounding software estate: the extension model inside the gateway, and the telemetry interface that exports operational data to external systems. ### Dynamic modules The gateway is built on a proxy component, and Agent Router–specific behaviour is delivered through **dynamic modules**: high-performance proxy extensions, written in Rust or Go, compiled to shared libraries that run inline in the filter chain. Dynamic modules implement the routing logic, credential handling, and provider-specific request translation that set Agent Router apart from a generic HTTP gateway. You don't configure them directly, but they're why the gateway behaves consistently across providers and why request-path overhead stays low. ### OpenTelemetry export Agent Router speaks **OpenTelemetry (OTEL)**, the open observability standard. Request-level metrics, traces, and logs are exported in OTEL format to whatever observability stack you already run: Grafana, Datadog, New Relic, Honeycomb, or any other OTEL-compatible backend. The platform operator owns the export configuration and applies it centrally, so individual applications don't need separate instrumentation to show up in your existing dashboards. ### Audit logging Alongside operational telemetry, Agent Router keeps an **audit log** of administrative actions: model and provider changes, user and role assignments, SSO configuration changes, API key issuance and revocation, and other state-modifying events in the Admin Dashboard. You can query the audit log from the Admin Dashboard, and it's the primary record reviewed during compliance investigations or post-incident analysis. The audit event schema is documented in the *Reference* section. ## Where to go next Architecture overview How these concepts assemble into the running system, with diagrams of the request path between application, data plane, management plane, and upstream provider. Solution capabilities Structured outputs, multimodal inputs, and function calling, passed through to any provider or translated across providers where mappings exist. Glossary The same vocabulary in one-line form for quick lookup. --- ## Network and Security Tetrate Agent Router is designed so that sensitive data stays inside the customer's own network and so that the customer's firewall never has to be opened to the internet on the platform's behalf. This page explains the network controls that make that possible: where the software runs, how it reaches AI providers privately, how network isolation is enforced, how traffic is encrypted, and how a Web Application Firewall can inspect requests inline. This page applies to Agent Router Enterprise Self-Hosted Data Plane, where you operate the data plane in your own infrastructure. On Agent Router Service and Agent Router Enterprise Fully Managed, Tetrate operates the data plane; see Deployment models for how the paths differ. This page assumes no prior networking background beyond a working familiarity with the platform's overall shape. Each control is introduced by describing what it is, why it matters, and how it fits the outbound-only design that keeps data inside the customer's environment. Terms are defined inline the first time they appear. For the architectural foundation these controls sit on (the split-plane model and the outbound-only connection), see Planes and core components. For a quick lookup of any single term, see the Glossary. ## The design goal: data stays in the customer's environment Before the individual controls, it helps to state the property they all serve. Tetrate Agent Router divides into two halves, called *planes*. The **management plane** is hosted by Tetrate and holds configuration (routing rules, policies, provider settings) and the web applications. The **data plane** runs inside the customer's own infrastructure and is the only part that live AI traffic passes through. Because every prompt and every response travels through the data plane and nowhere else, the sensitive content of requests (which can include proprietary code, customer records, or internal documents) stays inside the customer's network boundary. Two networking terms recur throughout this page: - **Ingress** is traffic entering an environment from outside it. An *inbound* connection is one that some external party opens toward the environment. - **Egress** is traffic leaving an environment toward the outside. An *outbound* connection is one the environment itself opens toward something external. The controls below are shaped by a single rule: the data plane opens outbound connections when it needs to, but nothing on the internet is allowed to open an inbound connection into it. This is the *outbound-only* design, and it is what lets the platform run inside a locked-down network with no new holes in the firewall. ## Intranet deployment An **intranet** is a private network internal to an organisation, reachable by its own systems and staff but not exposed to the public internet. Running the data plane on an intranet is the most restrictive and most common enterprise posture, and Tetrate Agent Router supports it directly. The data plane (the Controller and the Agent Router gateway) runs entirely inside the customer's Kubernetes cluster. Kubernetes is the industry-standard system for running containerised software; for the purposes of this page it can be read as "the customer's own compute environment". The only outbound connection the data plane requires is to the management plane, which the Controller uses to poll for configuration updates. That connection is a single outbound HTTPS link initiated from inside the customer environment. HTTPS is HTTP secured with encryption, described under [Transport security](#transport-security) below. Two consequences follow from this arrangement: - No inbound traffic from the internet to the data plane is required. Because the management plane never opens a connection *into* the customer environment, the customer's firewall needs no inbound rules for the platform. That is simpler to get approved by a security team, and it removes an entire class of attack surface (there is no exposed listener for an attacker on the internet to reach). - If every client application that calls the gateway is itself internal (in the same VPC, the same cluster, or reachable over a corporate network), no public **ingress** is needed at all. A **VPC**, or virtual private cloud, is a logically isolated private network inside a cloud provider's infrastructure that the customer controls. When callers and gateway share such a network, the gateway can operate fully privately and still receive its configuration over the one outbound link. The single outbound configuration link, and the fact that the data plane keeps routing even if that link is briefly interrupted, are covered in Planes and core components and Operations and reliability. ## Private provider connectivity The data plane still has to reach the AI providers that actually serve requests. By default that traffic travels over the public internet. When the data plane runs inside a cloud environment, each provider can instead be configured to be reached over *private connectivity*, so that model traffic stays on the cloud's internal backbone and never crosses the public internet. Private connectivity is a cloud feature that exposes a service (here, an AI provider's endpoint) as a private address reachable only from inside the customer's own VPC, rather than as a public address reachable by anyone. Each major cloud offers its own version of this feature: | Cloud | Private connectivity option | How to configure | |-------|----------------------------|------------------| | **Azure** | Azure Private Link | Set the provider endpoint URL to the private-link FQDN for the Azure OpenAI resource | | **AWS** | AWS PrivateLink / VPC Endpoints | Set the provider endpoint URL to the VPC endpoint DNS name for the AI service | | **GCP** | Private Service Connect | Set the provider endpoint URL to the Private Service Connect endpoint for the AI service | In each case the mechanism is the same in spirit: the cloud maps the provider's service to a private endpoint inside the customer's network, and the gateway dials that private endpoint instead of the public one. The value entered in the Admin Dashboard is a **FQDN** (fully qualified domain name, the complete host name of an endpoint, for example `my-resource.privatelink.openai.azure.com`) or a private DNS name that resolves only inside the customer's cloud network. To configure a provider with a private endpoint: 1. Establish the private connectivity in the cloud environment first (Azure Private Link, AWS PrivateLink or a VPC endpoint, or GCP Private Service Connect, depending on the cloud). The Admin Dashboard configures only the address the gateway dials; it does not provision the underlying network. 1. In the Admin Dashboard, open **[Providers Management](/agent-router-enterprise/guides/operate-and-govern/provision-models-and-providers/)** and add or edit the provider. 1. Set the **endpoint URL** to the private endpoint address instead of the provider's public URL. 1. Save and verify that the connection reports healthy. A private endpoint that resolves only inside the cloud network will fail verification if the data plane cannot reach it, which surfaces a misconfigured route or DNS record before any traffic depends on it. The gateway routes requests to whatever endpoint URL is configured for a provider, so private endpoints work transparently with every routing feature: fallback policies, traffic splitting, and Bring Your Own Key (BYOK, the pattern where a caller supplies their own provider credentials). Private and public entries can coexist in the same deployment, so one provider can be reached privately while another, not subject to the same constraint, uses its public address. The step-by-step operator flow for configuring private endpoints across several clouds and regions is in Connect provider subscriptions across clouds and regions. ## Private-network enforcement Private connectivity makes it *possible* to reach a provider privately. Enforcement is what makes it *guaranteed* that traffic cannot leak to a public provider. In Tetrate Agent Router, that enforcement reuses the same mechanism that controls routing: the **fallback policy**. A fallback policy is an ordered list of backends (provider-and-model combinations) attached to an API key. Under normal operation the gateway sends requests to the first backend in the list; if that backend fails with a recoverable error, the gateway walks to the next one, and so on down the chain. The important property for network isolation is this: **the gateway can only route to backends that appear in the chain**. Nothing outside the chain is a candidate, regardless of what the calling application asks for. That property turns the fallback chain into an enforceable network boundary. A team that requires all traffic to stay on private networks configures its fallback chains exclusively with private-endpoint providers. Because a public provider was never added to the chain, the gateway has no path to one; it cannot fall back to a provider that is not listed. There is no separate "private only" mode to enable and no flag that could be left off by mistake. The boundary is the composition of the chain itself. How to design and attach a fallback chain, and how the same property is used for data-residency and approved-provider constraints, is covered in Improve resilience with fallbacks. Region-level residency guarantees are described in Compliance. ## Transport security The controls above govern *where* traffic goes. Transport security governs whether that traffic can be read or tampered with on the way. Every hop the gateway participates in can be encrypted, and where identity matters, mutually authenticated. Two related terms are used here: - **TLS** (Transport Layer Security) is the standard protocol that encrypts a network connection and verifies the identity of the server being connected to. It is the "S" in HTTPS. TLS protects data in transit from being read or altered by anything on the network path. - **mTLS** (mutual TLS) extends TLS so that *both* ends of the connection present a certificate and prove their identity to each other, not just the server. This prevents an unauthorised client from connecting even if it can reach the endpoint on the network. The gateway supports encrypted transport at every hop: | Hop | Protection | What it achieves | | :---- | :---- | :---- | | **Client to gateway** | mTLS can be configured between client applications and the Agent Router | Mutual authentication at the ingress boundary, so only clients presenting a valid certificate are accepted | | **Gateway to self-hosted upstreams** | mTLS is supported for connections to self-hosted or on-premises model endpoints | End-to-end mutual authentication between the gateway and an organisation's own model servers | | **Gateway to public providers** | Traffic to public AI providers (OpenAI, Anthropic, Google, and others) is encrypted via TLS | Confidentiality and integrity of requests and responses as they travel to and from the provider | The pattern reflects the trust model at each hop. Between components the customer controls (client to gateway, gateway to self-hosted models), full mutual authentication is available, so each side can insist on the other's identity. Toward public providers, where the provider's own certificate establishes the server's identity, TLS encrypts the connection. ## WAF policies The final layer inspects the *content* of requests for known attack patterns before they reach an AI provider. This is the job of a **Web Application Firewall (WAF)**: a security control that examines incoming HTTP requests and blocks or logs those matching signatures of common web attacks, such as SQL injection (attempts to smuggle malicious database commands through an input field) or cross-site scripting (XSS, attempts to inject malicious script into a response). Tetrate Agent Router provides WAF protection at the data plane through the gateway's own extensibility, rather than as a separate appliance in front of it. Two open-source pieces make this work: - **Envoy Gateway** is the management layer for Envoy, the open-source proxy that the Agent Router gateway is built on. It exposes standard Kubernetes resources for extending the proxy's behaviour. - A **WebAssembly (Wasm) filter** is a small, sandboxed program that runs *inside* the proxy as part of processing each request. WebAssembly is a portable binary format that lets such a program run safely and efficiently inline, without being a separate network service. The specific filter used here is the [Coraza WAF Wasm filter](https://github.com/corazawaf/coraza-proxy-wasm), an open-source WAF engine. Together these allow standard **OWASP Core Rule Set (CRS)** policies to be applied directly at the gateway. The OWASP CRS is a widely used, community-maintained set of WAF rules that detect common attack categories; OWASP (the Open Worldwide Application Security Project) is the organisation that publishes it. Support is delivered through [Envoy Gateway Wasm extensibility](https://gateway.envoyproxy.io/docs/tasks/extensibility/wasm/) combined with the Coraza filter above. To enable WAF protection: 1. Deploy the Coraza Wasm filter alongside the data plane using the Envoy Gateway `EnvoyExtensionPolicy` resource. 1. Configure the desired rule sets, for example the OWASP CRS rules covering SQL injection, XSS, and other common attack vectors. 1. The filter then inspects requests inline before they reach the AI provider, blocking or logging matches according to the configured policy. Because the WAF runs as a Wasm filter inside the proxy rather than as a separate service in the request path, it adds no additional network hops, and it can be applied globally across the data plane or scoped to individual routes. ## Key terms The vocabulary introduced above, collected for reference: | Term | Meaning | | :---- | :---- | | **Intranet** | A private network internal to an organisation, not exposed to the public internet. | | **Ingress / egress** | Traffic entering an environment (ingress) and traffic leaving it (egress). | | **Inbound / outbound connection** | A connection opened toward an environment from outside (inbound) or opened by the environment toward the outside (outbound). | | **Outbound-only design** | The data plane opens connections outward but accepts no inbound connections from the internet, so no firewall openings are needed. | | **VPC (virtual private cloud)** | A logically isolated private network inside a cloud provider that the customer controls. | | **Private connectivity** | A cloud feature exposing a service as a private address reachable only inside the customer's network (Azure Private Link, AWS PrivateLink, GCP Private Service Connect). | | **FQDN (fully qualified domain name)** | The complete host name of an endpoint, used as the provider endpoint URL. | | **Fallback policy / chain** | An ordered list of backends attached to an API key; the gateway can route only to backends present in the chain. | | **TLS (Transport Layer Security)** | The standard protocol that encrypts a connection and verifies the server's identity. | | **mTLS (mutual TLS)** | TLS in which both ends present certificates and authenticate each other. | | **WAF (Web Application Firewall)** | A control that inspects HTTP requests and blocks or logs known attack patterns. | | **WebAssembly (Wasm) filter** | A sandboxed program that runs inline inside the proxy, adding no separate network hop. | | **OWASP Core Rule Set (CRS)** | A community-maintained set of WAF rules for common attack categories. | | **Envoy Gateway** | The management layer for Envoy, the proxy the gateway is built on, used to deploy the Wasm filter. | ## Where to go next - **Deployment models**: the Enterprise model that enables these private-connectivity options. See [Deployment models](/product-architecture/deployment-models/). - **Guardrails**: the content-safety layer that complements network security on the request path. See [Guardrails](/product-architecture/guardrails/). Where to go next Deployment models The Enterprise model that enables private connectivity and data residency. Provision models and providers Configure provider endpoints, including private connectivity, in the Admin Dashboard. --- ## Operations and Reliability Tetrate Agent Router routes every AI request through a data plane. In Agent Router Enterprise Self-Hosted Data Plane that data plane is customer-managed, and because it is the single path all traffic takes, its reliability is the reliability of the whole service. This page explains the mechanisms that keep it running: how it survives losing contact with the management plane, how it spreads across regions, how it heals and scales itself, how it is deployed reproducibly, and how it absorbs provider failures one request at a time. This page applies to Agent Router Enterprise Self-Hosted Data Plane, where you operate the data plane in your own Kubernetes cluster. On Agent Router Service and Agent Router Enterprise Fully Managed, Tetrate operates the data plane; see Deployment models for how the paths differ. This page assumes no prior familiarity with Tetrate Agent Router and defines each term as it appears. Every mechanism described here follows from one architectural choice: the platform is split into two halves that run in different places and have different jobs. That split is worth restating before the mechanisms make sense. The split-plane model is introduced in the Architecture overview and covered in depth in Planes and core components. This page builds on both and focuses on what the split means for staying available. ## The split that reliability follows from Tetrate Agent Router divides into two halves, called *planes*, that run in separate environments: - The **management plane** is hosted and operated by Tetrate. It stores all configuration (routing rules, policies, provider settings, user accounts) and exposes the two web applications used to manage the platform. No live AI traffic passes through it. This is sometimes also called the control plane, in the sense that it is where control decisions are recorded. - The **data plane** is deployed inside the customer's own Kubernetes cluster. (Kubernetes is the industry-standard system for running containerised software, meaning applications packaged with everything they need to run, across a group of machines.) One or more data planes handle all AI traffic: every prompt sent to a model and every response returned passes through a data plane and nowhere else. The two planes stay in sync over a single connection with a deliberate direction. The data plane reaches *out* to the management plane to pull down the latest configuration; the management plane never initiates a connection *into* the customer's environment. This is described more fully in [Planes and core components](/product-architecture/planes-and-components/). Two consequences of this split drive everything below. First, the component that keeps configuration current, the *Controller*, pulls configuration on a schedule rather than waiting to be pushed to. Second, because the data plane holds its own copy of the configuration and does the actual routing, it can keep working even when it cannot reach the management plane at all. ## Control plane resilience *Configuration polling* is the mechanism the data plane uses to stay current. The Controller (the data-plane component that bridges the two planes) repeatedly asks the management plane, on a fixed schedule, whether the configuration has changed, and applies any changes it finds. "Polling" simply means asking on a schedule rather than being notified. This design matters most when the connection between the planes is interrupted, for example by a network problem or a management-plane maintenance window. In that situation: - The Controller continues to poll on its normal schedule, retrying the connection rather than giving up. - All configuration already applied to the data plane remains in effect. The gateway keeps routing requests exactly as before, because it is acting on its own local copy of the rules, not on a live lookup against the management plane. - When connectivity is restored, the Controller picks up any configuration changes made in the meantime and applies them. The practical effect is that a management-plane interruption does not stop AI traffic. The configuration cannot be *changed* while the connection is down, but existing traffic continues to be served. This is the direct payoff of the split: the plane that carries traffic does not depend, moment to moment, on the plane that stores configuration. The same polling connection doubles as a health signal in the other direction. Because a healthy data plane polls the management plane on a known schedule, the management plane can treat a data plane that stops polling as unreachable, and operators can see that state. Configuration polling is therefore both the mechanism that keeps a data plane current and the way the management plane knows a data plane is alive. ## Multi-region deployment A single management plane can manage many data planes at once. Each data plane is an independent Kubernetes deployment, and each can be placed in any cloud, region, or availability zone. Two of those terms are worth defining: - A **region** is a geographically separate location operated by a cloud provider (for example a European or a North American region). Regions are designed to fail independently of one another. - An **availability zone** is an isolated section within a region, with its own power and networking, so that a fault in one zone does not take down the others in the same region. Because the data planes are independent of one another, they can be arranged to fit the organisation's needs rather than a fixed template: one data plane per region, one per business unit, or one per cloud provider are all valid. A common resilient arrangement runs several regional data planes at the same time, all serving traffic, so that the loss of one region degrades the service rather than stopping it. Sizing each region to carry its own share of traffic plus enough spare capacity to absorb a failed peer is a planning task covered in [Sizing and scale](/reference/sizing-and-scale/). Distributing traffic across several regional deployments of the same model, and draining a region for maintenance without dropping requests, are covered in Load balance across regional deployments. Planning recovery targets and topology for a production deployment is covered in Plan high availability and disaster recovery. ## Health checks For a deployment to heal itself, the system running it has to be able to tell healthy components from unhealthy ones. The data plane exposes two kinds of health signal, at two different levels. At the cluster level, data plane components expose Kubernetes *liveness* and *readiness probes*. A probe is a small check Kubernetes runs against a component on a schedule: - A **liveness probe** answers the question "is this component still working?" If it fails, Kubernetes restarts the component automatically, on the assumption that it has become stuck. - A **readiness probe** answers the question "is this component ready to receive traffic right now?" If it fails, Kubernetes stops sending new requests to that component until it recovers, without restarting it. This is what prevents traffic from being sent to a component that is still starting up. Together these probes let Kubernetes (the *orchestrator*, meaning the system that decides where and whether each component runs) detect and recover from unhealthy workloads without operator intervention. At the platform level, the management plane monitors data plane health through the configuration polling described above: a data plane that stops checking in is flagged as unreachable. The two levels are complementary. Kubernetes probes keep an individual data plane healthy from the inside; the polling signal tells the management plane whether a whole data plane is reachable from the outside. ## Scaling Traffic to an AI gateway is not steady. It concentrates into working hours, time zones, and bursts, and the peak can sit well above the monthly average. A deployment sized only for the average would be overwhelmed at peak, so the data plane is built to add and remove capacity as load changes. The data plane scales *horizontally*, meaning capacity is added by running more copies (called *replicas*) of a component behind a load balancer, rather than by making a single copy larger. This is the mechanism that also provides redundancy: running more than one replica of each component is what lets the data plane survive the loss of an individual replica or machine. Data plane components are configured with the Kubernetes **Horizontal Pod Autoscaler (HPA)**, a built-in Kubernetes feature that adds or removes replicas automatically in response to load. (A *pod* is Kubernetes' unit of running software, so "horizontal pod autoscaler" is literally the thing that scales the number of running copies out and in.) The HPA can be tuned to match a given traffic pattern and resource budget. The management plane is scaled and managed by Tetrate, so this scaling concern applies to the customer-managed data plane only. Deciding how many replicas a deployment needs (measuring the throughput of one replica, dividing peak concurrency by it, then adding headroom for failover and upgrades) is the subject of [Sizing and scale](/reference/sizing-and-scale/). ## Infrastructure as code *Infrastructure as code* is the practice of describing a deployment in text files kept under version control, rather than by clicking through consoles or running ad-hoc commands. The benefit is reproducibility: the same files produce the same deployment every time, changes are reviewed and tracked like any other code, and a lost environment can be rebuilt from its description. This is what makes a tested disaster-recovery procedure possible in the first place, because the configuration to restore is the committed source, not something reconstructed by hand. Tetrate Agent Router supports this in two layers: - **Helm charts** for installing and managing the data plane components on Kubernetes. A Helm chart is a packaged, parameterised template for a set of Kubernetes resources; Helm is the standard package manager for Kubernetes. The chart describes what the data plane should look like inside the cluster. - **The underlying cloud infrastructure** the data plane runs on (networks, subnets, access-control roles, and similar), where that infrastructure needs to be created as well. This layer is provisioned with whichever infrastructure-as-code tooling the organisation already uses; Agent Router places no requirement on that choice. Used together, these give a fully reproducible, version-controlled deployment: the cloud foundation is stood up first, and Helm installs the data plane onto it. Because both layers are declarative descriptions in version control, the committed files serve as the backup from which a region can be rebuilt. ## Resilience without full cutover The mechanisms above keep a data plane and its regions healthy. A separate mechanism handles the failures that come from *outside* the data plane: the AI providers it forwards requests to. Providers have bad hours (regional incidents, capacity exhaustion, rate-limit throttling, outright outages), and the data plane absorbs these one request at a time rather than by switching an entire cluster or region over. The tool for this is a **fallback policy**: an ordered list of backends (a *backend* being a specific model at a specific provider) to try for a given request. The gateway tries the first backend; if that call fails with a recoverable error, it moves to the next backend in the list and tries again, continuing until one succeeds or the list is exhausted. The calling application sees a single request that either returns a response or a final error; the failover happens inside the gateway and is invisible to the caller. Developers configure these chains as described in [Improve resilience with fallbacks](/agent-router-service/guides/improve-resilience-with-fallbacks/). Under the hood, each backend in a fallback chain is assigned a *priority level* in the AI gateway's `AIGatewayRoute` resource (the configuration object that defines how a route's requests are dispatched). Priority 0 is the primary and serves all traffic under normal conditions; priority 1 is the first fallback, priority 2 the second, and so on. When a request to the priority-0 backend fails (a rate-limit response, a server error, or a connection failure), the gateway retries at priority 1, then priority 2, and so on, all within the same HTTP request lifecycle. This is *per-request failover*: the retry happens inside the handling of one request, before any response is returned, rather than by taking a backend out of service globally. Two properties follow from this per-request model, and both matter for how the service behaves under provider trouble: - There is no provider-level health tracking behind the chain. Each request independently discovers failures and walks its own priority chain. A backend is not marked "down" for everyone; it is simply skipped, per request, whenever it fails for that request. - Intermittent errors are absorbed transparently, without any retry or branching logic in the application. A provider that fails one request in fifty costs those requests a walk down the chain, not an operator intervention. This is a different layer from the regional and cluster resilience above, and the layers stack. Health-aware balancing inside a regional pool absorbs a single unhealthy region silently; a fallback chain absorbs the case where a whole backend or pool is unavailable; and neither requires a manual, cluster-wide cutover. The relationship between in-pool balancing and chain-level fallback is covered in [Load balance across regional deployments](/agent-router-enterprise/guides/operate-and-govern/load-balance-across-regional-deployments/). ## Key terms The vocabulary introduced above, collected for reference: | Term | Meaning | | :---- | :---- | | **Management plane (control plane)** | The Tetrate-hosted half that stores configuration and serves the web applications. Carries no traffic. | | **Data plane** | The customer-hosted half, in Kubernetes, that all AI traffic flows through. | | **Configuration polling** | The Controller asking the management plane for configuration on a fixed schedule, rather than being pushed to. Doubles as a health signal. | | **Kubernetes** | The industry-standard system for running containerised software across a group of machines. | | **Liveness probe** | A Kubernetes check for "is this component still working?" A failure triggers a restart. | | **Readiness probe** | A Kubernetes check for "is this component ready for traffic?" A failure removes it from rotation without a restart. | | **Horizontal Pod Autoscaler (HPA)** | The Kubernetes feature that adds or removes replicas automatically as load changes. | | **Helm chart** | A packaged, parameterised template for a set of Kubernetes resources; used to install the data plane. | | **Infrastructure as code** | Describing a deployment in version-controlled text files so it is reproducible and reviewable. | | **Region / availability zone** | A geographically separate cloud location, and an isolated section within a region, both designed to fail independently. | | **Fallback policy (chain)** | An ordered list of backends the gateway tries in turn when one fails. | | **Priority level** | A backend's fixed position in a fallback chain; priority 0 is primary, higher numbers are fallbacks. | | **`AIGatewayRoute`** | The gateway resource that defines how a route's requests are dispatched, including the priority-ordered backends. | | **Per-request failover** | Retrying down the priority chain inside a single request, rather than taking a backend out of service globally. | ## Where to go next The operational guides build on this foundation: - [Plan high availability and disaster recovery](/agent-router-enterprise/guides/operate-and-govern/plan-high-availability-and-disaster-recovery/) covers topology and recovery planning. - [Load balance across regional deployments](/agent-router-enterprise/guides/operate-and-govern/load-balance-across-regional-deployments/) covers spreading traffic across multiple data planes. - [Improve resilience with fallbacks](/agent-router-service/guides/improve-resilience-with-fallbacks/) is the developer-side configuration of the fallback chain. - [Sizing and scale](/reference/sizing-and-scale/) covers how many replicas and regions a deployment needs. Where to go next Plan high availability and disaster recovery Topology and recovery planning for a production deployment. Load balance across regional deployments Spread traffic across multiple independent data planes. --- ## Planes and core components Tetrate Agent Router is built on a split-plane model. - A management plane hosted by Tetrate holds configuration and exposes the web applications. - A data plane handles all AI traffic. Where it runs depends on the deployment model: Tetrate operates it in Agent Router Service and Agent Router Enterprise Fully Managed; the customer deploys it in their Kubernetes cluster in Agent Router Enterprise Self-Hosted Data Plane. The two planes stay in sync over a connection the data plane opens. Live traffic stays on the data plane and never reaches the management plane. This page explains how Tetrate Agent Router is put together and defines each term as it appears. No prior familiarity with the platform is assumed. The starting point is a single idea that shapes everything else: the parts of the platform that decide what should happen are kept separate from the parts that carry live traffic. Where those traffic-carrying parts run is the choice described in [Deployment models](/product-architecture/deployment-models/). For a first-principles walk-through of what an AI gateway is and how a request travels end to end, start with the Architecture overview. This page goes deeper on the two planes and the components inside the data plane. ## System architecture A *plane* is a group of software components that share a single job. Tetrate Agent Router is organised into two planes, each with a distinct responsibility. - The **management plane** is the part hosted and operated by Tetrate in every deployment model. It holds configuration (routing rules, policies, model and provider definitions, user records) and exposes the two web applications used to manage the platform. It is the control room: decisions are recorded here, but no live AI traffic passes through it. - The **data plane** is the part that handles all AI traffic: every prompt sent to a model, and every response returned, passes through it and nowhere else. In [Agent Router Service](/agent-router-service/quickstarts/) and [Agent Router Enterprise Fully Managed](/agent-router-enterprise/fully-managed/), Tetrate operates it. In [Agent Router Enterprise Self-Hosted Data Plane](/agent-router-enterprise/self-hosted-data-plane/), it is deployed inside the customer's own Kubernetes cluster. *Kubernetes* is the industry-standard system for running containerised software across a group of machines; a *cluster* is one such group. The two planes communicate over a single outbound Hypertext Transfer Protocol Secure (HTTPS) connection initiated by the data plane. *Outbound* means the connection is opened from the data plane toward Tetrate, never the other way around. In Agent Router Enterprise Self-Hosted Data Plane that means no inbound connections from the internet to the customer environment are required. The diagram below shows the layout for **Agent Router Enterprise Self-Hosted Data Plane** (data plane in the customer domain). Fully hosted deployments keep the same components and flows, with the data plane in Tetrate's environment instead. ![data-paths](/img/screenshots/about/architecture/TARE_arch.png) The diagram traces two independent journeys. Configuration travels down the left: the management plane hands updates to the Controller, which applies them to the Agent Router. Live traffic travels along the bottom: client applications send requests to the Agent Router, which forwards them to the upstream AI providers and returns the responses. The two journeys never mix, which is the point of the split. ### Why the architecture is split this way The separation exists for one main reason: **live prompt and response content stays on the data plane and never reaches the management plane**. The management plane only ever holds configuration and receives telemetry (operational data such as latency and token counts); it never sees the content of a request. In Agent Router Enterprise Self-Hosted Data Plane, that also means **prompt and response data never leaves the customer's environment**, because the data plane runs in infrastructure the customer controls. The outbound-only link reinforces this: a firewall configured to allow a connection out is far easier for a security team to approve than one that must accept connections coming in, and refusing all inbound connections removes an entire class of attack. The result is data sovereignty (the customer keeps full control of its data) without the customer having to build and operate the control-plane software itself. In Agent Router Service and Agent Router Enterprise Fully Managed the same plane separation holds, but the data plane itself is operated by Tetrate. ## Operating model The split-plane model divides responsibilities cleanly between a centralized management environment controlled by Tetrate and a data plane that carries all AI traffic. In practice this means each side owns a different set of concerns, and each side can be changed, scaled, or recovered without disturbing the other. Tetrate managed Management plane Hosted and operated by Tetrate in every deployment model. Contains the Developer Console, the Admin Dashboard, and the central backend that stores routing rules, policies, and configuration. Customers interact with these applications to configure the platform but do not manage the underlying infrastructure, so there are no control-plane servers to patch, back up, or keep highly available on the customer side. Traffic path Data plane All prompt and response traffic flows exclusively through this environment. Tetrate operates it in Agent Router Service and Agent Router Enterprise Fully Managed; the customer deploys it in their own Kubernetes cluster in Agent Router Enterprise Self-Hosted Data Plane. The data plane receives configuration from the management plane but operates independently: it holds the most recent configuration locally and continues routing requests normally even if connectivity to the management plane is temporarily interrupted. A brief loss of the link pauses configuration updates, not live traffic. The two applications in the management plane are two views of the same system, each scoped to a different audience. | Application | Audience | What it is used for | | :---- | :---- | :---- | | **Developer Console** | Developers | Issue API keys, author routing configurations, assemble Model Context Protocol (MCP) profiles, and monitor their own traffic. | | **Admin Dashboard** | Platform operators (platform / IT) | Provision models and providers, manage users, set budgets and rate limits, review audit logs, and configure single sign-on. | The three journeys that cross these planes (configuration polling, the live request path, and telemetry export) are traced in detail on the Data flows page. ## Operating multiple data planes Tetrate Agent Router Enterprise supports running more than one data plane from a single management plane. Each data plane is an independent deployment (typically in its own Kubernetes cluster), and the data planes can sit in any combination of clouds, regions, or on-premises environments while the one management plane keeps their configuration in sync. This fleet model is most relevant to Agent Router Enterprise Self-Hosted Data Plane, where the customer chooses where each data plane runs. The diagram below illustrates a multi–data-plane layout for Agent Router Enterprise Self-Hosted Data Plane. ![data-paths](/img/screenshots/about/architecture/TARE_multi-dp.png) More than one data plane is useful whenever a single deployment cannot satisfy every requirement at once. A common pattern is one data plane per region, placing traffic handling close to the applications it serves to reduce latency and to keep requests within a required geography for data residency. Another is one data plane per cloud, so that models offered natively by a provider are reached over that provider's own network: an AWS data plane serving Amazon Bedrock models, for example, alongside an Azure data plane serving Azure OpenAI models. Data planes are also split per business unit or per environment (production, staging) when separate teams need to operate without affecting one another. Within this model, each data plane provisions and manages one or more *gateways*, where a gateway is a single inference endpoint (one customer-facing URL) mapped to exactly one *project*. A project is the logical boundary that owns a set of models, API keys, and MCP profiles, together with the members allowed to use them, and is defined in full under [Projects](/product-architecture/key-concepts/#projects). Because a gateway serves only its project, isolation is enforced at request time: a key issued for one project is refused on another project's gateway, and a model a project has not been granted cannot be called through it. Several projects can share one data plane, each with its own gateway, or be spread across data planes as the region and cloud patterns above require. Holding configuration centrally in the management plane while each data plane runs independently is what makes a fleet of data planes practical to operate. Data planes can be added as demand grows, and a gateway's runtime can be upgraded in place or moved to a different data plane without changing its customer-facing URL or dropping in-flight requests. The operator-facing procedures are covered in [Run multiple platform instances](/agent-router-enterprise/guides/operate-and-govern/run-multiple-platform-instances/) and [Load balance across regional deployments](/agent-router-enterprise/guides/operate-and-govern/load-balance-across-regional-deployments/). ## Core components of the data plane The data plane is not a single program. It is a small set of components that run together wherever the data plane is hosted. At the center is the AI Gateway, which intercepts and manages all traffic between client applications and upstream AI providers. A *gateway* is a single entry point that requests pass through on their way somewhere else, and this one is built as a *proxy*: a server that receives a request, does something with it (here: choosing a provider, attaching credentials, applying policy), and forwards it on. Controller Acts as the bridge between the data plane and the management plane. It polls the backend for configuration updates and translates them into a gateway configuration, keeping routing rules, policies, and provider credentials in sync without requiring the management plane to open inbound connections. Polling means the Controller repeatedly asks for updates on a schedule, so it is always the side that reaches out. Modules The AI gateway is extended with high-performance native modules for cost controls, guardrails, and custom extensions. A dynamic module is add-on logic loaded into the proxy at runtime, so it runs inline in the proxy filter chain (the ordered sequence of steps each request passes through) and adds processing without an extra network hop. Agent Router The proxy that routes every request to a provider and enforces policy inline. The Agent Router itself (the proxy every request passes through) is the gateway the other components extend and support. It is built on an Cloud Native Computing Foundation (CNCF) backed open source foundation, co-created and maintained by Tetrate. This open source component is a widely used open-source proxy, which is why the platform can route to many providers reliably and with low overhead. The Controller keeps the gateway configured, the dynamic modules add cost controls and guardrails inline, and Redis lets several copies of the gateway share one view of rate-limit usage. Why these pieces run inline rather than as separate services: every extra service a request must call adds a network hop, and every hop adds latency and a new thing that can fail. Running cost controls and guardrails as dynamic modules inside the proxy keeps that logic on the request path itself, so policy is enforced without leaving the gateway. The components divide across the two planes as follows. The management plane components are operated by Tetrate; the data plane components run wherever the data plane is hosted. | Component | Plane | Role | | :---- | :---- | :---- | | Developer Console, Admin Dashboard, central backend | Management | Configuration storage and the web applications operators and developers use | | Agent Router (AI Gateway) | Data | The proxy that routes every request to a provider and enforces policy inline | | Controller | Data | Polls the backend and translates configuration into gateway configuration | | Modules | Data | Inline gateway extensions for cost controls, guardrails, logs, telemetry and more | ## Key terms The vocabulary introduced above, collected for reference: | Term | Meaning | | :---- | :---- | | **Plane** | A group of software components that share one responsibility. The platform has two: management and data. | | **Management plane** | The Tetrate-hosted half that stores configuration and serves the web applications. Holds no live traffic. | | **Data plane** | The half that all AI traffic flows through. Tetrate-operated in Agent Router Service and Agent Router Enterprise Fully Managed; customer-hosted in Kubernetes for Agent Router Enterprise Self-Hosted Data Plane. | | **Kubernetes** | The industry-standard system for running containerised software across a group of machines (a cluster). | | **Controller** | The data-plane component that polls the management plane for configuration and applies it to the gateway. | | **Gateway** | The single entry point every request passes through before reaching a provider. | | **Proxy** | A server that receives a request, acts on it, and forwards it on. The gateway is built as one, on Envoy. | | **Dynamic module** | Add-on logic loaded into the proxy at runtime, running inline in the filter chain with no extra network hop. | | **Redis** | A fast in-memory datastore the gateway uses as a shared counter for rate limiting across instances. | | **Rate limiting** | Capping how much an API key may consume, enforced consistently across all gateway copies. | | **Outbound-only connection** | A link opened from the data plane toward Tetrate, never the reverse. In Agent Router Enterprise Self-Hosted Data Plane, that means no inbound firewall openings into the customer environment are needed. | ## Where to go next - **Data flows**: the three traffic paths that cross these planes. See [Data flows](/product-architecture/data-flows/). - **Guardrails**: how the Guardrails dynamic module inspects traffic inline. See [Guardrails](/product-architecture/guardrails/). - **Deployment models**: where each plane runs in Agent Router Service, Agent Router Enterprise Fully Managed, and Agent Router Enterprise Self-Hosted Data Plane. See [Deployment models](/product-architecture/deployment-models/). Where to go next Data flows The configuration, request, and observability paths that cross the two planes. Guardrails How the Guardrails dynamic module inspects traffic inline at the gateway. Deployment models Agent Router Service, Agent Router Enterprise Fully Managed, and Agent Router Enterprise Self-Hosted Data Plane compared. --- ## Set up Auth0 for evaluation Choose this path when you lack corporate IdP admin access (or prefer not to use the corporate IdP for a PoC). Complete every step below in order: this is the full Auth0 setup for [Step 1 of the fast track](/agent-router-enterprise/fully-managed/fast-track/#sign-in-and-onboard). When you finish, SSO works and the Admin Dashboard shows providers and models. :::info One linear path Do not treat Auth0 setup and Management plane onboarding as separate jobs. You will open the onboarding wizard mid-flow to copy the Redirect URI, finish Auth0, paste credentials back into the same wizard, then complete onboarding. ::: :::warning Only change the settings listed below. Leave everything else at its default. ::: ## What you will have when done | Value | Where it comes from | | :---- | :---- | | **Issuer** | `https:///` *(trailing slash)* from the Auth0 Domain | | **Discovery URL** | `https:///.well-known/openid-configuration` | | **Client ID** | Auth0 application Settings | | **Client Secret** | Auth0 application Settings | You also register in Auth0: **Allowed Callback URLs**, **Allowed Web Origins**, and optionally **Allowed Logout URLs**. ## Step 1: create an Auth0 tenant 1. Go to [https://auth0.com](https://auth0.com) and sign up (or sign in). 1. Create a tenant if prompted (any region is fine for evaluation). 1. Confirm you can open the **Auth0 Dashboard** and create applications. ## Step 2: open Management plane onboarding and copy the Redirect URI 1. Open your Admin Dashboard URL and start [Management plane onboarding](/agent-router-enterprise/fully-managed/onboarding/). 1. Complete organization details if prompted, then reach the **Configure SSO** (Corporate Login) step. 1. Click **IdP setup guide** at the top of the form. A panel opens with the tenant **Redirect URI**. 1. Use the copy icon. The URL looks like: ```text https://auth..tetrate.ai/api/auth/sso/callback/corporate ``` :::tip Always copy the Redirect URI from the IdP setup guide panel. Do not type it from memory or paste the placeholder from this page—any mismatch causes `redirect_uri_mismatch` at sign-in. ::: Leave the wizard open on the Configure SSO step. You will paste Auth0 values into this form after Step 6. ## Step 3: create the Auth0 application 1. In the **Auth0 Dashboard**, go to **Applications**. 1. Click **+ Create Application**. 1. Select **Create Manually**. 1. Enter a recognizable name, for example `Tetrate Agent Router Enterprise`. 1. Leave **"This application is owned by a third party"** toggled **off**. Select **Regular Web Application**. 1. Click **Create**. :::warning Do not enable third-party ownership Leave "This application is owned by a third party" off. Third-party applications require PKCE, which this integration doesn't support — sign-in fails immediately with `invalid_request: The PKCE protocol extension is required`. This can't be changed after creation; if set wrong, delete the app and create a new one. ::: ## Step 4: configure application settings 1. Open the **Settings** tab. 1. Under **Basic Information**, copy **Domain**, **Client ID**, and **Client Secret** into a notes app. 1. Under **Application Properties**, optionally set **Application Logo** to a logo URL of your choosing (this only affects the branding your team sees on the Auth0 login screen). 1. Under **Application URIs**, set **Allowed Callback URLs** to the Redirect URI you copied in Step 2 (exact match). 1. Set **Allowed Web Origins** to your tenant hostnames: ```text https://auth..tetrate.ai https://dashboard..tetrate.ai https://router..tetrate.ai ``` Replace `` with the same tenant segment as in your Redirect URI (for example, if the callback host is `auth.acme.tetrate.ai`, use `acme`). ## Step 5: set the logout URL (optional) 1. Still under **Application URIs**, set **Allowed Logout URLs** to: ```text https://auth..tetrate.ai ``` 1. Click **Save**. ## Step 6: create an Auth0 user and note Tetrate form values 1. In Auth0, go to **User Management** → **Users** → **+ Create User**. 1. Use an email you control; set a password Auth0 allows. This email must appear under **Admin emails** in the Tetrate SSO form. :::warning Admin emails is not optional in practice If you leave **Admin emails** empty or omit this user's email from it, your first SSO sign-in lands as a regular, non-admin user — and because there is no email/password fallback in TARE, nobody will be able to grant admin access from inside the product. Recovering from this requires contacting Tetrate Support. Set this field before you finish Step 7. ::: 1. From the application **Settings** tab, build: | Field | How to build it | | :---- | :---- | | **Issuer** | `https:///` with a trailing slash. Example: Domain `acme.us.auth0.com` → Issuer `https://acme.us.auth0.com/` | | **Discovery URL** | `/.well-known/openid-configuration` | | **Client ID** | From Basic Information | | **Client Secret** | From Basic Information | | **Scopes** | `openid, profile, email` | | **Admin emails** | The Auth0 user email from this step (plus any other admins) | | **Attribute mapping** | Default `email` → `email` | ## Step 7: paste into the wizard and finish onboarding 1. Return to the Configure SSO step left open in Step 2. 1. Fill in Issuer, Discovery URL (if required), Client ID, Client Secret, scopes, admin emails, and attribute mapping. 1. Click **Configure**. 1. Continue the wizard: connect at least one LLM provider and select models. 1. **Skip Step 5** (data-plane credential)—Fully Managed only. Do not create, regenerate, or download a data-plane credential. 1. Complete setup. If the wizard is already closed without SSO, configure SSO from the Admin Dashboard using [Configure SSO](/agent-router-enterprise/guides/identity-and-access-management/single-sign-on/sso-configuration-guide/) Part 1, then confirm providers and models under **Models** / **Providers**. ## Step 8: verify SSO sign-in 1. Sign out of the Admin Dashboard if you are still on initial admin credentials. 1. Sign in with the Auth0 user from Step 6 (**Sign in with SSO** / corporate login). 1. Confirm the Admin Dashboard loads and shows the providers and models you configured. 1. Confirm the default project is ready: its default gateway's **endpoint** is set to the assigned proxy host (set it if empty), your admin user is a project member, and the project-scoped models you plan to call are enabled. ## Verify (Step 1 complete) - Auth0 Regular Web Application exists with callback and web origins from the live Redirect URI. - SSO sign-in with the Auth0 user succeeds. - Admin Dashboard catalog shows providers and models, and the default project is ready for traffic. ## Team prep: add teammates in Auth0 The fast track's team step needs 2 to 5 people signed in through the same Auth0 tenant. Corporate IdP invites do not apply on this path. 1. In Auth0, go to **User Management** → **Users** → **+ Create User** for each teammate (use emails they control). 1. Share the Console URL (`https://router..tetrate.ai`) and tell them to **Sign in with SSO** (Auth0). 1. After each teammate’s first SSO login, their Agent Router user record appears in the Admin Dashboard **Users** list. 1. Continue with [Step 6 of the fast track](/agent-router-enterprise/fully-managed/fast-track/#bring-teammates): teammates connect passthrough credentials, point their agents at the gateway, and get team budgets. ## Where to go next Continue the [fast track](/agent-router-enterprise/fully-managed/fast-track/#first-request) to route your first request. When you reach the team step, return to [Team prep](#team-prep-add-teammates-in-auth0). --- ## Cost Control and Budgets *Feeds categories D2 and D3 (cost tracking, budget enforcement).* Starting state from the [fast track](/agent-router-enterprise/fully-managed/fast-track/): a personal key with traffic history, a hello-world budget tracking spend, and teams with budgets drawing down. Scenarios 6.3 and 6.4 build directly on that state rather than starting from scratch. Grade each scenario Pass (2 points), Pass with gaps (1), or Fail (0) against the [grading rubric](/product-architecture/product-evaluation/evaluation-scenarios/#grading-rubric), and carry the subtotal into the [summary scorecard](/product-architecture/product-evaluation/evaluation-scenarios/#summary-scoring). Scoresheet document {props.body ?? 'For your convenience, this page can be downloaded as PDF or printed directly. Site navigation is omitted, the scoring table keeps its rows intact with the header repeated on every page, and the Grade and Notes columns leave room for handwriting.'} | ID | Scenario | Guide | Grade | Notes | | :-- | :---- | :---- | :---- | :---- | | 6.1 | Pick the right cost control for two contrasting workloads (a production app and an evaluation key) using the decision guide. Verify: the chosen controls match the guide's situation table. | [Choose the right cost control](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/choose-the-right-cost-control/) | | | | 6.2 | Run a usage review: by-key breakdown, sorted by cost, over a chosen period. Verify: the top consumers match expectations, and one row can be drilled into its keys and models. | [Catch spend drift before the invoice does](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/catch-spend-drift-early/) | | | | 6.3 | Create a team budget with a deliberate scope (pooled or per-member). Verify: the budget appears with the intended scope, and two identities draw down as the scope dictates. *(Carried over from fast-track Step 6; grade from that evidence.)* | [Keep each team's spend inside an agreed ceiling](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/keep-team-spend-inside-a-ceiling/) | | | | 6.4 | Exhaust a small test budget and read the burn along the way. Verify: the meter shows used, remaining, and days left, and an alert is raised as the ceiling is approached and crossed while requests continue to be served. *(Head start from fast-track Step 5: exhaust the test budget and watch the meter and alert.)* | [See budget exhaustion coming](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/see-budget-exhaustion-coming/) | | | | 6.5 | Raise the exhausted budget's ceiling in place. Verify: the meter and alerting reflect the new ceiling immediately, and the change appears in the audit trail. | [Raise a spend ceiling without interrupting delivery](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/raise-a-spend-ceiling/) | | | | 6.6 | Export a spend statement grouped by team for a closed period. Verify: the download carries USD amounts joinable to cost-centre records. | [Bill AI spend back to the teams that incur it](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/bill-ai-spend-back-to-teams/) | | | | 6.7 | Attribute cost with caller-set tags. Verify: spend reports group by the supplied tags, and an untagged or mislabeled request is refused where policy requires the tag. | [Know what every app and project actually costs](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/know-what-every-app-and-project-costs/) | | | | 6.8 | Set a rate limit and exceed it deliberately. Verify: requests over the limit are rejected with a `429`, and the firing limit is identifiable in the traffic view. | [Stop runaway workloads](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/stop-runaway-workloads/) | | | | 6.9 | *(optional)* Simulate a leaked key: revoke it mid-traffic. Verify: the next request on the credential is rejected, and the revocation is recorded in the audit trail. | [Contain a leaked key](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/contain-a-leaked-key/) | | | | | **Subtotal** | | **/18** | | --- ## Enterprise Operations and Compliance *Feeds categories F2 and F3 (compliance, residency, HA and scale) and G1 (configuration as code).* Starting state from the [fast track](/agent-router-enterprise/fully-managed/fast-track/): an organization with real users, traffic, and budgets exists to run the compliance checks against. Grade each scenario Pass (2 points), Pass with gaps (1), or Fail (0) against the [grading rubric](/product-architecture/product-evaluation/evaluation-scenarios/#grading-rubric), and carry the subtotal into the [summary scorecard](/product-architecture/product-evaluation/evaluation-scenarios/#summary-scoring). Scoresheet document {props.body ?? 'For your convenience, this page can be downloaded as PDF or printed directly. Site navigation is omitted, the scoring table keeps its rows intact with the header repeated on every page, and the Grade and Notes columns leave room for handwriting.'} | ID | Scenario | Guide | Grade | Notes | | :-- | :---- | :---- | :---- | :---- | | 10.1 | Configure data residency and provider no-retention. Verify: the settings applied to the evaluation environment match the documented posture. | [Configure data residency and no-retention](/agent-router-enterprise/guides/operate-and-govern/compliance-and-governance/configure-data-residency-and-no-retention/) | | | | 10.2 | Manage log retention and run a purge. Verify: purged records are gone from queries and exports. | [Manage log retention and purge](/agent-router-enterprise/guides/operate-and-govern/compliance-and-governance/manage-log-retention-and-purge/) | | | | 10.3 | Publish an announcement to Console users. Verify: the notification appears for a developer account. | [Communicate with platform users](/agent-router-enterprise/guides/operate-and-govern/communicate-with-platform-users/) | | | | 10.4 | *(optional)* Review running multiple instances (for example, staging and production). Verify: the separation model and its operational cost are understood and noted. | [Run multiple platform instances](/agent-router-enterprise/guides/operate-and-govern/run-multiple-platform-instances/) | | | | 10.5 | Review high availability and disaster recovery planning against organizational RPO/RTO targets. Verify: the documented posture is recorded with any gaps. | [Plan high availability and disaster recovery](/agent-router-enterprise/guides/operate-and-govern/plan-high-availability-and-disaster-recovery/) | | | | 10.6 | *(optional)* Load balance across regional deployments. Verify: traffic distributes across regions as configured. | [Load balance across regional deployments](/agent-router-enterprise/guides/operate-and-govern/load-balance-across-regional-deployments/) | | | | 10.7 | Manage configuration as code: export or define configuration declaratively and apply a change through it. Verify: the change lands without manual UI edits and is auditable. | [Manage configuration as code](/agent-router-enterprise/guides/operate-and-govern/manage-configuration-as-code/) | | | | | **Subtotal** | | **/14** | | --- ## Evaluation Scenarios # Hands-On evaluation scenarios An Agent Router evaluation expressed as a sequence of hands-on scenarios. Each scenario follows one or more documentation guides, ends with a verifiable outcome, and receives a grade, so the evaluation produces evidence rather than impressions. How to use these pages: 1. Run the [fast track](/agent-router-enterprise/fully-managed/fast-track/) first: it builds the organization, SSO, keys, traffic, teams, and budgets the graded sections assume, and pre-completes several scenarios (see the [carry-over table](#fast-track-carry-over)). 1. Then work through the graded sections in order; each section is a separate page and notes which evaluation categories it feeds. 1. For each scenario, follow the linked guide, run the scenario, and check the verification. 1. Grade the scenario Pass, Pass with gaps, or Fail using the rubric below, and note anything surprising (gaps, workarounds, documentation problems). 1. Total each section and carry the results into the scorecard. The Installation and onboarding section applies to Self-Hosted Data Plane evaluations only; fully hosted evaluations (Service or Enterprise Fully Managed) start at First requests and developer onboarding. Scenarios marked *(optional)* depend on the evaluation scope. Scoresheet document {props.body ?? 'For your convenience, this page can be downloaded as PDF or printed directly. Site navigation is omitted, the scoring table keeps its rows intact with the header repeated on every page, and the Grade and Notes columns leave room for handwriting.'} ## Grading rubric | Grade | Points | Definition | | ----- | ----- | :---- | | **Pass** | 2 | Scenario completed as documented; at most minor polish items. | | **Pass with gaps** | 1 | Scenario completed, but the documentation was unclear or workarounds not in the documentation were needed. | | **Fail** | 0 | Scenario could not be completed, or only partially; capability missing or non-functional. | On a printed scoresheet, mark the Grade column P, G, or F; the points are tallied in each section's subtotal. ## Prerequisites checklist Before starting, confirm the following: - [ ] **Console access**: sign-in works for the Agent Router Console of the evaluation environment - [ ] **Admin Dashboard access**: sign-in works with the initial admin credentials - [ ] **At least one AI provider credential** on hand (for example, an OpenAI API key) - [ ] **Terminal or HTTP client** available (curl, Python, Postman) - [ ] **OpenID Connect (OIDC) identity provider (IdP) details**: issuer URL, client ID, client secret, and an IdP administrator reachable during setup For self-hosted data plane configurations: - [ ] **A Kubernetes cluster** meeting the [prerequisites](/agent-router-enterprise/self-hosted-data-plane/prerequisites/), plus cloud CLI access Optionally, depending on the evaluation scenario: - [ ] **Model Context Protocol (MCP) client** installed (Claude Code, Cursor, or VS Code) - [ ] **OpenTelemetry-capable observability backend** for telemetry export testing ## Evaluation sections The ungraded [Fast-Track Evaluation](/agent-router-enterprise/fully-managed/fast-track/) runs first and covers the end-to-end path in an afternoon, from first access to an onboarded team. The graded sections follow: - [Installation and onboarding (Self-Hosted Data Plane only)](/product-architecture/product-evaluation/installation-and-onboarding/) - [First requests and developer onboarding](/product-architecture/product-evaluation/first-requests-and-developer-onboarding/) - [Routing, resilience, and provider abstraction](/product-architecture/product-evaluation/routing-resilience-and-provider-abstraction/) - [MCP gateway](/product-architecture/product-evaluation/mcp-gateway/) - [Observability and audit](/product-architecture/product-evaluation/observability-and-audit/) - [Cost control and budgets](/product-architecture/product-evaluation/cost-control-and-budgets/) - [Safety and data protection](/product-architecture/product-evaluation/safety-and-data-protection/) - [Identity and developer governance](/product-architecture/product-evaluation/identity-and-developer-governance/) - [Models, providers, and projects](/product-architecture/product-evaluation/models-providers-and-projects/) - [Enterprise operations and compliance](/product-architecture/product-evaluation/enterprise-operations-and-compliance/) ## Fast-track carry-over Work done in the fast track counts here. Grades for fully covered scenarios can be claimed from fast-track evidence; partial rows list what remains. | Fast-track step | Pre-completes | Coverage | | :---- | :---- | :---- | | [Step 1: Sign in and onboard](/agent-router-enterprise/fully-managed/fast-track/#sign-in-and-onboard) | 8.1 Configure single sign-on | Full | | [Step 1: Sign in and onboard](/agent-router-enterprise/fully-managed/fast-track/#sign-in-and-onboard) | 9.1 Add a provider and enable models | Partial: verify the disabled-model error separately | | [Step 2: Route the first request](/agent-router-enterprise/fully-managed/fast-track/#first-request) | 2.1 Make a first API call | Full | | [Step 3: Put real traffic through it](/agent-router-enterprise/fully-managed/fast-track/#your-real-traffic) | 2.5 Connect a coding agent | Full | | [Step 4: See what it costs](/agent-router-enterprise/fully-managed/fast-track/#see-the-cost) | 5.1 Monitor traffic and usage | Partial: locate a specific fast-track request with its provider, tokens, and latency | | [Step 5: Set a budget](/agent-router-enterprise/fully-managed/fast-track/#set-a-budget) | 6.4 Exhaust a test budget | Partial: exhaust the test budget and watch the meter and alert | | [Step 6: Bring teammates](/agent-router-enterprise/fully-managed/fast-track/#bring-teammates) | 8.3 Onboard a developer and issue keys | Partial: key revocation still to test | | [Step 6: Bring teammates](/agent-router-enterprise/fully-managed/fast-track/#bring-teammates) | 6.3 Create a team budget | Full | ## Summary scoring Count only the scenarios in scope for the evaluation (skip Installation and onboarding for fully hosted paths; skip scenarios marked optional that were not attempted) and compute the percentage of the applicable maximum. | Section | Feeds categories | Scenarios | Max | Score | | :---- | :---- | ----- | ----- | ----- | | Installation and onboarding | F, G | 6 | 12 | | | First requests and developer onboarding | G2 | 5 | 10 | | | Routing, resilience, and provider abstraction | D | 7 | 14 | | | MCP gateway | E | 4 | 8 | | | Observability and audit | C, A3 | 5 | 10 | | | Cost control and budgets | D2, D3 | 9 | 18 | | | Safety and data protection | A, B | 6 | 12 | | | Identity and developer governance | F1 | 3 | 6 | | | Models, providers, and projects | D1, G1 | 11 | 22 | | | Enterprise operations and compliance | F2, F3, G1 | 7 | 14 | | | **Total** | | **63** | **126** | | | Percentage | Fit | Typical profile | | ---------- | :---- | :---- | | 90 to 100% | Excellent fit | Nearly all scenarios pass clean; none fail | | 70 to 89% | Strong fit | Most scenarios pass; some documentation gaps | | 50 to 69% | Acceptable fit | Scenarios complete, but with gaps throughout or a few failures | | below 50% | Not recommended | Failures on core scenarios | A Fail on any non-optional scenario caps the outcome at Acceptable fit, regardless of the percentage. ## Reporting issues When a scenario fails or a guide does not match product behavior, record: 1. **Scenario ID** and the guide followed 1. **Expected behavior** per the documentation 1. **Actual behavior**, with the exact error message or screenshot 1. **Environment**: deployment path (Service, Fully Managed, Self-Hosted), browser, and OS Share the record with the Tetrate contact for resolution, and grade the scenario on what was achieved, not on the promise. Where to go next Where to begin Set up the evaluation environment: the role and path router for all three deployment paths. Compliance Residency, retention, and the due-diligence answers behind the Enterprise operations and compliance section. Sizing and scale Throughput, resource sizing, and the deployment footprint for load planning. --- ## Evaluate Agent Router Two ways to evaluate Tetrate Agent Router, depending on how the decision is being made. Run the fast-track first: it stands up a working environment that the graded scenarios then assume. Fast track evaluation for Fully Managed A guided evaluation path on the Fully Managed deployment: onboard, route real traffic through the gateway, and leave with cost and efficiency numbers. No infrastructure to stand up. Graded evaluation scenarios A scored scorecard across installation, routing, MCP, observability, cost, safety, and governance, for a structured, comparable decision. New to Agent Router, or unsure which deployment fits? Start at Where to begin. --- ## First Requests and Developer Onboarding *Feeds category G2 (developer onboarding and SDK drop-in). Record time-to-first-call: the target is 30 minutes using only the documentation.* Starting state from the [fast track](/agent-router-enterprise/fully-managed/fast-track/): SSO sign-in works and a personal API key with a working agent already exists; 2.1 and 2.5 are carried over. Grade each scenario Pass (2 points), Pass with gaps (1), or Fail (0) against the [grading rubric](/product-architecture/product-evaluation/evaluation-scenarios/#grading-rubric), and carry the subtotal into the [summary scorecard](/product-architecture/product-evaluation/evaluation-scenarios/#summary-scoring). Scoresheet document {props.body ?? 'For your convenience, this page can be downloaded as PDF or printed directly. Site navigation is omitted, the scoring table keeps its rows intact with the header repeated on every page, and the Grade and Notes columns leave room for handwriting.'} | ID | Scenario | Guide | Grade | Notes | | :-- | :---- | :---- | :---- | :---- | | 2.1 | Make a first API call: authenticate, create an API key, and send a routed request. Verify: a model response returns, and the elapsed time from start is recorded. *(Carried over from fast-track Step 2; grade from that evidence.)* | [Make an API call](/agent-router-service/quickstarts/make-an-api-call/) | | | | 2.2 | Build the sample chat app against the gateway endpoint. Verify: the app completes a multi-turn conversation. | [Build a chat app](/agent-router-service/quickstarts/build-a-chat-app/) | | | | 2.3 | Test prompts in the Playground against two different configured providers. Verify: responses and token counts are shown per run. | [Test prompts in the Playground](/agent-router-service/guides/test-prompts-in-the-playground/) | | | | 2.4 | Integrate the gateway into an existing application by swapping the base URL and key in an unmodified OpenAI-compatible SDK. Verify: the application works unchanged. | [Integrate the gateway with an app](/agent-router-service/guides/integrate-the-gateway-with-an-app/) | | | | 2.5 | Connect one coding agent (Claude Code, Cursor, or Codex) through the gateway. Verify: the agent completes a coding request and the traffic appears in request logs. *(Carried over from fast-track Step 3.)* | [Coding agents and tools](/agent-router-service/guides/coding-agents/) | | | | | **Subtotal** | | **/10** | | --- ## Identity and Developer Governance *Feeds category F1 (SSO, IdP integration, RBAC). Skip 8.1 if SSO was completed in [Installation and onboarding](/product-architecture/product-evaluation/installation-and-onboarding/).* Starting state from the [fast track](/agent-router-enterprise/fully-managed/fast-track/): SSO is configured (8.1 is carried over) and 3 to 5 teammates are already onboarded. Grade each scenario Pass (2 points), Pass with gaps (1), or Fail (0) against the [grading rubric](/product-architecture/product-evaluation/evaluation-scenarios/#grading-rubric), and carry the subtotal into the [summary scorecard](/product-architecture/product-evaluation/evaluation-scenarios/#summary-scoring). Scoresheet document {props.body ?? 'For your convenience, this page can be downloaded as PDF or printed directly. Site navigation is omitted, the scoring table keeps its rows intact with the header repeated on every page, and the Grade and Notes columns leave room for handwriting.'} | ID | Scenario | Guide | Grade | Notes | | :-- | :---- | :---- | :---- | :---- | | 8.1 | Configure single sign-on from the Admin Dashboard. Verify: a directory user signs in; password login is not required. *(Carried over from fast-track Step 1; grade from that evidence.)* | [Configure SSO](/agent-router-enterprise/guides/identity-and-access-management/single-sign-on/sso-configuration-guide/) | | | | 8.2 | Map Entra ID (or equivalent IdP) groups to business functions. Verify: group membership drives the role seen on login. | [Map Entra ID groups to business functions](/agent-router-enterprise/guides/identity-and-access-management/single-sign-on/map-entra-id-groups-to-business-functions/) | | | | 8.3 | Onboard a developer and issue keys; then revoke one. Verify: the new developer reaches the Console with scoped access, and the revoked key stops working immediately. | [Onboard developers and issue keys](/agent-router-enterprise/guides/operate-and-govern/onboard-developers-and-issue-keys/) | | | | | **Subtotal** | | **/6** | | --- ## Onboarding *Feeds categories F (enterprise readiness) and G (admin experience). Time-box: half a day plus DNS and IdP turnaround.* This section applies to Self-Hosted Data Plane evaluations only; fully hosted evaluations (Service or Enterprise Fully Managed) start at [First requests and developer onboarding](/product-architecture/product-evaluation/first-requests-and-developer-onboarding/). Grade each scenario Pass (2 points), Pass with gaps (1), or Fail (0) against the [grading rubric](/product-architecture/product-evaluation/evaluation-scenarios/#grading-rubric), and carry the subtotal into the [summary scorecard](/product-architecture/product-evaluation/evaluation-scenarios/#summary-scoring). Scoresheet document {props.body ?? 'For your convenience, this page can be downloaded as PDF or printed directly. Site navigation is omitted, the scoring table keeps its rows intact with the header repeated on every page, and the Grade and Notes columns leave room for handwriting.'} | ID | Scenario | Guide | Grade | Notes | | :-- | :---- | :---- | :---- | :---- | | 1.1 | Complete the 5-step management plane onboarding wizard: organization details, providers, models, and the data plane credential. Verify: `service-account.json` downloaded and the Admin Dashboard is populated. | [Management plane onboarding](/agent-router-enterprise/self-hosted-data-plane/onboarding/) | | | | 1.2 | Configure single sign-on (SSO) against the corporate IdP. Verify: a directory user signs in to the Console; the initial-credentials recovery path is understood. | [SSO configuration](/agent-router-enterprise/guides/identity-and-access-management/single-sign-on/sso-configuration-guide/) | | | | 1.3 | *(optional)* Map IdP app roles or groups to Agent Router roles. Verify: a directory group change propagates to the assigned role on next login. | [SSO role mapping](/agent-router-enterprise/guides/identity-and-access-management/single-sign-on/sso-role-mapping-guide/) | | | | | **Subtotal** | | **/6** | | --- ## MCP Gateway *Feeds category E (MCP gateway).* Starting state from the [fast track](/agent-router-enterprise/fully-managed/fast-track/): Console access and a working API key already exist. Grade each scenario Pass (2 points), Pass with gaps (1), or Fail (0) against the [grading rubric](/product-architecture/product-evaluation/evaluation-scenarios/#grading-rubric), and carry the subtotal into the [summary scorecard](/product-architecture/product-evaluation/evaluation-scenarios/#summary-scoring). Scoresheet document {props.body ?? 'For your convenience, this page can be downloaded as PDF or printed directly. Site navigation is omitted, the scoring table keeps its rows intact with the header repeated on every page, and the Grade and Notes columns leave room for handwriting.'} | ID | Scenario | Guide | Grade | Notes | | :-- | :---- | :---- | :---- | :---- | | 4.1 | Aggregate two or more MCP servers into a profile and connect an MCP client to the single endpoint. Verify: tools from both servers are discoverable and callable through one URL. | [Aggregate MCP servers into a profile](/agent-router-service/guides/aggregate-mcp-servers-into-a-profile/) | | | | 4.2 | Govern MCP server access, including the OAuth client lifecycle. Verify: a denied server or tool is not reachable from the client. | [Govern MCP server access](/agent-router-enterprise/guides/operate-and-govern/govern-mcp-server-access/) | | | | 4.3 | Secure MCP secrets and identity. Verify: credentials are not stored in client config, and the end-user identity is visible on the tool call trail. | [Secure MCP secrets and identity](/agent-router-enterprise/guides/operate-and-govern/secure-mcp-secrets-and-identity/) | | | | 4.4 | Scope MCP servers and profiles to a project. Verify: another project's client cannot reach the scoped servers. | [Scope MCP to a project](/agent-router-enterprise/guides/operate-and-govern/project-and-data-plane-management/scope-mcp-to-a-project/) | | | | | **Subtotal** | | **/8** | | --- ## Models, Providers, and Projects *Feeds categories D1 (provider abstraction) and G1 (admin surface).* Starting state from the [fast track](/agent-router-enterprise/fully-managed/fast-track/): providers and models from the onboarding wizard already exist; 9.1 is partially carried over. Grade each scenario Pass (2 points), Pass with gaps (1), or Fail (0) against the [grading rubric](/product-architecture/product-evaluation/evaluation-scenarios/#grading-rubric), and carry the subtotal into the [summary scorecard](/product-architecture/product-evaluation/evaluation-scenarios/#summary-scoring). Scoresheet document {props.body ?? 'For your convenience, this page can be downloaded as PDF or printed directly. Site navigation is omitted, the scoring table keeps its rows intact with the header repeated on every page, and the Grade and Notes columns leave room for handwriting.'} | ID | Scenario | Guide | Grade | Notes | | :-- | :---- | :---- | :---- | :---- | | 9.1 | Add a provider and enable or disable models from the catalog. Verify: an enabled model becomes callable and a disabled one returns a clear error. *(Partially carried over from fast-track Step 1; the disabled-model error still needs verification.)* | [Managing models and providers](/agent-router-enterprise/guides/operate-and-govern/provision-models-and-providers/) | | | | 9.2 | *(optional)* Provision AWS Bedrock models. Verify: a Bedrock model answers through the gateway. | [Provisioning AWS Bedrock models](/agent-router-enterprise/guides/operate-and-govern/provision-models-and-providers/provisioning-aws-bedrock-models/) | | | | 9.3 | *(optional)* Connect provider subscriptions across clouds. Verify: the same logical model resolves to different provider subscriptions as configured. | [Connect provider subscriptions across clouds](/agent-router-enterprise/guides/operate-and-govern/provision-models-and-providers/connect-provider-subscriptions-across-clouds/) | | | | 9.4 | *(optional)* Provision a custom or self-hosted model behind an OpenAI-compatible endpoint. Verify: the custom model is callable through the gateway. | [Provision custom and self-hosted models](/agent-router-enterprise/guides/operate-and-govern/provision-models-and-providers/provision-custom-and-self-hosted-models/) | | | | 9.5 | Create a project as the isolation boundary for a team. Verify: the project owns its own models, keys, and members. | [Create a project](/agent-router-enterprise/guides/operate-and-govern/project-and-data-plane-management/create-a-project/) | | | | 9.6 | On the project, select a data plane and confirm the gateway is auto-provisioned (inference URL ready). Verify: there is no top-level Gateways menu; the project receives its own inference URL. *(Optional Enterprise HA)* Attach additional gateways only when evaluating multi-gateway DNS failover — see [Gateway sets and DNS-level failover](/agent-router-enterprise/guides/operate-and-govern/project-and-data-plane-management/gateway-sets-and-dns-failover/); that is a separate check, not part of the default path. | [Provision a gateway](/agent-router-enterprise/guides/operate-and-govern/project-and-data-plane-management/provision-a-gateway/) | | | | 9.7 | Manage project members and roles. Verify: a member sees only their project after the switcher re-scopes. | [Manage project members](/agent-router-enterprise/guides/operate-and-govern/project-and-data-plane-management/manage-project-members/) | | | | 9.8 | Issue a project API key and test isolation. Verify: the key works on its own project's gateway and is rejected (403) on another project's. | [Issue a project API key](/agent-router-enterprise/guides/operate-and-govern/project-and-data-plane-management/issue-a-project-api-key/) | | | | 9.9 | *(optional, self-hosted)* Deploy an additional data plane. Verify: the new data plane registers and can host gateways. | [Deploy a data plane](/agent-router-enterprise/guides/operate-and-govern/project-and-data-plane-management/deploy-a-data-plane/) | | | | 9.10 | *(optional)* Manage gateways on a data plane. Verify: gateway lifecycle actions complete without dropping other projects' traffic. | [Manage gateways on a data plane](/agent-router-enterprise/guides/operate-and-govern/project-and-data-plane-management/manage-gateways-on-a-data-plane/) | | | | 9.11 | *(optional)* Migrate a gateway between data planes. Verify: the project URL is unchanged and in-flight requests survive. | [Migrate a gateway](/agent-router-enterprise/guides/operate-and-govern/project-and-data-plane-management/migrate-a-gateway/) | | | | | **Subtotal** | | **/22** | | --- ## Observability and Audit *Feeds category C (usage observability and analytics) and A3 (audit trail).* Starting state from the [fast track](/agent-router-enterprise/fully-managed/fast-track/): several days of real traffic exist to search, from the evaluator and the onboarded teammates. Grade each scenario Pass (2 points), Pass with gaps (1), or Fail (0) against the [grading rubric](/product-architecture/product-evaluation/evaluation-scenarios/#grading-rubric), and carry the subtotal into the [summary scorecard](/product-architecture/product-evaluation/evaluation-scenarios/#summary-scoring). Scoresheet document {props.body ?? 'For your convenience, this page can be downloaded as PDF or printed directly. Site navigation is omitted, the scoring table keeps its rows intact with the header repeated on every page, and the Grade and Notes columns leave room for handwriting.'} | ID | Scenario | Guide | Grade | Notes | | :-- | :---- | :---- | :---- | :---- | | 5.1 | Monitor traffic and usage in the Console: request logs, per-key usage, latency, and cost. Verify: a request made in the First requests section can be found with its provider, tokens, and latency. | [Monitor traffic and usage](/agent-router-service/guides/monitor-traffic-and-usage/) | | | | 5.2 | Export telemetry to the existing observability stack over OpenTelemetry. Verify: traces and metrics arrive in the backend with LLM attributes. | [Export telemetry to an observability stack](/agent-router-enterprise/guides/observability-and-analytics/export-telemetry-to-an-observability-stack/) | | | | 5.3 | Configure request logging modes and confirm what is captured per mode. Verify: switching modes changes the stored payload as documented. | [Configuring request logs](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/configuring-request-logs/) | | | | 5.4 | Audit platform activity: locate the audit events for the admin changes made so far. Verify: model, provider, key, and SSO changes appear with actor and timestamp. | [Audit platform activity](/agent-router-enterprise/guides/operate-and-govern/audit-platform-activity/) | | | | 5.5 | Export audit and policy decisions to a SIEM. Verify: a guardrail decision from the Safety and data protection section is queryable in the SIEM with its correlation ID. | [Export audit and policy decisions to a SIEM](/agent-router-enterprise/guides/operate-and-govern/compliance-and-governance/export-audit-and-policy-decisions-to-a-siem/) | | | | | **Subtotal** | | **/10** | | --- ## Routing, Resilience, and Provider Abstraction *Feeds category D (multi-provider routing and spend optimization).* Starting state from the [fast track](/agent-router-enterprise/fully-managed/fast-track/): a personal API key with traffic history already exists; add a second provider from the model catalog as needed. Grade each scenario Pass (2 points), Pass with gaps (1), or Fail (0) against the [grading rubric](/product-architecture/product-evaluation/evaluation-scenarios/#grading-rubric), and carry the subtotal into the [summary scorecard](/product-architecture/product-evaluation/evaluation-scenarios/#summary-scoring). Scoresheet document {props.body ?? 'For your convenience, this page can be downloaded as PDF or printed directly. Site navigation is omitted, the scoring table keeps its rows intact with the header repeated on every page, and the Grade and Notes columns leave room for handwriting.'} | ID | Scenario | Guide | Grade | Notes | | :-- | :---- | :---- | :---- | :---- | | 3.1 | Route requests across two or more providers from one endpoint. Verify: the same client code reaches both providers by model selection alone. | [Route requests across providers](/agent-router-service/guides/route-requests-across-providers/) | | | | 3.2 | Configure a fallback chain and force a failure on the primary backend. Verify: the request succeeds via the fallback within the same call, and the failover event is logged. | [Improve resilience with fallbacks](/agent-router-service/guides/improve-resilience-with-fallbacks/) | | | | 3.3 | Split traffic between two backends by percentage. Verify: the observed distribution over a batch of requests approximates the configured split. | [Reduce cost with traffic splitting](/agent-router-service/guides/reduce-cost-with-traffic-splitting/) | | | | 3.4 | Apply an attribute-based routing rule (for example, a header that routes to a specific provider). Verify: tagged requests land on the intended backend. | [Apply advanced routing rules](/agent-router-service/guides/apply-advanced-routing-rules/) | | | | 3.5 | Bring Your Own Key (BYOK): register team-owned provider credentials in a routing chain alongside centrally managed ones. Verify: requests bill against the team-owned provider account. | [Use your own provider credentials](/agent-router-service/guides/use-your-own-provider-credentials/) | | | | 3.6 | Generate embeddings through the gateway. Verify: an embeddings response returns and is logged with token usage. | [Generate embeddings](/agent-router-service/guides/generate-embeddings/) | | | | 3.7 | *(optional)* Run a batch or long-running job through the gateway. Verify: the job completes and its usage is attributed correctly. | [Run batch and long-running jobs](/agent-router-service/guides/run-batch-and-long-running-jobs/) | | | | | **Subtotal** | | **/14** | | --- ## Safety and Data Protection(Product-evaluation) *Feeds categories A (data protection and DLP) and B (prompt injection, guardrails, tool safety).* Starting state from the [fast track](/agent-router-enterprise/fully-managed/fast-track/): a working key and live agent traffic exist to put behind guardrails. Grade each scenario Pass (2 points), Pass with gaps (1), or Fail (0) against the [grading rubric](/product-architecture/product-evaluation/evaluation-scenarios/#grading-rubric), and carry the subtotal into the [summary scorecard](/product-architecture/product-evaluation/evaluation-scenarios/#summary-scoring). Scoresheet document {props.body ?? 'For your convenience, this page can be downloaded as PDF or printed directly. Site navigation is omitted, the scoring table keeps its rows intact with the header repeated on every page, and the Grade and Notes columns leave room for handwriting.'} | ID | Scenario | Guide | Grade | Notes | | :-- | :---- | :---- | :---- | :---- | | 7.1 | Protect requests with guardrails from the developer side. Verify: a guarded request shows the applied policy in its log entry. | [Protect requests with guardrails](/agent-router-enterprise/guides/operate-and-govern/safety-and-data-protection/protect-requests-with-guardrails/) | | | | 7.2 | Configure a guardrail provider and a rule that uses it (for example `regexp` with a Patterns entry, or `tetrate` with a toxicity check). Verify: a violating prompt is blocked or redacted as configured, or the would-be action is recorded in monitor mode. | [Configure guardrail providers](/agent-router-enterprise/guides/operate-and-govern/safety-and-data-protection/configure-vendor-guardrails/) | | | | 7.3 | Configure a guardrail with a keyword or regex rule. Verify: a seeded pattern triggers the configured action. | [Configure guardrails](/agent-router-enterprise/guides/operate-and-govern/safety-and-data-protection/configure-custom-guardrails/) | | | | 7.4 | Detect and redact sensitive data: send a prompt containing seeded personally identifiable information (PII) and a fake API key. Verify: the response and logs show redaction, not the raw values. | [Detect and redact sensitive data](/agent-router-enterprise/guides/operate-and-govern/safety-and-data-protection/detect-and-redact-sensitive-data/) | | | | 7.5 | Detect and block prompt injection using a known jailbreak pattern. Verify: the request is blocked or sanitized and a decision event is recorded. | [Detect and block prompt injection](/agent-router-enterprise/guides/operate-and-govern/safety-and-data-protection/detect-and-block-prompt-injection/) | | | | 7.6 | Protect coding assistant traffic with a stricter policy bundle. Verify: a seeded secret in a coding-agent request is blocked or redacted while normal traffic is unaffected. | [Protect coding assistant traffic](/agent-router-enterprise/guides/operate-and-govern/safety-and-data-protection/protect-coding-assistant-traffic/) | | | | | **Subtotal** | | **/12** | | --- ## Product overview # One control point for all AI traffic Most organisations adopt AI faster than they can govern it. Tetrate Agent Router Enterprise is an AI gateway that puts every model request through a single governed path, so that cost, access, and reliability are managed centrally instead of application by application. Requests route to 200+ models, spend is attributed by team, guardrails are enforced inline, and traffic fails over when a provider goes down, without agents being rewritten. ## What Agent Router can do for you Routing every request through one shared path is what lets developers, finance, and security win at once. These goals usually pull against each other, because the controls that satisfy finance and security tend to slow developers down; on a shared path, the request a developer sends is the same request finance meters and security inspects. Ship agents faster An OpenAI-compatible endpoint fronts every model and provider, so existing code runs unchanged. Automatic failover reroutes traffic when a provider degrades, keeping applications available. Control AI spend Per-key token budgets and usage logs cap consumption in real time. Cross-team attribution, showback, and chargeback show where spend goes. Govern every request Enterprise SSO attaches an authenticated identity to each call. Inline guardrails redact PII and enforce policy on model traffic, whether the data plane runs on Tetrate-hosted infrastructure or in the customer's own cloud, on-premises, or hybrid environment. ## Product overview Tetrate Agent Router Enterprise sits between the applications, teams, and AI agents that consume AI models and the providers that supply them (OpenAI, Anthropic, Google, Azure, Amazon, and others). Every AI request travels through one path that can be governed, observed, and secured, regardless of which provider ultimately serves it. Agent Router is not a model, and it does not replace the providers. It is the layer that makes all of them usable on consistent terms: one interface for the people building with AI, one set of controls for the people accountable for cost, security, and compliance, and one record of what is actually used. Because requests are expressed against a single OpenAI-compatible interface, existing application code reaches any configured provider without per-provider integration work. In [Agent Router Enterprise Self-Hosted Data Plane](/agent-router-enterprise/self-hosted-data-plane/) deployments, all AI traffic runs inside the customer's own environment: prompts, responses, and provider credentials never leave the customer network. Tetrate hosts only the configuration and analytics that keep Agent Router manageable, and no request content crosses that boundary. In the fully hosted paths ([Agent Router Service](/agent-router-service/quickstarts/) and [Agent Router Enterprise Fully Managed](/agent-router-enterprise/fully-managed/)), Tetrate operates the data plane as well. This division is the split-plane model described under [How it fits together](#how-it-fits-together). ## What Agent Router changes Most organisations reach Agent Router Enterprise after AI is already in production, once an improvised, per-application approach has begun to cost them in spend, risk, and operational effort. The six areas below are where a governed path replaces that improvisation. Six areas that change Provider independence Each provider is integrated, priced, and maintained differently, and when one retires a model, every application that used it has to be updated. Agent Router hides those differences behind a single OpenAI-compatible endpoint. Providers are added, swapped, or combined through configuration rather than application changes, and a routing chain can span several providers at once. The choice of provider becomes a configuration decision rather than an engineering constraint. Visibility into cost and usage Without a gateway, AI usage is scattered across provider dashboards, application logs, and spreadsheets, so basic questions (which team spent the most this month, which model returns the most errors) become investigations. Agent Router records every request in one place and reports cost and usage as it happens, broken down by team, by key, and by model. The same per-request data is exported over OpenTelemetry (OTEL) into the monitoring tools already in use. Policy-based governance Left unmanaged, AI access sprawls: developers hold raw provider keys, no one decides which models are approved for which work, and budgets surface only after the spend has happened. Agent Router centralises the controls. Models, providers, and credentials are managed in one place; each team, application, or project is granted its own access and limits; rate limits are enforced inline at the gateway while budgets track spend and raise alerts; and every administrative action is captured in an audit log. Governance becomes configuration rather than convention. Resilience to provider failure Production AI cannot depend on a single provider staying healthy. When one slows down or begins returning errors, applications without a fallback fail or behave unpredictably. Agent Router reroutes automatically: each API key carries an ordered list of backends, and the gateway advances to the next one the moment a provider fails, within the same request. The application receives a clean response, and the failure is recorded for the operations team. Agent-ready infrastructure AI agents increasingly reach tools and data through Model Context Protocol (MCP) servers, one per file store, ticketing system, or documentation source. Wiring each agent to each server by hand produces unmanageable configuration and no control over what is reachable. Agent Router aggregates MCP servers behind a single endpoint and governs them with the same identity, audit, and access controls as model traffic, so agents are adopted without a loss of oversight. Central and team-owned credentials Most organisations run on a mix of central and team-owned AI accounts: a central team funds the bulk of usage, while a business unit keeps its own provider account for compliance or billing. Agent Router supports Bring Your Own Key (BYOK), where a team's own credentials sit alongside centrally managed ones. Policy, rather than application code, decides which credentials a given request uses, and both kinds can appear in the same routing chain. ## How it fits together Two structural ideas shape how Agent Router is run, and both matter to whoever is accountable for it. The first is the split between the plane that carries traffic and the plane that stores configuration. Data plane Where AI traffic runs Handles all live AI traffic. Every prompt and response passes through it. Tetrate operates it in Agent Router Service and Agent Router Enterprise Fully Managed; in Agent Router Enterprise Self-Hosted Data Plane it runs in the customer's Kubernetes cluster so sensitive data and credentials stay inside the customer's infrastructure. See Deployment models. Management plane Hosted by Tetrate Holds configuration, policies, audit history, and analytics. It publishes configuration to the data plane and receives telemetry back, but no request content passes between the two. The two planes are joined by a connection that the data plane opens outward to the management plane. In Agent Router Enterprise Self-Hosted Data Plane that outbound-only design means no inbound access to the customer environment is required. The mechanics of that connection, and the components inside the data plane, are covered in [Planes and core components](/product-architecture/planes-and-components/). The second idea is the split between Agent Router's two applications, the **Developer Console** and the **Admin Dashboard**, covered next. The [Architecture overview](/product-architecture/architecture-overview/) traces the data plane, the management plane, and the path an AI request takes between them. ## The two applications Agent Router is used through two web applications, each aimed at a different audience but built on the same foundation. Developers work in the **Developer Console** to build against the gateway: issuing API keys, shaping routing, and watching their own traffic. Platform operators work in the **Admin Dashboard** to govern Agent Router: provisioning the models and providers, users, and policies that everyone else works within. The two are not separate products but two views of one system, so a model an operator enables in the Admin Dashboard becomes available to a developer in the Console without any handoff. Both applications sit above the same **shared backend**, which stores configuration and serves it to each application, and both ultimately shape how the **AI Gateway** behaves. That layering is the key to reading the diagram below: the operator configures a resource, the developer consumes it, and the gateway enforces the result on every request. Nothing a developer sees in the Console exists independently of what an operator has set up behind it. ### Application roles The two applications exist because developers and platform operators come to Agent Router with different questions. A developer asks how to route a request, hold a key, or test a prompt; an operator asks which models are allowed, who may use them, and what it all costs. Splitting those concerns into two applications means each audience sees only the surface relevant to its work, rather than one crowded interface that tries to serve both. The division follows a single rule: the operator provisions and governs a resource in the Admin Dashboard, and the developer consumes it in the Console. Models, providers, credentials, and policies are set up on the operator side and become the options a developer works within on the developer side. The two cards below list what each application provides, and the same pairing runs through every row: what the Admin Dashboard manages, the Console uses. For developers Developer Console The Console is the developer-facing application. It provides: API key management: Create and configure keys that authenticate requests to the AI gateway Model catalog: Browse 200+ models across all supported providers Playground: Test model routing interactively before integrating MCP Profiles: Combine multiple MCP servers into unified endpoints Integrations: Setup guides for Claude Code, Cursor, LangChain, CrewAI, and more Usage monitoring: Track request volumes, costs, and latency in real time Routing configuration: Define fallback policies and traffic splitting rules per API key For platform operators Admin Dashboard The Admin Dashboard is the operator-facing application. It provides: Model management: Enable, disable, and configure which AI models are available Provider management: Manage AI provider credentials and connection settings User management: Create accounts, assign roles, and manage access MCP servers: Manage the catalog of MCP servers available to Console users SSO configuration: Set up OIDC single sign-on Audit logs: View a tamper-resistant history of administrative actions Announcements: Publish organization-wide notifications to Console users ### How the applications connect The two applications are not separate systems that hand data back and forth; they are two front ends on the same shared backend. That backend is the single source of truth for identity, models, keys, audit records, and usage, so there is nothing to export from one application and import into the other. A model enabled, a user assigned a role, or a key revoked in one place is immediately reflected wherever that resource appears. What connects the applications, then, is the set of resources they both draw on, each from its own side. The table below reads each shared resource across the two applications: the same underlying object, with the Admin Dashboard managing it and the Console consuming it. Both applications share a common authentication system. Users sign in once and can switch between applications using the **app switcher** in the navigation bar. | Shared Resource | Console | Admin Dashboard | |-----------------------|--------------------|------------------| | User identity and SSO | Uses | Configures | | Model registry | Browses, routes to | Manages | | API keys | Creates, uses | Manages | | Audit logs | Generates | Generates, views | | Usage metrics | Views own | Views all | The division is consistent: the Admin Dashboard configures and manages a resource, and the Console consumes it. Single sign-on is configured in [Configure SSO](/agent-router-enterprise/guides/identity-and-access-management/single-sign-on/sso-configuration-guide/), and the developer and operator entry points are covered in the [Console quickstart](/agent-router-service/quickstarts/quickstart-console/) and the [Admin quickstart](/agent-router-enterprise/quickstarts/quickstart-admin-dashboard/). ## How access is organised Access in Agent Router is organised into projects. A **project** is the boundary that owns a set of models, API keys, MCP servers and profiles, routing and policy, budgets, and members, and it is the unit that usage and audit are attributed to. It is the way a shared deployment is divided cleanly among the teams and applications that use it. Each project is served by its own **gateway**, a single inference URL, so one team's or one application's configuration stays isolated from the next. That isolation is enforced at request time: a key issued for one project cannot call another project's models, and a model a project has not been granted cannot be reached through it. A gateway runs on a **data plane**, and projects can share one data plane or be spread across several by region or cloud. The full model, and how the three objects relate, is defined under [Projects](/product-architecture/key-concepts/#projects). ## The path of a request Every request follows the same governed path through the data plane, whichever provider ultimately serves it: 1. An application sends a request to its project's gateway endpoint, using the OpenAI-compatible API and a project-scoped API key. 1. The gateway authenticates the key and identifies the project and team the request belongs to. 1. The routing chain on the key selects a backend (a model on a provider, together with the credentials to reach it), applying fallback, traffic-splitting, or attribute-based rules as configured. 1. Policy is applied: guardrails inspect the prompt and rate limits are checked inline before the request is forwarded, while budgets track cumulative spend from the request logs and raise alerts rather than blocking the request. 1. The request is sent to the selected provider. If that provider fails, the gateway advances to the next backend in the chain within the same request, so the application still receives a single clean response. 1. The response returns through the gateway, where guardrails can inspect it, and the cost and usage of the exchange are recorded before the response reaches the application. Every step runs inside the data plane. The management plane supplies the rules beforehand and receives telemetry afterwards, but never sees request content. Where that data plane runs depends on the [deployment model](/product-architecture/deployment-models/). The gateway's request-handling behaviour, including error normalisation and correlation IDs, is detailed in [Gateway behavior](/reference/gateway-behavior/). ## What Agent Router provides Agent Router's capabilities fall into four areas. | Area | What it includes | | :---- | :---- | | **Routing** | One endpoint to 200+ models across OpenAI, Anthropic, Google, Azure, Mistral, and more; automatic fallback between providers; weighted traffic splitting; and routing rules based on request attributes. | | **Identity and access** | Per-team and per-application access, each with its own limits; enterprise single sign-on (SSO); role-based access in both the Console and the Admin Dashboard; and Bring Your Own Key (BYOK) alongside centrally managed credentials. | | **Observability** | Request logs, usage analytics, and cost reporting in the Admin Dashboard; export to existing monitoring tools over OpenTelemetry; and audit logging for every administrative action. | | **Agent infrastructure** | MCP aggregation behind a single endpoint; tool catalogues usable by AI clients such as Claude Code, Cursor, and VS Code; and a Playground for testing prompts against any configured provider. | Each area is expanded, with the specifics that matter when evaluating the product or planning a rollout, in [Solution capabilities](/product-architecture/ai-capabilities/). The vocabulary used throughout is defined in [Key concepts](/product-architecture/key-concepts/). ## Ways to run it Agent Router is delivered in three get-started paths: [Agent Router Service](/agent-router-service/quickstarts/), [Agent Router Enterprise Fully Managed](/agent-router-enterprise/fully-managed/), and [Agent Router Enterprise Self-Hosted Data Plane](/agent-router-enterprise/self-hosted-data-plane/). The management plane is always Tetrate-hosted; the paths differ in where the data plane runs and who operates it. The trade-offs between them, and the networking, residency, and operational implications of each, are covered in [Deployment models](/product-architecture/deployment-models/). ## Where to go next Continue reading Who this documentation is for The readers the documentation serves (evaluators, developers, and operators) and where each should start. Key concepts Defines the vocabulary used across the documentation, including data plane, project, fallback policy, BYOK, and MCP profile. Solution capabilities How Agent Router passes structured outputs, multimodal inputs, and function calling through to any provider, translating across providers where mappings exist. Architecture overview Walks through the data plane, the management plane, and the path a request takes between them. Planes and core components Where the two applications sit relative to the data plane, and how hosting differs by deployment model. Deployment models How the applications are delivered in the fully hosted and self-hosted paths. Get started Covers prerequisites and installation on AWS, Azure, and GCP. Guides for developers The developer workflows, for readers whose role is already clear. Guides for platform operators The operator workflows, for readers whose role is already clear. --- ## Management API reference This section is the OpenAPI catalog of Agent Router HTTP services: control-plane and management APIs (clients, keys, catalog, projects, insights, and related services), plus a small set of inference management RPCs. It is the place to look up method paths, request and response schemas, and to try calls interactively. :::tip Gateway model calls live elsewhere To send prompts with OpenAI- or Anthropic-compatible request shapes (`/v1/chat/completions`, `/v1/responses`, `/v1/messages`, `/v1/models`, and related gateway paths), use [Gateway APIs](/reference/supported-apis/). Those wire formats are not listed as OpenAPI operations in this catalog. ::: ## Introduction There are two ways to call the APIs documented here: raw HTTP, or the typed Agent Router SDKs. Gateway requests that reach the inference data plane authenticate with a bearer API key; not every control-plane method that creates a key mints a credential the gateway will accept. See [Which API key for which surface](#which-api-key-for-which-surface). - **HTTP APIs.** Plain HTTPS against the documented paths: curl, an HTTP client, or a generated client from the OpenAPI spec. - **Language SDKs.** Typed clients for Go, Python, and TypeScript wrap the same HTTP APIs with helpers for authentication, retries, and streaming. The SDKs mirror the HTTP surface, so anything described in this reference is reachable either way. ## Which API key for which surface Not every control-plane "create a key" method produces a credential the inference data plane will accept. | Goal | Use | Endpoint | | :---- | :---- | :---- | | Call models through the inference data plane | [CreateClientWithKey](/reference/api/clients/agentrouter-clients-v-1-clients-service-create-client-with-key/) | `POST /v1/customers/{customer_id}/projects/{project_id}/clients` | | Mint a JWT user-token for a user | [CreateApiKey](/reference/api/api-keys/agentrouter-api-keys-v-1-api-keys-service-create-api-key/) | `POST /v1/api-keys` | Keys from **CreateApiKey** are rejected by the inference data plane. If gateway requests return **401** after you created a key with CreateApiKey, create a client key with CreateClientWithKey instead and use that bearer token. When calling CreateApiKey, `user_key_address` must equal the owner's `user_id`. Passing an email address is rejected. ## Which cost fields to read Request logs and Insights stats expose more than one cost representation. Use the decimal string fields for metering and billing; do not rely on the microdollar integers. | Use for | Fields | | :---- | :---- | | Canonical cost (includes platform fee/markup) | `cost`, `inputTokensCost`, `outputTokensCost`, `cachedInputTokensCost`, `cacheCreationInputTokensCost` | | Pre-fee provider cost | `inputTokensCostBeforeFee`, `outputTokensCostBeforeFee`, `cachedInputTokensCostBeforeFee`, `cacheCreationInputTokensCostBeforeFee` | | Avoid | `costMicrodollar`, `inputTokensCostMicrodollar`, `outputTokensCostMicrodollar`, and other `*CostMicrodollar` fields — these currently return 0 | Decimal cost fields include the platform fee/markup. The corresponding `*BeforeFee` fields carry the raw provider cost before that markup. Prefer summing or reading the decimal strings when attributing usage per API key or building a credit ledger. ## Get the full API specification The complete management API is published as a single OpenAPI 3.1 document covering every service and endpoint in this catalog. Import it into Postman, Insomnia, or Swagger UI, or feed it to a client generator such as `openapi-generator`. Download the OpenAPI spec (YAML) Always-current spec. The pinned agentrouter-openapi-0.1.1.yaml is also available at the same path. Browse the interactive reference Per-endpoint pages with request and response schemas and a built-in "try it" console. Every service is listed under Management API reference in the sidebar. ## Make your first call For a hands-on walkthrough, [Make an AI API call through Agent Router](/agent-router-service/quickstarts/make-an-api-call/) covers getting a key, pointing an SDK at the gateway, and streaming a response in a few minutes. That guide also covers the `tare` CLI and the typed Agent Router SDKs. ## Related Gateway APIs Gateway wire formats for model calls (OpenAI- and Anthropic-compatible), with request and streaming examples. Gateway behavior Error normalization, correlation IDs, and the request debugging workflow. --- ## API tags A tag in Tetrate Agent Router is a key/value label used to group API keys along a dimension the fixed user and team breakdowns cannot express, such as an application, an environment, or a cost centre. Tags are governed rather than free-form: the Admin Dashboard holds a catalog of allowed key/value pairs, and only pairs present in that catalog can be attached to API keys, targeted by policies, or used to filter the usage analytics. Keeping the tag space closed is what makes a tag-grouped cost report summable, because an ungoverned tag space accumulates near-duplicate labels that no report can add together. :::info Two tag surfaces This page covers the API key tag catalog managed in the Admin Dashboard, and, separately, the free-form `tags` map carried on secrets in the Secret API. The two are independent: the catalog constrains which pairs may be attached to API keys, while secret tags are arbitrary strings validated by no catalog. See [Tags on secrets](#tags-on-secrets) for the API surface. ::: ## The tag catalog The catalog is reached in the Admin Dashboard under **Directory** > **API Keys**, then the **Tags** tab. The page caption states the contract directly: allowed key/value pairs that can be attached to API keys and targeted by policies. ![The Tags tab of the API Keys page, listing nine allowed key/value pairs](/img/screenshots/admin/api-key-tags-catalog.jpeg) Each entry is one complete `key=value` pair, not a bare key. Declaring `env` as a dimension is not sufficient: `env=dev` and `env=prod` are two separate catalog entries, and a third value such as `env=sandbox` has to be added before any API key can carry it. Keys are assigned a colour automatically and consistently, so entries sharing a key read as a group in the list and on the filter chips. ### Catalog list The list is sorted by key, then by value. Each row carries the following. | Element | Description | | --- | --- | | Tag chip | The pair, rendered as `key=value` with the colour dot of its key | | Description | The optional description recorded with the pair, or `—` when none was given | | Date | When the pair was added to the catalog | | Delete | Trash icon that removes the pair from the catalog | Above the list, a row of filter chips summarises the catalog: an **All** chip with the total number of pairs, then one chip per key with the number of values defined for that key. The header inside the list card repeats the total, for example `9 tags`. ### Filtering the list by key Selecting a key chip narrows the list to the values defined for that key. The card header changes to name the key and its value count, and a **Show all** control clears the filter. ![The catalog filtered to the team key, showing three values](/img/screenshots/admin/api-key-tags-filtered.jpeg) **Refresh** re-reads the catalog from the management plane. The filter chip counts and the list are refreshed independently, so a count may update a moment before the rows do. ## Add a tag **Add tag** opens the **New tag** panel, described as adding an allowed key/value pair to the tag catalog. 1. Select or type a **Tag key**. The field is a combo box, placeholder `Choose or create a key…`, that lists the keys already in the catalog with the number of values defined for each. An existing key can be picked from the list, or a new key typed to open a new dimension. 1. Enter a **Tag value**. The value field stays disabled until a key is chosen, showing the placeholder `pick a key first`. Once a key is set, the field is prefixed with a `key=` chip so the pair being created is visible while typing. 1. Optionally expand **Add description (optional)** and record what the pair means. Descriptions are shown in the catalog list and are the only place the intent of a value is written down. 1. Select **Create**. The button reports progress while the pair is written, then a toast confirms the result, for example `Tag "env=sandbox" created`, and the new pair appears in the list with its key count incremented. ![The New tag panel with the tag key combo box open, listing existing keys and their value counts](/img/screenshots/admin/api-key-tags-new-tag.jpeg) ### Field reference | Field | Required | Notes | | --- | --- | --- | | Tag key | Yes | Chosen from the existing keys or typed to create a new dimension | | Tag value | Yes | Enabled only after a key is set; suggestions offered from the values already defined for that key | | Description | No | Free text, shown in the catalog list | ### Validation The panel footer states why creation is currently blocked, and the **Create** button stays unavailable until both fields are set. | Footer message | Meaning | | --- | --- | | `Blocked: key required` | No key has been chosen or typed | | `Blocked: value required` | A key is set but the value is empty | | `Creating key=value` | Both fields are set and the pair shown will be created | Two further checks guard the shape of the catalog rather than the shape of the input. **Near-duplicate keys.** Typing a key close to one already in the catalog raises a warning that near-duplicate keys fragment the catalog, naming the existing key and offering a **Use** control that replaces the typed text with it. This is the guard against `app` and `appp` both existing as dimensions, which would split a cost report in two along a typo. ![The near-duplicate key warning, offering to use the existing app key instead of the typed appp](/img/screenshots/admin/api-key-tags-duplicate.jpeg) The warning is advisory: a genuinely new key that merely resembles an existing one can still be created by ignoring it. **Existing keys and taken values.** When the key already exists, a hint below the field reports how many values are defined for it. The value field then suggests those values, each marked `TAKEN`, so an already-catalogued pair is not submitted a second time. ![The tag value field suggesting the existing dev and prod values, both marked TAKEN](/img/screenshots/admin/api-key-tags-value-taken.jpeg) ## Delete a tag The trash icon on a row removes that key/value pair from the catalog. When the last value of a key is removed, the key stops being a dimension and its filter chip disappears. :::caution Deletion narrows what policies can target The catalog is the set of pairs that policies may target and that API keys may carry. Removing a pair that a budget or rate limit policy selects on, or that keys in service already carry, changes which traffic that policy matches. Confirm that a pair is unused before removing it, and record the change alongside the other governance changes described in [Audit Agent Router activity](/agent-router-enterprise/guides/operate-and-govern/audit-platform-activity/). ::: ## Filter usage by a key tag The usage analytics draw on the catalog directly. The Analytics page under **Usage** carries a **Key tag** filter chip alongside the existing **User**, **Team**, and **Group** chips. Selecting one catalogued pair narrows the view to the API keys carrying that tag, which is what turns a tag dimension into a cost report rather than a label. ### Applying the filter 1. Open the Analytics page under **Usage** and select the **Cost** or **Tokens** tab. 1. Select the **Key tag** chip. A picker opens with a **Search catalog tags…** field above the pairs available from the catalog, and typing narrows that list. 1. Select a pair. The chip becomes active and shows the selected pair next to its label, for example `env=dev`, and a **Reset** control appears at the end of the filter row. ![The Key tag picker open on the Cost tab, listing the env=dev and env=prod pairs from the catalog](/img/screenshots/admin/usage-key-tag-picker.jpeg) Everything on the tab narrows to keys carrying the selected pair: the chart, the **Total cost**, **Total requests**, and **Total tokens** summary tiles, the per-key table and its result count, and the **Download CSV** export. ![The Cost tab filtered to env=dev, with the chart and summary tiles narrowed and a Reset control alongside the chip](/img/screenshots/admin/usage-key-tag-applied.jpeg) The filter is cleared by selecting the same pair again, by the clear icon on the chip itself, or by **Reset**. ### Where the filter applies | Tab | Key tag filter | | --- | --- | | Cost | Applied | | Tokens | Applied | | API Key, Errors, Clients, Model Metrics, MCP | Not applied; the chip is dimmed | On the tabs where it does not apply, the chip reports that the key-tag filter is not applied on that tab yet. Those tabs are built from data that does not record which API key served the request, so there is nothing for the filter to match on. A selection made on **Cost** or **Tokens** is retained while another tab is open, and takes effect again on return. ### Sharing a filtered view The selection is held in the page URL as two separate parameters, `tagKey` and `tagValue`: ```text /usage?tagKey=env&tagValue=prod ``` A filtered view therefore survives a page reload, can be bookmarked, and can be sent to a colleague. Where the parameters are hand-edited into something invalid, the filter is dropped and unfiltered data is shown rather than an error. ### How matching works - **Current tags, not historical.** Matching uses a key's tags as they stand now, so tagging a key today brings that key's entire usage history into the filter, including usage recorded before the tag existed. There is no view of what a key was tagged at the time a request was served. - **One pair at a time.** Pairs cannot yet be combined into a compound filter. - **Tags are per project, usage is organisation-wide.** Filtering on `env=prod` returns usage from every key carrying that pair, across all projects visible to the operator. - **Reserved tags are excluded.** Tags in the reserved `internal:` namespace cannot be used as a filter, consistent with the rest of the product. ### Permissions Applying the filter needs only the usual usage read access. Browsing the catalog to pick a pair needs tag catalog read access, and the two are checked separately. That split has a useful consequence: a shared filtered link opened by someone without catalog access still shows correctly filtered figures. The chip reports that the filter is applied but that the catalog is not available, and offers a control to clear it, so the numbers can be read but a different pair cannot be chosen. Where the tags feature is not enabled in an environment, or a role carries no tag permissions at all, the chip is not shown. ## Why the catalog is governed A tag is only useful for chargeback if everyone producing requests and everyone reading reports uses the same names. Where the tag space is open, the long tail of near-duplicate labels arrives quickly: `env=prod` beside `env=production`, `app` beside `application`. No report can sum across them, and a cost line item becomes arguable rather than billable. Closing the tag space is what removes that argument. Because a pair has to exist in the catalog before an API key can carry it, the set of values a report can group by is known in advance, and the near-duplicate warning stops the catalog fragmenting as it grows. A report grouped by a tag that Agent Router required and validated carries the same authority as the user dimension behind it. The governance intent behind these dimensions, and the reports they feed, are covered in [Know what every app and project costs](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/know-what-every-app-and-project-costs/). Where a tag dimension sits among the other cost controls is covered in [Choose the right cost control](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/choose-the-right-cost-control/). ### Naming conventions The catalog constrains which pairs exist; it does not choose good ones. The following conventions keep a catalog readable as it grows. - Use lower case keys with hyphens, so that `cost-center` is never competing with `CostCenter` and `costCenter`. - Prefer one spelling of a dimension and keep it: `environment` or `env`, `app` or `application`, not both. - Keep values short and enumerable, so a key such as `env` carries a handful of known values rather than free text. - Record a description on each pair. The catalog is the only place the meaning of `tier=external` is written down. - Add a value when it is needed rather than pre-declaring dimensions, since every catalogued pair is a pair a report may have to account for. ## Tags on secrets Secrets carry their own `tags` map, independent of the API key tag catalog. The map is free-form: keys and values are arbitrary strings, and no catalog constrains them. Replace `https://api.` with the gateway base URL for the deployment. For the hosted service this is `https://api.tetrate.ai`. ### Setting tags on a secret `POST /v1/secrets` creates or updates a secret. The `tags` field is an object whose values are strings. ```bash curl -X POST https://api./v1/secrets \ -H "Authorization: Bearer $API_KEY" \ -H "Content-Type: application/json" \ -d '{ "name": "openai-prod-key", "realm": "llm-providers", "value": "sk-...", "description": "OpenAI API key for production", "tags": { "app": "chatbot", "environment": "production", "cost-center": "engineering" } }' ``` The `realm` field scopes the secret. The values documented in the specification are `llm-providers`, `byok`, `mcp`, `telemetry`, and `kek`. Because the same operation creates and updates, the `tags` map submitted is the map stored. A request that omits a key previously present does not merge with what was there. ### Reading tags `GET /v1/secrets/{name}` returns the secret's metadata, including its `tags`. Secret responses never include the plaintext value, which is retrieved separately through `POST /v1/secrets/{name}/value`. ```json { "secret": { "id": "sec_abc123", "name": "openai-prod-key", "realm": "llm-providers", "description": "OpenAI API key for production", "tags": { "app": "chatbot", "environment": "production", "cost-center": "engineering" }, "status": "SECRET_STATUS_ACTIVE", "createdAt": "2026-03-15T10:30:00Z", "updatedAt": "2026-03-15T10:30:00Z" } } ``` ### Filtering by tag `GET /v1/secrets` accepts `tags.key` and `tags.value` as query parameters. These name the key and the value of a single pair to match on; they are not a `tags.=` family of parameters, and one pair is filtered per request. ```bash # All secrets carrying environment=production curl -X GET "https://api./v1/secrets?tags.key=environment&tags.value=production" \ -H "Authorization: Bearer $API_KEY" # Narrowed to one realm curl -X GET "https://api./v1/secrets?realm=llm-providers&tags.key=cost-center&tags.value=engineering" \ -H "Authorization: Bearer $API_KEY" ``` `realm`, `status`, `pageSize`, and `pageToken` may be combined with the tag filter. ### Operation summary | Operation | Method and path | Tag behaviour | | --- | --- | --- | | ListSecrets | `GET /v1/secrets` | Filters on one pair through `tags.key` and `tags.value` | | CreateSecret | `POST /v1/secrets` | Accepts a `tags` map; creates or updates the secret | | GetSecret | `GET /v1/secrets/{name}` | Returns the stored `tags` in the secret metadata | | RotateSecret | `POST /v1/secrets/{name}/rotate` | Takes `newValue`; returns the rotated secret's metadata, including its `tags` | | DeleteSecret | `DELETE /v1/secrets/{name}` | Removes the secret and its tags | Full request and response schemas are in the [Secret API reference](/reference/api/secret/agentrouter-secret-v-1/). ## Related Know what every app and project costs Agreeing tag dimensions with finance, then exporting the reports grouped along them. Choose the right cost control Where a tag dimension sits among budgets, rate limits, and quotas. Onboard developers and issue keys The API Keys page the tag catalog sits alongside. Audit log events The event reference for tracking catalog and policy changes. --- ## Audit log events The audit log captures a structured record of every state-modifying administrative action in Agent Router. Each entry is immutable, cannot be edited or deleted by any user, and supports compliance frameworks such as SOC 2 and ISO 27001. This page documents the wire schema returned by the management API and shown in Admin Dashboard exports: the fields on every entry, how actions and resources are named, and how to join related rows. For the surface that browses the log, see Audit Agent Router activity. For the OpenAPI operations that query entries, see QueryAuditLogs and GetAuditLog. ## Event envelope Every audit entry mirrors a row of the `audit_logs` table. The management API exposes that row as `AuditLogEntry` (JSON field names are camelCase; OpenAPI also documents snake_case titles such as `action_type`). | Field | Type | Description | | --- | --- | --- | | `id` | string | Server-generated audit entry UUID (primary key) | | `timestamp` | string (RFC 3339, UTC) | When the audited action occurred | | `createdAt` | string (RFC 3339, UTC) | When the audit row was persisted (defaults to the event time) | | `userId` | string | Id of the user that performed the action; empty for system actions | | `userEmail` | string | Email of the acting user, captured at write time | | `actorType` | string | Actor category (for example `user`, `system`, or `service_account`) | | `actionType` | string | Action performed (for example `CREATE`, `UPDATE`, `DELETE`, `CREATE_SECRET`, or `SET_GUARDRAIL_STATUS`) | | `resourceType` | string | Type of the affected resource (for example `secret`, `provider`, `guardrail`, or `api_key`) | | `resourceId` | string | Id of the affected resource | | `success` | boolean | Whether the action succeeded | | `statusCode` | integer | HTTP status code of the response | | `errorMessage` | string | Error detail when the action failed; empty on success | | `sourceIp` | string | Client IP the request originated from | | `userAgent` | string | User-Agent header of the originating request | | `httpMethod` | string | HTTP method of the originating request (for example `POST`) | | `endpoint` | string | Request path or RPC endpoint that triggered the action | | `correlationId` | string | Correlation id grouping related entries from one operation | | `metadata` | object | Additional structured context attached to the entry | | `requestParams` | object | Request path or query parameters captured for the action | | `requestBody` | object | Request body captured for the action (sensitive values may be redacted) | | `responseData` | object | Response payload captured for the action | `userId`, `userEmail`, and `actorType` answer *who*. `actionType`, `resourceType`, and `resourceId` answer *what* and *to which thing*. `metadata`, `requestBody`, `requestParams`, and `responseData` carry operation-specific context. `correlationId` joins related rows from one administrative call, and can be used with other telemetry that shares the same identifier. The Admin Dashboard **Audit Logs** table projects a subset of these fields (timestamp, actor, action, resource, source, and result). The UI may show a human-readable resource label next to `resourceId`; that label is a presentation detail, not a separate wire field on `AuditLogEntry`. --- ## Action and resource vocabulary `actionType` and `resourceType` are stable strings that dashboard filters and SIEM consumers use. They are not dotted `.` names. ### Action types Many mutations use a verb-plus-resource form such as `CREATE_SECRET`, `UPDATE_GUARDRAIL`, or `DELETE_DATAPLANE`. Shorter verbs such as `CREATE`, `UPDATE`, `DELETE`, and `UPGRADE` also appear. Representative values include: | `actionType` | Typical when | | --- | --- | | `CREATE_SECRET` / `DELETE_SECRET` / `ROTATE_SECRET` | Secret lifecycle | | `CREATE_GUARDRAIL` / `UPDATE_GUARDRAIL` / `DELETE_GUARDRAIL` / `SET_GUARDRAIL_STATUS` | Guardrail lifecycle and enablement | | `CREATE_GUARDRAIL_RULE` / `UPDATE_GUARDRAIL_RULE` / `DELETE_GUARDRAIL_RULE` | Guardrail rule changes | | `CREATE_GUARDRAIL_PROVIDER` / `UPDATE_GUARDRAIL_PROVIDER` / `DELETE_GUARDRAIL_PROVIDER` | Guardrail provider changes | | `ADD_PROJECT_MODEL` / `REMOVE_PROJECT_MODEL` | Project model catalogue changes | | `ADD_PROJECT_PROVIDER` / `REMOVE_PROJECT_PROVIDER` | Project provider attachment | | `CREATE_CLIENT_KEY` / `ADD_CLIENT_KEY` / `ROTATE_CLIENT_KEY` / `DISABLE_CLIENT_KEY` / `ENABLE_CLIENT_KEY` / `DESTROY_CLIENT_KEY` | API key and client-key lifecycle | | `CREATE_PROJECT` / `UPDATE_PROJECT` / `DELETE_PROJECT` | Project lifecycle | | `ADD_PROJECT_MEMBER` / `UPDATE_PROJECT_MEMBER` / `REMOVE_PROJECT_MEMBER` | Project membership | | `CREATE_DATAPLANE` / `UPDATE_DATAPLANE` / `DELETE_DATAPLANE` / `SET_DATAPLANE_URL` | Data plane lifecycle | | `CREATE_PROJECT_GATEWAY` / `UPDATE_PROJECT_GATEWAY` / `DELETE_PROJECT_GATEWAY` | Project gateway lifecycle | | `ATTACH_PROJECT_GATEWAY_PROJECT` / `DETACH_PROJECT_GATEWAY_PROJECT` | Gateway-to-project binding | | `CREATE_TELEMETRY_CONFIG` / `UPDATE_TELEMETRY_CONFIG` / `DELETE_TELEMETRY_CONFIG` | Telemetry and SIEM forward configuration | | `UPSERT_SYSTEM_SETTING` / `UPSERT_PROJECT_SETTING` / `DELETE_PROJECT_SETTING` | Settings changes | This table is representative, not exhaustive. New management APIs can introduce additional `actionType` values; filter on observed values from [QueryAuditLogs](/reference/api/admin/agentrouter-admin-v-1-admin-service-query-audit-logs/) or the Audit Logs UI when building SIEM rules. ### Resource types | `resourceType` | Resource family | | --- | --- | | `secret` | Stored secrets | | `guardrail` / `guardrail_rule` / `guardrail_provider` | Guardrails | | `project` / `project_member` / `project_model` / `project_provider` / `project_setting` / `project_mcp_server` | Project-scoped resources | | `project_gateway` | Project gateways | | `dataplane` | Data planes | | `client` / `client_key` | Clients and API keys | | `user_token` | User tokens | | `service_account` | Service accounts | | `customer` | Customers | | `telemetry_config` | Telemetry forward configs | | `system_setting` | Deployment-global settings | | `mcp_route` | MCP route status | --- ## Example event ```json { "id": "8f14e45f-ceea-467f-a8f0-6b2a1c3d4e5f", "timestamp": "2026-07-25T19:11:11Z", "createdAt": "2026-07-25T19:11:11Z", "userId": "c631d6d5-afb2-41fe-a284-eaf7be377c14", "userEmail": "alice@example.com", "actorType": "user", "actionType": "CREATE_SECRET", "resourceType": "secret", "resourceId": "a1b2c3d4-e5f6-7890-abcd-ef1234567890", "success": true, "statusCode": 200, "errorMessage": "", "sourceIp": "203.0.113.10", "userAgent": "Mozilla/5.0", "httpMethod": "POST", "endpoint": "/v1/secrets", "correlationId": "corr-7c9e6679-7425-40de-944b-e07fc1f90ae7", "metadata": { "name": "0-salesforce" }, "requestParams": {}, "requestBody": {}, "responseData": {} } ``` Sensitive values in `requestBody`, `responseData`, and `metadata` may be redacted. Credential material is not written in cleartext to the audit log. --- ## Coverage and immutability Two properties of the audit log are worth asserting for compliance and integrity purposes. - **Coverage.** Every state-modifying administrative action generates an audit event. There is no path through the Admin Dashboard that changes Agent Router state without leaving a record. - **Immutability.** Audit entries cannot be edited or deleted by any user, regardless of role. There is no "delete from the log" operation anywhere in Agent Router. Signed Merkle checkpoints support tamper-evidence for sealed windows of the log; see the Admin API's audit checkpoint operations. Retention is configured per deployment. For long-horizon retention beyond the in-Console window, export the audit stream to an external SIEM or log store; see [Export audit decisions to SIEM](/agent-router-enterprise/guides/operate-and-govern/compliance-and-governance/export-audit-and-policy-decisions-to-a-siem/). --- ## Related - [Audit Agent Router activity](/agent-router-enterprise/guides/operate-and-govern/audit-platform-activity/), the surface for browsing audit events and usage analytics - [QueryAuditLogs](/reference/api/admin/agentrouter-admin-v-1-admin-service-query-audit-logs/) and [GetAuditLog](/reference/api/admin/agentrouter-admin-v-1-admin-service-get-audit-log/), management API operations that return `AuditLogEntry` - [OTel Metrics](/reference/otel-metrics/), metric stream that complements the audit stream Where to go next Audit Agent Router activity Review and export the audit log alongside usage analytics. Compliance Certifications, data residency, and retention posture. --- ## Guardrails best practices & FAQ Guardrails in Agent Router Enterprise run inline on model traffic that crosses the gateway — on both inputs and outputs to large language models (LLMs). Model Context Protocol (MCP) traffic is governed separately through identity, access control, and audit; content guardrails do not evaluate standalone MCP profile requests today. Rolling guardrails out safely means understanding how the provider kinds differ in latency and capability, promoting rules through monitor mode before enforcement, and choosing fail-open versus fail-closed behavior deliberately. This page collects the operational practices and frequently asked questions that recur when teams deploy guardrails in production: how to avoid self-inflicted outages, how to investigate a blocked request, and what the runtime layer protects — and what falls outside its scope. **Persona:** Platform operator or security stakeholder rolling out guardrails in production. **Estimated time:** 15 to 20 minutes to read; ongoing for rollout and tuning. ## What you will learn - How the provider kinds (`regexp`, `builtin`, `tetrate`) work and how they differ in latency and capability. - How to roll out guardrails without causing outages, using monitor mode and deliberate fail-open and fail-closed settings. - How to investigate a blocked request and which role to grant for read-only review. - What guardrails protect at runtime, and what falls outside their scope. ## Best practices ### 1. Roll out new guardrails in monitor mode first Each rule has an **action** and a **mode**: - **Action: redact** — modifies the request or response inline, substituting anonymized placeholders or masking the matched content. - **Action: block** — stops the request and returns an HTTP `40x` with an optional message (often including a correlation identifier). - **Mode: monitor** — audit-only. The gateway records what it *would* have done without changing the request or response. Use this as a shadow mode to validate a rule against real traffic. - **Mode: enforce** — applies the configured action. The most common cause of self-inflicted outages is promoting a rule straight to enforce with block before validating it. Start in monitor mode, confirm the trigger rate and false-positive rate are acceptable, then switch mode to enforce. The step-by-step configuration workflow is covered in [Configure guardrails](/agent-router-enterprise/guides/operate-and-govern/safety-and-data-protection/configure-custom-guardrails/). ### 2. Order guardrails so cheap checks run first Latency varies by provider kind: - **`regexp`** — near zero; in-process pattern matching against the Patterns catalog. - **`builtin`** — near zero; in-process multi-layer keyword and regex pipeline. - **`tetrate`** (and any remote / external guardrail service where one is configured) — typically 100 ms or more, driven mostly by the network hop and inference. Stack rules so cheap local providers evaluate first. A pattern match can short-circuit evaluation before a remote ML call ever runs, which keeps tail latency down on the common path. See [Configure guardrail providers](/agent-router-enterprise/guides/operate-and-govern/safety-and-data-protection/configure-vendor-guardrails/). ### 3. Set fail-open vs. fail-closed deliberately Guardrail rules default to **fail-closed**. Both the failure behavior and the timeout are configurable per rule. Choose based on the path: fail-closed when safety dominates, fail-open when availability dominates. Set an explicit timeout rather than relying on the default. ### 4. Scope code-execution guardrails narrowly Guardrails that reject responses containing executable code will break any coding assistant if applied across the fleet. Apply them to contexts where generated code is never expected — a support agent, for example — rather than globally. See [Protect coding assistant traffic](/agent-router-enterprise/guides/operate-and-govern/safety-and-data-protection/protect-coding-assistant-traffic/) for scoping guidance. ### 5. Treat tuning as a controlled change Guardrails expose real configuration knobs. Tightening sensitivity too far produces false-positive blocks that affect end users. Validate threshold changes in monitor mode before applying them to live traffic, and restrict who can modify guardrail configuration (see the guardrail-reader role below). ## FAQ ### Where do guardrails run? Does my data leave my network? Local providers (`regexp` and `builtin`) run co-located with the router, inside your network. Data they evaluate does not leave your network. Data leaves only if you explicitly use a remote provider such as `tetrate`, or an external guardrail service where one is configured, in which case you incur the network hop to that endpoint. See [Configure guardrail providers](/agent-router-enterprise/guides/operate-and-govern/safety-and-data-protection/configure-vendor-guardrails/). ### Where does Agent Router Enterprise sit relative to an agent? An agent is the combination of the client (the programmatic harness), the tools, and the LLM. Agent Router Enterprise sits between the client and the LLM and the MCP servers — in front of the model and the tools. Content guardrails inspect what the client sends to the model and what comes back on that model path. MCP profile traffic (`/mcp/*`) is subject to identity, access control, and audit, but not to content guardrails today. For the developer-side view of which traffic is protected, see [Protect requests with guardrails](/agent-router-enterprise/guides/operate-and-govern/safety-and-data-protection/protect-requests-with-guardrails/). ### What guardrail providers are supported? Three provider kinds ship by default, arranged as a cost-and-capability ladder: 1. **`regexp`** — fast pattern recognition for known formats such as email addresses and national identifiers. Uses the Patterns library; custom patterns are supported. 2. **`builtin`** — a five-layer local keyword and regex pipeline (blocked keywords, allowlist gate, subject-plus-term co-occurrence, detection patterns, category keywords). 3. **`tetrate`** — remote ML detection (embeddings, classifiers, NLI) for PII, toxicity, jailbreak, prompt injection, banned topics, and related check types. Where an external guardrail service is configured, it appears as another provider instance rules can reference. Configuration details are in [Configure guardrails](/agent-router-enterprise/guides/operate-and-govern/safety-and-data-protection/configure-custom-guardrails/) and [Configure guardrail providers](/agent-router-enterprise/guides/operate-and-govern/safety-and-data-protection/configure-vendor-guardrails/). ### Can I plug in my own models? The `tetrate` provider is backed by Tetrate's semantic router and ships with task-specific models for the supported check types. Where an external guardrail service is configured, its capabilities depend on that service's contract. See [Configure guardrail providers](/agent-router-enterprise/guides/operate-and-govern/safety-and-data-protection/configure-vendor-guardrails/). ### Do guardrails run on outputs as well as inputs? Yes. Guardrails evaluate both inputs and outputs on model traffic. You can, for example, redact PII on the way in and block disallowed content on the way out. Standalone MCP profile traffic is not evaluated by content guardrails. Note that not every check runs on both stages for every provider kind: some (provider, check) pairs are input-only or output-only, and a rule that leaves its **Run on** setting unset is asking for *both* stages. See the [capability matrix](/reference/guardrails/#provider-support-by-check-type). ### Why was my rule or provider rejected with an invalid-argument error? The API validates every guardrail rule and provider write against the platform's capability matrix: the check type must be supported by the provider's kind, and the requested stage (unset means both) must be within the stages that (kind, check) pair supports. Unsupported combinations are rejected at write time with an error naming the supported set, so a rule that could never fire cannot be created. Provider creation likewise rejects unknown kinds, and external-service providers must declare a valid, non-empty check list. See [Write-time validation](/reference/guardrails/#write-time-validation) for the full contract. ### An existing guardrail rule never fires. Why? Rules created before write-time validation existed are grandfathered: they still run, but an unsupported (provider kind, check type, stage) combination scores zero and never fires. The control plane logs a warning (with the rule ID, provider kind, check type, and stage) whenever it ships such a rule, and the dashboard shows an unsupported stored check as disabled in the rule form. The troubleshooting steps are in [Rules that never fire](/reference/guardrails/#troubleshooting-rules-that-never-fire). ### A user was blocked and believes they shouldn't have been. How do I investigate? Blocked responses carry a correlation identifier. The user provides that identifier, and an operator with the appropriate privileges can look up which rules triggered, the confidence score, and the actual input and output that caused the block. Use the correlation identifier to join against your own logging system for the full request context. Request Logs and audit workflows are covered in [Audit Agent Router activity](/agent-router-enterprise/guides/operate-and-govern/audit-platform-activity/). ### What is the guardrail-reader role? A read-only role for reviewing guardrail logs and events. It lets security teams read which rules triggered and review guardrail activity without granting permission to modify guardrail settings. Reading unredacted inputs and outputs requires sufficient privilege. ### Can guardrails enforce a budget or cost cap? Budgets are an observability feature, not a blocking control. This is intentional: blocking on a budget would let a single typo cause an outage. Set budgets per team or per user, export them as metrics, and drive alerts from your own alerting stack. To constrain abuse or runaway usage, use rate limiting instead — by source address or by account, on either request count or token count. See [Working with budgets](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/choose-the-right-cost-control/). ### How are logs and metrics handled? Agent Router Enterprise is designed so it never has to handle your sensitive request and response data. Request and response logs go directly to your own logging tenant. The product sees only the metadata required to drive Agent Router behavior, such as token usage and timing. Metrics are exposed over OpenTelemetry for ingestion into your own monitoring system, and audit logs (configuration changes, administrative interface access, user management) can be written out for ingestion into your security information and event management (SIEM) system. ### Can guardrails handle non-English or obfuscated attacks? Shipped providers cover common cases. A language-allowlist rule (`language` check type) can block any request that is not in an approved language, which is a useful control for high-risk locales. Exotic evasion — rare languages, novel encodings, or techniques like poetry injection — may require a remote or external provider beyond local pattern matching. For jailbreak and injection detection specifically, see [Detect and block prompt injection](/agent-router-enterprise/guides/operate-and-govern/safety-and-data-protection/detect-and-block-prompt-injection/). This is an ongoing area: new evasion techniques appear regularly, and coverage is a moving target. ### What do guardrails protect, and what is out of scope? Agent Router Enterprise is a runtime layer. Content guardrails protect the model request and response path — traffic to and from an LLM. Think of that path as data loss prevention (DLP) that does not itself require inference. Standalone MCP profile traffic (`/mcp/*`) is out of content-guardrail scope today; it is governed by identity, access control, and audit instead. Out of scope: - **Training-time and pre-deployment concerns** — data poisoning, manipulation of model weights, and anything in the artificial intelligence software development lifecycle before the model is served. - **Client-side activity** — lateral movement that manipulates the client itself, or data that never crosses the network (for example, a local file read by a coding agent). - **Direct client-to-retrieval access** — the product protects data on its way to the model, but not the gap between a retrieval-augmented generation (RAG) or vector store and the client that queries it directly. If the same data is exposed over MCP instead, Agent Router can apply access-layer controls (identity, ACL, audit) on the MCP path; content guardrails apply only when that data later rides the model path. - **Standalone MCP profile traffic** — client calls to `/mcp/*` are not inspected by content guardrails. Govern which servers a profile may reach through [Govern MCP server access](/agent-router-enterprise/guides/operate-and-govern/govern-mcp-server-access/). - **Supply chain** — Agent Router Enterprise ships as versioned Helm charts with pinned component versions. The rest of the supply chain — the agents, application programming interfaces (APIs), and MCP servers you choose to deploy — remains your responsibility. Version Agent Router configuration through automation as described in [Manage configuration as code](/agent-router-enterprise/guides/operate-and-govern/manage-configuration-as-code/). ## What to do next - **Configure guardrails**: define project policies and rules and pilot them in monitor mode before enforcement. See [Configure guardrails](/agent-router-enterprise/guides/operate-and-govern/safety-and-data-protection/configure-custom-guardrails/). - **Configure guardrail providers**: choose `regexp`, `builtin`, or `tetrate` (or another configured engine) for each check type. See [Configure guardrail providers](/agent-router-enterprise/guides/operate-and-govern/safety-and-data-protection/configure-vendor-guardrails/). - **Protect requests with guardrails**: the developer-side view of how blocks and redactions appear. See [Protect requests with guardrails](/agent-router-enterprise/guides/operate-and-govern/safety-and-data-protection/protect-requests-with-guardrails/). - **Detect and block prompt injection**: handle jailbreak and indirect injection with the dedicated check types. See [Detect and block prompt injection](/agent-router-enterprise/guides/operate-and-govern/safety-and-data-protection/detect-and-block-prompt-injection/). Where to go next Configure guardrails Define project policies and rules and pilot them in monitor mode before enforcement. Configure guardrail providers Choose regexp, builtin, or tetrate for each check type. Protect requests with guardrails The developer-side view of attaching a guardrail to a specific request path. Detect and block prompt injection Handle jailbreak and indirect injection separately from content guardrails. --- ## Best Practices Recommended patterns for running Agent Router Enterprise in production. This section collects the operational practices that recur across deployments: resilience, cost control, safety, and configuration management. Each links to the guide that covers it in full. Resilience and routing Improve resilience with fallbacks Define ordered fallback policies so requests survive a provider outage. Plan high availability and disaster recovery Size the deployment and plan recovery for a production gateway. Cost and governance Working with budgets Enforce spend limits with per-key rate limits and monitoring. Reduce cost with traffic splitting Distribute traffic across backends by weight to manage cost. Safety and configuration Protect requests with guardrails Apply inline content filtering to inbound and outbound traffic. Guardrails best practices & FAQ Roll out, tune, and operate guardrails without outages; investigate blocks and understand scope. Manage configuration as code Version Agent Router configuration and apply it through automation. --- ## Compliance This page is a reference for the compliance posture of Tetrate Agent Router. It records the certifications and attestations Agent Router carries, how the underlying reports are obtained, the data-residency options enforced at the routing level, and the upstream-provider data-retention settings that apply in an evaluation environment. It is a lookup surface for an operator or security reviewer; the how-to procedures it points to live in the operations guides. The exact set of certifications, their current status, and provider-specific no-retention terms change over time and are governed by contract. None of the values on this page should be treated as a current attestation of fact. Certification status and provider terms must be confirmed with the Tetrate field and legal team for the specific engagement. ## Certifications and attestations Agent Router is operated against recognised security and controls frameworks. The table below lists the attestations typically discussed during an evaluation. Presence in this table is descriptive, not an assertion that a given certification is currently held or current; the live status is confirmed by Tetrate per engagement. | Attestation | Scope | Status source | | --- | --- | --- | | SOC 2 Type II | Operating effectiveness of security, availability, and confidentiality controls over a defined audit period | Tetrate contact | | ISO/IEC 27001 | Information security management system certification, where available | Tetrate contact | The underlying audit reports, the SOC 2 Type II report and any ISO/IEC 27001 certificate and statement of applicability, are not published in this documentation. They are made available under a non-disclosure agreement (NDA) through the Tetrate contact. Requests for the reports, the audit period covered, and the scope boundary should be directed to that contact rather than inferred from this page. --- ## Data residency Data residency determines the geographic region in which request data is processed and stored. Agent Router supports selectable residency so that traffic for a given engagement is confined to an approved region. | Residency option | Description | | --- | --- | | US | Request processing and data storage confined to United States regions | | EU | Request processing and data storage confined to European Union regions | Residency is enforced at the policy and routing level rather than as a single account-wide flag. A routing policy directs matching traffic to the data plane and upstream targets that satisfy the required region, so different policies can carry different residency guarantees within the same deployment. The available regions for an engagement, and any region beyond US and EU, are confirmed with the Tetrate contact. Configuration of residency at the policy level is described in [Configure Data Residency and No-Retention](/agent-router-enterprise/guides/operate-and-govern/compliance-and-governance/configure-data-residency-and-no-retention/). This page records what the options are; that guide records how they are applied. --- ## Upstream-provider data retention Each upstream provider has its own data-retention posture for the prompts and completions it receives. Where a provider supports a no-retention or zero-data-retention mode, a configuration under which submitted content is not retained or used for training, that mode can be applied to the provider integration. The following points govern how retention is handled: - No-retention settings are provider-specific. They can be applied only where the provider offers the capability, and the available terms differ between providers. - The applied setting must be recorded per provider. For an evaluation environment, the retention mode in effect for each upstream provider is documented as part of the provisioning record so that a reviewer can verify which providers are running under no-retention terms. - A provider that does not support a no-retention mode retains data according to its own default policy. The absence of a no-retention setting for such a provider is itself recorded. The exact no-retention terms, the providers that support them, and any contractual conditions attached must be confirmed with the Tetrate field and legal team. This page does not assert that a specific provider is running under no-retention terms in any given environment. The procedure for applying and recording these settings is covered in [Configure Data Residency and No-Retention](/agent-router-enterprise/guides/operate-and-govern/compliance-and-governance/configure-data-residency-and-no-retention/). --- ## Audit evidence Evidence that residency and retention controls are in effect, and that administrative actions are accounted for, is drawn from Agent Router's audit surfaces rather than from this document. | Evidence source | What it provides | | --- | --- | | Audit log | A record of administrative actions (policy changes, provider and model configuration, and key management) with actor, `actionType`, resource, and timestamp | | Policy-decision export | A record of routing and policy decisions applied to traffic, used to demonstrate that residency and retention policies were enforced for matching requests | The audit log is reviewed and exported through the workflow in [Audit Agent Router Activity](/agent-router-enterprise/guides/operate-and-govern/audit-platform-activity/). The event schema for administrative actions, the fields available for each recorded event, is documented in [Audit Log Events](/reference/audit-log-events/). --- ## Related - [Configure Data Residency and No-Retention](/agent-router-enterprise/guides/operate-and-govern/compliance-and-governance/configure-data-residency-and-no-retention/), applying residency and provider no-retention settings - [Audit Agent Router Activity](/agent-router-enterprise/guides/operate-and-govern/audit-platform-activity/), reviewing and exporting the audit log - [Audit Log Events](/reference/audit-log-events/), event schema for administrative actions - [Gateway Behavior](/reference/gateway-behavior/), per-request data captured by the gateway Where to go next Configure data residency and no-retention Apply residency and provider no-retention settings. Sizing and scale Capacity planning for the data plane. --- ## External links A curated list of external references that come up across Agent Router documentation. The links below are external to Tetrate and are subject to change at the upstream sources; if a link breaks, the project name in the surrounding text is usually enough to find the current location through a search engine. ## Standards and protocols | Reference | Purpose | | --- | --- | | [OpenAPI Specification](https://www.openapis.org/) | The API description standard used in adjacent integration material | | [OpenID Connect](https://openid.net/connect/) | The OIDC standard used for SSO and SSO-driven role mapping | | [SAML 2.0](https://docs.oasis-open.org/security/saml/v2.0/) | The federated authentication standard, for background only; Agent Router supports OIDC and not SAML | | [OpenTelemetry](https://opentelemetry.io/) | The observability standard used for trace and metric export | | [OTLP Specification](https://opentelemetry.io/docs/specs/otlp/) | The wire protocol for OpenTelemetry data | | [Prometheus](https://prometheus.io/) | The metrics format exposed on the gateway's scrape endpoint | | [Server-Sent Events](https://html.spec.whatwg.org/multipage/server-sent-events.html) | The streaming format used by all three supported API surfaces | | [Model Context Protocol](https://modelcontextprotocol.io/) | The MCP standard for exposing tools, data sources, and context to AI clients | ## Provider documentation For provider-side concerns (model pricing, rate-limit tiers, API quirks not handled by the gateway's translation layer, and account management), the canonical source is each provider's own documentation. | Provider | Documentation | | --- | --- | | OpenAI | [platform.openai.com/docs](https://platform.openai.com/docs) | | Anthropic | [docs.anthropic.com](https://docs.anthropic.com/) | | Google (Gemini) | [ai.google.dev/docs](https://ai.google.dev/docs) | | Azure OpenAI | [learn.microsoft.com/azure/ai-services/openai](https://learn.microsoft.com/azure/ai-services/openai/) | | Mistral | [docs.mistral.ai](https://docs.mistral.ai/) | | Cohere | [docs.cohere.com](https://docs.cohere.com/) | | Together | [docs.together.ai](https://docs.together.ai/) | | Perplexity | [docs.perplexity.ai](https://docs.perplexity.ai/) | ## SDKs and frameworks For the SDKs and frameworks that integrate with the gateway through its OpenAI-compatible surface: | Project | Documentation | | --- | --- | | OpenAI Python SDK | [github.com/openai/openai-python](https://github.com/openai/openai-python) | | OpenAI JavaScript SDK | [github.com/openai/openai-node](https://github.com/openai/openai-node) | | Anthropic Python SDK | [github.com/anthropics/anthropic-sdk-python](https://github.com/anthropics/anthropic-sdk-python) | | LangChain | [python.langchain.com](https://python.langchain.com/) | | Vercel AI SDK | [ai-sdk.dev](https://ai-sdk.dev/) | | Pydantic AI | [ai.pydantic.dev](https://ai.pydantic.dev/) | | CrewAI | [docs.crewai.com](https://docs.crewai.com/) | | Open WebUI | [docs.openwebui.com](https://docs.openwebui.com/) | | Goose | [github.com/block/goose](https://github.com/block/goose) | ## MCP clients The MCP clients most often configured against MCP profiles: | Client | Documentation | | --- | --- | | Claude Code | [docs.claude.com/en/docs/claude-code](https://docs.claude.com/en/docs/claude-code) | | Cursor | [docs.cursor.com](https://docs.cursor.com/) | | VS Code | [code.visualstudio.com/docs](https://code.visualstudio.com/docs) | ## Observability backends The OTLP-compatible backends most commonly used for trace export, and the Prometheus-compatible metrics agents: | Backend | Documentation | | --- | --- | | Jaeger | [jaegertracing.io/docs](https://www.jaegertracing.io/docs/) | | Grafana Tempo | [grafana.com/docs/tempo](https://grafana.com/docs/tempo/latest/) | | Grafana Alloy | [grafana.com/docs/alloy](https://grafana.com/docs/alloy/latest/) | | Datadog | [docs.datadoghq.com](https://docs.datadoghq.com/) | | Honeycomb | [docs.honeycomb.io](https://docs.honeycomb.io/) | | New Relic | [docs.newrelic.com](https://docs.newrelic.com/) | ## Cloud platform documentation For the underlying cloud platform that hosts the data plane: | Platform | Documentation | | --- | --- | | Kubernetes | [kubernetes.io/docs](https://kubernetes.io/docs/) | | AWS EKS | [docs.aws.amazon.com/eks](https://docs.aws.amazon.com/eks/) | | Azure AKS | [learn.microsoft.com/azure/aks](https://learn.microsoft.com/azure/aks/) | | GCP GKE | [cloud.google.com/kubernetes-engine/docs](https://cloud.google.com/kubernetes-engine/docs) | | Helm | [helm.sh/docs](https://helm.sh/docs/) | --- ## Related - [Gateway APIs](/reference/supported-apis/), the API formats backed by the standards above - [Gateway Behavior](/reference/gateway-behavior/), the error handling and correlation model that complements the provider documentation - [Glossary](/reference/glossary/), one-line definitions of Agent Router–specific terms Where to go next Glossary One-line definitions of Agent Router–specific terms. Product overview What Agent Router is and what it does. --- ## Gateway behavior The gateway provides consistent, predictable behaviour across all AI providers. It normalises provider errors into a single format, attaches correlation IDs to every request, and emits structured observability data regardless of which upstream provider handles the request. This page covers the error semantics, the correlation-ID model, and the data captured per request. ## Error handling Every AI provider has its own error schema, status code semantics, and message vocabulary. When a request routes through the gateway (whether to a primary provider, a fallback target, or one leg of a traffic split), every error response is normalised into a single OpenAI-compatible format before being returned to the client. A single error-handling path is sufficient on the application side even as providers change behind the scenes. ### Error response format All errors, whether they originate at the gateway itself or at an upstream provider, are returned with the following JSON structure: ```json { "error": { "message": "A human-readable description of the error", "type": "error_type", "param": null, "code": "error_code" } } ``` | Field | Description | | --- | --- | | `message` | A human-readable explanation. Useful for logging and surfacing context during development. | | `type` | A machine-readable error category (`rate_limit_error`, `timeout_error`, `invalid_request_error`, and similar). Use this field for programmatic error handling. | | `param` | The specific request parameter that caused the error, when applicable. Often `null` for provider-level and gateway-level errors. | | `code` | A specific error code within the error type. Distinguishes subtypes (`rate_limit_exceeded` vs. `model_not_found`, for example). | ### Error origin Errors originate at two distinct points in the request lifecycle: - **Gateway errors** are produced by the gateway before the request reaches a provider. These include authentication failures, malformed request bodies, unknown model names, and policy violations. HTTP status codes are typically in the 4xx range. - **Provider errors** are returned by an upstream provider and normalised by the gateway before being forwarded to the client. These include rate limits, model overload conditions, and provider outages. The HTTP status code reflects the nature of the provider failure. In both cases, the response body uses the same JSON format and the same `X-Request-ID` correlation header is present. ### HTTP status codes | HTTP Status | Meaning | Typical cause | | --- | --- | --- | | 400 | Bad Request | Malformed request body, unsupported parameters, or missing required fields | | 401 | Unauthorized | Invalid or missing API key | | 403 | Forbidden | API key lacks permission for the requested model or endpoint | | 404 | Not Found | Unknown endpoint path or model not present in the catalogue | | 429 | Too Many Requests | Rate limit exceeded at the client level or by the upstream provider | | 500 | Internal Server Error | Unexpected error within the gateway | | 502 | Bad Gateway | The upstream provider returned an invalid or unparseable response | | 503 | Service Unavailable | The upstream provider is temporarily unavailable or returning server errors | | 504 | Gateway Timeout | The upstream provider did not respond within the configured timeout window | ### Retryable vs. non-retryable errors | Status | Retryable? | Recommended action | | --- | --- | --- | | 400 | No | Inspect the `message` and `param` fields; fix the request before retrying | | 401 | No | Verify and rotate the API key | | 403 | No | Confirm the model or endpoint is enabled for the API key | | 404 | No | Verify the model id with `GET /v1/models` for the same API key ([Models API](/reference/supported-apis/#models-api-v1models)), and confirm the model is enabled for that key | | 429 | Yes | Retry with exponential backoff; honour any `Retry-After` header if present | | 500 | Maybe | Retry once; if the error persists, investigate using Request Logs | | 502 | Yes | Retry; the provider response was malformed but may succeed on a subsequent attempt | | 503 | Yes | Retry with backoff; the provider is temporarily unavailable | | 504 | Yes | Retry; consider adjusting timeout settings if this error occurs consistently | :::tip When [fallback policies](/agent-router-service/guides/improve-resilience-with-fallbacks/) are configured, the gateway automatically retries retryable errors (5xx and 429) against the next provider in the fallback chain, transparent to the calling application. The application only receives an error response if every provider in the chain is exhausted or if the error is non-retryable. ::: ### Error examples Upstream rate limit (provider returns 429): ```json { "error": { "message": "Rate limit exceeded. Please retry after 30 seconds.", "type": "rate_limit_error", "param": null, "code": "rate_limit_exceeded" } } ``` Provider timeout (no response within the timeout window): ```json { "error": { "message": "Upstream provider did not respond in time.", "type": "timeout_error", "param": null, "code": "timeout" } } ``` Invalid or missing API key: ```json { "error": { "message": "Invalid API key. Please check your credentials.", "type": "authentication_error", "param": null, "code": "invalid_api_key" } } ``` Model not present in the catalogue: ```json { "error": { "message": "The model 'example-model' is not available in this gateway.", "type": "not_found_error", "param": "model", "code": "model_not_found" } } ``` ### Streaming errors For streaming requests, the gateway attempts to detect and translate provider errors before the stream is opened. If a provider returns an error mid-stream, the gateway closes the stream and, where possible, emits a final error event. Client applications should handle stream interruptions and check for error payloads at stream termination. --- ## Request tracking The gateway attaches correlation IDs to every request. These IDs link the HTTP response the application receives to the detailed record stored in Request Logs and to the spans emitted to the OpenTelemetry backend. ### Correlation headers | Header | Direction | Description | | --- | --- | --- | | `X-Request-ID` | Response | A gateway-generated UUID attached to every response. The primary identifier for looking up the request in Request Logs and OTel traces. | | `X-Client-Request-ID` | Response | Echoed back from the `X-Request-ID` header sent by the client, if present. Allows correlation between gateway records and application-side request identifiers. | When a client sends an `X-Request-ID` header, the gateway preserves it as `X-Client-Request-ID` in the response and generates its own `X-Request-ID`. Both IDs appear in Request Logs and OpenTelemetry traces. ### Sending a correlation ID `PROXY_URL` stands for the proxy endpoint from the Console Dashboard, scheme included and ending in `/v1`. ```bash curl -i PROXY_URL/chat/completions \ -H "Authorization: Bearer YOUR_API_KEY" \ -H "Content-Type: application/json" \ -H "X-Request-ID: my-session-abc-123" \ -d '{ "model": "gpt-4o", "messages": [{"role": "user", "content": "Hello"}] }' ``` Response headers: ```text HTTP/2 200 x-request-id: 8f14e45f-ceea-467f-a8f0-6b2a1c3d4e5f x-client-request-id: my-session-abc-123 content-type: application/json ``` The `x-request-id` value (`8f14e45f-...`) is the gateway's canonical identifier. The `x-client-request-id` (`my-session-abc-123`) is the original client ID, preserved for joining gateway records with application-side logs. ### Data captured per request Every request processed by the gateway produces a record containing: | Field | Description | | --- | --- | | Request ID | The gateway-assigned `X-Request-ID` UUID | | Client Request ID | The echoed `X-Client-Request-ID`, if provided by the client | | Timestamp | When the request was received by the gateway | | Model | The model identifier specified in the request | | Provider | The upstream provider that served the response | | HTTP Status | The final HTTP status code returned to the client | | Latency | Total round-trip time from gateway receipt to completed response | | Token usage | Prompt tokens, completion tokens, and total tokens | | Request / response body | Full payloads, subject to the deployment's data retention settings | This data is searchable in [Request Logs](/agent-router-service/guides/monitor-traffic-and-usage/) and is also available as OpenTelemetry span attributes in the tracing backend. ### Correlating in observability tools - **Request Logs.** Search by `X-Request-ID` (gateway-assigned) or `X-Client-Request-ID` (the application's ID) to retrieve the complete request record: provider used, latency breakdown, token counts, and raw request and response payloads. - **OTel traces.** Both IDs are emitted as span attributes on every trace. Search for them in the tracing backend (Jaeger, Grafana Tempo, Honeycomb, Datadog, and others) to view the full execution path including any fallback retries. :::tip Custom application-defined headers (`agent-session-id`, `user-id`, `workflow-run-id`, and similar) are forwarded as OpenTelemetry span attributes. This allows traces to be grouped or filtered by any application-level concept (conversation ID, agent run, team, deployment) without affecting how the gateway routes requests. ::: ### Debugging workflow A typical end-to-end debugging workflow using correlation IDs: 1. **Capture the response header.** Log the `x-request-id` value from every response in the application. For error responses, also capture the response body. 1. **Search Request Logs.** Paste the `x-request-id` value into the search field in [Request Logs](/agent-router-service/guides/monitor-traffic-and-usage/) to retrieve the full request record: which provider was used, total latency, token counts, fallback attempts, and raw payloads. 1. **Inspect the trace.** If OTel export is configured, search for the same `x-request-id` as a span attribute in the tracing backend to see provider-level timing and any retry hops. 1. **Join with application logs.** If a custom `X-Request-ID` was sent, use the echoed `x-client-request-id` in the gateway record to join against application logs and reconstruct the full request context. --- ## Related - [Gateway APIs](/reference/supported-apis/): inference wire formats and endpoint reference - [OTel Metrics](/reference/otel-metrics/): metric names, types, and labels exported by the gateway - [Audit Log Events](/reference/audit-log-events/): event schema for administrative actions Where to go next Gateway APIs Inference wire formats and endpoint reference. Monitor traffic and usage Find requests in Request Logs by correlation ID. --- ## Glossary A consolidated definition list for the Agent Router–specific vocabulary used across the documentation. For a narrative introduction to the same concepts, see Key Concepts. Product and marketing materials on tetrate.io use some alternate names for the same concepts: for example, control plane for management plane, token budgets for Agent Router's per-key rate limits, and AI Gateway for the data-plane proxy. This glossary uses the names that appear in these docs; cross-references note the marketing equivalents where they differ. ### Active fallback chain The ordered, priority-ranked list of backends that the gateway walks when a request fails with a retryable error (connection failure, 5xx, or 429). The gateway tries the first backend; on failure it moves to the next until one succeeds or the chain is exhausted, the standard **provider fallback** pattern used by AI gateways. In Agent Router, the chain is the priority-ordered portion of an API key's **Routing configuration**; it is **active** only when that configuration's Active toggle is on (inactive configurations are saved but not enforced). See **Fallback policy**, **Routing chain**, and [Improve Resilience with Fallbacks](/agent-router-service/guides/improve-resilience-with-fallbacks/). ### Admin Dashboard The operator-facing application. Where models, providers, users, budgets, SSO, audit logs, and instance settings are managed. ### Advanced routing rules Routing decisions that depend on attributes of the request itself (a custom header, the requested model name, or request metadata) rather than only on static priority or weight. See [Apply Advanced Routing Rules](/agent-router-service/guides/apply-advanced-routing-rules/). ### Developer Console The developer-facing application. Where API keys, routing configurations, MCP profiles, integrations, the Playground, request logs, and personal usage analytics are accessed. ### Agent Router Enterprise The full Agent Router deployment: data plane, management plane, Console, and Admin Dashboard, considered as a single system. Often referred to as "Agent Router" after first mention. ### Agent Router Service The self-serve product tier: sign up with a GitHub or Google account and route through a Tetrate-managed service without a dedicated Enterprise instance. Includes multi-model routing, MCP access, and per-key usage logs. Agent Router Enterprise adds cross-team cost attribution, admin access controls, runtime AI Guardrails, enterprise SSO, and distributed data-plane deployment. This documentation set focuses on **Agent Router Enterprise**; tier comparison is on the [welcome page](/). ### AI Gateway The model-routing component of the data plane: the proxy that receives LLM requests from applications, applies routing configuration, attaches credentials, and forwards traffic to upstream providers. On [tetrate.io](https://tetrate.io/agent-router-product) this is one of three product pillars (alongside MCP Gateway and AI Guardrails). In these docs it is also called the **Gateway**; older materials may use **LLM Gateway**. Built on [**Envoy AI Gateway**](/reference/glossary/#envoy-ai-gateway). See [Architecture overview](/product-architecture/architecture-overview/). ### AI Guardrails Runtime policy enforcement on model traffic for a **project**: PII detection and redaction, prompt and response filtering, and request blocking (HTTP `403`) before traffic reaches upstream model providers. Implemented inline in the gateway through **Dynamic modules**. A guardrail is a collection of rules; each rule is a check type scored by a provider (`regexp`, `builtin`, `tetrate`, or another configured engine). An Agent Router Enterprise capability; configured in the Admin Dashboard. See [Configure guardrails](/agent-router-enterprise/guides/operate-and-govern/safety-and-data-protection/configure-custom-guardrails/) and [Protect requests with guardrails](/agent-router-enterprise/guides/operate-and-govern/safety-and-data-protection/protect-requests-with-guardrails/). ### API key The credential an application presents to the gateway. Each key is associated with a Console user, a routing configuration, and optional per-key rate limits. ### Backend In routing terms, a model-on-a-provider combination plus the credentials needed to reach it. Routing decisions resolve to a choice of backend. ### Bring your own key (BYOK) The mechanism by which a consumer's own upstream provider credentials are used in place of centrally managed credentials. BYOK credentials are configured at the Console account level and are slotted into routing chains alongside centrally managed credentials. See [Use Your Own Provider Credentials](/agent-router-service/guides/use-your-own-provider-credentials/). ### Budget An operator-facing spending control implemented through **Rate limits** on API keys plus **Usage analytics** for visibility. There is no separate "budget object" in Agent Router; budgets are the combination of per-key token ceilings (enforced inline with HTTP 429) and regular review of usage by key or team. Tetrate marketing often calls the same mechanism **token budgets**. See [Working with budgets](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/choose-the-right-cost-control/). ### Canary deployment A routing pattern in which a new model version is rolled out gradually, by traffic splitting from a small weight on the new version to progressively larger weights as confidence grows. ### Console Short form for the Developer Console. ### Control plane Synonym for **Management plane**. Tetrate-hosted; stores routing rules, policies, user records, API key metadata, audit history, and analytics rollups. Governs one or more data planes (Tetrate-operated or customer-managed, depending on the [deployment model](/product-architecture/deployment-models/)). Used on [tetrate.io](https://tetrate.io/faq) and in the public FAQ; these docs prefer **management plane**. ### Controller The data-plane component that bridges the **Data plane** and **Management plane**. Polls the management plane for configuration updates and translates them into gateway configuration, keeping routing rules, policies, and provider credentials in sync without inbound connections from the internet. ### Correlation ID A UUID attached by the gateway to every request, exposed as the `X-Request-ID` response header and emitted as a span attribute on the corresponding OpenTelemetry trace. The primary identifier for joining application logs with Request Logs and traces. ### Cost attribution The practice of tying AI spend and usage to a team, application, or agent. In Agent Router Enterprise, attribution relies on per-purpose API keys, **Usage analytics**, optional export to an observability stack, and (with SSO) authenticated identity on every request. Supports showback and chargeback workflows. An Enterprise-tier capability. ### Data plane The request-path half of Agent Router: a deployment containing the **Controller** and an **AI Gateway** proxy built on [**Envoy AI Gateway**](/reference/glossary/#envoy-ai-gateway). All AI traffic flows through it. Where it runs depends on the [deployment model](/product-architecture/deployment-models/): Tetrate operates it in Agent Router Service and Agent Router Enterprise Fully Managed; the customer runs it in their Kubernetes cluster in Agent Router Enterprise Self-Hosted Data Plane (where prompts and responses stay inside the customer's infrastructure). Only configuration and telemetry cross to the management plane. ### Dynamic module A high-performance proxy extension (written in Rust or Go and compiled to a shared library) that runs inline in the gateway's filter chain. Dynamic modules implement Agent Router's routing logic, credential handling, and provider-specific translation. ### Endpoint picker A pluggable component (Endpoint Picker Provider, EPP) that selects the best upstream inference endpoint per request from a pool of candidates, using real-time signals rather than static weights or round-robin. Standard signals include KV-cache utilisation, queue depth, and prefix-cache affinity. Defined by the [Kubernetes Gateway API Inference Extension](https://gateway-api-inference-extension.sigs.k8s.io/); integrated in [**Envoy AI Gateway**](/reference/glossary/#envoy-ai-gateway) via `InferencePool` + Endpoint Picker Provider. In Agent Router, the Endpoint Picker operates within the eligible set defined by the routing configuration. It does not override policy boundaries. See [Apply Advanced Routing Rules](/agent-router-service/guides/apply-advanced-routing-rules/) and [Load Balance Across Regional Deployments](/agent-router-enterprise/guides/operate-and-govern/load-balance-across-regional-deployments/). ### Envoy AI Gateway The open-source, CNCF-backed AI gateway co-created and maintained by Tetrate and Bloomberg. The data-plane proxy technology underneath **Agent Router Enterprise**. Tetrate Agent Router adds the **Management plane**, model and MCP catalogues, cost attribution, **AI Guardrails**, SSO, and multi-gateway governance on top. See the [Envoy AI Gateway project](https://aigateway.envoyproxy.io/) for upstream documentation, including the [endpoint-picker guide](https://aigateway.envoyproxy.io/blog/endpoint-picker-for-inference-routing/). ### Fallback policy An ordered list of backends per API key. The gateway tries the first backend; if it fails with a retryable error, the gateway walks to the next backend, continuing until a backend succeeds or the chain is exhausted. See [Improve Resilience with Fallbacks](/agent-router-service/guides/improve-resilience-with-fallbacks/). ### Gateway The proxy component of the data plane that receives application requests, applies routing rules, attaches provider credentials, and translates between API formats. In architecture diagrams and on tetrate.io this component is also called the **AI Gateway** (formerly **LLM Gateway** in some older materials). See **AI Gateway** and [**Envoy AI Gateway**](/reference/glossary/#envoy-ai-gateway). ### Logical model name A name exposed by a routing configuration that the gateway resolves to a specific provider model identifier. Decouples application code from provider model version strings. Configured through model-name overrides; see [Apply Advanced Routing Rules](/agent-router-service/guides/apply-advanced-routing-rules/). ### Management plane The Tetrate-hosted control surface. Stores routing rules, policies, user records, API key metadata, audit history, and analytics rollups. Communicates with the data plane through a small, well-defined interface; no application payloads cross the boundary. Tetrate marketing and the public FAQ refer to the same component as the **control plane**; the terms are interchangeable. ### Model context protocol (MCP) An emerging standard for exposing tools, data sources, and context to AI clients. MCP servers act as adapters between AI clients (Claude Code, Cursor, VS Code, and others) and the systems those clients need to reach. ### MCP catalogue The set of MCP servers registered in Agent Router by operators. The catalogue is the universe from which developers assemble MCP profiles in the Console. Managed in the Admin Dashboard under MCP Servers. ### MCP Gateway The MCP-routing and governance component of Agent Router: operators curate an **MCP catalogue** centrally, developers assemble **MCP profiles** in the Console, and agents reach approved tools through governed endpoints. On tetrate.io this is a distinct product pillar alongside the **AI Gateway**. Tool calls receive the same attribution and logging as LLM requests. ### MCP profile A named collection of MCP servers exposed through a single Agent Router URL. AI clients connect to the profile URL once and receive access to every server included in the profile. See [Aggregate MCP Servers into a Profile](/agent-router-service/guides/aggregate-mcp-servers-into-a-profile/). ### OAuth client (MCP) A configuration record holding the credentials and endpoints needed for OAuth-authenticated MCP servers. Managed in the Admin Dashboard under MCP OAuth Clients. ### OpenTelemetry (OTel / OTLP) The open observability standard Agent Router uses for trace and metric export. OTLP is the wire protocol. The gateway exports traces over OTLP; metrics are exposed on a Prometheus scrape endpoint that can be forwarded as OTLP by an external collector. ### Persona The audience profile used to organise documentation. Agent Router serves two primary personas: Developer (Console) and Platform operator (Admin Dashboard), with the Installer treated as a phase of the operator persona during initial setup. ### Solution capabilities The catalogue of features active on a deployment. Visible on the Instances surface under Enabled Features. ### Passthrough mode Passthrough mode lets you keep paying a flat Anthropic subscription fee (for example Claude Pro or Max) while Agent Router Enterprise monitors the traffic—Request Logs, usage, and the Admin Dashboard Spend overview. Anthropic still authenticates you with Claude Code's stored OAuth token; the gateway routes and observes without putting those tokens on Agent Router billing. In the Spend overview, passthrough usage is the theoretical API-rate cost of that traffic if you did not have the subscription—it does not count toward the budget. This is separate from BYOK: you do not paste a Claude subscription into Bring Your Own Key. ### Platform operator The persona responsible for running Agent Router: provisioning models and providers, governing access, configuring SSO, auditing usage, and operating the system across environments. ### Playground An interactive testing surface inside the Console. Sends messages to any enabled model and renders responses, token counts, and latency in real time. Playground traffic flows through the same gateway as application traffic. See [Test Prompts in the Playground](/agent-router-service/guides/test-prompts-in-the-playground/). ### Provider An upstream AI service (OpenAI, Anthropic, Google, Azure OpenAI, Mistral, and others) that the gateway routes requests to. Provider configurations carry credentials and connectivity details. ### Provider translation The transparent conversion the gateway performs between its OpenAI-compatible request surface and the provider-specific APIs of upstream services. Includes header mutations, body mutations, path rewriting, model-name override, and response normalisation. See [Gateway APIs](/reference/supported-apis/). ### Rate limit A per-API-key cap on token consumption within a rolling one-hour window. Available on input tokens, output tokens, total tokens, or any combination. Enforced inline by the gateway; requests that would exceed an active limit receive HTTP 429. On tetrate.io and in operator guides, the same mechanism is often described as a **token budget** or **budget**; see **Budget**. ### Redirect URI The tenant-specific callback URL that the identity provider returns the user to after authentication, in the form `https://auth..tetrate.ai/api/auth/sso/callback/corporate`. Surfaced in the **IdP setup guide** panel of the Configure SSO step, and registered on the OIDC client in the identity provider. Most providers require a character-for-character match. See [Configure SSO](/agent-router-enterprise/guides/identity-and-access-management/single-sign-on/sso-configuration-guide/). ### Request logs The per-request record in the Console. Captures the resolved model, provider, token counts, latency, cost, and full request and response payloads for every gateway call under the user's API keys. ### Resolved model The model that actually served a request, after the routing configuration has been evaluated. May differ from the model the application asked for (the `requested_model`) due to fallback walks, traffic splits, or logical-name overrides. ### Role An Agent Router-level permission set. The role model is small: `super_admin`, `model_admin`, `provider_admin`, `mcp_admin`, `user_admin`, `billing_admin`, and `user`. Roles can be assigned manually or driven by SSO claim mapping. ### Role mapping The mechanism by which OIDC claims (app roles, group memberships) are translated into Agent Router roles on every login. Configured on the SSO provider in the Admin Dashboard. See the [SSO role mapping guide](/agent-router-enterprise/guides/identity-and-access-management/single-sign-on/sso-role-mapping-guide/). ### Routing chain The ordered or weighted list of backends attached to an API key. May be a fallback chain (priority-based), a traffic split (weight-based), or a combination. ### Routing configuration The settings that govern how requests on a given API key are dispatched: routing strategy, backend list, weights or priorities, advanced rules, and active/inactive state. ### Routing policy Synonym for **Routing configuration**. Used in quickstarts and some operator guides. The canonical term in the Console UI and developer guides is **routing configuration**; both refer to the settings on an API key that govern dispatch strategy, backends, weights, and advanced rules. Guardrails are scoped to a **project**, not to a routing policy. ### Session affinity The gateway property that keeps a multi-turn conversation, agent workflow, or MCP session pinned to a consistent processing path for its entire lifetime. Applied automatically; no configuration required. ### Single sign-on (SSO) Delegated authentication through a corporate identity provider over OpenID Connect (OIDC). Configured during onboarding, and thereafter in the Admin Dashboard under Settings → SSO. SSO is the only way to sign in; there is no email and password path. See [Configure SSO](/agent-router-enterprise/guides/identity-and-access-management/single-sign-on/sso-configuration-guide/). In Agent Router Enterprise, SSO also means every gateway request can carry authenticated user and team identity, enabling per-team cost attribution and audit; this is an Enterprise-tier capability. ### Streaming Server-sent events (SSE) delivery of incremental response tokens. Supported across all three API formats; see [Gateway APIs](/reference/supported-apis/) for the per-format SSE shape. ### Tetrate Agent Router The umbrella product name for Tetrate's AI gateway offering. Includes **Agent Router Service** (self-serve) and **Agent Router Enterprise** (dedicated instance with full governance). Sits in front of existing agent frameworks and provider SDKs via an OpenAI-compatible API; built on [**Envoy AI Gateway**](/reference/glossary/#envoy-ai-gateway). This documentation set documents **Agent Router Enterprise** unless a page explicitly compares tiers. ### Traffic splitting A routing strategy in which all backends sit at the same priority and share traffic by weight. Used for cost reduction, A/B evaluation, and gradual migration. See [Reduce Cost with Traffic Splitting](/agent-router-service/guides/reduce-cost-with-traffic-splitting/). ### Usage analytics The aggregated traffic surface. Two variants exist: the Console version, scoped to the signed-in user's own data, and the Admin Dashboard version, aggregated across all users and API keys in Agent Router. Where to go next Key concepts A narrative introduction to the vocabulary defined here. Product overview How Agent Router fits together end to end. --- ## Guardrails(Reference) Guardrails in Agent Router Enterprise run inline in the gateway data plane as a dynamic module, inspecting the prompt on its way to a model provider and the response on its way back. A guardrail is a project-scoped policy: a set of rules, each pairing a check type (what to detect) with a provider (how it is detected) and a stage (where on the request path it runs). This page is the capability reference: guardrail and rule settings, the 23 check types, the five provider kinds with their latency and cost profiles, the full matrix of which provider supports which check type on which stage, how write-time validation rejects combinations that could never fire, the 14 built-in templates, and the design characteristics of the runtime. ## Where guardrails run Evaluation happens in process, as a dynamic module in the gateway data plane, with no sidecar and no extra network hop. Every request that crosses the gateway is inspected on the way in (the prompt, before it reaches any model provider) and on the way out (the model's response, before it reaches the application). Guardrails inspect large language model (LLM) traffic. Model Context Protocol (MCP) traffic is also proxied by the gateway, but standalone MCP profile requests do not pass through content guardrails; they are governed by identity, access control, and audit. See [Guardrails](/product-architecture/guardrails/) for the request-path detail. Running at the gateway rather than in application code has two consequences: - **Centralised governance:** one policy protects every application in the project, with no per-application safety code and no software development kit (SDK) integration. - **Cost at the edge:** a prompt blocked at the gateway never reaches the model provider, so no charge is incurred for a call that would have failed policy. The core design split drives everything below: - A **check type** states what to look for (personally identifiable information, jailbreak, toxicity, and others). There are 23. - A **provider** states how it is detected (regular expression engine, keyword pipeline, machine-learning classifiers, external service). There are five provider kinds. - A **stage** states where on the request path the rule runs: input, output, or both. - Any check type can in principle be served by different providers, so the cost and accuracy point is selected per rule. **Providers score, the gateway decides:** a provider returns a normalised score between 0.0 and 1.0, and the gateway compares it to the rule's threshold and applies the action. - Not every combination of the three is implemented. The [capability matrix](#provider-support-by-check-type) is the authority, and the API validates every rule against it at write time. ## Anatomy of a guardrail A guardrail is a named, project-scoped policy: a collection of rules plus shared settings. A rule is one check type, one provider, an action, a mode, and a stage. ### Guardrail-level settings | Setting | Values | Meaning | | :--- | :--- | :--- | | Type | `compliance`, `security`, `policy`, `quality`, `custom`, `evaluator` | What class of concern the guardrail addresses | | Severity | `low`, `medium`, `high`, `critical` | Operational severity of a trigger | | Category | `data_privacy`, `financial`, `healthcare`, `government`, `custom` | Domain grouping | | Failure mode | `fail_close` (default), `fail_open` | If evaluation itself fails (provider down, timeout): fail-close blocks the request, fail-open lets it pass. The setting is per path: fail-close where safety dominates, fail-open where availability dominates | | Evaluation timeout | milliseconds | Per-guardrail evaluation budget (the built-in templates use 3000 to 5000 ms) | | Triggered response | message template | What the caller sees on a violation; supports `{{policy_name}}`, `{{rule_name}}`, and `{{violation_details}}` substitution | ### Rule-level settings | Setting | Values | Meaning | | :--- | :--- | :--- | | Check type | one of the 23 listed below | What to detect | | Provider | one of the configured provider instances | The engine that scores the content | | Action | `block`, `redact` | **Block** stops the request with HTTP `403`, plus an optional message and a correlation ID for investigation. **Redact** masks the matched content (for example `[CREDENTIAL_REDACTED]`) and lets the request continue | | Mode | `enforce`, `monitor` | **Enforce** applies the action. **Monitor** is shadow mode: it records what would have happened and passes content unchanged, which surfaces trigger and false-positive rates on live traffic before enforcement. Monitor rules never break the request flow and are always effectively fail-open | | Stage | `input`, `output`, or both | Run against the prompt, the response, or both. Labelled **Run on** in the Admin Dashboard and `execute_on` in the API. **Leaving it unset requests both stages**, so a check supported on only one stage must have the stage set explicitly or the write is rejected | | Content type | all, text, image, document, tool, thinking, context | Which kind of content an evaluation ran against. Carried on the evaluation and on the resulting trigger record, not set on the rule in the Admin Dashboard | | Target | project | Rules apply to the guardrail's whole project | | Threshold | 0.0 to 1.0 | Violation sensitivity: the gateway flags a violation when the provider score is at or above the threshold (deterministic engines score 1.0 or 0.0) | ### Streaming behaviour - If every rule on a stage is monitor mode, evaluation is asynchronous and streaming is unaffected. - If any rule on the output stage enforces, the response is buffered for evaluation before delivery, so enforcing output guardrails de-stream the response. This is a deliberate correctness trade-off; the operational implications are covered in [Guardrails best practices and FAQ](/reference/best-practices/guardrails-best-practices-and-faq/). ## Check types 23 check types, in four groups. Each name below is the value used in the Admin Dashboard; the corresponding API constant is `CHECK_TYPE_` followed by the upper-cased name, so `pii` is `CHECK_TYPE_PII` and `factual_consistency` is `CHECK_TYPE_FACTUAL_CONSISTENCY`. ### Privacy and secrets | Check type | Detects | | :--- | :--- | | `pii` | Personally identifiable information: names, emails, phone numbers, national identifiers, payment cards, IBANs, and similar | | `secrets` | Credentials and API keys (AWS, GitHub, Slack, generic API keys, and others) | | `protected_material` | Copyrighted or otherwise protected content in outputs | ### Security | Check type | Detects | | :--- | :--- | | `prompt_injection` | Instructions smuggled into prompts or context to subvert the model, including system-prompt extraction and data exfiltration | | `jailbreak` | Direct "ignore previous instructions" and role-play attacks in the user's own prompt | | `ban_code` | Executable or malicious code in prompts or responses, including SQL injection payloads, detected by denylist and pattern. Available on the deterministic engines only | | `code` | Executable or malicious code, additionally available on the `tetrate` classifier | ### Content safety | Check type | Detects | | :--- | :--- | | `toxicity` | Toxic, profane, abusive language | | `hate_speech` | Hate speech targeting protected groups | | `sexual_content` | Sexual content, including child-safety violations | | `violence` | Violent content and threats | | `self_harm` | Self-harm and suicide-related content | | `bias` | Biased or discriminatory language (gender, age, disability, and others) | ### Policy, quality, and control | Check type | Detects | | :--- | :--- | | `compliance` | Regulation-specific prohibited practices (see the compliance templates below) | | `banned_topics` | Configurable off-limits subject matter (for example medical, legal, or financial advice) | | `ban_competitors` | Competitor names, recommendations, and comparisons | | `banned_substrings` | Literal string denylists | | `regexp` | Arbitrary custom patterns from the shared Patterns catalog | | `language` | Language allow and deny lists (28 languages) | | `gibberish` | Nonsense or garbled input | | `sentiment` | Negative-sentiment content | | `relevance` | Response relevance to the prompt | | `factual_consistency` | Groundedness of a response against source material | ## Detection engines (providers) Five provider kinds, compared by mechanism, deployment, and cost: | Provider kind | Detection mechanism | ML-based? | Where it runs | Data leaves the customer environment? | Latency and cost profile | | :--- | :--- | :--- | :--- | :--- | :--- | | `regexp` | RE2 pattern matching against a named, reusable Patterns catalog (custom patterns supported) | No, deterministic | In process, inside the gateway | No | **Near zero, free** (about 0 ms) | | `builtin` | Five-layer keyword and regex pipeline: blocked keywords, then an allowlist gate, then subject and term co-occurrence within a sentence, then detection patterns, then category keywords, severity-banded | No, deterministic | In process, inside the gateway | No | **Near zero, free** (sub-millisecond) | | `tetrate` | Tetrate Semantic Router: an embedding model plus task-specific small-model classifiers, comprising a domain classifier (violence, self-harm, hate speech, sexual content, weapons, crime), a PII classifier, a jailbreak classifier, a natural language inference (NLI) entailment fallback (ModernBERT class), a gibberish detector, and language identification. Cascading evaluation: local known patterns, then the ML classifier, then the NLI fallback, short-circuiting at the first firing layer | **Yes, small-language-model and classifier based** | **In the customer cluster:** a Helm-deployed workload in the customer data plane; CPU inference, no GPU required | **No**, the callout stays inside the customer network | Slower: in-cluster network hop plus inference, typically 40 to 120 ms; compute cost is the customer's own cluster capacity | | `azurecontentsafety` | Azure AI Content Safety REST API: text analysis (hate, sexual, violence, and self-harm severity), Prompt Shields (jailbreak), protected-material detection, groundedness detection | Yes, hosted ML | **Third-party SaaS** (Microsoft) | **Yes**, content is sent to Azure | Slowest and metered: external network round trip (about 200 ms class) plus per-call vendor pricing | | `externalservice` | Generic HTTP callout to any guardrail service the customer runs or subscribes to; its supported checks are declared per instance | Depends on the service | Wherever it is pointed | Yes, to that endpoint | Depends on the service | ### Provider availability and evaluation order - `regexp`, `builtin`, and `tetrate` ship by default as globally seeded providers, immutable through the API. `azurecontentsafety` and `externalservice` are operator-configured integrations. - The kinds form a cost and capability ladder: deterministic engines catch known formats and phrasings for free, ML engines catch paraphrase, novel phrasing, and semantic evasion at a latency and compute cost, and third-party engines add vendor capabilities at per-call prices and a data-egress trade-off. - **Cheap-decisive-first ordering:** the gateway hoists enforced `block` rules on the in-process engines ahead of everything else, so a local pattern hit short-circuits evaluation before any remote or ML call runs. ML latency and vendor fees are not incurred for requests a regular expression would have blocked anyway. - **One batched call per provider:** N rules on one provider cost one evaluation call, not N. Results are cached per content snapshot and invalidated only when a redaction rewrites the content. - Some combinations of check type, provider, and stage are not implemented, and are restricted by the capability matrix below. ### Provider support by check type A rule is valid only if its provider kind supports its check type **on the stage the rule requests**. The constraint is enforced at write time, not at request time, and the Admin Dashboard mirrors it by scoping the check-type dropdown to the selected provider. Each cell gives the stages the (provider kind, check type) pair supports: - **Both:** the check runs on input, output, or both, so the stage may be left unset. - **Input only** or **Output only:** the check runs on that stage alone. Because an unset stage requests both, such a rule must set **Run on** explicitly or it is rejected. - **No:** the provider kind does not implement the check, and rules combining them are rejected. The four kinds with fixed capabilities are shown. `externalservice` declares its capabilities per instance and is covered [below](#external-guardrail-services-per-instance-capabilities). | Check type | `regexp` | `builtin` | `tetrate` | `azurecontentsafety` | `tetrate` detection mechanism | | :---- | :----: | :----: | :----: | :----: | :---- | | `ban_code` | Both | Both | No | No | Not supported | | `ban_competitors` | Both | Both | No | No | Not supported | | `banned_substrings` | Both | Both | Both | No | Keyword signal | | `banned_topics` | Both | Both | Both | No | NLI entailment | | `bias` | Both | Both | Output only | No | NLI entailment | | `code` | Both | Both | Both | No | Classifier | | `compliance` | Both | Both | No | No | Not supported | | `factual_consistency` | Both | No | No | Output only | Not supported | | `gibberish` | Both | No | Both | No | NLI or classifier | | `hate_speech` | Both | Both | Both | Both | Signal evaluation | | `jailbreak` | Both | Both | Input only | Input only | Security classifier, with NLI fallback | | `language` | Both | Both | Both | No | Language signal | | `pii` | Both | Both | Both | No | PII model, returning entities and masked text | | `prompt_injection` | Both | Both | Input only | Input only | Known patterns, then classifier, then NLI | | `protected_material` | Both | Both | No | Both | Not supported | | `regexp` | Both | Both | Both | No | Catalog regex signal | | `relevance` | Both | No | Output only | No | NLI against the original prompt | | `secrets` | Both | Both | Input only | No | Keyword signal | | `self_harm` | Both | Both | Both | Both | Signal evaluation | | `sentiment` | Both | No | Both | No | NLI entailment | | `sexual_content` | Both | Both | Both | Both | Signal evaluation | | `toxicity` | Both | Both | Both | Both | Signal evaluation (violence and hate) | | `violence` | Both | Both | Both | Both | Signal evaluation | Five qualifications apply when reading the table: - `regexp` accepts every check type on both stages only because a regular expression is check-type-agnostic. The check type is a label, and the referenced patterns are the real definition, so a `pii` rule and a `secrets` rule on this provider behave identically given identical patterns. - `builtin` runs its keyword and regex pipeline on both stages for every check it supports. It does not implement the meaning-based checks (`gibberish`, `relevance`, `sentiment`) or `factual_consistency`. - `tetrate` carries stage limits on five checks: `prompt_injection`, `jailbreak`, and `secrets` on input only, `bias` and `relevance` on output only. - `azurecontentsafety` covers the nine checks shown, with `prompt_injection` and `jailbreak` on input only and `factual_consistency` (groundedness) on output only. - `factual_consistency` is therefore not available on any in-cluster provider except nominally through `regexp`. Groundedness checking requires `azurecontentsafety` or an external service that implements it. The table is derived from the canonical capability declaration that the API validates against and the Admin Dashboard uses to scope its forms. Where this page and the product disagree, the API's error message is authoritative. ### External guardrail services: per-instance capabilities An `externalservice` provider wraps a remote guardrail service the customer operates. Its capabilities are declared **per instance**, not per kind: - **Checks** are declared in the provider instance's configuration as a list of canonical check names. A rule write against that provider is validated against the instance's declared list rather than against a kind-wide set. - **Stages** are unrestricted at write time. The gateway forwards the stage to the remote service, which decides what it evaluates. Provider creation enforces the declaration: an `externalservice` provider with no valid, non-empty check list, drawn from the check-type vocabulary above, is rejected. ### Write-time validation Creating or updating a guardrail rule or provider is validated against the matrix. The API rejects the write with an **invalid-argument** error naming the supported set: | Write | Rejected when | | :---- | :---- | | Create or update a rule | The check type is not supported by the provider's kind or, for `externalservice`, is absent from the instance's declared check list. The error lists the checks the provider does support | | Create or update a rule | The requested stage, where unset means both, falls outside the (kind, check) pair's supported stages. The error names the supported stages | | Create or update a rule | The referenced provider's kind is unknown | | Create a provider | The `kind` is not one of `regexp`, `builtin`, `tetrate`, `azurecontentsafety`, or `externalservice` | | Create an `externalservice` provider | The configuration's check list is missing, empty, or contains names outside the canonical check vocabulary | Updates re-validate whenever the provider, check type, stage, or configuration changes. Edits confined to unrelated fields (name, description, active) do not re-trigger validation, which is what keeps legacy rules editable. Because the Admin Dashboard scopes the rule form to the same matrix, these rejections surface mainly through direct API use and automation, where the error fails an apply fast instead of shipping a rule that can never fire. ### Troubleshooting: rules that never fire Rules created before write-time validation existed were never checked against the matrix, and they are **grandfathered**: they keep working exactly as before. A grandfathered rule whose (provider kind, check type, stage) combination is unsupported nevertheless **never fires**, because the provider returns a zero score for the unsupported check or stage and the rule is a silent no-op. Where a guardrail does not fire on content that should trigger it: 1. **Check the rule against the matrix above.** Confirm that the provider kind supports the check type, and that the rule's stage, remembering that unset means both, falls within the supported stages. 1. **Check the control-plane logs.** The control plane logs a warning, including the rule ID, provider kind, check type, and stage, every time it ships a rule to the data plane that the matrix says can never fire. Searching for that warning is the fastest way to find pre-existing no-op rules. 1. **Check the Admin Dashboard.** A stored check type the provider does not support is shown as disabled in the rule's form. Editing a grandfathered rule repairs it: changing its provider, check type, stage, or configuration re-triggers validation, so the edit is held to the matrix and the corrected rule is guaranteed to be a combination that can fire. Edits limited to name, description, or active status leave the rule untouched by validation. ## Built-in guardrail templates Agent Router Enterprise ships 14 ready-made template guardrails (`is_template=true`). Creating a guardrail from a template clones its rules into the project, where every setting can then be tuned. All template rules ship with **action = redact, mode = monitor**, an observe-first default: a template rule records what it would have done until it is promoted to enforce, and to block where appropriate. The rollout procedure is covered in [Configure guardrails](/agent-router-enterprise/guides/operate-and-govern/safety-and-data-protection/configure-custom-guardrails/). ### Data privacy and PII (regex-based: deterministic, in process, about 0 ms) All rules use the `pii` and `secrets` check types on the `regexp` provider with named pattern catalogs, on both input and output, redacting matches in place (for example `[CREDENTIAL_REDACTED]`, `[{pattern_name}_REDACTED]`). | Template | Severity and failure mode | Coverage | | :--- | :--- | :--- | | **Baseline PII Protection** | critical, fail-close | Minimal set for internal tools: tax identifiers, credentials and API keys (AWS, GitHub, Slack, generic), payment cards and IBAN | | **GDPR EU PII Protection** | critical, fail-close | EU national identifiers, financial data, contact information, business identifiers (GDPR Art. 32) | | **Canadian PIPEDA PII Protection** | critical, fail-close | SIN, OHIP and driver licenses, passports and immigration documents, institutional IDs, contact and financial data (PIPEDA, FIPPA) | | **Australian PII Protection** | critical, fail-close | TFN, ABN, Medicare, passports, international IDs, financial data, credentials, network infrastructure identifiers, protected-class information | | **Singapore PDPA Compliance** (PII half) | critical, fail-close | NRIC-class identifiers, contact, financial, and business identifiers | ### Regulatory compliance (builtin keyword pipeline: deterministic, in process, sub-millisecond) All rules use the `compliance` or `banned_topics` check type on the `builtin` provider's five-layer keyword pipeline, on both input and output unless noted. | Template | Severity and failure mode | Coverage | | :--- | :--- | :--- | | **EU AI Act Article 5 Compliance** | critical, fail-close | Prohibited practices: manipulative AI, vulnerability exploitation, social scoring, emotion recognition, biometric profiling, with parallel rule sets in **English and French** | | **Singapore MAS AI Governance** | critical, fail-close | MAS AI risk-management guidelines for financial institutions: fairness and bias assessment, transparency and explainability, human oversight, data governance, model security controls | | **Singapore PDPA Compliance** (policy half) | critical, fail-close | Sensitive data categories, do-not-call registry, cross-border data transfer, profiling and automated decisions | | **UAE Regulatory Compliance** | high, fail-close | Anti-discrimination law and cultural-sensitivity requirements | | **Healthcare Claims Compliance** | critical, fail-close | Claims-processing safety: fraud coaching, disclosure of protected health information (PHI), prior-authorization gaming, system-override attempts, unauthorized medical advice | ### Security (builtin, input stage, deterministic, in process) | Template | Severity and failure mode | Coverage | | :--- | :--- | :--- | | **Prompt Injection Detection** | critical, fail-close | Input-only rules for prompt injection, system-prompt extraction, data exfiltration, jailbreaks, malicious code injection, and SQL injection (`prompt_injection`, `jailbreak`, and `ban_code` check types) | ### Content and business policy (builtin, deterministic, in process) | Template | Severity and failure mode | Coverage | | :--- | :--- | :--- | | **NSFW Content Safety (Global)** | high, fail-close | Toxicity in **five language variants** (English, Spanish, French, German, Australian English), plus violence, self-harm, and child-safety rules (`toxicity`, `violence`, `self_harm`, `sexual_content`) | | **Bias & Discrimination Protection** | high, fail-open | Gender, racial, religious, sexual-orientation, age, and disability bias (`bias`, `hate_speech`) | | **Professional Advice Guardrails** | high, fail-open | Output-only rules preventing unauthorized legal, financial, and medical advice (`banned_topics`) | | **Competitor Mention Detection** | medium, fail-open | Paired input and output rules for competitor references, recommendations, and comparisons (`ban_competitors`) | The same check types can be re-pointed at the ML providers where semantic coverage is needed. A `jailbreak` rule on the `tetrate` provider uses its jailbreak classifier with NLI fallback instead of, or layered behind, the pattern-based template rule. ## Design characteristics | Characteristic | Description | | :--- | :--- | | In-gateway, in-process evaluation | a dynamic module in the data plane, rather than an SDK embedded per application or a separate proxy-of-a-proxy service hop | | ML detection without data egress | The `tetrate` provider runs embedding and classifier models inside the customer cluster on CPU, so semantic detection (jailbreak, PII, toxicity, and the NLI-backed checks) sends no prompt to a third party. Third-party engines (Azure AI Content Safety, external services) are available and are an explicit opt-in per rule | | Combinations validated before they ship | The API rejects a rule whose provider kind cannot serve its check type on its stage, so a rule that could never fire cannot be created. The Admin Dashboard scopes its forms to the same matrix | | A cost ladder controlled per rule | The same check type can be served by a free in-process engine or by an ML engine. The gateway auto-orders cheap decisive checks first so local hits short-circuit expensive calls, and batches all of a provider's checks into one call | | Redaction as well as blocking | Rules can mask matched content in place, with configurable redaction formats, and let traffic flow, on inputs and on outputs, instead of only rejecting requests | | Shadow mode on every rule | Any rule can run in audit-only mode against live traffic before enforcement. Monitor evaluation is asynchronous and never affects latency or streaming | | Deliberate failure semantics | Fail-close, fail-open, and the evaluation timeout are explicit per-guardrail settings, so safety-critical paths can fail closed while availability-critical paths fail open | | Investigation support | Blocked responses carry a correlation ID. Operators holding the read-only guardrail-reader role can look up which rules fired, the scores, and the offending content, and join against their own logging by correlation ID | | 14 ready-made templates | Region-specific and regulation-specific packs (GDPR, PIPEDA, PDPA, MAS, EU AI Act Article 5, UAE, Australian privacy, healthcare claims) plus security and content-safety packs, multilingual where relevant (EU AI Act in English and French, toxicity in five language variants), cloneable and fully editable per project | Where to go next About guardrails The object model: policies, rules, providers, patterns, and thresholds. Configure guardrails Create project policies and rules, and validate in monitor mode before enforcing. Configure guardrail providers How the scoring engines differ in detection approach, latency, and configuration. Guardrails best practices and FAQ Rollout order, latency budgets, failure modes, and investigation. Guardrails architecture Where guardrail evaluation sits on the request path through the gateway. Guardrails API reference Request and response detail for the guardrail and rule management API. Providers, patterns, and triggers have their own API references. --- ## References Reference material for looking things up rather than reading end to end: the SDK and API surfaces, the gateway's documented behavior, telemetry and audit schemas, and the planning and evaluation pages that support a rollout. Programming surfaces Agent Router exposes two different HTTP surfaces. They are related to the [management plane and data plane](/product-architecture/planes-and-components/) split, but they are not the same thing: the management plane is where Agent Router is hosted and configured; the Management API is the HTTP catalog for that configuration. Live model traffic uses the Gateway APIs on the data plane. Agent Router SDK Quickstarts for Go, Python, TypeScript, and curl. Gateway APIs Use when your app sends prompts through the data-plane gateway: OpenAI- or Anthropic-compatible paths such as /v1/chat/completions, with request and streaming examples. Management API reference Use when you provision or automate Agent Router itself: clients, keys, catalog, projects, insights, and related OpenAPI services. Gateway behavior and schemas Gateway behavior Routing, fallback, and error semantics as the gateway implements them. OTEL metrics The OpenTelemetry metrics and attributes the gateway exports. Audit log events The audit event catalog with actors, targets, and payloads. Planning and evaluation Sizing and scale Throughput, resource sizing, and deployment footprint. Compliance Residency, retention, and due-diligence answers. Evaluation scenarios Graded, verifiable exercises for a hands-on product evaluation. Lookup Glossary Definitions of the terms used throughout the documentation. External links Provider documentation and related Tetrate resources. --- ## Integrations Agent Router presents an OpenAI-compatible surface, so most tools that already speak to an AI provider connect by changing only a base URL and an API key. This page is the map of everything that plugs into the gateway: the SDKs and frameworks that send inference traffic, the coding agents that generate against it, the guardrail providers and observability backends it can call out to, and the MCP clients that consume aggregated tools. Each entry links to the guide or reference that documents the connection. ## SDKs and frameworks Any client that targets the OpenAI Chat Completions, OpenAI Responses, or Anthropic Messages format works against the gateway. The base URL points at the proxy endpoint and the API key is an Agent Router key; no provider-specific client is required. Agent Router SDK Language quickstarts for Go, Python, TypeScript, and curl. Integrate the gateway with an app Point OpenAI, Anthropic, LangChain, and other SDKs at the gateway. The current documentation URLs for the supported SDKs and frameworks (OpenAI, Anthropic, LangChain, Vercel AI SDK, Pydantic AI, CrewAI, and others) are listed in [External links](/reference/external-links/). ## Coding agents Coding agents receive a ready-to-paste config that teaches them the gateway base URL, authentication, and endpoints, so generated code routes through Agent Router from the first request. Coding agents & tools Config for Claude Code, Cursor, Codex CLI, Lovable, and Coder workspaces. Playground Compare models and iterate on prompts before wiring an agent. ## Guardrail providers Guardrails run inline in the gateway. Rules are scored by providers such as `regexp`, `builtin`, and `tetrate`, and can call out to an external guardrail service where one is configured. Providers and rules are configured by a platform operator for a project. Configure guardrails Create project policies and rules; validate in monitor mode before enforce. Configure guardrail providers Choose regexp, builtin, or tetrate for each check type. ## Observability backends The gateway exports traces and metrics over OpenTelemetry (OTLP) and exposes a Prometheus scrape endpoint, so any OTLP-compatible or Prometheus-compatible backend can receive gateway telemetry. Export telemetry to an observability stack Send traces and metrics to Grafana, Datadog, New Relic, and similar backends. OpenTelemetry metrics The metric names and attributes the gateway emits. ## MCP clients Multiple MCP servers can be aggregated into a single profile and exposed to AI clients through one endpoint, with access governed centrally. Aggregate MCP servers into a profile Combine MCP servers into one endpoint for AI clients. Govern MCP server access Control which MCP servers and tools are reachable, and by whom. ## Related - [Gateway APIs](/reference/supported-apis/): the request formats every integration uses - [Supported providers](/reference/supported-providers/): the upstream services the gateway routes to - [External links](/reference/external-links/): documentation URLs for every SDK, backend, and client named above --- ## OpenTelemetry traces and metrics The gateway emits two complementary streams of observability data. Trace data flows over OTLP to any OpenTelemetry-compatible backend; metrics are exposed by each data plane component on a Prometheus-compatible scrape endpoint, where they can be pulled by an existing metrics agent and forwarded as OTLP metrics by a collector if the destination requires it. This page documents the structure of both streams: the span model used for traces, the attributes and events actually stamped on those spans, and the metric families exposed by the gateway. For the configuration mechanics of trace export and the supported authentication modes, see Export Telemetry to an Observability Stack. :::note Every span name, attribute, event, and metric instrument on this page corresponds to a specific construct in the shipped data plane, based on Envoy AI Gateway v0.7.0 with Tetrate patches applied. Instrumentation changes between data plane releases. Where a dashboard or alert rule depends on an exact string, confirm it against the running deployment: spans through a test request to the configured backend, metrics through the scrape endpoint described under [Metric names on the Prometheus endpoint](#metric-names-on-the-prometheus-endpoint). ::: ## Trace structure Spans are emitted by a single component: the **ai-gateway extproc**, the external-processing service that Envoy calls for each AI request. Each request produces **one flat span**. There is no nesting. Three consequences follow, and each matters when dashboards or alert rules are being authored against this data: - **There are no gateway, routing, or inference child spans.** The extproc span covers the request end to end. A fallback walk across several backends does not add spans, and the individual attempts are not separately visible in the trace stream. - **There are no Envoy proxy spans.** The `EnvoyProxy` resource rendered by the data plane does not configure a tracing provider, so the Envoy hop is not instrumented. The extproc span is the whole of the gateway's contribution to the trace. - **Client trace context is honoured.** Incoming request headers are run through the configured propagator (W3C `traceparent` by default) before the span is started, so the extproc span attaches to the caller's trace as a child rather than starting a new one. Where a client is instrumented, the gateway span appears inside the application's existing trace. The span carries the OTel span kind `Internal` for all inference endpoints, and `Client` for MCP spans. ### Span names The span name identifies the endpoint that served the request. Names are fixed strings supplied by the per-endpoint recorder, not derived from the request path: | Span name | Endpoint | | --- | --- | | `ChatCompletion` | Chat completions | | `Completion` | Legacy text completions | | `Responses` | Responses API | | `CreateEmbeddings` | Embeddings | | `ImagesResponse` | Image generation | | `AudioSpeech` | Speech synthesis | | `Transcription` | Audio transcription | | `Translation` | Audio translation | | `Rerank` | Rerank (Cohere schema) | | `Message` | Anthropic Messages, including token-counting requests, which share the Messages recorder | MCP traffic is named from the JSON-RPC method rather than the endpoint: `Initialize`, `ListTools`, `CallTool`, `ListPrompts`, `GetPrompt`, `ListResources`, `ReadResource`, `Subscribe`, `Unsubscribe`, `ListResourceTemplates`, `SetLoggingLevel`, `Complete`, and `Ping`. Any method without a mapping is used verbatim as the span name. ### Resource attributes Resource attributes are set from the environment by the data plane chart and are identical on every span: | Attribute | Value | | --- | --- | | `service.name` | `ai-gateway-extproc` | | `service.layer` | `ENVOY_AI_GATEWAY` | | `job_name` | `envoy-ai-gateway` | ### Span attributes Attributes follow the [OpenInference semantic conventions](https://github.com/Arize-ai/openinference/blob/main/spec/semantic_conventions.md), the open standard for large-language-model trace attributes. This is what allows an LLM-aware backend, such as Arize Phoenix or any other OpenInference-compatible viewer, to render the span as a model invocation rather than a generic HTTP span. Identity and parameters: | Attribute | Type | Description | | --- | --- | --- | | `openinference.span.kind` | string | `LLM` for inference endpoints, `EMBEDDING` for embeddings, `RERANKER` for rerank | | `llm.system` | string | `openai`, `anthropic`, or `cohere`, reflecting the request schema. Not set on embedding spans, where the convention excludes it | | `llm.model_name` | string | The model name. Set from the request at span start, then overwritten from the response model where the provider returns one | | `llm.invocation_parameters` | string | JSON of the request parameters excluding messages and tools, which have their own attributes | | `embedding.model_name` | string | The embeddings equivalent of `llm.model_name`, on `CreateEmbeddings` spans | | `embedding.invocation_parameters` | string | JSON of `model`, `encoding_format`, `dimensions`, and `user`, on `CreateEmbeddings` spans | Input and output payloads: | Attribute | Type | Description | | --- | --- | --- | | `input.value` | string | The raw request body as JSON, or `__REDACTED__` when input capture is disabled | | `input.mime_type` | string | `application/json`, set alongside a captured `input.value`. On `Responses` spans it is set unconditionally | | `output.value` | string | The serialised response body, or `__REDACTED__` when output capture is disabled | | `output.mime_type` | string | `application/json`. Omitted when output capture is disabled | | `output.audio_duration` | double | Audio duration reported by a transcription response, where present | | `output.language` | string | Language detected by a transcription response, where present | Token counts. These are treated as metadata and are still recorded when prompt and response capture is disabled: | Attribute | Type | Description | | --- | --- | --- | | `llm.token_count.prompt` | integer | Prompt (input) tokens | | `llm.token_count.completion` | integer | Completion (output) tokens | | `llm.token_count.total` | integer | Combined prompt and completion tokens | | `llm.token_count.prompt_details.cache_read` | integer | Prompt tokens served from the provider's prompt cache | | `llm.token_count.prompt_details.cache_creation` | integer | Prompt tokens written to the provider's prompt cache | | `llm.token_count.prompt_details.audio` | integer | Audio tokens in the prompt | | `llm.token_count.completion_details.reasoning` | integer | Tokens spent on reasoning or chain-of-thought | | `llm.token_count.completion_details.audio` | integer | Audio tokens in the completion | Each count is recorded only when the provider reports a value greater than zero, so absence of an attribute indicates absence of usage rather than a gap in instrumentation. Message content. These are indexed families rather than single keys, and they are present only when the active logging mode permits prompt and response capture. See [Configuring Request Logs](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/configuring-request-logs/) for the controls: | Attribute family | Description | | --- | --- | | `llm.input_messages.{i}.message.role`, `.message.content` | Role and content per input message | | `llm.input_messages.{i}.message.contents.{j}.message_content.{text,type,image.image.url}` | Multi-part input content, including images | | `llm.output_messages.{i}.message.role`, `.message.content` | Role and content per output message | | `llm.output_messages.{i}.message.tool_calls.{j}.tool_call.{id,function.name,function.arguments}` | Tool calls returned by the model | | `llm.tools.{i}.tool.json_schema` | JSON schema of each tool offered in the request | | `llm.prompts.{i}.prompt.text`, `llm.choices.{i}.completion.text` | Prompt and choice text on legacy `Completion` spans | | `embedding.embeddings.{i}.embedding.text`, `.embedding.vector` | Embedding input text and output vector on `CreateEmbeddings` spans | Because indexed message attributes scale with conversation length, the default OTel attribute count limit of 128 is lifted when message capture is enabled, and retained otherwise. MCP spans carry their own set instead, including `mcp.protocol.version`, `mcp.transport`, `mcp.request.id`, `mcp.method.name`, and, according to the method, `mcp.tool.name`, `mcp.prompt.name`, `mcp.resource.uri`, `mcp.session.id`, and `mcp.client.{name,title,version}`. ### Header-mapped attributes The data plane configures a header-to-attribute mapping on the extproc. Each listed request header, when present, is copied onto the span under the mapped attribute name: | Attribute | Source header | Description | | --- | --- | --- | | `request.id` | `x-request-id` | The per-request correlation ID | | `tars.user` | `x-tars-user` | The identity that issued the request | | `tars.customer` | `x-tars-customer` | The owning customer | | `tars.workspace` | `x-tars-workspace` | The owning workspace | | `tars.router` | `x-tars-router` | The router that served the request | Two further internal mappings (`tars.signature` and `bellhop.user.id`) are configured alongside these and are not intended as a query surface. This mapping is applied to every endpoint span except `ImagesResponse`, whose tracer is constructed without a header mapping. Image-generation spans therefore carry no `request.id` and no `tars.*` attributes, which is worth accounting for in any dashboard that groups by workspace or customer. ### Span events The event set is small: lifecycle markers on streaming responses, the standard OTel error event, and one event specific to MCP. | Event | Recorded when | | --- | --- | | `First Token Stream Event` | The first chunk of a streaming response is observed. Emitted on `ChatCompletion`, `Completion`, `Responses`, and `Message` spans | | `Response Completed Event` | A streaming Responses API request reaches its completion event. Emitted on `Responses` spans only | | `exception` | The upstream returned a non-success status. Carries `exception.type` (`BadRequestError`, `AuthenticationError`, `PermissionDeniedError`, `NotFoundError`, `RateLimitError`, `InternalServerError`, or `Error`, mapped from the status code) and `exception.message`. The span status is also set to error | | `route to backend` | An MCP request is routed to a backend. Carries `mcp.backend.name` and `mcp.session.id`. Emitted on MCP spans only | There is no first-byte, request-received, routing-resolved, or backend-attempted event. Non-streaming requests emit no lifecycle event at all: the span's start and end timestamps are the only timing signal. ### Fields not present on spans The following are absent from the trace stream and cannot be queried, alerted on, or charted from span data. Several are available elsewhere, as noted: | Expected field | Reality | | --- | --- | | `gateway.*` (`request_id`, `api_key_id`, `requested_model`, `resolved_model`, `resolved_provider`, `fallback_attempts`) | No `gateway.*` namespace is emitted. Correlation is available as `request.id`; the model appears as `llm.model_name` | | `llm.usage.*` | Not emitted. Token counts use the `llm.token_count.*` keys above | | `llm.provider` | Not emitted. The OpenInference embedding spec excludes it, and no recorder sets it. The nearest equivalent is `llm.system`, which names the request schema rather than the resolved upstream | | `gateway.latency_ms`, `gateway.time_to_first_token_ms` | Not emitted as attributes. Latency is derivable from span duration, and both are available as metrics (`gen_ai.server.request.duration`, `gen_ai.server.time_to_first_token`) | | `gen_ai.*` | These are metric attributes only. No `gen_ai.*` attribute is set on any span | | `http.method`, `http.target`, `http.status_code` | Not set by the extproc recorders. The endpoint is identifiable from the span name, and failures from the `exception` event | | Fallback attempt detail | Not represented in the trace stream. Fallback behaviour is observable through metrics and Request Logs | For the export configuration that delivers these spans to a backend, see [Export Telemetry to an Observability Stack](/agent-router-enterprise/guides/observability-and-analytics/export-telemetry-to-an-observability-stack/). --- ## Metric families Metrics are registered by the ai-gateway extproc, the same component that emits spans. Two delivery paths exist and are independent of each other: - **Prometheus scrape.** The extproc admin listener serves `/metrics` in Prometheus exposition format, on port 1064 by default. The Prometheus reader is always registered, irrespective of the configured OTLP metrics exporter, so this endpoint is available whenever the extproc is running. - **OTLP push.** When metric export is enabled, the same instruments are also pushed over OTLP to the configured collector. Envoy's own connection and HTTP statistics are a separate stream, published by the proxy's Prometheus stats sink and documented by Envoy rather than here. The instrument set is small and follows the [OpenTelemetry semantic conventions for generative AI metrics](https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-metrics/). Every instrument is a histogram or a counter; **no gauges are registered**, so there is no in-flight or active-request metric to chart. ### Inference metrics Four instruments cover all inference traffic. All four are histograms: | Metric | Unit | Records | | --- | --- | --- | | `gen_ai.client.token.usage` | token | Token counts, one observation per token category. Partitioned by `gen_ai.token.type` | | `gen_ai.server.request.duration` | s | Total request duration, measured from receipt of request headers in the extproc to the end of response-body processing. Recorded for every request, successful or not | | `gen_ai.server.time_to_first_token` | s | Time from request-header receipt to the first token of the response. Streaming responses only | | `gen_ai.server.time_per_output_token` | s | Mean time per output token after the first, calculated as `(request_duration - time_to_first_token) / (output_tokens - 1)`. Recorded once at end of stream, and only when more than one output token was produced | Because `gen_ai.server.request.duration` spans the extproc's whole involvement, it does not separate gateway-internal processing from time spent waiting on the upstream provider. There is no backend-latency instrument, so that split is not available from the metric stream. `gen_ai.client.token.usage` records a separate observation per token category rather than a single total. Summing across all values of `gen_ai.token.type` double counts, because cached and cache-creation input tokens overlap with the input total as the provider reports them: | `gen_ai.token.type` | Meaning | | --- | --- | | `input` | Prompt tokens | | `output` | Completion tokens | | `cached_input` | Prompt tokens served from the provider's cache | | `cache_creation_input` | Prompt tokens written to the provider's cache | | `reasoning` | Tokens spent on reasoning | The `cached_input` and `cache_creation_input` values are not part of the upstream specification yet; they are permitted as custom values and are used here pending standardisation. ### MCP metrics MCP traffic is instrumented separately: | Metric | Type | Records | | --- | --- | --- | | `mcp.request.duration` | Histogram | Duration of an MCP request. Carries `error.type` on the error path | | `mcp.initialization.duration` | Histogram | Duration of MCP session initialisation | | `mcp.method.count` | Counter | MCP method invocations. Carries `mcp.method.name` and `status` (`success`, `failed`, or `error`) | | `mcp.capabilities.negotiated` | Counter | Capabilities agreed during initialisation. Carries `capability.type` (`tools`, `resources`, `prompts`, `sampling`, `roots`, `experimental`, `elicitation`, `completions`, `logging`) and `capability.side` (`client` or `server`) | | `mcp.progress.notifications` | Counter | Progress notifications relayed | MCP instruments additionally carry `mcp.backend`, identifying the upstream server that handled the request. :::note `mcp.initialization.duration` is registered with the unit `token` despite measuring seconds. The value is a duration; only the declared unit is wrong. This matters because the Prometheus exporter appends the unit to the metric name, so the scraped series is named for tokens rather than seconds. Confirm the rendered name against a live scrape before writing a query against it. ::: ### Metric names on the Prometheus endpoint The names above are the OTLP instrument names. The Prometheus exporter rewrites them on the way out: dots become underscores, the unit is appended as a suffix, and histograms expand into `_bucket`, `_sum`, and `_count` series. `gen_ai.server.request.duration` is therefore not the string a PromQL query uses. Rather than reproduce the mapping here, where it would drift, query the live endpoint for the exact series names. The extproc runs as a sidecar container in the Envoy proxy pods, so the admin listener is reached from inside one of those pods: ```bash kubectl port-forward -n NAMESPACE POD_NAME 1064:1064 curl -s localhost:1064/metrics | grep -E '^# (HELP|TYPE)' ``` Substitute the namespace and pod of the gateway being inspected; the proxy namespace is set at install time and is not fixed. The `# TYPE` lines give the exposed series name and instrument type for each metric. --- ## Metric attributes Every inference metric carries the same five base attributes, plus the header-mapped set below. Two further attributes are conditional. | Attribute | Cardinality | Description | | --- | --- | --- | | `gen_ai.operation.name` | Low | The endpoint that served the request: `chat`, `completion`, `embeddings`, `messages`, `image_generation`, `responses`, `speech`, `transcription`, `translation`, or `rerank` | | `gen_ai.provider.name` | Low | The backend's API schema: `openai`, `azure.openai`, `aws.bedrock`, `aws.anthropic`, `gcp.vertex_ai`, `gcp.anthropic`, `anthropic`, or `cohere`. Any other schema falls back to the configured backend name | | `gen_ai.original.model` | Medium | The model named in the incoming request body, before any virtualisation is applied | | `gen_ai.request.model` | Medium | The model sent upstream after resolution | | `gen_ai.response.model` | Medium | The model the provider reports as having generated the response, which is often a dated build of the requested model | | `gen_ai.token.type` | Low | Token category. Present on `gen_ai.client.token.usage` only | | `error.type` | Low | Present on `gen_ai.server.request.duration` only, and only for failed requests. The value is always the `_OTHER` placeholder; the gateway does not yet classify error types on this metric | The three model attributes are the requested-versus-resolved distinction: `gen_ai.original.model` is what the caller asked for, `gen_ai.request.model` is what routing selected, and `gen_ai.response.model` is what actually answered. Any of the three reports `unknown` when the value could not be determined, which is the expected reading for requests that failed before model resolution. Because `error.type` is the only failure dimension and it carries a single placeholder value, the metric stream distinguishes failed from successful requests but not one failure mode from another. Error classification comes from Request Logs, or from the `exception.type` attribute on the corresponding span. ### Header-mapped attributes {#metric-header-mapped-attributes} As with spans, request headers are copied onto metrics under mapped attribute names. **The metric mapping is not the same as the span mapping**, which is worth attention when building queries that span both: | Attribute | Source header | On spans | On metrics | | --- | --- | :---: | :---: | | `tars.user` | `x-tars-user` | Yes | Yes | | `tars.customer` | `x-tars-customer` | Yes | Yes | | `tars.workspace` | `x-tars-workspace` | Yes | Yes | | `tars.router` | `x-tars-router` | Yes | Yes | | `router.project.id` | `x-router-project-id` | No | Yes | | `router.gateway.id` | `x-router-gateway-id` | No | Yes | | `request.id` | `x-request-id` | Yes | No | Two consequences follow: - **There is no per-request join between metrics and traces.** `request.id` is a span attribute only, by design: it would be an unbounded label on a metric. Per-request investigation belongs to the trace stream and Request Logs. - **Chargeback and per-tenant reporting are available from metrics**, at customer, workspace, project, and router granularity. There is no API key dimension on any metric, so per-key attribution is not obtainable from the metric stream; Request Logs and the in-Console Usage Analytics surface are the sources for that. --- ## Sampling The data plane does not set a sampler, so the OpenTelemetry SDK default applies: **`parentbased_always_on`**. The consequences differ according to whether the caller is instrumented, and the second case is a common source of confusion: - **Unparented requests**, where the client sends no `traceparent` header, are always sampled. Every such request produces a span. - **Parented requests**, where the client does send trace context, inherit the caller's sampling decision. If the calling application samples at 1 %, then 99 % of its gateway requests arrive marked as not sampled, and the gateway records no span for them. The second case means trace volume is partly outside the gateway's control. An instrumented application that samples aggressively will appear to be missing from the trace stream, and no gateway-side setting recovers those spans: the decision was made upstream and propagated. Where a service's gateway traffic is expected in the trace stream but absent, the caller's own sampling configuration is the first thing to check. Metrics are not sampled. Every request is recorded in every applicable instrument regardless of the trace sampling decision, so the metric stream is a complete record where the trace stream may not be. This shapes which stream answers which question: | Question | Stream | | --- | --- | | What did this specific request send and receive? | Traces, or Request Logs | | Why did this request fail, and with what error type? | Traces (`exception` event), or Request Logs | | What is the p95 latency for this model? | Metrics | | How many tokens did this workspace consume? | Metrics | | Did every request get counted for billing? | Metrics or Request Logs, never traces | Because the gateway contributes a single flat span, the trace stream does not break latency down by stage and does not show which backends a fallback chain attempted. Neither is available from metrics either; fallback behaviour is visible only in Request Logs. --- ## Related - [Gateway Behavior](/reference/gateway-behavior/): correlation IDs and per-request data - [Audit Log Events](/reference/audit-log-events/): event schema for administrative actions - [Export Telemetry to an Observability Stack](/agent-router-enterprise/guides/observability-and-analytics/export-telemetry-to-an-observability-stack/): configuration walkthrough for trace and metric export Where to go next Export telemetry to an observability stack Configure trace and metric export to your backend. Monitor traffic and usage Read request volume, latency, and token usage in the Console. --- ## Release Notes Version history for the customer-managed parts of Agent Router. The management plane is operated by Tetrate and updated continuously, so it is not versioned here. This page tracks the components an operator installs and upgrades: the self-hosted data plane, the tare CLI, and the gateway. Entries are listed newest first. ## What is tracked | Component | Why it is versioned | Where it runs | | --- | --- | --- | | Data plane | Installed and upgraded by the operator; the version determines available gateway behavior | Customer-managed Kubernetes | | `tare` CLI | Drives installation and upgrades; new subcommands and flags arrive here | Operator workstation or CI | | Gateway | The Envoy-based proxy and its dynamic modules that enforce routing, guardrails, and cost | Inside the data plane | The management plane, the Developer Console, and the Admin Dashboard are hosted by Tetrate and roll out continuously. Behavior changes that affect operators or developers are noted in the relevant release entry below. ## How to read an entry Each release is a second-level heading with its version and date. Changes are grouped under a fixed set of categories so a reader can scan for what matters: - **Added**: new capabilities or endpoints. - **Changed**: behavior that differs from the previous release. - **Fixed**: resolved defects. - **Security**: fixes and hardening with a security impact. - **Deprecated**: capabilities scheduled for removal, with the target release. - **Upgrade notes**: manual steps required to move from the previous version. Upgrades are performed with `tare upgrade`. Any release that needs manual intervention says so in **Upgrade notes**; releases without that section upgrade in place with no additional steps. ## Releases :::note This page is the template for tracking releases. Replace the illustrative entry below with real releases as they ship, keeping the newest at the top and reusing the category headings above. ::: ### 0.0.0 — YYYY-MM-DD The format for a release entry. Only the categories that apply to a release are included. **Added** - New capability, with a link to the guide or reference page that documents it. **Changed** - Behavior that differs from the previous release, stated so an operator can tell whether action is needed. **Fixed** - Defect that is now resolved. **Upgrade notes** - Any manual step required before or after running `tare upgrade`. Omit this section when the upgrade is in place. ## Related - [tare CLI reference](/reference/tare-cli/): the commands used to install and upgrade the data plane - [Sizing and scale](/reference/sizing-and-scale/): capacity guidance to review before an upgrade - [Gateway behavior](/reference/gateway-behavior/): the request-handling contract that release entries reference Where to go next tare CLI reference Commands for installing and upgrading the data plane. Sizing and scale Capacity guidance to review before an upgrade. --- ## SDK Reference Because the gateway is OpenAI-compatible and Anthropic-compatible, the existing provider SDKs work unchanged: point the SDK's base URL at the gateway and pass an Agent Router API key. No Agent Router-specific SDK is required. For teams that prefer a first-class client, typed Agent Router SDKs are also published for Go, Python, and TypeScript. Release v0.1.1 Stable channel ## Use an existing provider SDK The lowest-friction path is to keep the SDK an application already uses. The OpenAI and Anthropic client libraries call the same API shapes the gateway serves, so only two things change: the base URL is set to the gateway endpoint, and the API key is an Agent Router key rather than a provider key. Existing request and response handling is untouched. This approach is covered end to end, with per-language snippets, in [Integrate the gateway with an app](/agent-router-service/guides/integrate-the-gateway-with-an-app/). ## Typed Agent Router SDKs A typed client for Go, Python, or TypeScript can be downloaded when a generated, strongly typed surface over the management API is preferred. Each artifact is the Agent Router SDK source tree at the release shown above, distributed from `https://tare.tetrate.ai/tools/sdks/stable/`. Go Typed client, source tree.Download .tar.gz Python Typed client, source tree.Download .tar.gz TypeScript Typed client, source tree.Download .tgz Verify downloads | Artifact | Version | Size | SHA-256 | | :---- | :---- | :---- | :---- | | `agentrouter-go-0.1.1.tar.gz` | 0.1.1 | 438.8 KB | `a83023263c0b…` | | `agentrouter-python-0.1.1.tar.gz` | 0.1.1 | 132.7 KB | `bef21b0a9585…` | | `agentrouter-typescript-0.1.1.tgz` | 0.1.1 | 232.9 KB | `d7889ef26ec4…` | ## Quickstarts Each quickstart installs the client for one language, then makes a first read-only identity call to confirm the key and base URL are wired correctly. The curl quickstart needs no SDK at all. Go Install from the local tarball and run a first whoami call. Python Install the sdist and run a first whoami call. TypeScript Install the local pack and run a first whoami call. curl Make a first whoami call with no SDK, using only an API key. ## API specifications Machine-readable descriptions of the management API are available as a combined OpenAPI 3.1 document and a Postman v2.1 collection. The OpenAPI document supports client generation and contract tooling; the Postman collection supports interactive exploration and testing. OpenAPI spec OpenAPI 3.1, YAML.Download .yaml Postman collection Postman v2.1, JSON.Download .json Verify downloads | Artifact | Version | Size | SHA-256 | | :---- | :---- | :---- | :---- | | `agentrouter-openapi-0.1.1.yaml` | 0.1.1 | 554.8 KB | `1dfa0ffd16bb…` | | `agentrouter-postman-0.1.1.json` | 0.1.1 | 184.5 KB | `8be489ab856c…` | Versions, archive sizes, and full SHA-256 digests are published on the Agent Router developer portal, which is the authoritative source for the current release. The checksums above are abbreviated; the full digest is available there for integrity verification. Related references Management API reference The HTTP APIs the gateway exposes, which the provider SDKs and typed clients call. Integrate the gateway with an app Point SDKs, frameworks, and code assistants at the gateway. --- ## SDK quickstart for curl This quickstart makes a first read-only identity lookup (whoami) against the gateway using nothing but curl and an API key. The lookup resolves the identity behind an API key with a GET /v1/me call and is safe to run as-is, so it confirms that the key and the base URL are wired together correctly before any billable request is sent. ## Prerequisites Two values and a tool are required before starting: - **An Agent Router API key.** Keys are minted in the Console or with the `tare` CLI. The secret is shown once, in the form `.`, so it must be copied immediately. See [Make an API call](/agent-router-service/quickstarts/make-an-api-call/) for the key-minting steps. - **The gateway base URL** for the deployment, in the form `https://api.`. For the hosted service this is `https://api.tetrate.ai`. - **curl**, which is available by default on macOS and most Linux distributions. Both values are read from the environment in the call below: ```bash export AGENTROUTER_BASE_URL=https://api. export AGENTROUTER_API_KEY=. ``` ## Install the SDK No installation is required. curl and an Agent Router API key are sufficient, so there is no SDK or package to download for this path. A typed client is available for teams that prefer one: see the [Go](/reference/sdk/quickstart-go/), [Python](/reference/sdk/quickstart-python/), and [TypeScript](/reference/sdk/quickstart-typescript/) quickstarts. ## Make the first call The request below calls the read-only `GET /v1/me` endpoint, passing the API key as a bearer token. It returns a JSON object describing the identity associated with the key. ```bash curl "${AGENTROUTER_BASE_URL}/v1/me" \ -H "Authorization: Bearer ${AGENTROUTER_API_KEY}" ``` Piping the output to a formatter such as `jq` makes the JSON easier to read: ```bash curl -sS "${AGENTROUTER_BASE_URL}/v1/me" \ -H "Authorization: Bearer ${AGENTROUTER_API_KEY}" | jq ``` A successful call returns the identity associated with the API key. Because `GET /v1/me` only reads identity metadata, the call incurs no model usage and can be repeated freely while verifying setup. Related references SDK Reference Download the SDK archives and API specifications, with versions and checksums. Make an API call Mint a key with the tare CLI and make a first routed request across languages. Management API reference The HTTP endpoints the SDK calls, including the identity and inference APIs. --- ## SDK quickstart for Go This quickstart installs the typed Agent Router SDK for Go, then makes a first read-only identity lookup (whoami) against the gateway. The lookup resolves the identity behind an API key with a GET /v1/me call and is safe to run as-is, so it confirms that the SDK, the key, and the base URL are wired together correctly before any billable request is sent. ## Prerequisites Two values and a toolchain are required before starting: - **An Agent Router API key.** Keys are minted in the Console or with the `tare` CLI. The secret is shown once, in the form `.`, so it must be copied immediately. See [Make an API call](/agent-router-service/quickstarts/make-an-api-call/) for the key-minting steps. - **The gateway base URL** for the deployment, in the form `https://api.`. For the hosted service this is `https://api.tetrate.ai`. - **Go 1.26 or later**, which the SDK module requires. The SDK reads both values from the environment: ```bash export AGENTROUTER_BASE_URL=https://api. export AGENTROUTER_API_KEY=. ``` ## Install the SDK The Go SDK is distributed as a source tarball from the public stable channel and is installed from the local file system. This is the only supported install path: `go get` is not used. Replace `` with the version shown on the [SDKs](/reference/sdk/) page. Download and extract the tarball into `./third_party`: ```bash curl -fLO https://tare.tetrate.ai/tools/sdks/stable/agentrouter-go-.tar.gz mkdir -p third_party && tar -xzf agentrouter-go-.tar.gz -C third_party/ ``` Add `require` and `replace` directives so the module resolves to the extracted copy rather than a public registry, then tidy the module graph: ```text title="go.mod" module example.com/agentrouter-quickstart go 1.26 require github.com/tetrateio/agentrouter-go v0.1.0 // Point replace at the extracted tarball directory. // Its name matches the tarball stem from the SDKs download page. replace github.com/tetrateio/agentrouter-go => ./third_party/agentrouter-go- ``` ```bash go mod tidy ``` ## Make the first call Save the program below as `main.go`. It constructs a client from the two environment variables, calls `client.Me().Get(ctx)` (the read-only `GET /v1/me` endpoint), and prints the resolved identity. ```go title="main.go" // Command example is a runnable example for the AgentRouter Go SDK. // Set AGENTROUTER_BASE_URL and AGENTROUTER_API_KEY in the environment, then `go run .`. package main import ( "context" "fmt" "log" "os" agentrouter "github.com/tetrateio/agentrouter-go" ) func main() { ctx := context.Background() client, err := agentrouter.New(ctx, agentrouter.WithBaseURL(os.Getenv("AGENTROUTER_BASE_URL")), agentrouter.WithAPIKey(os.Getenv("AGENTROUTER_API_KEY")), ) if err != nil { log.Fatalf("client: %v", err) } result, err := client.Me().Get(ctx) if err != nil { log.Fatalf("call: %v", err) } fmt.Printf("%+v\n", result) } ``` Run the program from the directory containing `main.go` and `go.mod`: ```bash go run . ``` A successful run prints the identity associated with the API key. Because `GET /v1/me` only reads identity metadata, the call incurs no model usage and can be repeated freely while verifying setup. Related references SDK Reference Download the SDK archives and API specifications, with versions and checksums. Make an API call Mint a key with the tare CLI and make a first routed request across languages. Management API reference The HTTP endpoints the SDK calls, including the identity and inference APIs. --- ## SDK quickstart for Python This quickstart installs the typed Agent Router SDK for Python, then makes a first read-only identity lookup (whoami) against the gateway. The lookup resolves the identity behind an API key with a GET /v1/me call and is safe to run as-is, so it confirms that the SDK, the key, and the base URL are wired together correctly before any billable request is sent. ## Prerequisites Two values and a runtime are required before starting: - **An Agent Router API key.** Keys are minted in the Console or with the `tare` CLI. The secret is shown once, in the form `.`, so it must be copied immediately. See [Make an API call](/agent-router-service/quickstarts/make-an-api-call/) for the key-minting steps. - **The gateway base URL** for the deployment, in the form `https://api.`. For the hosted service this is `https://api.tetrate.ai`. - **A currently supported Python 3 runtime**, with `pip` available. The SDK reads both values from the environment: ```bash export AGENTROUTER_BASE_URL=https://api. export AGENTROUTER_API_KEY=. ``` ## Install the SDK The Python SDK is distributed as a source distribution (sdist) from the public stable channel and is installed from the local file rather than from PyPI. Replace `` with the version shown on the [SDKs](/reference/sdk/) page. ```bash curl -fLO https://tare.tetrate.ai/tools/sdks/stable/agentrouter-python-.tar.gz pip install ./agentrouter-python-.tar.gz ``` Installing into a virtual environment is recommended, so the SDK and its dependencies are isolated from other projects: ```bash python -m venv .venv && source .venv/bin/activate pip install ./agentrouter-python-.tar.gz ``` ## Make the first call Save the program below as `main.py`. It constructs a client from the two environment variables, calls `client.me.get()` (the read-only `GET /v1/me` endpoint), and prints the resolved identity. ```python title="main.py" """Runnable example for the Agent Router Python SDK. Set AGENTROUTER_BASE_URL and AGENTROUTER_API_KEY in the environment, then run `python main.py`. """ import os from agentrouter_sdk import Client client = Client( base_url=os.environ["AGENTROUTER_BASE_URL"], api_key=os.environ["AGENTROUTER_API_KEY"], ) try: result = client.me.get() print(result) except Exception as err: print("Error:", err) ``` Run the program from the directory containing `main.py`: ```bash python main.py ``` A successful run prints the identity associated with the API key. Because `GET /v1/me` only reads identity metadata, the call incurs no model usage and can be repeated freely while verifying setup. Related references SDK Reference Download the SDK archives and API specifications, with versions and checksums. Make an API call Mint a key with the tare CLI and make a first routed request across languages. Management API reference The HTTP endpoints the SDK calls, including the identity and inference APIs. --- ## SDK quickstart for TypeScript This quickstart installs the typed Agent Router SDK for TypeScript, then makes a first read-only identity lookup (whoami) against the gateway. The lookup resolves the identity behind an API key with a GET /v1/me call and is safe to run as-is, so it confirms that the SDK, the key, and the base URL are wired together correctly before any billable request is sent. ## Prerequisites Two values and a runtime are required before starting: - **An Agent Router API key.** Keys are minted in the Console or with the `tare` CLI. The secret is shown once, in the form `.`, so it must be copied immediately. See [Make an API call](/agent-router-service/quickstarts/make-an-api-call/) for the key-minting steps. - **The gateway base URL** for the deployment, in the form `https://api.`. For the hosted service this is `https://api.tetrate.ai`. - **Node.js 18 or later**, which provides the `npm` and `npx` commands used below. The SDK reads both values from the environment: ```bash export AGENTROUTER_BASE_URL=https://api. export AGENTROUTER_API_KEY=. ``` ## Install the SDK The TypeScript SDK is distributed as an npm pack (`.tgz`) from the public stable channel and is recorded as a local `file:` dependency rather than installed from the public npm registry. Replace `` with the version shown on the [SDKs](/reference/sdk/) page. Download the pack into the project directory: ```bash curl -fLO https://tare.tetrate.ai/tools/sdks/stable/agentrouter-typescript-.tgz ``` Create `package.json` and `tsconfig.json` alongside it. The SDK is referenced as a `file:` dependency, and `tsx` is included to run the example without a separate build step: ```json title="package.json" { "name": "agentrouter-quickstart", "version": "0.1.0", "private": true, "type": "module", "dependencies": { "@tetrate/agentrouter-sdk": "file:agentrouter-typescript-.tgz" }, "devDependencies": { "@types/node": "^20.0.0", "tsx": "^4.0.0", "typescript": "^5.4.0" } } ``` ```json title="tsconfig.json" { "compilerOptions": { "target": "ES2020", "module": "ESNext", "moduleResolution": "bundler", "strict": true, "esModuleInterop": true, "skipLibCheck": true } } ``` Install the dependencies: ```bash npm install ``` ## Make the first call Save the program below as `index.ts`. It constructs a client from the two environment variables, calls `client.me.get()` (the read-only `GET /v1/me` endpoint), and prints the resolved identity. ```typescript title="index.ts" // Runnable example for the Agent Router TypeScript SDK. // Set AGENTROUTER_BASE_URL and AGENTROUTER_API_KEY in the environment, then run `npx tsx index.ts`. import { Client } from '@tetrate/agentrouter-sdk' const client = new Client({ baseUrl: process.env.AGENTROUTER_BASE_URL, apiKey: process.env.AGENTROUTER_API_KEY, }) try { const result = await client.me.get() console.log(result) } catch (err) { console.error('Error:', err) } ``` Run the program from the project directory: ```bash npx tsx index.ts ``` A successful run prints the identity associated with the API key. Because `GET /v1/me` only reads identity metadata, the call incurs no model usage and can be repeated freely while verifying setup. Related references SDK Reference Download the SDK archives and API specifications, with versions and checksums. Make an API call Mint a key with the tare CLI and make a first routed request across languages. Management API reference The HTTP endpoints the SDK calls, including the identity and inference APIs. --- ## Sizing and scale This page is a capacity reference for planning an Tetrate Agent Router deployment against a concrete scale target. It derives an approximate workload from a stated set of non-functional requirements, lists the dimensions that drive sizing, and describes the method used to translate those dimensions into a data-plane footprint. The figures are planning approximations intended to frame a load test; they are not capacity guarantees. Exact pod counts, replica sizes, and provider quotas should be validated against load testing of Agent Router on the target infrastructure before any number is committed. The architecture referenced throughout is described in Architecture Overview; the customer-managed data plane contains the Controller and the request proxy, while the Tetrate-hosted management plane stores routing rules, policies, and user configuration. ## Reference workload The reference target for this deployment is: | Dimension | Target | | --- | --- | | Users | ~60,000 | | Models | ~25 | | LLM calls per month | ~1.9 million | A monthly call volume converts to an average request rate as follows. A 30-day month contains roughly 2.6 million seconds (`30 × 24 × 3600`). Dividing 1.9 million calls by that figure gives an average of approximately 0.7 requests per second across the whole month. | Derived quantity | Approximate value | | --- | --- | | Calls per month | 1,900,000 | | Seconds per 30-day month | 2,592,000 | | Average requests per second | ~0.7 | An average of under one request per second is a modest steady-state load. Provisioning to the average would be a mistake. Real traffic from 60,000 users is not uniform across the month; it concentrates into working hours, time zones, and bursts driven by application behaviour. Peak concurrency is the figure that sizes the deployment, and it can sit one to two orders of magnitude above the monthly average. The peaking factor, the ratio of peak rate to average rate, is the single most important unknown in this exercise. It depends on usage patterns that only the field and SME team can confirm for this engagement. A common planning approach is to assume traffic concentrates into a fraction of the day and to size against that window rather than the 30-day average: | Concentration assumption | Effective window | Approximate peak requests per second | | --- | --- | --- | | Traffic spread evenly across 30 days | 2,592,000 s | ~0.7 | | Traffic within an 8-hour working day, 22 working days | 633,600 s | ~3 | | Traffic within a 2-hour daily peak, 22 working days | 158,400 s | ~12 | | Bursty interactive load (short spikes) | Not applicable | well above the figures above | These rows illustrate sensitivity to assumptions, not a prediction. The correct peaking factor for this engagement must be supplied by the field/SME team and confirmed against observed traffic before sizing is finalised. --- ## Sizing dimensions Request rate alone does not size a deployment. The dimensions below jointly determine the data-plane footprint and the provider-side quota required. | Dimension | Why it matters | | --- | --- | | Concurrent requests | The number of in-flight requests at peak, not the monthly count, drives memory and connection-pool sizing. Long-running streaming requests hold resources for their full duration | | Tokens per request | Larger prompts and completions increase per-request CPU, memory, and provider latency. Token distribution, not just request count, determines provider throughput and cost | | Streaming vs. non-streaming | Streaming responses hold a connection open for the full generation. A workload that is predominantly streaming sustains far more concurrent connections than the same request rate served as discrete responses | | Number of models and providers | The ~25 models map to a set of upstream providers, each with its own rate limits, latency profile, and quota. Routing and fallback configuration grows with this count | | Regions | A multi-region deployment multiplies the data-plane footprint and introduces cross-region latency. Each region is sized for its own share of peak concurrency | --- ## Data-plane sizing The data plane scales horizontally. Both the Controller and the routing proxy add capacity by adding replicas rather than by enlarging a single instance. - The Agent Router (the proxy component) handles the request data path. Its capacity is governed by peak concurrent connections and per-request token volume. Replicas are added behind a load balancer to absorb peak concurrency, with headroom held in reserve above the expected peak. - The Controller manages configuration and the control path. It scales with the number of distinct routing rules, API keys, models, and providers rather than with raw request rate. The sizing method is the same regardless of the target: 1. Establish the peak concurrent-request figure from the average rate and the agreed peaking factor. 1. Characterise the workload mix (token distribution and the streaming/non-streaming split) because these set per-connection resource cost. 1. Measure the throughput of a single Agent Router replica on the target infrastructure under a representative load profile. 1. Divide peak concurrency by per-replica throughput to obtain a replica count, then add headroom for failover, rolling upgrades, and traffic above the modelled peak. Per-replica throughput is a function of the instance type, the workload mix, and Agent Router version. For that reason this page does not state a per-pod request rate. The figure must be measured against the deployment, not assumed. --- ## Throughput and latency The gateway adds minimal overhead to each request. End-to-end latency is dominated by the upstream provider's model-inference time, which is typically orders of magnitude larger than the gateway's internal processing. Sizing for latency is therefore mostly a question of provider behaviour and provider-side quota, not of gateway capacity. | Latency component | Relative contribution | Notes | | --- | --- | --- | | Gateway processing | Small | Routing resolution, policy evaluation, and normalisation. Not separately measured: the gateway's request-duration metric covers its whole involvement, including the upstream wait | | Provider inference | Dominant | Time the upstream provider spends generating the response. Scales with output token count and provider load | | Network | Variable | Cross-region hops add to this component in a multi-region deployment | The latency instruments available for this analysis (`gen_ai.server.request.duration`, `gen_ai.server.time_to_first_token`, and `gen_ai.server.time_per_output_token`) are documented in [OpenTelemetry traces and metrics](/reference/otel-metrics/). None of them isolates the gateway's own processing from the upstream wait, so the components in the table above are a model for reasoning about latency rather than a set of separately measurable quantities. Because provider latency dominates, the practical latency ceiling at a given concurrency is set by how many in-flight requests the chosen providers can serve without queuing, which ties back to provider-side rate limits. The gateway's normalisation, fallback, and error semantics under load are described in [Gateway Behavior](/reference/gateway-behavior/). Fallback chains add latency on the failed leg of a walk and should be accounted for when modelling worst-case latency at peak. --- ## Headroom for rate limits and budgets At 60,000 users Agent Router's own rate-limit and budget controls are a sizing concern in their own right, separate from data-plane capacity. - Per-key and per-group rate limits should be set with headroom above expected peak so that legitimate peak traffic is not throttled, while still bounding runaway clients. Limit configuration is covered in [Set Rate Limits](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/stop-runaway-workloads/). - Upstream provider quotas are a hard ceiling that Agent Router cannot exceed. The sum of configured per-key limits across all keys should be reconciled against the aggregate provider quota for each provider, with headroom retained for fallback traffic redirected from a degraded provider. - Budget controls track spend rather than throughput, and they alert rather than reject traffic, so at this user count budgets and rate limits should be modelled together: the rate limits bound throughput inline while the budgets flag cumulative spend for someone to act on. --- ## Observability at scale Telemetry volume and metric cardinality grow with users, models, and API keys. The metric and trace schema is documented in [OTel Metrics](/reference/otel-metrics/); the export configuration is covered in [Export Telemetry to an Observability Stack](/agent-router-enterprise/guides/observability-and-analytics/export-telemetry-to-an-observability-stack/). Two cardinality drivers matter at this scale: - Metric cardinality is driven by the model and tenant dimensions. The three model attributes (`gen_ai.original.model`, `gen_ai.request.model`, and `gen_ai.response.model`) multiply against each other, and `gen_ai.response.model` is the largest contributor because providers report dated model builds that change without notice. The tenant attributes (`tars.customer`, `tars.workspace`, `router.project.id`, `router.gateway.id`) then multiply against those. No metric carries an API key dimension, so key population does not drive metric cardinality; per-key reporting comes from Request Logs and the in-Console Usage Analytics surface instead. - Trace volume scales directly with request rate. At peak, exporting every trace can overwhelm the destination backend and inflate cost. Because the data plane sets no sampler, unparented requests are always sampled, while requests carrying client trace context inherit the caller's sampling decision. Reducing trace volume therefore depends on the calling applications' sampling configuration or on tail sampling in the collector, rather than on a gateway-side rate. Metrics are not sampled and remain the complete record. Sampling behaviour is detailed in [OpenTelemetry traces and metrics](/reference/otel-metrics/). The observability backend should be sized for peak time-series count and peak trace ingest, using the same peaking factor applied to the data plane. --- ## Capacity questions to confirm The figures on this page rest on assumptions that the field and SME team should confirm before any number is committed: 1. What is the peaking factor (the ratio of peak request rate to the monthly average) for this engagement, and over what window does traffic concentrate? 1. What fraction of requests is streaming, and what is the expected token distribution per request? 1. How many upstream providers serve the ~25 models, and what is the rate limit and quota for each? 1. Is the deployment single-region or multi-region, and how is the 60,000-user population distributed across regions? 1. What infrastructure (instance types, node sizing) will host the data plane, so that per-replica throughput can be measured rather than assumed? 1. What is the metrics backend's tolerance for high-cardinality labels, and what trace sampling rate is acceptable at peak? 1. What headroom margin above modelled peak is required for failover and rolling upgrades? Each answer narrows a planning approximation toward a number that can be validated by a load test against the target environment. --- ## Related - [Architecture Overview](/product-architecture/architecture-overview/): data plane and management plane components - [Gateway Behavior](/reference/gateway-behavior/): error semantics and per-request data under load - [OTel Metrics](/reference/otel-metrics/): metric families, labels, and sampling - [Set Rate Limits](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/stop-runaway-workloads/): per-key and per-group limit configuration - [Export Telemetry to an Observability Stack](/agent-router-enterprise/guides/observability-and-analytics/export-telemetry-to-an-observability-stack/): telemetry export configuration Where to go next Plan high availability and disaster recovery Build resilience and failover into the sized deployment. Data plane installation Deploy the data plane on the target infrastructure. --- ## Gateway APIs Agent Router supports three inference API formats: the OpenAI Chat Completions API, the OpenAI Responses API, and the Anthropic Messages API, plus an OpenAI-compatible Models endpoint for discovering which models are routable for your API key. All gateway features (routing, fallback policies, traffic splitting, cost tracking, and observability) apply equally regardless of the format chosen. Applications send requests in one format, and the gateway handles provider translation transparently, normalising responses and errors back to the format that was requested. For new projects with no existing SDK preference, Chat Completions offers the widest ecosystem compatibility. :::tip Management APIs live elsewhere These are the gateway inference endpoints that applications and SDKs hit. The OpenAPI catalog does not enumerate `/v1/chat/completions` and the other paths on this page. For provisioning keys, clients, models, and other control-plane operations, use the [Management API reference](/reference/api/). ::: ## Endpoint summary | Format | Path | SDK Method | Streaming | | --- | --- | --- | --- | | OpenAI Chat Completions | `/v1/chat/completions` | `client.chat.completions.create()` | `stream=True` | | OpenAI Responses | `/v1/responses` | `client.responses.create()` | `stream=True` | | Anthropic Messages | `/v1/messages` | `client.messages.create()` | `stream=True` | | Models | `/v1/models` | `client.models.list()` | n/a | In the examples below, replace `PROXY_URL` with the proxy endpoint shown on the Console Dashboard (for example, `https://proxy.poc.tetrate.ai/v1`) and `YOUR_API_KEY` with a key from [API Keys](/agent-router-service/guides/route-requests-across-providers/). --- ## Chat Completions API (`/v1/chat/completions`) The most widely supported format, compatible with OpenAI and most third-party SDKs. ### Non-streaming ```bash curl PROXY_URL/chat/completions \ -H "Authorization: Bearer YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "gpt-4o", "messages": [{"role": "user", "content": "Hello, world!"}] }' ``` ```python from openai import OpenAI client = OpenAI( base_url="PROXY_URL", api_key="YOUR_API_KEY", ) response = client.chat.completions.create( model="gpt-4o", messages=[{"role": "user", "content": "Hello, world!"}], ) print(response.choices[0].message.content) ``` ### Streaming ```bash curl PROXY_URL/chat/completions \ -H "Authorization: Bearer YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "gpt-4o", "messages": [{"role": "user", "content": "Hello, world!"}], "stream": true }' ``` ```python from openai import OpenAI client = OpenAI( base_url="PROXY_URL", api_key="YOUR_API_KEY", ) stream = client.chat.completions.create( model="gpt-4o", messages=[{"role": "user", "content": "Hello, world!"}], stream=True, ) for chunk in stream: content = chunk.choices[0].delta.content if content: print(content, end="", flush=True) ``` ### SSE format Chat Completions streaming uses data-only SSE. Each event is a `data:` line containing a JSON object, terminated by `data: [DONE]`: ```text data: {"id":"chatcmpl-...","object":"chat.completion.chunk","choices":[{"delta":{"content":"Hello"}}]} data: {"id":"chatcmpl-...","object":"chat.completion.chunk","choices":[{"delta":{"content":" world"}}]} data: [DONE] ``` To receive token usage in the stream, add `"stream_options": {"include_usage": true}` to the request. Usage appears in the final chunk before `[DONE]`: ```json {"prompt_tokens": 10, "completion_tokens": 5, "total_tokens": 15} ``` --- ## Responses API (`/v1/responses`) The newer OpenAI Responses API provides a simplified interface with semantic streaming events. ### Non-streaming ```bash curl PROXY_URL/responses \ -H "Authorization: Bearer YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "gpt-4o", "input": "Hello, world!" }' ``` ```python from openai import OpenAI client = OpenAI( base_url="PROXY_URL", api_key="YOUR_API_KEY", ) response = client.responses.create( model="gpt-4o", input="Hello, world!", ) print(response.output_text) ``` ### Streaming ```bash curl PROXY_URL/responses \ -H "Authorization: Bearer YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "gpt-4o", "input": "Hello, world!", "stream": true }' ``` ```python from openai import OpenAI client = OpenAI( base_url="PROXY_URL", api_key="YOUR_API_KEY", ) stream = client.responses.create( model="gpt-4o", input="Hello, world!", stream=True, ) for event in stream: if event.type == "response.output_text.delta": print(event.delta, end="", flush=True) ``` ### SSE format Responses API streaming uses semantic `event:` plus `data:` lines. Each event has a named type describing what happened: ```text event: response.created data: {"id":"resp_...","object":"response","status":"in_progress"} event: response.output_item.added data: {"item":{"id":"msg_...","type":"message","role":"assistant"}} event: response.output_text.delta data: {"delta":"Hello"} event: response.output_text.delta data: {"delta":" world"} event: response.output_text.done data: {"text":"Hello world"} event: response.completed data: {"id":"resp_...","status":"completed","usage":{"input_tokens":10,"output_tokens":5}} ``` ### Differences from Chat Completions | Aspect | Chat Completions | Responses API | | --- | --- | --- | | Input field | `messages` array | `input` (string or array) | | Usage fields | `prompt_tokens` / `completion_tokens` | `input_tokens` / `output_tokens` | | SSE format | Data-only (`data: {...}`) with `data: [DONE]` sentinel | Semantic events (`event: response.created`, etc.) | | Stream usage | Opt-in via `stream_options.include_usage` | Always in `response.completed` event | | Response access | `response.choices[0].message.content` | `response.output_text` | --- ## Anthropic Messages API (`/v1/messages`) For applications built with the Anthropic SDK. The gateway accepts standard `Authorization: Bearer` headers; the Anthropic-native `x-api-key` header is not required. ### Non-streaming ```bash curl PROXY_URL/messages \ -H "Authorization: Bearer YOUR_API_KEY" \ -H "Content-Type: application/json" \ -H "anthropic-version: 2023-06-01" \ -d '{ "model": "claude-sonnet-4-20250514", "max_tokens": 1024, "messages": [{"role": "user", "content": "Hello, world!"}] }' ``` ```python from anthropic import Anthropic client = Anthropic( base_url="PROXY_URL", auth_token="YOUR_API_KEY", ) response = client.messages.create( model="claude-sonnet-4-20250514", max_tokens=1024, messages=[{"role": "user", "content": "Hello, world!"}], ) print(response.content[0].text) ``` ### Streaming ```bash curl PROXY_URL/messages \ -H "Authorization: Bearer YOUR_API_KEY" \ -H "Content-Type: application/json" \ -H "anthropic-version: 2023-06-01" \ -d '{ "model": "claude-sonnet-4-20250514", "max_tokens": 1024, "messages": [{"role": "user", "content": "Hello, world!"}], "stream": true }' ``` ```python from anthropic import Anthropic client = Anthropic( base_url="PROXY_URL", auth_token="YOUR_API_KEY", ) with client.messages.stream( model="claude-sonnet-4-20250514", max_tokens=1024, messages=[{"role": "user", "content": "Hello, world!"}], ) as stream: for text in stream.text_stream: print(text, end="", flush=True) ``` ### SSE format Anthropic Messages streaming uses semantic `event:` plus `data:` lines with block-level granularity: ```text event: message_start data: {"type":"message_start","message":{"id":"msg_...","role":"assistant","model":"claude-sonnet-4-20250514"}} event: content_block_start data: {"type":"content_block_start","index":0,"content_block":{"type":"text","text":""}} event: content_block_delta data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"Hello"}} event: content_block_delta data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":" world"}} event: content_block_stop data: {"type":"content_block_stop","index":0} event: message_delta data: {"type":"message_delta","delta":{"stop_reason":"end_turn"},"usage":{"output_tokens":5}} event: message_stop data: {"type":"message_stop"} ``` :::note When using the gateway, authenticate with `Authorization: Bearer YOUR_API_KEY` instead of the Anthropic-native `x-api-key` header. The gateway translates the auth header before forwarding to the provider. ::: --- ## Models API (`/v1/models`) List the models available to your API key. The endpoint is OpenAI-compatible and requires authentication. Use each returned `id` as the `model` value in Chat Completions, Responses, Messages, and other inference calls. The response follows the OpenAI list shape (`object: "list"` with a `data` array) and extends each model object with pricing and capability fields used for routing and cost awareness. ### Request ```bash curl PROXY_URL/models \ -H "Authorization: Bearer YOUR_API_KEY" ``` ```python from openai import OpenAI client = OpenAI( base_url="PROXY_URL", api_key="YOUR_API_KEY", ) models = client.models.list() for model in models.data: print(model.id) ``` ### Response The example below is truncated. A live response returns every model routable for the key. ```json { "object": "list", "data": [ { "id": "claude-sonnet-4-6", "object": "model", "created": 1771393580, "owned_by": "system", "input_price": "0.000003", "caching_price": "0.00000375", "cached_price": "0.0000003", "output_price": "0.000015", "max_output_tokens": 64000, "context_window": 1000000, "supports_caching": true, "supports_vision": true, "supports_computer_use": true, "supports_reasoning": true }, { "id": "gpt-4o-mini", "object": "model", "created": 1773126109, "owned_by": "system", "input_price": "0.00000015", "caching_price": "0", "cached_price": "0.000000075", "output_price": "0.0000006", "max_output_tokens": 16384, "context_window": 128000, "supports_caching": false, "supports_vision": true, "supports_computer_use": true, "supports_reasoning": false } ] } ``` :::note `GET /v1/models` returns the models routable for the calling key. For the unauthenticated public catalog with richer metadata, see `https://router.tetrate.ai/api/public/models`. For management-plane catalog CRUD, use `GET /v1/catalog/models` (documented in the [API reference](/reference/api/catalog/agentrouter-catalog-v-1-catalog-service-list-models/)). ::: --- ## Supported endpoint types The gateway handles nine endpoint types. All use the same model-based routing logic; fallback policies and traffic splitting apply equally across every endpoint. | Endpoint | Path | Description | | --- | --- | --- | | Chat Completions | `/v1/chat/completions` | Standard chat interface (OpenAI-compatible) | | Completions | `/v1/completions` | Legacy text completions | | Responses | `/v1/responses` | OpenAI Responses API | | Messages | `/v1/messages` | Anthropic Messages API | | Embeddings | `/v1/embeddings` | Text embeddings | | Images | `/v1/images` | Image generation | | Audio speech | `/v1/audio/speech` | Text-to-speech (OpenAI-compatible) | | Rerank | `/v1/rerank` | Reranking | | Models | `/v1/models` | List available models | :::note Speech-to-text is not supported `/v1/audio/transcriptions` returns "Unsupported endpoint". Speech-to-text (Whisper-style) models cannot be called through the gateway. ::: --- ## Provider translation The gateway automatically translates between the canonical (OpenAI-compatible) schema and 25+ provider-specific APIs. Applications send requests in one format, and the gateway handles all conversions transparently. **Translated elements:** - **Request body**: field names, structure, and defaults adjusted per provider - **Path**: endpoint paths mapped to provider conventions - **Headers**: authentication and provider-specific headers set automatically - **Response format**: provider responses normalised back to the format that was requested For example, an OpenAI Chat Completions request that routes to Anthropic Claude is translated to the Anthropic Messages format before being forwarded, and the response is translated back to Chat Completions format. The application never sees the difference. No configuration is needed; translation is built into the gateway. For how errors are normalised across providers, see [Gateway Behavior](/reference/gateway-behavior/). --- ## Protocols The gateway supports REST (HTTPS) for all inference traffic. This is the only protocol needed to use any endpoint. gRPC is used internally for OpenTelemetry (OTLP) telemetry export. See [OpenTelemetry Export](/agent-router-enterprise/guides/observability-and-analytics/export-telemetry-to-an-observability-stack/) for the configuration surface. WebSocket and gRPC for inference are not currently supported. --- ## Choosing an API format | Use case | Recommended format | | --- | --- | | Widest SDK and tool compatibility | Chat Completions | | New OpenAI projects with the simplified interface | Responses API | | Anthropic Claude-native applications | Anthropic Messages | | Agent frameworks (LangChain, CrewAI, and similar) | Chat Completions | | Code assistants (Cursor, Cline, Aider) | Chat Completions | | Streaming with semantic events | Responses API or Anthropic Messages | All three formats support the same gateway features. The choice is driven by SDK preference and provider ecosystem alignment. ## Related - [Integrate the Gateway with an App](/agent-router-service/guides/integrate-the-gateway-with-an-app/): developer-side integration patterns for each SDK - [Gateway Behavior](/reference/gateway-behavior/): request handling, error semantics, and routing resolution rules Where to go next Management API reference Control-plane OpenAPI catalog: clients, keys, catalog, and related services. Gateway behavior How requests are handled, normalised, and routed. --- ## Supported Providers Agent Router routes to 200+ models across the major commercial providers, the hyperscaler AI platforms, and self-hosted or open-model backends through a single OpenAI-compatible endpoint. This page lists the provider families that can be connected, how each one is authenticated, and where the authoritative, always-current list of routable models is published. Providers are configured once in the Admin Dashboard by a platform operator; developers then route to any enabled model without handling provider credentials directly. ## How providers are connected A provider is an upstream AI service the gateway is permitted to reach. Each provider carries its own credentials and connectivity, and each exposes a set of models that a platform operator enables for developers. The connection style differs by provider: | Connection style | What is supplied | Example providers | | --- | --- | --- | | API key | A single secret token | OpenAI, Anthropic, Mistral, Cohere, Together, Perplexity, Groq, xAI | | Cloud credentials | A regional endpoint plus cloud IAM or a deployment key | Azure OpenAI, AWS Bedrock, Google Vertex AI | | Native API key | A provider-issued key for the provider's own API surface | Google Gemini | | Custom endpoint | A base URL for a self-hosted or compatible service | Self-hosted and OpenAI-compatible backends | Provisioning, credential rotation, and model enablement are covered in [Provision models and providers](/agent-router-enterprise/guides/operate-and-govern/provision-models-and-providers/). AWS Bedrock has region-specific requirements documented separately in [Provision AWS Bedrock models](/agent-router-enterprise/guides/operate-and-govern/provision-models-and-providers/provisioning-aws-bedrock-models/). Provider families OpenAI GPT model family. Connected with an API key. Anthropic Claude Opus, Sonnet, and Haiku families. Connected with an API key. Google Gemini Gemini models through the native Google AI API. Google Vertex AI Gemini and partner models through a Google Cloud project. Azure OpenAI Azure-hosted OpenAI deployments with enterprise Azure credentials. AWS Bedrock Bedrock-hosted models with region-specific endpoints and IAM. Mistral Mistral model family. Connected with an API key. Cohere Command models and Cohere embeddings. Connected with an API key. Together Open-source models hosted on Together AI. Connected with an API key. Perplexity Search-augmented models. Connected with an API key. Groq Low-latency inference for Llama, Qwen, and open models. Self-hosted OpenAI-compatible and custom backends reached at a configured base URL. The provider set grows over time, and the dropdown in **Providers Management** in the Admin Dashboard is the authoritative list of provider types available in a given release. ## The authoritative model list Individual model identifiers, context windows, and pricing tiers change frequently as providers ship new versions. Rather than duplicate that list here, three live sources hold the current state: - **Console Model Catalog**: the models a platform operator has enabled for a deployment, visible to developers in the Console. - **Model Catalog - APIs**: the same enabled catalog, accessed programmatically with authenticated `GET /v1/models` for the calling API key. See [Models API](/reference/supported-apis/#models-api-v1models). - **Public models endpoint**: the machine-readable catalog of routable models, published at `https://router.tetrate.ai/api/public/models`. Only models that a platform operator has explicitly enabled are routable. A model offered by a connected provider is not reachable until it has been enabled in **Models Management**. ## Provider translation Applications send requests in one of the [supported API formats](/reference/supported-apis/), and the gateway translates transparently to each provider's native API. An OpenAI Chat Completions request that routes to Anthropic Claude is converted to the Anthropic Messages format before it is forwarded, and the response is converted back. No per-provider client code is required; the base URL and API key are the only application-side changes. For provider-side concerns that fall outside the gateway (model pricing, rate-limit tiers, and account management), each provider's own documentation is the canonical source. See [External links](/reference/external-links/) for the current documentation URLs. ## Related - [Provision models and providers](/agent-router-enterprise/guides/operate-and-govern/provision-models-and-providers/): the operator workflow that connects a provider and enables its models - [Gateway APIs](/reference/supported-apis/): the request formats that route to every provider - [Gateway behavior](/reference/gateway-behavior/): how requests are resolved, normalised, and routed across providers Where to go next Route requests across providers Create a key, attach a routing configuration, and send the first request. External links Documentation URLs for every supported provider. --- ## `azure-gateway.json` schema (`tare gateway install --type azure`) The JSON config file accepted by tare gateway install --type azure, consumed through the --config flag. This document describes the JSON config file accepted by: ```bash tare gateway install identity.json --type azure --config azure-gateway.json ``` The Azure path covers AKS clusters on traditional Azure CNI. The install enables the AGIC ingress-appgw addon (which provisions an Application Gateway in the cluster's `MC___` resource group) and applies an Ingress in `tars-dataplane` with AGIC's required health-probe annotations. See [`tare gateway install`](./install-gateway.mdx) for the end-to-end flow. > **Tip:** Use [`tare gateway config init --type azure`](./gateway-config-init.mdx) > to bootstrap a starter file from prompts or flags. The remaining > fields documented below are pre-filled with resolver defaults that you > can edit in place. ## Supported keys All keys are optional at the JSON level. Required-ness is enforced at install time by the resolver and the Azure pre-flight. | Key | Type | Used for | Required when | |---|---|---|---| | `customer` | string | Identifies the customer in the management plane | unless provided by identity / `--customer` | | `environment` | string | Optional environment label | required when customer is `saas` | | `serveDomain` | string | DNS host the customer points at the AGIC frontend IP | optional but recommended (drives the DNS hint output) | | `serveUrl` | string | Public URL the data plane reports to the MP | optional (falls back to identity fields) | | `azure.subscriptionId` | string | Azure subscription scoping every `az` call | required for `--apply-prereqs` and the AGIC compat pre-flight | | `azure.resourceGroup` | string | Resource group holding the AKS cluster | required for `--apply-prereqs` and the AGIC compat pre-flight | | `azure.aksClusterName` | string | AKS cluster name | required for `--apply-prereqs` and the AGIC compat pre-flight | | `azure.appGateway.name` | string | Application Gateway name AGIC creates | optional (default: `-appgw`) | | `azure.appGateway.subnetCidr` | string | Address range for the App Gateway subnet | optional (default: `10.225.0.0/24`) | | `azure.skipExisting` | bool | Continue when `az aks enable-addons` reports "already enabled" | optional (`false` default) | | `namespaces.dataplane` | string | Dataplane namespace the Ingress lives in | optional (default `tars-dataplane`) | | `lint` | object | Per-config rule overrides (shared with the GCP schema) | optional | ## Example ```json { "customer": "acme", "serveDomain": "proxy.acme.tetrate.ai", "serveUrl": "https://proxy.acme.tetrate.ai", "azure": { "subscriptionId": "00000000-0000-0000-0000-000000000000", "resourceGroup": "acme-aks-rg", "aksClusterName": "acme-aks", "appGateway": { "name": "acme-aks-appgw", "subnetCidr": "10.225.0.0/24" }, "skipExisting": true } } ``` ## Field precedence For most values, precedence is: 1. explicit CLI flag (`--azure-subscription-id`, `--azure-resource-group`, `--aks-cluster-name`, `--appgw-name`, `--appgw-subnet-cidr`) 2. `azure-gateway.json` (`--config`) 3. resolver defaults (`-appgw`, `10.225.0.0/24`) ## Why no `staticIpName` / `certificateMap` / `dnsAuthorization`? These are GCP-specific concepts. On Azure: - **Static IP**: AGIC auto-provisions the Application Gateway's frontend IP. No name to pin; customer DNS is updated to the dynamically-assigned address surfaced in the Ingress status. - **Certificate map / managed certificates**: The runbook AGIC path is HTTP-only on port 80 by default. HTTPS termination via AGIC listener TLS is configured separately and is not in scope of this config. - **DNS authorization**: GCP cert-manager mechanic; no equivalent. ## Minimal config (with `--apply-prereqs`) ```json { "customer": "acme", "serveDomain": "proxy.acme.tetrate.ai", "azure": { "subscriptionId": "00000000-0000-0000-0000-000000000000", "resourceGroup": "acme-aks-rg", "aksClusterName": "acme-aks" } } ``` Then run: ```bash tare gateway install identity.json \ --type azure \ --config azure-gateway.json \ --apply-prereqs --wait ``` ## Cluster compatibility `tare gateway install --type azure` requires AGIC-compatible AKS networking: ```bash az aks show -n -g \ --query 'networkProfile.{plugin: networkPlugin, mode: networkPluginMode, dataplane: networkDataplane}' ``` The compatible combination is `{plugin: "azure", mode: null, dataplane: "azure"}`. Anything else (mode `overlay`, dataplane `cilium`) is rejected by the pre-flight. AGC (Application Gateway for Containers) is the path for those clusters; `tare` support for it is planned. ## Required Azure roles The signed-in identity needs: - `Azure Kubernetes Service Contributor Role` on the AKS resource group: for `az aks enable-addons` - `Network Contributor` on the cluster's `MC___` node-RG: for AGIC to manage the Application Gateway `tare gateway install --type azure` checks these and prints the exact `az role assignment create` commands when a role is missing. When the signed-in identity cannot read its own role assignments (Forbidden), the pre-flight degrades gracefully: prints what's required, proceeds, and lets `az aks enable-addons` surface AuthorizationFailed if applicable. Where to go next tare gateway install Consume this config to enable AGIC and deploy the Azure gateway. gcp-gateway.json schema The equivalent config file for the GCP gateway flow. --- ## `tare gateway config init` Generate a starter gcp-gateway.json from a serve domain and a GCP project ID. The remaining fields are derived from the domain or written as documented defaults so the operator can edit the file directly without consulting the schema reference. ## Synopsis ```bash tare gateway config init [flags] ``` When stdin is a terminal and either `--serve-domain` or `--project-id` is omitted, the command runs a two-question wizard prompting for the missing values. With both flags supplied the command is fully non-interactive and safe to run in CI or `make` targets. ## Description `tare gateway config init` is the quickest way to bootstrap the config file consumed by `tare gateway install` and validated by `tare gateway config lint`. It writes a single file (no live cloud or cluster reach-out) and prints a per-field provenance table to stderr so the operator can see which values came from input, which were derived, and which are defaults that may need overriding. Generated values for a tetrate.ai SaaS-style serve domain pass `tare gateway config lint` with zero error findings at default severity. For domains that do not follow the SaaS convention (no `proxy..tetrate.ai` shape), the customer derivation may need overriding via `--customer` or a direct edit. `--type gcp` (default) and `--type azure` are both supported; `aws` returns an explicit not-implemented error. There is nothing to generate for AWS: [`tare gateway install --type aws`](./install-gateway.mdx) takes no config file and is configured entirely by flags. ## Flags | Flag | Default | Description | |---|---|---| | `--type` | `gcp` | Gateway provider type (`gcp\|azure`). | | `--serve-domain` | _(prompted)_ | Serve domain, e.g. `proxy.acme.tetrate.ai`. Wizard prompts when omitted on a TTY. | | `--project-id` | _(prompted)_ | (`--type gcp`) GCP project ID. Wizard prompts when omitted on a TTY. | | `--customer` | _(derived)_ | Customer slug. On GCP, defaults to a heuristic derived from `--serve-domain` (see [Derivation rules](#derivation-rules)). On Azure, prompted by the wizard (no domain-derived default). | | `--azure-subscription-id` | _(prompted)_ | (`--type azure`) Azure subscription UUID. Wizard prompts when omitted on a TTY. | | `--azure-resource-group` | _(prompted)_ | (`--type azure`) Resource group holding the AKS cluster. Wizard prompts when omitted on a TTY. | | `--aks-cluster-name` | _(prompted)_ | (`--type azure`) AKS cluster name. Wizard prompts when omitted on a TTY. | | `--output` | `gcp-gateway.json` or `azure-gateway.json` | Output path. Default depends on `--type`. Matches the default `--config` of `lint` / `install`. | | `--force` | `false` | Overwrite the output file if it already exists. Without this the command refuses to clobber existing files. | ## Derivation rules For a serve domain like `proxy.pocnt.tetrate.ai`: | Output field | Source | Example | |---|---|---| | `projectId` | `--project-id` | `withfraser` | | `serveDomain` | `--serve-domain` | `proxy.pocnt.tetrate.ai` | | `customer` | derived (or `--customer`) | `pocnt` | | `certificateMap.name` | slug of domain with `proxy.` stripped | `pocnt-tetrate-ai` | | `certificate.name` | `-serve-cert` | `pocnt-serve-cert` | | `dnsAuthorization.name` | slug of full domain + `-dns-auth` | `proxy-pocnt-tetrate-ai-dns-auth` | | `gateway.staticIpName` | slug of domain with `proxy.` stripped + `-ip` | `pocnt-tetrate-ai-ip` | | `namespaces.gateway` | default | `tars-gateway` | | `namespaces.system` | default | `tars-system` | | `namespaces.dataplane` | default | `tars-dataplane` | Customer derivation: - `proxy..tetrate.ai` → `` - `.tetrate.ai` → `` - Any other domain → first DNS label (before the first `.`) - Empty / unparsable → empty (lint rule TAREL001 will flag it) `serveUrl` is intentionally **not** emitted; the resolver falls back to `serveDomain` automatically, and emitting both with the same value triggers the informational TAREL030 finding. ## Examples ```bash # Wizard: prompts for serve domain and project ID tare gateway config init # Fully non-interactive (CI / Make targets) tare gateway config init \ --serve-domain proxy.acme.tetrate.ai \ --project-id acme-prod # Custom output path; overwrite an existing file tare gateway config init \ --serve-domain proxy.acme.tetrate.ai \ --project-id acme-prod \ --output ./envs/acme/gcp-gateway.json \ --force # Override the derived customer slug tare gateway config init \ --serve-domain proxy.acme.tetrate.ai \ --project-id acme-prod \ --customer acme-internal # Azure: wizard (prompts for customer, serve domain, subscription ID, RG, cluster) tare gateway config init --type azure # Azure: fully non-interactive tare gateway config init --type azure \ --customer acme \ --serve-domain proxy.acme.tetrate.test \ --azure-subscription-id 00000000-0000-0000-0000-000000000000 \ --azure-resource-group acme-aks-rg \ --aks-cluster-name acme-aks ``` ## Output The file written to `--output` is strict JSON (no comments) so it can be consumed unchanged by `lint` and `install`. The stderr stream carries a human-readable summary, e.g. after running with `--serve-domain proxy.pocnt.tetrate.ai --project-id withfraser`: ``` Wrote /abs/path/to/gcp-gateway.json Generated values: projectId [input] withfraser serveDomain [input] proxy.pocnt.tetrate.ai customer [derived] pocnt certificateMap.name [derived] pocnt-tetrate-ai certificate.name [derived] pocnt-serve-cert dnsAuthorization.name [derived] proxy-pocnt-tetrate-ai-dns-auth gateway.staticIpName [derived] pocnt-tetrate-ai-ip namespaces.gateway [default] tars-gateway namespaces.system [default] tars-system namespaces.dataplane [default] tars-dataplane Next: - Review gcp-gateway.json and edit any field as needed. - Validate: tare gateway config lint --config gcp-gateway.json - Install: tare gateway install --type gcp --config gcp-gateway.json ``` ## Exit codes - **0**: File written successfully. - **non-zero**: Provider not supported, output file already exists without `--force`, missing required values when stdin is not a terminal, or I/O failure. ## See also - [`gcp-gateway.json` schema](./gcp-gateway-config.mdx): full field reference for hand-editing the generated file. - [`tare gateway config lint`](./gateway-config-lint.mdx): validate the generated file before install. - [`tare gateway install`](./install-gateway.mdx): consume the file to provision GCP prereqs and deploy the gateway chart. Where to go next tare gateway config lint Validate the generated config file before install. gcp-gateway.json schema Full field reference for hand-editing the generated file. --- ## `tare gateway config lint` Lint a gateway config file (gcp-gateway.json or azure-gateway.json) against the v1 rule set. ## Synopsis ```bash tare gateway config lint [flags] ``` ## Description `tare gateway config lint` applies the registered lint rules to a gateway config file. The engine is hermetic: it never contacts GCP, Azure, or Kubernetes, so it is safe to run in pre-merge CI on a config repo. Rules cover correctness (required values, internally-consistent references), safety (latent operational risks like ephemeral gateway IPs), convention (env-mismatched resource names, identity/customer divergence), and hygiene (deprecated fields). Provider-specific rules self-gate on `--type` so an Azure config does not trip GCP-only rules (e.g. TAREL010 only fires when the resolved provider is `gcp`). ## Flags | Flag | Default | Description | |---|---|---| | `--type` | `gcp` | Gateway provider type (`gcp\|azure`). Controls which provider-gated rules fire. | | `--config` | `gcp-gateway.json` | Path to the config file (`gcp-gateway.json` or `azure-gateway.json`). | | `--identity` | _(unset)_ | Optional identity file. Enables identity-aware rules (e.g. TAREL021). | | `--format` | `text` | Output format: `text` or `json`. | | `--severity` | `info` | Minimum severity to display: `info`, `warn`, or `error`. | | `--fail-on-warn` | `false` | Exit non-zero on warnings (CI gating). | | `--list-rules` | `false` | Print the registered rules and exit 0. | ## Exit codes - **0**: No findings, or only info/warn findings (use `--fail-on-warn` to promote warnings to non-zero exits in CI). - **1**: At least one error finding. - **2**: Misuse (bad flags, missing config file, etc.). ## Examples ```bash # Default: text output, show all findings (GCP) tare gateway config lint --config gcp-gateway.json # Azure config tare gateway config lint --type azure --config azure-gateway.json # CI gating — fail the pipeline on warnings or errors tare gateway config lint --config gcp-gateway.json --fail-on-warn # JSON output for ingestion by other tooling tare gateway config lint --config gcp-gateway.json --format json # Identity-aware mode (enables TAREL021 customer-vs-identity check) tare gateway config lint \ --config gcp-gateway.json \ --identity identity.json # Inspect the rule registry tare gateway config lint --list-rules ``` ## v1 rule set Six rules ship in v1. The seventh (`TAREL011`, wildcard intent) is deferred to v1.1 pending a specification of the downstream wildcard contract. | ID | Severity | Category | What it catches | |---|---|---|---| | **TAREL001** | Error | Correctness | Required value cannot be resolved after merging flags + config + identity + defaults. | | **TAREL002** | Error | Correctness | `certificateMap.name` set but `certificate.name` empty: cert-map-entry step will fail at GCP. | | **TAREL010** | Warn | Safety | `gateway.staticIpName` empty: gateway gets an ephemeral address; customer DNS will eventually break. | | **TAREL020** | Warn | Convention | Resource names embed an env-like substring (`stag`, `dev`, `test`) that disagrees with `environment`. | | **TAREL021** | Warn | Convention | Config `customer` differs from `identity.customerId`: usually a wrong-identity-for-this-config install. | | **TAREL030** | Info | Hygiene | Redundant/deprecated fields (e.g. `serveDomain == serveUrl`; legacy `proxy` in identity). | Each rule has a docs anchor at `https://docs.tetrate.ai/tare/lint/` and is overridable per config via the `lint` block in `gcp-gateway.json`: ```json { "lint": { "ignore": ["TAREL020"], "rules": { "TAREL021": { "expectedPattern": "^pocnt-(stag|prod)-" } } } } ``` ## Output formats ### Text (default) ``` ⚠ [TAREL010] gateway.staticIpName — gateway will receive an ephemeral GCP forwarding-rule address; customer DNS A records may break on reschedule. Remediate: set gateway.staticIpName in gcp-gateway.json or pass --static-ip-name on install. Summary: 0 error, 1 warn, 0 info ``` ### JSON The JSON schema is a **public contract** versioned via the `apiVersion: tare.tetrate.io/v1alpha1` header. CI integrations and external tooling can depend on this shape; breaking changes will bump the version. ```json { "apiVersion": "tare.tetrate.io/v1alpha1", "kind": "LintReport", "config": "gcp-gateway.json", "summary": { "error": 0, "warn": 1, "info": 0 }, "findings": [ { "ruleId": "TAREL010", "severity": "warn", "field": "gateway.staticIpName", "message": "...", "remediate": "...", "docsUrl": "https://docs.tetrate.ai/tare/lint/TAREL010" } ] } ``` Where to go next tare gateway config init Generate a starter config file that passes these rules. gcp-gateway.json schema Full field reference for the config file being linted. --- ## `gcp-gateway.json` schema (`tare gateway install`) The JSON config file accepted by tare gateway install --type gcp, consumed through the --config flag. This document describes the JSON config file accepted by: ```bash tare gateway install identity.json --type gcp --config gcp-gateway.json ``` The parser is implemented in `svc/cli/internal/tare/gateway/config/types.go` (`Config`). > **Tip:** Use [`tare gateway config init`](./gateway-config-init.mdx) to > bootstrap a starter file from just a serve domain and a GCP project > ID: the remaining fields documented below are pre-filled with > sensible derivations and defaults that you can edit in place. > **Azure:** for the `--type azure` flow, see > [`azure-gateway.json` schema](./azure-gateway-config.mdx). ## Supported keys All keys are optional in JSON, but some become required by the selected workflow. | Key | Type | Used for | Required when | |---|---|---|---| | `projectId` | string | GCP project for prerequisite automation | `--dry-run-prereqs` or `--apply-prereqs` | | `serveDomain` | string | DNS hostname/domain for DNS authorization + cert map entry | creating cert/dns prereqs from CLI | | `customer` | string | Helm value `customer` | unless provided by identity / `--customer` | | `environment` | string | Helm value `environment` | required when customer is `saas` | | `serveUrl` | string | Helm value `serveUrl` | optional (falls back to identity fields) | | `securityPolicy` | string | Helm value `securityPolicy` | optional | | `certificateMap.name` | string | GCP cert map name + Helm `certificateMap.name` | prereq workflow and recommended for deploy | | `gateway.staticIpName` | string | GCP global static IP name + Helm `gateway.staticIpName` | optional but recommended | | `certificate.name` | string | Managed cert resource name (gcloud automation) | when automating cert creation | | `dnsAuthorization.name` | string | DNS authorization resource name (gcloud automation) | optional (auto-derived if omitted) | | `namespaces.gateway` | string | Gateway release namespace | optional (default `tars-gateway`) | | `namespaces.system` | string | System namespace reference | optional (default `tars-system`) | | `namespaces.dataplane` | string | Dataplane namespace reference | optional (default `tars-dataplane`) | | `gcloud.skipExisting` | bool | Continue when gcloud create hits "already exists" | optional (`false` default) | ## Example `examples/gcp-gateway.pocnt.json`: ```json { "projectId": "withfraser", "serveDomain": "proxy.pocnt.tetrate.ai", "customer": "pocnt", "environment": "production", "serveUrl": "proxy.pocnt.tetrate.ai", "certificateMap": { "name": "pocnt-tetrate-ai" }, "certificate": { "name": "pocnt-serve-cert" }, "dnsAuthorization": { "name": "proxy-pocnt-tetrate-ai-dns-auth" }, "namespaces": { "gateway": "tars-gateway", "system": "tars-system", "dataplane": "tars-dataplane" }, "gcloud": { "skipExisting": false } } ``` ## Field precedence For most values, precedence is: 1. explicit CLI flag 2. `gcp-gateway.json` (`--config`) 3. identity file defaults / built-in defaults Notable defaults: - `--serve-url`: falls back to `serveUrl` (config), then `serveDomain` (config), then identity `serveUrl`, then identity `proxy` (legacy). - `--dns-authorization-name`: if omitted and cert automation is enabled, default becomes `-dns-auth`. ## Minimal configs by use case ### A) Install only (prereqs handled outside CLI) Usually enough: ```json { "customer": "acme", "serveUrl": "proxy.acme.example.com", "certificateMap": { "name": "acme-cert-map" } } ``` Then run: ```bash tare gateway install identity.json \ --type gcp \ --config gcp-gateway.json \ --ack-prereqs ``` ### B) Use `--apply-prereqs` / `--dry-run-prereqs` Provide at least: - `projectId` - `certificateMap.name` And for certificate + DNS automation: - `certificate.name` - `serveDomain` `gateway.staticIpName` is optional (CLI only creates static IP when set). Where to go next tare gateway install Consume this config to provision GCP prereqs and deploy the gateway. azure-gateway.json schema The equivalent config file for the Azure gateway flow. --- ## Agent Router CLI Reference The tare command-line tool deploys and manages the Agent Router data plane, collapsing image sync, pull-secret creation, and Helm deployment into a single invocation. This section documents the command surface and the configuration file schemas. In this section CLI overview The full command list, quick start, identity file formats, and config schemas. tare install Install the data plane on a target cluster. tare gateway install Install serve gateway resources. tare upgrade Upgrade an existing data plane release. --- ## `tare gateway install` Install serve gateway resources. The provider is selected with --type (gcp, azure, or aws). ## Synopsis ```bash tare gateway install [flags] ``` ## Description `tare gateway install` deploys the gateway chart used to expose serve traffic. High-level workflow: 1. Load identity file 2. Load optional `gcp-gateway.json` config (`--config`) 3. Resolve values from flags/config/identity (with provenance: every resolved value carries the source it came from) 4. **Plan**: render a diff between the resolved config and live GCP/k8s state, with inline lint findings and a customer-action block; in a TTY, prompt `Proceed? [y/N]`. CI callers pass `--yes`. 5. Optional prereq workflow (`--dry-run-prereqs` or `--apply-prereqs`) 6. Run preflight checks 7. Install chart with Helm 8. Optional wait for Gateway address (`--wait`) and print DNS A-record action The Plan view is the same JSON contract (`apiVersion: tare.tetrate.io/v1alpha1`) used by `tare gateway config lint`; use `--plan-only --output json` to consume it from other tooling. ### Plan / Confirm / Apply flags | Flag | Description | |---|---| | `--plan-only` | Render the Plan and exit without applying changes. | | `--yes` | Skip the interactive `Proceed?` prompt. Required in non-TTY contexts (CI). | | `--accept-warnings` | Proceed even when lint warnings are present. Errors still block. | | `--output text\|json` | Plan output format. Default: `text`. | Provider support: - `gcp`: supported (GCP Cloud Load Balancer + certificate-manager + Helm chart) - `azure`: supported (AKS + AGIC addon + Ingress): traditional Azure CNI only - `aws`: supported (EKS + in-tree Network Load Balancer; no AWS Load Balancer Controller, no `aws` CLI) ## Prerequisites Before install, you must either: - run `--apply-prereqs`, or - confirm prerequisites are already done with `--ack-prereqs` ### GCP prerequisites Expected prerequisites: - wildcard DNS prepared for your serve hostname - TLS certificate + certificate map prepared - static IP / cloud policy resources prepared If `--apply-prereqs` is used, `gcloud` must be installed and authenticated. ### Azure prerequisites Expected prerequisites: - AKS cluster exists with AGIC-compatible networking (`networkPlugin: azure`, `networkPluginMode: null`, `networkDataplane: azure`) - `az` CLI installed and `az login` completed - signed-in identity has the required Azure roles (see below) - `kubectl` context points at the target AKS cluster If `--apply-prereqs` is used, the install runs `az aks enable-addons --addons ingress-appgw` (which provisions an Application Gateway, ~5min) before applying the Ingress. Pre-flights that run before any side effects: - **AGIC networking compatibility**: `az aks show ... --query networkProfile` must report `{plugin: azure, mode: null, dataplane: azure}`. If the cluster is Cilium/Overlay, fails fast with a pointer to AGC follow-up. - **Azure RBAC**: checks `Azure Kubernetes Service Contributor Role` on the AKS RG and `Network Contributor` on the `MC___` node-RG. Missing roles fail with the exact `az role assignment create` command to fix. If `Microsoft.Authorization/roleAssignments/read` is itself denied, the pre-flight degrades to a warning and proceeds (the install will surface AuthorizationFailed itself if a role is genuinely missing). - **Kubernetes RBAC**: `kubectl auth can-i create ingress -n tars-dataplane`. Fails with a hint pointing at the `edit` ClusterRole if denied. ### AWS prerequisites The AWS path is ingress-only and the lightest of the three. It exposes the data plane's egress as a Network Load Balancer by setting `envoyService.type=LoadBalancer` plus the NLB annotation on the egress **EnvoyProxy** (`tars-egress-proxy` in `tars-system`). The `egress` Service is Envoy-Gateway-managed, so driving the load balancer through the EnvoyProxy is durable: Envoy Gateway re-asserts the NLB across pod restarts, Envoy Gateway restarts, and `tare upgrade`. A direct Service patch reverts to `ClusterIP` on the next reconcile. The in-tree EKS cloud provider provisions the NLB from the annotation, so no AWS Load Balancer Controller and no `aws` CLI are required, and there is no prereq workflow (`--apply-prereqs` and `--ack-prereqs` do not apply). Expected prerequisites: - EKS cluster with the data plane already installed (`tare install`), so the egress EnvoyProxy exists in `tars-system` - `kubectl` context points at the target EKS cluster Pre-flights that run after the Plan and confirmation, before any mutation: - **kubectl / cluster reachable** - **egress EnvoyProxy exists** in the system namespace. Fails with an "is the data plane installed?" hint; override the name with `--aws-envoyproxy` if it differs. - **Kubernetes RBAC**: `kubectl auth can-i patch envoyproxies.gateway.envoyproxy.io -n tars-system`. Fails with a hint pointing at the needed role if denied. Identity for credential-less upstream auth (Bedrock, for example) is a separate step; see [Bedrock workload identity](/agent-router-enterprise/guides/operate-and-govern/provision-models-and-providers/bedrock-workload-identity/). This command only exposes the gateway. ## Config file Use `--config` to provide gateway settings: ```bash tare gateway install identity.json --type gcp --config gcp-gateway.json tare gateway install identity.json --type azure --config azure-gateway.json ``` Full schemas: - [`gcp-gateway.json` schema](./gcp-gateway-config.mdx) - [`azure-gateway.json` schema](./azure-gateway-config.mdx) `--type aws` takes **no config file**. It is configured entirely by flags (`--aws-internal`, `--aws-service`, `--aws-envoyproxy`). ## Usage ### Render the Plan and exit ```bash tare gateway install identity.json \ --type gcp \ --config gcp-gateway.json \ --plan-only ``` Outputs the resolved values (with provenance), the GCP/k8s deltas against live state, inline lint findings, and the customer-action block. No side effects. ### Print generated Helm values ```bash tare gateway install identity.json \ --type gcp \ --config gcp-gateway.json \ --print-helm-values ``` ### Preview prerequisite `gcloud` commands ```bash tare gateway install identity.json \ --type gcp \ --config gcp-gateway.json \ --dry-run-prereqs ``` ### Apply prerequisites and install ```bash tare gateway install identity.json \ --type gcp \ --config gcp-gateway.json \ --apply-prereqs \ --wait ``` ### Install when prereqs are already managed externally ```bash tare gateway install identity.json \ --type gcp \ --config gcp-gateway.json \ --ack-prereqs \ --wait ``` ### Print rendered manifests without applying ```bash tare gateway install identity.json \ --type gcp \ --config gcp-gateway.json \ --print-resources ``` On the Azure path, `--print-resources` emits the `tars-ingress` YAML (with the AGIC health-probe annotations) instead of helm-templated chart output. On the AWS path, it emits the EnvoyProxy merge patch (the `spec.provider.kubernetes.envoyService` block: `type: LoadBalancer` plus the NLB annotation). ### Azure: preview the `az` plan without applying ```bash tare gateway install identity.json \ --type azure \ --config azure-gateway.json \ --dry-run-prereqs ``` ### Azure: apply prerequisites and install ```bash tare gateway install identity.json \ --type azure \ --config azure-gateway.json \ --apply-prereqs --wait ``` Runs the AGIC compat pre-flight, the Azure RBAC pre-flight (printing remediation commands when needed), and the K8s `can-i create ingress` pre-flight; then enables the AGIC addon, applies the Ingress with health-probe annotations, waits for the Ingress to report an address, and prints the DNS A-record customer action. ### AWS: expose the egress as an NLB ```bash tare gateway install identity.json \ --type aws \ --wait ``` Renders the Plan (the EnvoyProxy patch), confirms (or `--yes` in CI), runs the kubectl, cluster, and EnvoyProxy pre-flights, patches the egress EnvoyProxy with `envoyService.type=LoadBalancer` plus `service.beta.kubernetes.io/aws-load-balancer-type=nlb`, waits for Envoy Gateway to reconcile the `egress` Service and the in-tree EKS cloud provider to publish the NLB hostname, then prints the DNS **CNAME** customer action and a smoke-test `curl` (expect HTTP 401). ### AWS: provision an internal NLB ```bash tare gateway install identity.json \ --type aws \ --aws-internal \ --wait ``` Adds `service.beta.kubernetes.io/aws-load-balancer-internal=true` so the NLB is internal-only. ## Flags ### Main | Flag | Default | Description | |---|---|---| | `--type` | `gcp` | Gateway provider type (`gcp\|azure\|aws`) | | `--config ` | none | Path to gateway config (`gcp-gateway.json` or `azure-gateway.json`; not used by `aws`) | | `--project-id ` | from config | (GCP) project for prereq automation | | `--serve-domain ` | from config/serve-url | Hostname/domain used by prereq automation | | `--certificate-name ` | from config | (GCP) certificate resource name for managed cert workflow | | `--dns-authorization-name ` | from config/derived | (GCP) DNS authorization resource name | | `--serve-url ` | config → identity | Gateway host override | | `--certificate-map-name ` | from config | (GCP) certificate map name | | `--static-ip-name ` | from config | (GCP) optional static IP resource name | | `--security-policy ` | from config | (GCP) optional Cloud Armor security policy name | | `--customer ` | identity/config | Customer label | | `--environment ` | from config | Environment label (required when customer is `saas`) | | `--ack-prereqs` | `false` | Confirm prereqs are already handled | | `--dry-run-prereqs` | `false` | Print prereq cloud-CLI plan (`gcloud` or `az`) | | `--apply-prereqs` | `false` | Run prereq cloud-CLI plan | | `--wait` | `false` | Wait until Gateway has an address (Azure: Ingress address; AWS: NLB hostname) | ### Azure (only consulted under `--type azure`) | Flag | Default | Description | |---|---|---| | `--azure-subscription-id ` | from config | Azure subscription scoping `az` calls | | `--azure-resource-group ` | from config | Resource group holding the AKS cluster | | `--aks-cluster-name ` | from config | AKS cluster name | | `--appgw-name ` | `-appgw` | Application Gateway name AGIC creates | | `--appgw-subnet-cidr ` | `10.225.0.0/24` | App Gateway subnet CIDR | ### AWS (only consulted under `--type aws`) | Flag | Default | Description | |---|---|---| | `--aws-internal` | `false` | Provision an internal NLB (adds the `aws-load-balancer-internal` annotation) | | `--aws-service ` | `egress` | Egress Service Envoy Gateway exposes as an NLB | | `--aws-envoyproxy ` | `tars-egress-proxy` | Egress EnvoyProxy resource to drive the NLB through | ### Inspect | Flag | Default | Description | |---|---|---| | `--print-helm-values` | `false` | Print generated Helm values to stdout | | `--print-resources` | `false` | Print rendered Kubernetes manifests to stdout | ### Advanced / hidden | Flag | Default | Description | |---|---|---| | `--timeout` | `10m` | Timeout for `--wait` | | `--release-name` | `tars-gateway` | Helm release name | | `--namespace` | `tars-gateway` | Helm release namespace | | `--system-namespace` | `tars-system` | Namespace referenced by gateway routes | | `--dataplane-namespace` | `tars-dataplane` | Dataplane namespace referenced by gateway routes | | `--chart-path` | embedded | Override chart source path/OCI/HTTP | | `--chart-version` | none | Chart version (needed for remote charts) | | `--helm-values` | none | Extra Helm values file to merge | | `--skip-preflight` | `false` | Skip preflight checks | ## Output behavior - Progress and operator guidance are written to `stderr` - `--print-helm-values` and `--print-resources` write generated artifacts to `stdout` When `--wait` succeeds, CLI prints: - resolved Gateway address - customer DNS A-record action (`Host`, `Type A`, `Value`, `TTL`) - certificate provisioning reminder On the AWS path the resolved address is the NLB hostname, and the customer action is a **CNAME** record rather than an A record, followed by the smoke-test `curl`. ## Verification ```bash kubectl get gateway -n tars-gateway kubectl get httproute -n tars-system kubectl get httproute -n tars-dataplane kubectl get gtwpc -n tars-gateway ``` If using managed certificates, also verify certificate state: ```bash gcloud certificate-manager certificates describe \ --project \ --format="yaml(name,managed.state,managed.domainStatus)" ``` On AWS, verify the egress Service has an NLB hostname and smoke-test it: ```bash kubectl get service egress -n tars-dataplane \ -o jsonpath='{.status.loadBalancer.ingress[0].hostname}' curl -sk -o /dev/null -w 'HTTP %{http_code}\n' http://:10080/v1/chat/completions ``` `401` means the ingress is wired through to the `tars-proxy` filter (it is asking for an API key). `200` means the filter is not loaded; `502` means the load balancer health check is failing. ## Troubleshooting - `missing projectId`: set `--project-id` or `projectId` in config when using prereq workflow - `missing certificate map name`: set `--certificate-map-name` or `certificateMap.name` in config - wait timeout: check Gateway events and cloud LB provisioning status - certificate still `PROVISIONING`: ensure DNS authorization record and final A record are correct - (AWS) `EnvoyProxy "tars-egress-proxy" not found`: run `tare install` first so the egress EnvoyProxy exists in `tars-system`, or pass `--aws-envoyproxy ` if it is named differently - (AWS) the egress Service stays `ClusterIP` after patching: confirm `kubectl get envoyproxy tars-egress-proxy -n tars-system -o jsonpath='{.spec.provider.kubernetes.envoyService.type}'` reads `LoadBalancer`, then check the `envoy-gateway` controller logs; Envoy Gateway must reconcile the EnvoyProxy into the Service - (AWS) NLB hostname never appears: run `kubectl describe service egress -n tars-dataplane` for events and check subnet discovery tags for the in-tree cloud provider Where to go next gcp-gateway.json schema Full field reference for the GCP gateway config file. azure-gateway.json schema Full field reference for the Azure gateway config file. --- ## `tare install` Install the complete Agent Router dataplane in a single command. ## Synopsis ``` tare install [flags] ``` ## Description `tare install` performs the full operator workflow: 1. Load and validate the identity (service-account) file 2. Optionally sync pinned images **and the `serve-helm` OCI chart** to a private registry (`--image-sync`) 3. Optionally create a Kubernetes `dockerconfigjson` pull secret (`--image-pull-secret-stdin`) 4. Generate Helm values 5. `helm upgrade --install` the Agent Router dataplane chart 6. Optionally wait for pods to be ready (`--wait`) `helm` and `kubectl` are downloaded automatically on first run into `~/.tare/tools/` and reused on subsequent runs. By default (`--ha=true`) the data-plane Envoy proxy + sidecar'd ExtProc deploy with HA-safe values: HPA `minReplicas: 2`, PDB `minAvailable: 1`, and `EnvoyProxy.spec.shutdown.drainTimeout: 300s` for graceful drain of in-flight LLM streams. If the target cluster has no `metrics.k8s.io` APIService (metrics-server absent), `tare install` prints a non-fatal warning: the install-time floor of 2 still applies but the HPA cannot autoscale beyond it. Pass `--ha=false` for single-replica installs (lab / CI). `--serve-url` is the customer-facing gateway URL for the data plane being installed. It is **optional**: if omitted, the helm chart leaves `tars-config[proxy-url]` unrendered and the CLI does not register a URL with the management plane. Admins can register the URL later from the Admin Dashboard under **System → Settings → Data planes**, which fires a `PROXY_URL` configuration event that updates the running data plane's `tars-config`. When supplied, `tare install` calls the management plane's `ConfiguratorService.SetDataplaneURL` before running `helm upgrade --install` so the URL is persisted in `system_settings.dataplane-url:` for the data plane's workspace (looked up from the identity-file SA's `workspaceId`). The chart then seeds `tars-config[proxy-url]` from `global.serveUrl` at install. No `TriggerUpdate` is fired: the value will be re-pushed via the standard event flow if an admin later edits it. ## Usage ### Basic install ```bash tare install identity.json --serve-url https://proxy.acme.com ``` `--serve-url` sets the data plane's customer-facing gateway URL. It flows into `global.serveUrl` in helm values; the controller-helm chart seeds `tars-config[proxy-url]` from that value at install. (The separate `controller.config.proxyUrl` value no longer exists: the controller and worker pods read the URL from `tars-config[proxy-url]` at runtime, not from a `PROXY_URL` env var.) Deploys using the embedded Helm chart with defaults: | Setting | Default | |---|---| | Release name | `tars` | | System namespace | `tars-system` | | Dataplane namespace | `tars-dataplane` | | Customer | inferred from identity file | ### Print Helm values Print the generated Helm values to stdout without touching the cluster: ```bash tare install identity.json --serve-url https://proxy.acme.com --print-helm-values ``` Useful for inspecting values or piping into GitOps tooling. `--dry-run` is accepted as a hidden alias. ### Image sync Sync all pinned images **and the `serve-helm` OCI chart** from the source registry to a private registry before installing: ```bash tare install identity.json --serve-url https://proxy.acme.com \ --image-sync myregistry.example.com ``` - The SA JWT token is used to authenticate to the source registry. - After the sync, `--image-registry` (the Helm `global.imageRegistry`) points at the sync destination so the cluster pulls from where the images now live. - Source registry priority: `--image-registry` (when `--image-sync` is also set) → `REGISTRY_URL` env → `registry.tetrate.ai` (default). - The `serve-helm` chart is mirrored as an OCI artifact alongside the images (e.g. `myregistry.example.com/serve-helm:0.1.0-rc.4_`). The tag encodes a SemVer build identifier: the `+` build metadata is normalized to `_` because OCI tags forbid `+`. The counter lives in the prerelease segment (`rc.4`, `alpha.347`) so versions are orderable; `tare versions list --channel rc` finds the newest release candidate. `tare install` itself still uses the chart embedded in the CLI binary; the mirrored chart exists so downstream `helm upgrade oci://myregistry.example.com/serve-helm` workflows succeed without reaching back to `registry.tetrate.ai`. - For release-tagged `tare` builds the chart also gets an **extra tag** equal to the running CLI version (e.g. `myregistry.example.com/serve-helm:v1.2.3`) so a downstream `helm install --version ` resolves to the chart that shipped with that CLI. Dev / dirty builds skip the extra tag to keep the mirror free of ambiguous tags. Sync only (no deploy), `--serve-url` is not required: ```bash tare install identity.json \ --image-sync myregistry.example.com --sync-only ``` Mirror-to-mirror sync: pull from an existing private registry and push to a new one in a single invocation. When paired with `--image-sync`, `--image-registry` is interpreted as the **source** registry to pull from; without `--image-sync` it keeps its original meaning of "set the Helm `global.imageRegistry`". ```bash tare install identity.json \ --image-registry acme.registry.example.com \ --image-sync acme1.registry.example.com \ --sync-only ``` Print the image list without copying, `--serve-url` is not required: ```bash # Just the source references tare install identity.json --print-images # Source → destination pairs (requires --image-sync) tare install identity.json \ --image-sync myregistry.example.com --print-images ``` ### Apply CRDs only Apply the chart's CRDs to the cluster and exit before namespaces, pull secret, and Helm install. `--serve-url` is not required. Useful for three operational patterns: - **Cluster-admin handoff**: a privileged operator pre-applies CRDs (which are cluster-scoped), then a less-privileged operator runs the rest of `tare install` against a namespace they own. - **Race-free upgrades**: apply CRDs first so subsequent `helm upgrade` runs render against the new schema with no chicken-and-egg ordering. - **Raw `helm install` workflows**: operators who drive Helm directly (GitOps, Argo CD, custom CI) hand the CRD lifecycle to `tare` and let Helm own only the workload resources. ```bash tare install identity.json --crds-only ``` After the CRDs land, re-run without `--crds-only` (and with `--serve-url`) to complete the install. `--crds-only` cannot be combined with `--sync-only`, `--image-sync`, `--image-pull-secret-stdin`, or any `--print-*` flag. #### Followed by a raw `helm install` If you intend to skip `tare install` for the workload step and run Helm directly, you must tell Helm not to render or adopt the CRDs that `tare` already applied via server-side apply. Two flags are needed: ```bash # Step 1: apply CRDs with tare (kubectl server-side apply; no Helm ownership labels) tare install identity.json --crds-only # Step 2: install the workload with helm, skipping CRDs in both places they live helm install tars oci:///serve-helm \ --version "" \ -f values.yaml -n tars-system \ --skip-crds \ --set ai-gateway-crds.enabled=false ``` - `--skip-crds` tells Helm to skip CRDs in the chart's `crds/` directory. - `--set ai-gateway-crds.enabled=false` disables the `ai-gateway-crds` subchart, which renders CRDs as templates (not as `crds/` files) and would otherwise fail Helm's ownership-adoption check against the kubectl-applied CRDs. Without both flags, Helm errors with: ``` CustomResourceDefinition "aigatewayroutes.aigateway.envoyproxy.io" ... exists and cannot be imported into the current release: invalid ownership metadata; label validation error: missing key "app.kubernetes.io/managed-by" ``` This split (`tare` owns CRDs, Helm owns the workload) is the recommended pattern for GitOps and raw-Helm consumers. CRDs survive `helm uninstall tars` because Helm does not manage them, so customer data (Gateways, AIGatewayRoutes, MCPRoutes, …) is preserved. ### Pull secret Create a Kubernetes `dockerconfigjson` secret from credentials read on stdin: ```bash echo "user:password" | tare install identity.json \ --serve-url https://proxy.acme.com \ --image-sync myregistry.example.com \ --image-pull-secret-stdin ``` - The secret is created (or updated) in both `tars-system` and `tars-dataplane` namespaces. - Re-runs are idempotent (`kubectl apply` under the hood). - Default secret name: `tars-image-pull-secret`. Override with `--image-pull-secret-name`. To reference a pre-existing secret without creating one: ```bash tare install identity.json --serve-url https://proxy.acme.com \ --image-pull-secret-name my-existing-secret ``` ### Full workflow ```bash echo "user:password" | tare install identity.json \ --serve-url https://proxy.acme.com \ --image-sync myregistry.example.com \ --image-pull-secret-stdin \ --wait ``` ### With guardrails Include `llm-guard-api` and `envoy-dm` images and enable guardrails configuration: ```bash echo "user:password" | tare install identity.json \ --serve-url https://proxy.acme.com \ --image-sync myregistry.example.com \ --image-pull-secret-stdin \ --enable-guardrails \ --wait ``` The guardrails container (`llm-guard-api`) requires a node with sufficient memory and CPU. Use `--guardrails-config` to pin it to the right node pool: ```bash # guardrails.yaml # nodeSelector: # node-type: gpu-large # # tolerations: # - key: large-node # operator: Exists # effect: NoSchedule echo "user:password" | tare install identity.json \ --serve-url https://proxy.acme.com \ --image-sync myregistry.example.com \ --image-pull-secret-stdin \ --enable-guardrails \ --guardrails-config guardrails.yaml \ --wait ``` The config file uses natural Kubernetes YAML, no escaped JSON strings: ```yaml # guardrails.yaml nodeSelector: node-type: gpu-large tolerations: - key: large-node operator: Exists effect: NoSchedule # Optional: full affinity rules affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: cloud.google.com/gke-nodepool operator: In values: [llm-guard-pool] ``` Pass `--guardrails-config -` to read the config from stdin instead of a file (follows the `kubectl`/`helm` convention): ```bash cat <<'EOF' | tare install identity.json \ --serve-url https://proxy.acme.com \ --enable-guardrails \ --guardrails-config - \ --wait nodeSelector: node-type: gpu-large tolerations: - key: large-node operator: Exists effect: NoSchedule EOF ``` > **Note:** `--guardrails-config -` and `--image-pull-secret-stdin` both read > stdin; use a file when combining both. ### Print Kubernetes resources (GitOps) Print namespace and pull-secret manifests without touching the cluster. `--serve-url` is not required: ```bash echo "user:password" | tare install identity.json \ --image-sync myregistry.example.com \ --image-pull-secret-stdin \ --print-resources ``` Pipe directly into `kubectl apply` or commit to a GitOps repo. ## Flags ### Main | Flag | Default | Description | |---|---|---| | `--serve-url ` | optional | Data plane gateway URL; sets `global.serveUrl` (which seeds `tars-config[proxy-url]`) and registers the URL with the management plane. Omit to defer URL registration to the Admin Dashboard under **System → Settings → Data planes**. | | `--ha` | `true` | HA-safe defaults for the data-plane Envoy proxy: HPA `minReplicas: 2`, PDB `minAvailable: 1`. Pass `--ha=false` for single-replica lab/CI installs. | | `--drain-timeout-seconds` | `300` | `EnvoyProxy.spec.shutdown.drainTimeout` (seconds). Maximum time Envoy waits for in-flight LLM streams to finish before SIGKILL. The Envoy Gateway controller propagates this into the data-plane Pod's `terminationGracePeriodSeconds`. | | `--enable-guardrails` | `false` | Include guardrails images and configuration | | `--guardrails-config ` | none | YAML file with Guardrails pod scheduling constraints (nodeSelector, affinity, tolerations), or `-` to read from stdin; requires `--enable-guardrails` | | `--wait` | `false` | Wait for pods ready after deploy | ### Registry | Flag | Default | Description | |---|---|---| | `--image-sync ` | none | Sync images and the `serve-helm` OCI chart to `` before install; sets `--image-registry` to `` after the sync (unless `--image-registry` was used to override the source) | | `--image-pull-secret-stdin` | `false` | Read `user:password` from stdin and create a pull secret | | `--image-pull-secret-name ` | `tars-image-pull-secret` | Name of the pull secret to create or reference | | `--sync-only` | `false` | Sync images then exit without deploying (requires `--image-sync`) | ### Inspect | Flag | Default | Description | |---|---|---| | `--print-helm-values` | `false` | Print generated Helm values to stdout without deploying | | `--print-resources` | `false` | Print Kubernetes manifests (namespaces, pull secret) to stdout without applying | | `--print-images` | `false` | Print image list and exit (add `--image-sync` for src→dst pairs) | | `--crds-only` | `false` | Apply CRDs to the cluster and exit; skip namespaces, pull secret, Helm install, and wait. Incompatible with `--sync-only`, `--image-sync`, `--image-pull-secret-stdin`, and any `--print-*` flag. | ### Advanced / hidden These flags are hidden from `--help` but accepted for advanced use. | Flag | Default | Description | |---|---|---| | `--dry-run` | `false` | Alias for `--print-helm-values` | | `--release-name` | `tars` | Helm release name | | `--system-namespace` | `tars-system` | Kubernetes system namespace | | `--dataplane-namespace` | `tars-dataplane` | Kubernetes dataplane namespace | | `--chart-path` | embedded | Path or OCI/HTTP reference to Helm chart | | `--chart-version` | none | Chart version (required for remote/OCI charts) | | `--image-registry` | none | Container image registry. Without `--image-sync`: sets Helm `global.imageRegistry`. With `--image-sync`: source registry to pull from (push destination is `--image-sync`). Falls back to `REGISTRY_URL` env, then `registry.tetrate.ai` | | `--image-tag` | none | Image tag override | | `--helm-values` | none | Additional Helm values file to merge | | `--deployment-mode` | `enterprise` | `enterprise` or `saas` | | `--environment` | none | Environment label | | `--customer` | from identity | Customer label override | | `--redis-type` | `cluster` | `cluster` or `hosted` | | `--rate-limiting` | `true` | Enable rate limiting | | `--timeout` | `5m` | Timeout for `--wait` | | `--guardrails-node-selector` | none | Node selector for Guardrails pods as `key=value` pairs (repeatable); prefer `--guardrails-config` | | `--guardrails-affinity` | none | Affinity rules for Guardrails pods as a JSON object; prefer `--guardrails-config` | | `--guardrails-tolerations` | none | Tolerations for Guardrails pods as a JSON array; prefer `--guardrails-config` | ## Examples ### Production deployment with private registry ```bash # 1. Sync images (run once per release upgrade) tare install identity.json \ --image-sync gcr.io/acme/tars \ --sync-only # 2. Install (or upgrade) echo "user:password" | tare install identity.json \ --serve-url https://proxy.acme.com \ --image-sync gcr.io/acme/tars \ --image-pull-secret-stdin \ --release-name acme-serve \ --wait ``` ### GitOps: generate values only ```bash tare install identity.json \ --serve-url https://proxy.acme.com \ --image-registry gcr.io/acme/tars \ --image-pull-secret-name acme-pull-secret \ --guardrails-config guardrails.yaml \ --enable-guardrails \ --print-helm-values > helm/tars-values.yaml ``` Commit `helm/tars-values.yaml` and apply separately: ```bash helm upgrade --install tars charts/valet/serve \ --namespace tars-system --create-namespace \ -f helm/tars-values.yaml ``` ### Preview images before syncing ```bash tare install identity.json \ --image-sync myregistry.example.com \ --enable-guardrails \ --print-images ``` Output: ``` registry.tetrate.ai/gateway:v1.6.5 → myregistry.example.com/gateway:v1.6.5 registry.tetrate.ai/ratelimit:v1.6.5 → myregistry.example.com/ratelimit:v1.6.5 ... registry.tetrate.ai/llm-guard-api:v0.3.1 → myregistry.example.com/llm-guard-api:v0.3.1 registry.tetrate.ai/envoy-dm:v0.3.1 → myregistry.example.com/envoy-dm:v0.3.1 registry.tetrate.ai/serve-helm:0.1.0-rc.4_abcd123 → myregistry.example.com/serve-helm:0.1.0-rc.4_abcd123 ``` ## Output All progress is written to **stderr**; only `--print-helm-values`, `--print-images`, and `--print-resources` write to stdout, making them safe to redirect: ``` Syncing images registry.tetrate.ai → myregistry.example.com Copying registry.tetrate.ai/gateway:v1.6.5 → myregistry.example.com/gateway:v1.6.5 2009/11/10 23:00:00 existing blob: sha256:abc123... ... Image sync complete. Creating pull secret "tars-image-pull-secret" in namespace tars-system... pull secret "tars-image-pull-secret" ok Creating pull secret "tars-image-pull-secret" in namespace tars-dataplane... pull secret "tars-image-pull-secret" ok Installing Agent Router dataplane... release: tars system namespace: tars-system dataplane namespace: tars-dataplane Creating Kubernetes namespaces... namespace tars-system: ok namespace tars-dataplane: ok Deploying with Helm... Helm deployment complete Installation complete. Next steps: kubectl get pods -n tars-system kubectl logs -n tars-system -l app=tars ``` ## Verification ```bash kubectl get pods -n tars-system kubectl get pods -n tars-dataplane kubectl logs -n tars-system -l app=tars ``` ## Management-plane event reporting After the install completes, the CLI writes a ConfigMap labelled `tars.io/component=install-event` to `tars-system`. The `tare-doctor` CronJob (deployed with this chart) forwards that ConfigMap to the management plane on its next tick (default: every 5 minutes) so the workspace's deployment history reflects the new install. On a successful forward the ConfigMap is deleted; on a 5xx / network failure it is retained for the next tick; on a 4xx it is dropped (the same payload would re-reject). If you install the chart without going through the `tare` CLI (raw `helm install`, ArgoCD / Flux, customer GitOps), the same dashboard row appears via the `tare-doctor` reconciler which observes the helm release secret and emits an event with `source: tare-doctor-poll`. Operator attribution is empty on the non-CLI path (rendered as a dash in the timeline). ## Rollback and uninstall ```bash # View history helm history tars -n tars-system # Rollback helm rollback tars -n tars-system # Uninstall helm uninstall tars -n tars-system kubectl delete namespace tars-system tars-dataplane ``` Where to go next tare upgrade Move an installed dataplane to a newer version with zero downtime. tare uninstall Tear down the dataplane and remove its namespaces and CRDs. --- ## TARE CLI tare (Tetrate Agent Router) is a single-command operator tool for deploying the Agent Router dataplane. It collapses three manual steps (image sync, pull-secret creation, and Helm deployment) into one invocation. ## Commands | Command | Description | |---|---| | [`tare install`](./install.mdx) | Install the Agent Router dataplane | | [`tare uninstall`](./uninstall.mdx) | Tear down the Agent Router dataplane (cluster-only; reverses install) | | [`tare upgrade`](./upgrade.mdx) | Upgrade an existing Agent Router dataplane release (HA-safe, atomic) | | [`tare gateway install`](./install-gateway.mdx) | Install serve gateway resources (GCP, Azure, or AWS) | | [`tare gateway config init`](./gateway-config-init.mdx) | Generate a starter `gcp-gateway.json` from a domain + project ID | | [`tare gateway config lint`](./gateway-config-lint.mdx) | Lint a `gcp-gateway.json` config (hermetic, CI-safe) | | [`tare mcp install`](.) | Deploy MCP server pods on a customer cluster | | [`tare mcp list`](.) | Show deployed MCP server status | | [`tare mcp enable`](.) | Register a deployed MCP server in the management plane's MCP catalog | | [`tare mcp uninstall`](.) | Remove the `tars-mcp` Helm release | | [`tare integrate`](/agent-router-service/guides/coding-agents/configure-tools-with-tare-cli/) | Configure local coding tools (Claude Code, Codex, and others) against your Agent Router gateway | ## Quick start ```bash # Print Helm values without touching the cluster tare install identity.json --serve-url https://proxy.acme.com --print-helm-values # Full install tare install identity.json --serve-url https://proxy.acme.com # Sync images to a private registry then install tare install identity.json --serve-url https://proxy.acme.com \ --image-sync myregistry.example.com # Sync + create pull secret + install in one command echo "user:password" | tare install identity.json \ --serve-url https://proxy.acme.com \ --image-sync myregistry.example.com \ --image-pull-secret-stdin ``` ## Identity file `tare` accepts the service-account file in both formats produced by the management plane: **Wrapped format** (standard): ```json { "serviceAccount": { ... }, "checksum": "664b355c...", "checksumAlgorithm": "sha256" } ``` **Naked format** (direct service account JSON): ```json { "type": "service_account", "clientEmail": "default.prod@acme.sa.iam.tetrate.ai", ... } ``` Use `tars sa wrap` to convert a naked file to the wrapped format. ## Gateway config schema For `tare gateway install`, the config file shape depends on the provider: - [`gcp-gateway.json` schema](./gcp-gateway-config.mdx): `--type gcp` - [`azure-gateway.json` schema](./azure-gateway-config.mdx): `--type azure` - `--type aws` takes no config file; it is configured entirely by flags ## Version ```bash tare --version ``` ## CLI reference docs The `docs/tare/cli/ref/` directory contains auto-generated per-command reference files derived directly from each command's `--help` output. Regenerate after any flag or help-text change: ```bash make cli.docs ``` CI verifies the directory is up to date on every PR: ```bash make cli.docs && git diff --exit-code docs/tare/cli/ref/ ``` `make cli.docs` depends on `cli.embed-charts` and `cli.manifest` (both run automatically as prerequisites). The generator binary lives at `svc/cli/cmd/tare-docgen`. Where to go next tare install Full reference for installing the data plane on a target cluster. tare gateway config init Generate a starter gateway config from a domain and project ID. --- ## `tare uninstall` Tear down the Agent Router dataplane from a Kubernetes cluster. ## Synopsis ``` tare uninstall [flags] ``` ## Description `tare uninstall` reverses `tare install` in a single command. After a successful run, the Helm release is gone, the `tars-system` and `tars-dataplane` namespaces have been removed, and the CRDs the install brought have been deleted. This is **mostly a cluster-only operation**, with one management-plane side channel: after `helm uninstall` runs (phase 2), the CLI reports the uninstall event to MP so the workspace's deployment history reflects it. See [Management-plane event reporting](#management-plane-event-reporting) below. `tare uninstall` does not: - Deregister the dataplane URL from your workspace. - Revoke the service-account identity. - Touch the management-plane database directly (beyond the install-event side channel, which is bounded to the `dataplane_install_events` table). If you need to clean up the dataplane URL registration or the SA identity, use the dashboard or the `tars sa` command. `helm` and `kubectl` are downloaded automatically on first run into `~/.tare/tools/` if they are not already on PATH. ## Phases ``` [1/6] Preflight (Helm 3+, kubectl, cluster) [2/6] helm uninstall -n └── after helm uninstall: sync-POST uninstall event to MP (CM fallback on 5xx; see "Management-plane event reporting" below) [3/6] Patch finalizers in tars-system / tars-dataplane + delete GatewayClass tars-egress [4/6] Delete namespaces (system, dataplane) [5/6] Wait for namespaces to disappear [6/6] Delete TARS-owned CRDs (skipped under the multi-DP guard; see --force) ``` Phase ordering is deliberate: - **Phase 3 runs before phase 4** so the namespace delete in phase 4 has no blockers. We enumerate the namespaced resource kinds in the chart's API groups (`aigateway.envoyproxy.io`, `tars.tetrate.ai`, `gateway.networking.k8s.io`, `gateway.envoyproxy.io`), list any in both `tars-system` and `tars-dataplane` that carry finalizers, and patch `metadata.finalizers: []` on them. Both namespaces are covered: `tars-system` holds the bulk of the chart's CRs and controllers, but `tars-dataplane` can also hold finalizer-bearing resources such as `EnvoyPatchPolicy`. - **Phase 4 deletes the namespaces** which sweeps every namespaced resource in one stroke. Immediately after issuing each delete, we clear the namespace's `.spec.finalizers` via the `/finalize` subresource: the standard "force a stuck namespace through" technique. Safe here because we have already patched all resource finalizers and the goal of the command is to tear the namespaces down. - **Phase 5 polls** for the namespaces to disappear. - **Phase 6 deletes CRDs**, by which point no CRs of any owned kind exist, so CRD deletion has nothing left to wait on. ## Examples ```bash # Standard teardown with confirmation prompt. tare uninstall identity.json # Non-interactive (CI). tare uninstall identity.json --yes # Stuck finalizers: bypass controller cleanup and patch them empty. tare uninstall identity.json --yes --force # Multi-DP cluster: leave CRDs in place so other releases keep working. tare uninstall identity.json --yes --keep-crds # Preview without touching the cluster. tare uninstall identity.json --dry-run ``` ## Flags | Flag | Default | Description | |---|---|---| | `--keep-crds` | false | Leave CRDs in place. Use on shared / multi-DP clusters. | | `--force` | false | Override the multi-DP guard and delete CRDs even when other releases share them. | | `--yes` | false | Skip the typed-confirmation prompt. | | `--timeout` | `5m` | Per-phase wait timeout. | | `--dry-run` | false | Print the plan and run `helm uninstall --dry-run`; no cluster mutation. | | `--skip-preflight` (hidden) | false | Skip Helm/kubectl/cluster preflight (also `TARE_SKIP_PREFLIGHT=1`). | | `--release-name` (hidden) | `tars` | Helm release name. | | `--system-namespace` (hidden) | `tars-system` | Kubernetes system namespace. | | `--dataplane-namespace` (hidden) | `tars-dataplane` | Kubernetes dataplane namespace. | Inherited from root: `--verbose`, `--quiet`. ## What gets deleted Namespaced resources (cleared via the namespace delete in phase 4): - All objects in `tars-system` and `tars-dataplane`, including the image-pull secret created by `tare install --image-pull-secret-stdin`. Cluster-scoped resources (explicit deletes): - `GatewayClass/tars-egress` - CRDs in groups `aigateway.envoyproxy.io` and `tars.tetrate.ai` ## What is preserved - CRDs in `gateway.networking.k8s.io` (commonly centrally managed; GKE, distro bundles, etc.). - CRDs in `gateway.envoyproxy.io` (Envoy Gateway distribution, often shared). - Any namespaces, secrets, or ConfigMaps outside `tars-system` and `tars-dataplane` that an operator created independently. - Management-plane state, except for the single new row appended to `dataplane_install_events` (one row per uninstall, see [Management-plane event reporting](#management-plane-event-reporting)). ## Management-plane event reporting Between phase [2/6] (`helm uninstall`) and phase [3/6] (finalizer sweep), `tare uninstall` reports the operation to the management plane: 1. Read the pre-uninstall Helm revision (`preRev`) so the event payload can carry the `helm_revision` that matches the install / upgrade events in the workspace's deployment history. The PK is unrecoverable once `helm uninstall` has run, which is why the read happens before phase 2. 2. After `helm uninstall` returns, sync-POST the event to `POST /v1/dataplane-install-events` on the management plane, signed with the identity file's SA token. 3. On `2xx`: bump the `tars-doctor-install-event-state` ConfigMap so the doctor's reconciler doesn't re-emit a synthetic event on its next tick, then continue with phases 3–6. 4. On `5xx` or network failure: write a ConfigMap to `tars-system` labelled `tars.io/component=install-event`, same shape as the install / upgrade outbox CMs. The `tare-doctor` cron in that cluster would normally pick it up on the next tick, **but** the namespace is about to be deleted in phase 4, so the CM may not survive long enough to be forwarded. This is a known limitation: the dashboard timeline may not show an `uninstall` row in this failure case. 5. On `4xx`: log a warning and skip the fallback (the next retry would re-reject with the same error). The CLI continues teardown. The event POST **never blocks teardown, never changes the CLI's exit code, and never re-prompts**. The dataplane is removed regardless of whether the management plane could be reached. Configured via the identity file's `apiUri`. No env var override is honored on this code path (the doctor's `TARE_DOCTOR_API_URI` only affects in-cluster forwarding, not CLI-side uninstall reporting). ## Confirmation prompt By default `tare uninstall` prints the plan and asks the operator to type the system namespace name to confirm. Use `--yes` to skip in automation. ``` About to uninstall Agent Router dataplane: Identity: identity.json (client_email: tetrate@example.sa.iam.tetrate.ai) Release: tars System namespace: tars-system Dataplane: tars-dataplane CRDs: will be deleted (aigateway.envoyproxy.io, tars.tetrate.ai) Touches MP: appends one row to dataplane_install_events (operation=uninstall) after helm uninstall. NOT touched: dataplane URL registration, SA identity revocation, image-pull secrets outside the namespaces above. Type the system namespace name (tars-system) to confirm: _ ``` ## Stuck finalizers The Agent Router CRs (`AIGatewayRoute`, `MCPRoute`, `BackendSecurityPolicy`, etc.) and `GatewayClass/tars-egress` carry finalizers that the dataplane controllers normally clear during a graceful delete. After phase 2's `helm uninstall` removes those controllers, the finalizers can never clear on their own: the only reconciler that could is gone. For this reason, `tare uninstall` automatically patches `metadata.finalizers: []` on any CR or `GatewayClass` that remains after a brief poll. This is the only path to completion at that point and does not require `--force`. The patch bypasses controller cleanup hooks (egress backends may remain registered with external services, telemetry exports may not be flushed). That is the unavoidable cost of teardown; the same cost applies whether you patch via `tare uninstall` or via `kubectl patch` by hand. ## Multi-DP guard Before deleting CRDs in phase 6, `tare uninstall` runs `helm list -A` and checks for other releases of the same chart. Behaviour: | Situation | Effect | |---|---| | No other Agent Router releases | CRDs are deleted. | | Other releases found, `--force` not set | CRDs **skipped**. Other releases remain functional. Command exits 0 with a warning. | | Other releases found, `--force` set | CRDs deleted anyway. Other releases will break. | | `--keep-crds` set | Phase 6 skipped unconditionally. No detection run. | ## Dry run `--dry-run` prints the plan, runs `helm uninstall --dry-run`, lists the CRs that would be drained, lists the CRDs that would be deleted (or notes the multi-DP skip), and names the namespaces that would be removed. No mutating calls (`kubectl delete`, `kubectl patch`, `helm uninstall` without `--dry-run`) are made. ## Exit codes - `0`: Success, or "nothing to do" (release already absent). - `1`: Phase failed; see error message. Where to go next tare install Install the dataplane that this command tears down. tare upgrade Move an installed dataplane to a newer version instead of reinstalling. --- ## `tare upgrade` Upgrade an installed Agent Router dataplane to the version embedded in the current tare binary, with zero downtime when the data plane is HA. ## Synopsis ``` tare upgrade [flags] ``` ## Description `tare upgrade` performs the operator workflow for moving an existing Agent Router dataplane release to a newer version. Unlike `tare install`, it does not fresh-install and does not re-prompt for configuration the operator already supplied at install time, it reads the live release's Helm values via `helm get values` and carries forward the operator-supplied config. Phases: 1. Preflight (Helm 3+, `kubectl`, cluster connectivity; warns when metrics-server is absent and HA is on) 2. Discover existing release in the target namespace (refuses to run if none is found: that path is `tare install`) 3. Read existing release values and carry forward registry, customer, environment, deployment mode, serve URL, image pull secret name, and the OTel collector enable + endpoint + auth-header config 4. HA preflight: read the `EnvoyProxy` CRD's `envoyHpa.minReplicas`; refuse to upgrade if it is `< 2` unless `--allow-downtime` is set 5. Apply CRDs from the embedded chart (Helm's `crds/` directory is install-only, so an upgrade has to upsert CRDs explicitly) 6. `helm upgrade --install --atomic --wait` against the embedded chart `helm` and `kubectl` are downloaded automatically on first run into `~/.tare/tools/` and reused on subsequent runs. The new tare binary's embedded chart supplies the new chart defaults: notably the HA-by-default flip and the `EnvoyProxy.spec.shutdown` block. Existing operator-supplied values override chart defaults where they overlap. > **Image sync is out of scope.** `tare upgrade` does not pull new images > into your registry. Sync first, then upgrade: > ``` > tare install identity.json --image-sync --sync-only > tare upgrade identity.json > ``` ## Usage ### Standard upgrade After downloading a newer `tare` binary and placing it in `~/.tare/bin/`: ```bash tare upgrade identity.json ``` `tare upgrade` reads the existing release values, carries them forward, applies CRD changes from the new chart, and runs `helm upgrade --install --atomic --wait`. A failed rollout auto-rolls back to the previous revision. ### Single-replica install (lab / non-production) If the existing install is single-replica (an install from a release that predates the HA default, or `tare install --ha=false`), `tare upgrade` refuses to proceed because rolling the single data-plane Envoy pod would drop in-flight LLM streams: ``` data-plane Envoy is single-replica (envoyHpa.minReplicas=1). Upgrading now will RST in-flight LLM streams. Fix HA first (idempotent, takes ~1m): tare install --ha identity.json Or force this upgrade with downtime: tare upgrade identity.json --allow-downtime ``` The recommended path is to migrate to HA first via `tare install --ha` (which is idempotent: it calls `helm upgrade --install` and bumps the HPA `minReplicas` from 1 to 2). That migration step itself causes a brief one-time disruption while the second pod comes up, but every subsequent `tare upgrade` is zero-downtime. ### Override carried-forward values CLI flags take precedence over carryover. Override only what you need to change, e.g., to move to a new image registry as part of the upgrade: ```bash # 1. Sync to the new registry tare install identity.json --image-sync newregistry.example.com --sync-only # 2. Upgrade using the new registry tare upgrade identity.json --image-registry newregistry.example.com ``` ### Disable atomic rollback When debugging a stuck release, run without `--atomic` to leave the half-deployed release in place: ```bash tare upgrade identity.json --no-atomic ``` The default `--atomic` is the safer behavior for production. Use this flag only when you need to inspect a failed rollout. ## Flags ### Main | Flag | Default | Description | |---|---|---| | `--allow-downtime` | `false` | Proceed even when the data plane is single-replica. In-flight LLM streams will be dropped. | | `--ha` | `true` | HA-safe defaults for the data-plane Envoy proxy (HPA min 2, PDB min 1). Pass `--ha=false` to keep single-replica. | | `--drain-timeout-seconds` | `300` | `EnvoyProxy.spec.shutdown.drainTimeout` (seconds). Maximum time Envoy waits for in-flight requests (long LLM streams) to finish before SIGKILL. | | `--no-atomic` | `false` | Disable `helm --atomic` (no auto-rollback on failure). Leaves stuck releases in place for debugging. | | `--timeout` | `10m` | `helm upgrade --wait` timeout. Should exceed `drainTimeout × replicas` to allow serial drain. | ### Telemetry | Flag | Default | Description | |---|---|---| | `--enable-otel-collector` | carry-forward | Override the OTel collector enable state. Default carries the existing release's value. | | `--otel-collector-endpoint ` | carry-forward | Override OTel OTLP endpoint. | | `--otel-exporter-auth-headers ` | carry-forward | Override OTel Authorization header value. | ### Advanced / hidden Hidden from `--help` but accepted for advanced use; defaults all come from carryover unless overridden. | Flag | Default | Description | |---|---|---| | `--release-name` | `tars` | Helm release name. | | `--system-namespace` | `tars-system` | Kubernetes system namespace. | | `--dataplane-namespace` | `tars-dataplane` | Kubernetes dataplane namespace. | | `--chart-path` | embedded | Path or OCI/HTTP reference to chart. | | `--chart-version` | none | Chart version (for remote/OCI charts). | | `--image-registry` | carry-forward | Image registry override. | | `--image-tag` | embedded manifest | Image tag override. | | `--deployment-mode` | carry-forward | `enterprise` or `saas`. | | `--customer` | carry-forward | Customer label override. | | `--environment` | carry-forward | Environment label override. | | `--serve-url` | carry-forward | Serve URL override. | | `--redis-type` | `cluster` | `cluster` or `hosted`. | | `--no-rate-limiting` | `false` | Disable rate limiting. | | `--skip-preflight` | `false` | Skip Helm/kubectl/cluster preflight (also `TARE_SKIP_PREFLIGHT=1`). | ## Carryover behavior `tare upgrade` reads the existing release's user-supplied values (`helm get values` returns only what was passed via `-f` or `--set` at install time, not chart defaults). The following fields are carried forward into the regenerated values: | Helm values path | CLI flag override | |---|---| | `global.imageRegistry` | `--image-registry` | | `global.customer` | `--customer` | | `global.environment` | `--environment` | | `global.deploymentMode` | `--deployment-mode` | | `global.serveUrl` | `--serve-url` | | `global.imagePullSecretName` | (no override; lives in operator-supplied pull secret) | | `otelCollector.enabled` | `--enable-otel-collector` | | `otelCollector.exporters.otlp.endpoint` | `--otel-collector-endpoint` | | `otelCollector.exporters.otlp.headers.Authorization` | `--otel-exporter-auth-headers` | Fields not in the carryover list fall back to embedded chart defaults (from the new tare binary). Pass the corresponding flag to override. ## Single-replica preflight `tare upgrade` reads `spec.provider.kubernetes.envoyHpa.minReplicas` from the `EnvoyProxy` CRD in the system namespace. The check has three outcomes: - `minReplicas ≥ 2`: proceed. - `minReplicas < 2` and `--allow-downtime` not set: abort with the migration message. - `EnvoyProxy` resource not found, or kubectl call fails: print a warning and skip the preflight (operator's responsibility to verify HA). The preflight does not currently inspect live `Deployment.spec.replicas` because the EnvoyProxy CRD HPA configuration is the source-of-truth for how the data plane scales over time. Inspecting both would tighten the check; that is a future enhancement. ## Atomic rollback The default `helm upgrade --atomic` makes the upgrade transactional: if any pod fails its readiness probe within `--timeout`, Helm rolls the release back to the previous revision and exits non-zero. The data plane returns to its previous state without operator intervention. `--no-atomic` disables this: useful for inspecting a stuck release mid-rollout, at the cost of leaving traffic potentially affected. ## Output `tare upgrade` writes phase-prefixed status lines (`[i/N] `) to stderr. Final lines include the rollback hint and inspection commands: ``` [1/5] Preflight (Helm / kubectl / cluster) [2/5] Discover existing release release tars found in tars-system carried forward: registry="gcr.io/acme/tars" customer="acme" env="" otel=true [3/5] HA preflight EnvoyProxy minReplicas=2 (HA) Using embedded chart [4/5] Apply / update CRDs 3 created, 12 updated, 4 skipped [5/5] Helm upgrade ... Upgrade completed. Rollback (if needed): helm rollback tars -n tars-system Inspect: kubectl get pods -n tars-system kubectl get pods -n tars-dataplane ``` ## Verification ```bash # Pods in both namespaces should be Running with the new image tag kubectl get pods -n tars-system kubectl get pods -n tars-dataplane # EnvoyProxy HA configuration kubectl get envoyproxy -n tars-system -o yaml | yq .spec.provider.kubernetes.envoyHpa # Active rollout status kubectl get deploy -n tars-dataplane -o wide ``` ## Management-plane event reporting `tare upgrade` reports the upgrade as an event to the management plane via the same in-cluster ConfigMap outbox that `tare install` uses. After a successful `helm upgrade`, the CLI writes a ConfigMap labelled `tars.io/component=install-event` to `tars-system` carrying `operation=upgrade` and the new Helm revision. The `tare-doctor` CronJob forwards it on its next tick (default every 5 minutes); the dashboard's deployment-history timeline shows the row at `helm_revision = N+1` with `from_version` and `to_version` populated when chart versions differ. The event POST never blocks the upgrade and never changes the exit code. Failed atomic-rollback upgrades (`helm upgrade --atomic` that triggers a rollback) produce **two** dashboard rows: - One for the failed upgrade attempt: `operation: upgrade, status: failed`, `error_summary` populated. - One for the auto-rollback: `operation: rollback, status: success`, on the new helm revision the rollback created. This is the `tare-doctor` reconciler recognising helm's "Rollback to N" description and labelling the artefact correctly, instead of attributing the system reversion to the operator who triggered the upgrade. ## Rollback `tare upgrade` defaults to `--atomic`, so failed upgrades roll back automatically. To roll back a successful but problematic upgrade: ```bash # View revision history helm history tars -n tars-system # Roll back to the previous revision helm rollback tars -n tars-system # Or to a specific revision number helm rollback tars 5 -n tars-system ``` Note: `helm rollback` does not roll back CRDs. Any CRD schema changes applied during the upgrade remain in place. This is rarely an issue in practice because CRD changes are additive (new fields, no removed fields) in supported upgrade paths. ## Migration: single-replica → HA Customers whose `tare install` ran before the HA default flip are on single-replica installs. The migration path is one idempotent install call that bumps the EnvoyProxy CRD's HPA `minReplicas` from 1 to 2: ```bash tare install --ha identity.json ``` This calls `helm upgrade --install`, so it does not require uninstall and does not change any other configuration. The Envoy Gateway controller scales the data-plane Deployment up to 2 pods (a one-time brief disruption while the second pod comes up). After this migration, `tare upgrade` is zero-downtime. Where to go next tare install Fresh-install the dataplane and sync images before upgrading. tare uninstall Tear down the dataplane when an upgrade is not the goal.