Routing policy reference
Enterprise Tier
Smart routing in Tetrate Agent Router covers two mechanisms, model aliases and attribute-based routing. The object model behind the attribute-based half is a small set of objects, presented in the order they relate to one another: the request attribute an application declares, the routing policy that holds an ordered list of rules, the scope that decides which policy applies to a key, the mode that decides whether it acts, and the record every request leaves behind. Each entry states what the object holds and how it is resolved. There are no procedures here: the scenarios that show these objects in use are on the smart routing page, and each links back here.
Attribute-based routing shipped in v0.7.0 as a preview. It is off until a platform operator enables it on the Admin Console, the management plane, and each data plane; the Admin Console pages answer 404 while the Console switch is off. The three switches are listed under Enable the preview on the smart routing page. String operators and a published decision contract are not part of the preview.
The shapes shown in text blocks on this page are illustrative, to show what an object holds. None is a schema. The X-Tars-Metadata header format and its limits are fixed: the header is the half of this feature an application ships, and it can be built against before a policy exists to match it.
Persona: Platform operator working in the Admin Console.
Estimated time: 15 minutes to read.
The objects
Three objects, and each is defined below in this order.
| Object | What it is | Where it lives |
|---|---|---|
| Request attribute | A named thing an application may declare about a request | Defined centrally in the Admin Console |
| Routing policy | An ordered list of routing rules with a default, attached to a scope | The Policies area |
| Routing rule | One condition and one destination, inside a policy | Inside a policy |
The division is between the attribute definition, which is reusable and referenced by many policies, and the policy itself, which is an attachment: its rule list is its configuration, so it holds no reusable definition of its own.
Request attribute
A request attribute is a reusable definition that many policies reference, defined centrally in the Admin Console. Defining one changes no traffic.
Request attribute
name workload, task, plan
source where the value comes from
value type one of a named set, or free text
allowed values when the value type is a named set
| Field | Holds |
|---|---|
| Name | The attribute's identifier, used unprefixed by the calling application and prefixed by its source in a rule |
| Source | Where the value comes from, and therefore whether the caller can set it |
| Value type | Either a named set of permitted values, or free text |
| Allowed values | The permitted values, when the value type is a named set |
An attribute whose value type is a named set is meant to be matched against values from that set. A request may still carry a value outside it: the value is recorded on the request as out of set, and it is compared against each condition's own values like any other. It therefore matches no eq or in condition that names a value from the set, but it does satisfy a ne or nin condition, because those are met by any present value other than the ones named. See Routing rule.
Source, and the name a rule uses
Source records where an attribute's value comes from. One source exists today, metadata, which means the value is set by the calling application on each request. A rule names an attribute by its source, so a rule reads metadata.workload rather than workload. The attribute definition and the rule builder state the source on every attribute, so a rule author can see what a condition depends on.
The prefix distinguishes values the caller sets from values the caller does not, so a source that is not caller-supplied can be added later without an existing rule becoming ambiguous.
The name is unprefixed in the header the application sends, because the header is the source. See how a routing rule knows what a request is for the wire form.
The X-Tars-Metadata header
Attribute values reach the gateway in one request header. That header is the only input attribute-based routing reads: nothing is taken from the request body, and nothing is inferred.
X-Tars-Metadata: workload=batch,task=summarization
The format is a comma-separated list of key=value pairs. Each key is an attribute name as defined in the Admin Console, written unprefixed, because the header itself is the source.
Limits
These limits are fixed, and they are applied to every request before any rule is evaluated. An entry outside them is dropped, never refused: the request is routed on the entries that remain, and each drop is counted on the data plane metric router_attribute_routing_header_dropped_total, labelled with its reason.
| Limit | Value | What happens outside it | reason |
|---|---|---|---|
| Entries per request | 8 | Every valid entry past the eighth is dropped | over_cap |
| Key | [a-z][a-z0-9_]{0,31}, so 1 to 32 characters beginning with a lowercase letter | The entry is dropped | bad_key |
| Value | [A-Za-z0-9._:-]{1,64}, so 1 to 64 characters | The entry is dropped | bad_value |
| Whole header | 2 KiB, measured across every header line before parsing | The whole header is dropped and the request carries no attributes | oversize |
An entry with no = is dropped the same way, under bad_value. Dropped entries are not written to the request log; the counter is the only record of them. The limits are read only where a routing policy governs the request: a request to a project and key that no routing policy covers is never affected by what the header contains.
Five parsing rules complete the format.
- An uppercase ASCII letter in a key is folded to lowercase before the key is validated, so
Workload=batchmatches theworkloadattribute. Values are never folded: they are compared byte for byte against the values a condition names. - Whitespace around
=and around,is trimmed. - Where a request carries the header on more than one line, the lines are joined with a comma and parsed as one, so two lines and a single line of two entries are equivalent.
- Where a key appears more than once, the last occurrence among the first 8 valid entries wins. A repeated key still spends one of the 8.
- An empty segment, such as a trailing comma or a blank header line, is ignored. It is not malformed and is not counted.
A value cannot itself contain , or =. Both are separators, and the value grammar excludes them.
A malformed entry is dropped, and nothing fails the request
Nothing about this header can fail a request. An entry that breaks the format above is dropped and counted, and the request goes on with whatever entries parsed. With nothing usable left, the request takes the policy's default and its response carries x-tars-attribute-routing-outcome: default_no_attributes (see Response headers). A caller keeps its traffic even when its intent is unreadable. In monitor mode the request is still served by the model it named.
The same holds for a well-formed header, whatever it says. A key naming an attribute that is not defined, an attribute the caller did not send, no rule matching, a paused policy: each sends the request to the policy's default, exactly as a request with no header at all would be. Routing itself never refuses a request.
The practical consequence is that a mistake is silent. A team that ships workload=btch, a key nobody defined, or an entry the grammar rejects sees its requests served by the default, which is indistinguishable from a team that has not shipped the header at all. Two things tell them apart: the response headers on a test request, and reviewing a policy in monitor before it is enforced. An operator who scrapes the data plane's Prometheus endpoint can also alert on router_attribute_routing_header_dropped_total (exposed there as envoy_dynamicmodulescustom_router_attribute_routing_header_dropped_total from 0.7.0; see the data plane metrics).
Attribute values are visible to administrators
The attribute values a policy matched on are written to the request log, where any administrator of the organization can read them. They are visible per request in the Admin Console alongside the rest of the decision record.
Attribute values are therefore not a place for user identifiers, account numbers, email addresses, or any other personal or sensitive data. An attribute is a low-cardinality label describing a class of request, such as workload=batch or plan=wealth. Attributing spend to a person or a team is a different mechanism: see Know what every app and project costs.
Only the attributes the winning policy names are recorded, not everything the caller sent. Telemetry exported over OpenTelemetry carries the policy identifier, the position of the matched rule, and whether any attribute was present at all, never the attribute names or their values.
Routing metadata is also a separate namespace from the tags an administrator assigns to an API key. A calling application cannot set a tag, and routing metadata is never read as one, so a caller cannot make its requests look as though they carry an administrator's tag.
Routing policy
A routing policy is an ordered list of routing rules with a required default, attached to a scope and held in one of two modes.
Routing policy
scope project | key
mode monitor | enforce
rules an ordered list, evaluated top to bottom
default required, applied when no rule matches
A routing policy is a policy type, so it is created, targeted, monitored, enforced, and listed the same way every other policy type is.
Routing rule
Routing rule
condition attribute, operator, value or values
destination a model available to the project, and the provider that serves it
A condition compares one attribute against one value, or against a set of values, using one of four operators.
| Operator | Takes | Matches when the value declared is |
|---|---|---|
eq | One value | The value the condition names |
ne | One value | Present, and anything other than the value the condition names |
in | A set of values | One of the values the condition names |
nin | A set of values | Present, and none of the values the condition names |
eq and ne take exactly one value. Saving a rule with an eq or ne condition on more than one value is refused; a set is written with in or nin. The refusal is there because the data plane rejects such a policy as malformed, and a malformed policy leaves its whole project with no attribute routing at all.
An attribute the request did not declare matches no operator, ne and nin included. A negative condition is satisfied only by a value that is present and differs from the ones named. A request that sent no workload at all does not match metadata.workload ne batch; a request that sent workload=interactive does, and so does one that sent workload=btch, a value outside the attribute's allowed set. The allowed set bounds what a condition can be written against; it does not filter what a request may send.
Conditions may be combined within a rule, and all of them must hold for the rule to match, so one rule can require both that metadata.workload is batch and that metadata.task is summarization.
Every rule carries at least one condition. A rule with none would match every request and shadow everything below it, which is what the default already does, so a rule without a condition cannot be saved. A policy that should send every request from its target to one model is written as a default with no rules at all, which is 9: a degraded provider.
A rule's destination is a model available to the project, together with the provider that serves it. Naming the provider matters where several providers serve the same model name: the rule then reaches the one it names rather than whichever the catalog loaded last. Routing chooses between the models a project already has, and cannot reach one the project does not. A destination that is not served on the endpoint the request arrived on, in that endpoint's mode, makes the rule fall through to the next one; see Evaluation order.
The model the caller named
A matched rule overrides the model named in the request. An application that asks for opus in its request body and matches a rule whose destination is haiku is served by haiku. The model the caller named is not a condition, and it is not consulted when a rule is chosen.
That is the behavior the feature exists for: the applications worth moving to a cheaper model overnight are exactly the ones that name a model directly in their code, and a policy that yielded to them would move nothing.
The override is bounded in one direction only. A policy chooses among the models the key can already reach, so an override moves a request sideways within that set and never widens it. Both the model the caller named and the destination the policy chose are kept on the decision record, and the response identifies the model that actually served the request.
Two kinds of request are outside routing altogether, whatever policy covers the key.
| Request | Why routing is skipped |
|---|---|
| One authenticated with the caller's own provider credential | The credential reaches one provider, so the request cannot be moved to a model held anywhere else |
| One carrying no model in its body | MCP, health, well-known, and OAuth paths are answered before a body is read, so there is nothing to rewrite |
In both cases the model the caller named stands, no rule is evaluated, and the request carries no routing decision.
The default
Every policy has a default, and it is required. A request that matches no rule is served by the default. A routing policy cannot reject a request, so a caller that sends a well-formed header and declares nothing, declares a value the attribute definition does not allow, or declares an attribute that does not exist is served the default rather than an error.
The default overrides the model the caller named, exactly as a matched rule does. An enforcing policy therefore gives every request it applies to a destination, whether a rule matched or not, subject to the two carve-outs under the model the caller named. Choosing a default is consequently a decision about all the traffic no rule describes: setting it to the model those callers are served today keeps them where they are, and setting it to anything else moves them the moment the policy is enforced.
Evaluation order
The first matching rule wins. Rules are evaluated top to bottom, and evaluation stops at the first match.
Three things cause a rule not to match, and all three have the same consequence: evaluation continues at the next rule, and reaches the default only when every rule has been tried.
| Cause | Example |
|---|---|
| The attribute the condition names was not declared on the request, whatever the operator | A rule on metadata.workload against a request that sent no attributes, including a rule written with ne or nin |
| The value declared does not satisfy the condition | A rule on metadata.workload eq batch against workload=interactive |
| The rule's destination is not available on the endpoint the request arrived on | The model and provider named by the rule are not served in that endpoint's mode. The rule's position is recorded as skipped, and evaluation continues |
Scope resolution decides which rule list applies to a key. The rule list is evaluated per request. A policy resolves to one rule list for a key, and which rule inside that list wins varies request by request.
Scope and resolution
The scope set
A routing policy targets one of two scopes.
| Scope | Selects |
|---|---|
| Project | Every key issued in that project |
| Key | One API key |
Organization and tag selector scopes are not offered in this release. A routing policy cannot be created against either, and an attempt to do so is refused; listing policies with a scope or status filter the API cannot apply is refused too, rather than answered with a broader list. A key-scoped policy stands on its own: it applies whether or not the project holds a project-wide policy. An organization-scoped policy is intended to bind every project beneath it, which is the opposite of how every other policy family resolves, and the scope is withheld until that precedence rule is implemented rather than shipped with behavior a reader would have to guess at.
There is no team scope and no user scope. A policy selects keys by where they are issued, never by key ownership. Routing traffic per team is covered under things a routing rule cannot do.
Precedence
A key resolves to exactly one routing policy, because two rule lists cannot be merged. Where both a key-scoped and a project-scoped policy could apply, the narrower target wins.
| Order | Scope | Note |
|---|---|---|
| 1 | Key | Wins for the one key it targets |
| 2 | Project | Applies to every key in the project that no key-scoped policy targets |
The narrower policy replaces the wider one rather than combining with it. A key-scoped policy substitutes its whole rule list for the project's, and the project's rules are not consulted for that key, including the project's default. Resolution ignores mode: whichever policy wins is the one whose mode then decides whether traffic moves.
Coverage
The coverage figure shown against a policy counts the API keys that resolve to a rule list. It counts keys covered, not routing decisions taken, so it rises the moment a policy is saved and says nothing about whether any request has matched a rule. What each rule would have matched is the monitor readout, described under Modes.
The inheritance view and the effective policy view that apply to every policy type apply to routing policies too, and answer what a given key resolves to.
Modes
A routing policy is held in one of two modes.
| Mode | What happens to traffic | What is recorded |
|---|---|---|
| Monitor | Nothing changes. Every request goes where it would have gone with no policy at all, and the request body is not rewritten | The rules are evaluated against real traffic and the decision record is written with the mode marked monitor, so the readout shows how many requests each rule would have matched over a recent window, with the model each would have moved to. The response headers carry the decision too |
| Enforce | The rules decide the destination, and the model in the request body is rewritten to it | The full decision record, including the model the caller named |
Every policy is created in monitor. Enforce is reached only by changing the mode on a policy that already exists, never as part of creating or editing one, so the audit log carries the moment traffic began to move as an event of its own. Reviewing a policy in monitor before it is enforced is covered in A policy has to be reviewed before it takes effect.
Mode is not status. A policy separately carries an enabled or disabled status, kept apart from mode so that a policy sitting in monitor can be paused and resumed without losing which of the two modes it held.
Monitor proves matching, not quality. The destination a rule names is never actually called in monitor, so nothing recorded there says whether its answers would have been suitable.
The decision record
Every request records why it went where it went.
| Recorded | Detail |
|---|---|
| The routing that applied | Which policy, which scope it was targeted at, and which mode it was in |
| The outcome | matched when a rule served the request. A request that matched no rule records which of the three reasons the default applied for: default_no_attributes (the request carried no usable value for any attribute the policy reads, whether the header was absent, dropped, or named only attributes the policy ignores), default_no_match (values were present and no rule matched), or default_rules_skipped (every rule whose conditions matched named a destination not available on the endpoint the request arrived on). A request no policy governs records nothing |
| The rule that matched | The rule's identifier and where in the rule order the match happened, on matched only. The positions of rules skipped for an unavailable destination are recorded alongside, so a skip followed by a later match is still visible |
| The attributes evaluated | The attributes the winning policy names, with the value declared for each, and any whose value fell outside the allowed set |
| The model the caller named | Recorded whenever the policy replaced it |
| The destination the policy chose | The model the rule or the default named |
| The model that served | What answered after any fallback or traffic split moved the request again |
| How it turned out | Whether it succeeded, how long it took, and what it cost |
Three model identities are kept apart, because they can all differ on one request: what the application asked for, what routing chose, and what finally served it after an availability fallback, a traffic split, or a budget action moved it again. Reading the destination alone hides that.
The response itself identifies the model that actually served the request, so a caller can tell what served a call without opening anything. In the Admin Console request log, a request routed by a rule is labelled Routing rule and one served by the default Routing default, separately from Budget fallback and Provider fallback, and the routed model stays visible when a fallback then moved the request off it.
Response headers
A routed response names the decision, so the developer who sent the header can confirm on a test request which policy and rule served it, without administrator access to the request log.
| Header | Value | Present |
|---|---|---|
x-tars-attribute-routing-outcome | matched, default_no_attributes, default_no_match, or default_rules_skipped, as in the record above | On every request a routing policy governs, in monitor as in enforce |
x-tars-attribute-routing-policy | The identifier of the policy that applied | On every request a routing policy governs |
x-tars-attribute-routing-rule | The identifier of the rule that matched | On matched only |
A request no policy governs carries none of them. Only identifiers and the outcome go out: the caller already knows the attributes it sent, and attribute names and values stay in the request log. The headers say what routing decided, not what served: a fallback or a traffic split can still move the request afterwards, and the model that answered is named by the response as before.
The values sit on the per-request record beside what that request cost and how long it took, so a routing decision can be reviewed against its own outcome. They are not aggregated. Attribute values never enter the usage rollups, and routing metadata is a separate namespace from the API key tags spend is grouped by, so reading spend along an attribute is not something this record supports. What the record holds, and who can read it, is covered under attribute values are visible to administrators.
Where each object is managed
| Object | Surface | Operations |
|---|---|---|
| Request attribute | Defined centrally in the Admin Console | Create, edit, remove, and see which policies reference one before changing it. The definition view shows how many policies hold a condition on the attribute and which rules compare against it, so an attribute no policy references is visible as removable |
| Routing policy | The Policies area | Create, target, monitor, enforce, list |
An attribute definition is organization-wide and shared by every project's rules, while a routing policy is targeted at one project or one key. Renaming an attribute or withdrawing one of its allowed values therefore reaches every project at once, which is why the two are held at different permission levels: defining attributes is an administrator operation, and writing a policy against them belongs to the project.
An attribute that a live rule condition still compares against cannot be deleted. The attempt is refused and names the policies blocking it, so the rules have to be changed first.
Everything available in the Admin Console is available in the admin API, so a routing policy can be managed as code.
A saved policy, a mode change, or a pause reaches live traffic within the configuration sync interval, under a minute, with no gateway restart. Connections already open can keep the previous rules for a few minutes more, until they drain. Removing a model or provider from a project, or adding a key, updates the delivered routing the same way. See Configuration propagation.
Roles
| Role | On routing |
|---|---|
| Administrator | Defines attributes centrally in the Admin Console, and creates, targets, and enforces routing policies |
| Developer | Declares attributes on the requests their application sends. Developers hold no Admin Console role in routing: they do not create or see routing policies |
A rule matching on an attribute is inert until the application team that owns the calling code sends it.
What a routing policy is not
Routing decides which of a project's models serves a request. Four nearby things are decided elsewhere, and each is a separate mechanism with its own guide.
| Not a routing policy | What decides it | Where it is covered |
|---|---|---|
| Which models a key may use at all | The project's model set | Create a project |
| Where a request goes when the destination fails | A fallback policy | Set up fallback |
| A fixed percentage across two backends | A traffic split | Reduce cost with traffic splitting |
| What happens when a spend limit is reached | A budget enforcement action | Enforce a budget cap |
The cases a routing rule cannot cover, with what to do instead, are listed under things a routing rule cannot do.
Where to go next
Smart routing
These objects worked through ten scenarios, from a hardcoded model name to a degraded provider at 2am.
Create a project
The model set that bounds every destination a routing rule can name.
Glossary
The routing terms used across these pages, including the ones that name mechanisms Agent Router deliberately does not have.