Skip to main content

Gateway sets and DNS-level failover

Enterprise Tier

A project can attach more than one gateway, and every attached gateway serves the identical project configuration by construction. That turns a set of member gateways into interchangeable failover targets: point customer-managed DNS at the member URLs, health-check each member's membership probe endpoint, and a client that moves from one member to another encounters the same authentication, guardrails, and catalogs it left behind. This guide covers the membership model, the probe contract, a reference DNS failover pattern, and the runbook for detaching a member safely.


Availability

Multi-gateway projects require an Agent Router Enterprise release that includes the multi-gateway projects capability, and the deployment itself must declare the Enterprise product line (DEPLOYMENT_PRODUCT=enterprise, derived from the installation charts; the gate fails closed, so a deployment that does not declare it behaves as Agent Router Service). On earlier releases a project maps to exactly one gateway and this guide does not apply. This surface does not exist on Agent Router Service (SaaS).

Agent Router provides the consistency guarantee, the membership lifecycle, per-member visibility, and per-data-plane request attribution. It ships no traffic steering: there is no built-in load balancer, GSLB, or failover executor in front of a project's gateways. Failover runs entirely under the operator's control, at the DNS (or equivalent routing) layer the operator already owns. This is deliberate — the DNS name clients call is the operator's artifact, and the failover policy (active/passive, weighted, geo) is an infrastructure decision Agent Router does not make.

Persona: Platform operator working in the Admin Dashboard and the organisation's DNS provider (the examples use AWS Route 53).

Estimated time: 30--45 minutes to set up the reference pattern for one project; 10--15 minutes to run the detach runbook.

Outcomes

By the end of this guide:

  • The membership model is understood: like-configured members, the four membership states, the zero-gateway state, and per-data-plane request attribution.
  • Customer-managed DNS failover is configured against the member URLs, health-checked on the membership probe endpoint.
  • The project's endpoint URL is set, so developers are given the client-facing name rather than a single member's URL.
  • A member has been detached safely: DNS shifted, drain verified through attribution, and the terminal state interpreted correctly.
  • The one documented limitation — per-gateway rate limits — is understood and communicated to application teams.

Prerequisites

  • Administrator access to the Admin Dashboard, with project and data-plane operations permissions.
  • A project with at least one gateway attached and ready. See Provision a gateway for a project.
  • Control of the DNS zone that clients resolve, and a DNS provider that supports health-checked records (Route 53 failover or weighted records, or equivalent).

Concepts: a project is a set of like-configured gateways

Identical by construction

A project is the single configuration authority; its attached gateways are interchangeable members. Every member receives the identical project configuration — authentication and authorization, guardrails, and the model and MCP catalogs — generated from the same project state. Per-member variance is limited to identity, URL, name, and labels. Divergence of desired configuration between members is not reconciled after the fact; it is unrepresentable — there is no per-member override surface at all. Delivery, though, is by data-plane-initiated polling: at any instant the guarantee is identical desired configuration plus a bounded convergence window, during which members can serve different revisions of the same project configuration. Each member's configuration version and sync state are surfaced on the Gateways tab.

For security review this means the project, not the gateway, is the unit of review: no failover event can reach a boundary that was never part of the project's configuration, because there is no member on which a different rule set could exist. Reviewing the project's policy once covers every member it will ever have, including members attached after the review. The convergence window is the one nuance: after a security-tightening change, a client failing over to a member still catching up can observe the previous (looser) boundary until that member converges. Confirm every member reports the new configuration version before treating a tightened boundary as enforced fleet-wide — a check that is meaningful only for attached, reachable members with management-plane connectivity. Treat an empty configuration-version column as convergence unconfirmed, not as converged.

Membership states

Each member is in exactly one state, shown in the Status column of the project's Gateways tab:

StateMeaningOperational significance
syncingThe member is converging on the project's current configuration version.A freshly attached member that has never converged serves nothing and probes red — it cannot receive traffic prematurely. A previously ready member that is catching up to a config change keeps serving its last-known-good configuration and probes green.
readyThe member has confirmed convergence on the project's current configuration version.A valid traffic and failover target. Advertised in the project view with its configuration version and last-seen time.
detached — zeroizedDetach completed and the gateway acknowledged the final, emptied configuration: routes, credentials, and catalogs removed.Teardown is confirmed. The listener may still be up, but project traffic gets no-route 404s and the probe is red. Nothing further is required.
detached — unreachableDetach was requested but the gateway never acknowledged the zeroized configuration; the management plane revoked its identity as the backstop.The gateway can no longer fetch project configuration or authenticate to the management plane, but its local teardown is unconfirmed — a partitioned-but-reachable gateway may still be serving its last-known configuration and answering its probe green. Remove its DNS record manually and verify decommissioning (see the runbook below).

A member is promoted from syncing to ready only on confirmed convergence, so no client ever reaches a half-configured gateway: until a new member is ready, its probe stays red and health-checked DNS never routes to it.

Zero gateways is a valid state

A project with zero attached gateways is a fully valid, fully configured project — keys, models, guardrails, and members are all retained. The Gateways tab shows a "nothing serving" state, which is a description, not an error. Use it deliberately:

  • Stage before capacity exists. Configure a project completely — policy reviewed, keys issued — before any gateway is attached. Attaching the first gateway later materializes the already-reviewed configuration.
  • Drain to zero safely. Detaching the last member leaves the project intact and re-attachable; nothing about the project is torn down with its capacity.

Per-data-plane request attribution

Every request is attributed to the data plane that served it — the reporting data plane's identity, stamped server-side when its telemetry reaches the management plane — and the usage and request-log surfaces are filterable by data plane. Attribution is data-plane granularity, not per-gateway: a data plane can host gateways from several projects, and gateways co-located on one data plane share an attribution bucket. This is what makes failover auditable and drains verifiable:

  • An audit trail survives failover — the record shows which data plane served each request, not just which project. When each member runs on its own data plane, that resolves the record to the member.
  • An incident can be localized: filter usage or request logs by data plane and compare the members' data planes.
  • A drain can be verified empirically before a detach — this is the backbone of the runbook below — provided the member's data plane hosts no other serving gateway, from any project. A shared data plane's counts include every co-located gateway; give a member a dedicated data plane if drain verification matters.

Two caveats. Attribution follows the data plane: migrating a gateway to a different data plane splits its request history across the two data-plane buckets. And requests recorded before data-plane attribution existed appear under an unattributed "" bucket rather than any data plane's.

The membership probe endpoint

Every member gateway exposes a health endpoint whose meaning is ready member of this project — not process liveness. This is the endpoint DNS health checks must target.

Provenance

This section is condensed from the normative probe contract in the Agent Router data-plane documentation — docs/project-gateways.md, "Membership probe endpoint", in the fraser repository at commit 48f0b01f2 (multi-gateway projects). The endpoint, the green/red definition, and the state table are quoted verbatim; the properties are summarised. Where this page and the normative contract disagree, the contract wins.

The endpoint, on the member's own hostname:

GET https://<gateway-hostname>/healthz/membership

The scheme, host, and port are the member's URL exactly as configured on the gateway — the same listener your clients use. No separate port, no admin interface. Standard HTTPS port (443) only: hostname matching for the gateway's host-stamped routes (the probe included) is port-exact, and health checkers send Host: host:port for a non-standard port — so a member URL on a non-standard port is outside this contract. Front a non-443 listener with a standard-port endpoint before pointing DNS health checks at it.

Green and red, exactly as specified:

  • Green (healthy): HTTP 200 with body {"status":"ready","gateway":"<gateway-uuid>"} (Content-Type: application/json). The UUID identifies which member answered — useful when DNS fronts several members.
  • Red (unhealthy): any non-2xx response — concretely HTTP 404 — or a connection failure/timeout. There is no "degraded" body; absence of the green answer is the signal.

Semantics: "ready member of this project" — not process liveness. The probe is green if and only if the gateway currently holds the project's materialized, non-zeroized configuration:

Membership stateProbeWhy
ready (config synced)200 greenThe probe route is materialized together with the member's project routes.
syncing, never synced (fresh attach)404 redNothing materialized yet — the probe route does not exist until the first successful config rollout.
syncing, previously synced (catching up to new config)200 greenThe member still serves its last-known-good config; it remains a valid failover target.
detached — zeroized404 redDetach removes the member's config and its probe route; the listener stays up, so client traffic gets no-route 404s while the probe goes red and DNS health checks drain the member.
detached — unreachable (from the DP's side)red once the zeroize lands or the gateway is down — but may stay GREEN while the gateway is partitioned from the management plane yet still client-reachableThe probe is data-plane-local configuration; a zeroize that never reaches the DP cannot turn it red. DNS will NOT drain this member automatically: remove the member's DNS record manually and verify decommissioning (ADR 052's detached-unreachable runbook).
Gateway/listener downred (connection failure)Handled by the health checker's own timeout.

The properties behind the table, summarised from the contract:

  • Answered by the data plane itself. The probe is pure gateway configuration (a native direct-response route) — no management-plane round-trip, no backend, no AI processing. It keeps answering during a full management-plane outage; a dashboard or management-plane failure can never fail your DNS health checks. The flip side is the detached-unreachable row above: because the probe is data-plane-local, a zeroize that never reaches the gateway cannot turn its probe red.
  • Red by construction on delivered teardown. The probe route exists only as part of the member's materialized project configuration and is removed with it when the zeroized configuration lands — there is no separate health state machine that can drift out of sync with what the gateway actually serves.
  • Host-scoped to the member. The probe answers only on the member's configured hostname — never on another member's hostname sharing the same data plane. A member with no URL configured has no probe (red; it cannot be in a DNS rotation anyway). On a hostname change, the probe on the old hostname goes red as soon as the new configuration rollout begins, and re-greens on the new hostname once that rollout is ready — a stalled rollout can never leave the old hostname green.
  • Unauthenticated for GET/HEAD only. Quoting the contract's security claim precisely: "DNS health checkers cannot send API keys, so the data plane exempts exactly GET/HEAD on this path from gateway API-key authentication. Any other method — and any other path — still requires an API key (401 otherwise), so the probe cannot be used as an unauthenticated request surface."
  • What the probe does not tell you: provider/upstream health, spend or rate-limit state, or per-request data-path success. A ready member of a project with no assigned models still probes green — it is faithfully serving its (empty) configuration. The probe answers "is this a ready member", nothing more.

Two consequences worth calling out:

  • Zeroized members drain themselves; unreachable ones do not. When a detach completes — the gateway acknowledges the zeroized configuration — the probe route is removed with the config, the health check fails, and any health-checked DNS record pointing at the member goes unhealthy automatically. But a member that lands in detached — unreachable while still client-reachable (partitioned from the management plane, gateway process up) keeps answering its probe green: DNS will not drain it. This is why the runbook below shifts DNS first — the DNS shift is the drain; the probe going red is the automatic backstop only for the zeroized terminal.
  • A newly attached member cannot receive traffic early. Its probe stays red until first convergence, so health-checked DNS never routes to it half-configured.

Verify a member by hand at any time:

$ curl -s https://ai-gw-usw2.payments.examplebank.com/healthz/membership
{"status":"ready","gateway":"1c9f4a6e-8d21-4f0b-9b3a-5e7d2c840a17"}

Reference pattern: DNS-level failover

The pattern is: one client-facing DNS name the operator owns, resolving to member gateway URLs, with per-member health checks on the membership probe. The examples use Route 53; any DNS provider with health-checked failover or weighted records works the same way.

The running example is a project payments-assist with two ready members:

GatewayURLData plane
gw-use1-paymentshttps://ai-gw-use1.payments.examplebank.comdp-prod-use1
gw-usw2-paymentshttps://ai-gw-usw2.payments.examplebank.comdp-prod-usw2

Clients call one name: ai.payments.examplebank.com. That name is the operator's artifact: the operator creates it, owns it, and Agent Router never resolves or routes on it. Agent Router can be told the name so it shows developers the right URL — see Step 3 — but that is a label, not configuration.

Whatever produces the name is the operator's choice. This guide's example is a health-checked failover record, which is the common case; a load balancer or global traffic manager fronting the members works identically as far as Agent Router is concerned, because Agent Router is not involved either way.

Step 1: Create a health check per member

For each member, create a health check against the membership probe on the member's own hostname:

  1. In Route 53, create an HTTPS health check with domain ai-gw-use1.payments.examplebank.com, port 443, and path /healthz/membership. Count only 2xx as healthy, and enable string matching on "status":"ready" — an edge-level redirect (3xx) must never be mistaken for green.
  2. Repeat for ai-gw-usw2.payments.examplebank.com.
  3. Route 53 knobs: fast interval (10s; standard is 30s) and failure threshold 3; the request timeout is fixed by Route 53, not configurable. Expect roughly 30 seconds to mark a member unhealthy, plus the record TTL for clients to drain to the survivor.

The health check must target the member's standard HTTPS port: the probe contract is 443-only (see above), so front any non-443 listener with a standard-port endpoint before health-checking it.

Check the member's hostname directly, never the shared client-facing name — a health check on ai.payments.examplebank.com would follow the failover itself and observe nothing useful.

On providers other than Route 53, use the tightest check interval offered, require 2xx (never 2xx/3xx), and enable response-body matching on "status":"ready" where supported.

Step 2: Create the failover records

Active/passive with Route 53 failover routing:

  1. Create a record ai.payments.examplebank.comai-gw-use1.payments.examplebank.com, routing policy Failover, type Primary, associated with the gw-use1-payments health check.
  2. Create a second record ai.payments.examplebank.comai-gw-usw2.payments.examplebank.com, routing policy Failover, type Secondary, associated with the gw-usw2-payments health check.
  3. Keep the record TTL short (30--60 seconds): the TTL, not the health check, usually dominates total failover time for clients that respect it.

Active/active is the same shape with Weighted records instead — one weighted, health-checked record per member — and degrades to the surviving members when a health check fails.

TLS note: clients connect to ai.payments.examplebank.com but the record resolves to a member; the certificate presented on each member's listener must cover the name clients dial (a SAN for the client-facing name, or terminate TLS at an edge the operator controls). This is edge infrastructure the operator owns, like the DNS name itself.

Step 3: Tell developers the name

Nothing so far tells developers the name exists. Left alone, the developer console shows them one member's URL — because until now that is the only URL it has — and an application that hardcodes it is pinned to that member and will not fail over. Everything in Steps 1 and 2 is then bypassed by the client.

Set the project's endpoint URL to the client-facing name:

  1. In the admin dashboard, open the project, go to the Gateways tab, and use the Project endpoint card.
  2. Enter the name clients call — https://ai.payments.examplebank.com — and save.

The developer console then shows that name as the project's Base URL, in code samples, in API-key setup, and in generated client configuration, in place of any member URL. Clearing the field reverts to showing a member URL.

Two things this field is not:

  • It is not routing configuration. Agent Router stores it and displays it. It does not resolve it, does not health-check it, and it never reaches a gateway: routes, authorization, and the membership probes are all still keyed on member hostnames, exactly as in Steps 1 and 2. Setting it changes nothing about how traffic is served.
  • It is not verified. It names infrastructure the operator owns and Agent Router cannot reach, so the only check applied is that the value is a well-formed http:// or https:// URL. If the record is wrong, Agent Router will faithfully show the wrong URL. The console labels it as administrator-provided for exactly this reason.

Because Agent Router does not route on the name, the members must accept it themselves.

Members accept it already. Each member gateway runs in its own cluster, as its own install, and a gateway that is the only one in its install serves its routes without a hostname match — it answers on whatever Host arrives. A client dialling ai.payments.examplebank.com reaches whichever member DNS resolved to and is served normally. Authorization is scoped the same way. This is what makes the whole pattern work: no edge rewriting, no per-name configuration on the Agent Router side, nothing to enable.

That leaves one genuine operator-side requirement: TLS.

Clients dial the client-facing name, so whatever terminates TLS in front of each member must present a certificate valid for it. Health checks dial the member's own hostname, so that name must be covered too — two names per member, from two different callers. How much work this is depends entirely on how the certificates are named:

  • A wildcard on the shared domain (*.payments.examplebank.com) covers every member hostname and the client-facing name at once. Adding a member later is DNS and an attach; no certificate work on the existing members. This is the shape to aim for.
  • An explicit domain list means every new member requires a new certificate for each load balancer whose certificate must now also carry the client-facing name. Managed certificates are generally immutable — a name cannot be added to an issued certificate, only reissued.
Decide the client-facing name before the second member exists

Under an explicit domain list, adding the shared name to each member's certificate after the fact is a reissue per member, on your PKI's timeline, while the failover configuration is half-built. Getting the name onto every certificate up front makes it one coordinated change — and it has to be decided anyway, since Step 3 publishes it to developers.

Certificates are also typically regional: each member's load balancer needs one issued in its own region, so even a wildcard is requested once per region rather than shared across them.

Where TLS terminates depends on how the data plane is exposed — for a self-hosted deployment on AWS, see Gateway installation on EKS.

Health checks still target member hostnames, never the client-facing name — Step 1 already covers why. The membership probe is host-scoped to the member's own hostname even though routes are not, so probing a member always tells you about that member specifically.

One-gateway-per-cluster is the shape this assumes

Members are clones of one another in separate clusters — that is what makes them failover targets, and it is what the managed platform installs. Two gateways sharing a single install is not a failover topology (a lost cluster takes both), and in that shape each gateway's routes become matched to its own hostname, so a request carrying the client-facing name would be rejected. Keep one gateway per cluster and the question does not arise.

Running one gateway per cluster is also what makes drain verification authoritative — see Per-data-plane request attribution.

Step 4: Confirm the failover semantics

What this pattern gives, and what it deliberately does not:

  • A newly attached member cannot receive traffic early. Its probe is red until first convergence, so its record stays unhealthy — attach order and DNS order do not need coordinating.
  • A member detached to zeroized drains automatically. The delivered zeroize removes the probe route; the health check fails; DNS steers away. A member that lands in detached — unreachable does not self-drain — its probe can stay green while the gateway is partitioned from the management plane but still client-reachable — which is one more reason the runbook below shifts DNS before detaching.
  • A member catching up on a config change keeps serving. syncing after a previous sync probes green and serves last-known-good configuration — a routine config rollout never triggers failover.
  • Failover is invisible to clients beyond the URL resolution. API keys, model access, MCP access, and guardrail behaviour are identical on every converged member, by construction; a member catching up on a rollout serves the project's last-known-good revision until it converges (see the convergence note under Identical by construction). See the rate-limit limitation below for the one per-member exception.
  • No steering from Agent Router. If the health checks, records, or TTLs are wrong, Agent Router will not compensate. Test the failover (take the primary's data plane offline in a rehearsal window and watch the secondary take traffic) before relying on it.

Richer alarm-driven variants — for example a CloudWatch alarm on the health-check status feeding notifications, or a Lambda that shifts weights on custom conditions — layer on top of the same probe without any change on the Agent Router side.

Runbook: detach a member gateway

Detaching is how a member is decommissioned: the member is actively emptied — routes, credentials, and catalogs removed — so a decommissioned gateway cannot keep serving project traffic once the emptied configuration lands. The detached — unreachable terminal below is the exception that keeps verification on the operator. Detach is not delete: the gateway record survives and the project is untouched. Run the steps in this order; each verifies the one before it.

Step 1: Shift DNS away from the member (act)

  1. In the DNS provider, remove the member's record, zero its weight, or demote it from the failover set, so ai.payments.examplebank.com no longer resolves to it.
  2. Wait out the record TTL so cached resolutions expire.

Do this before detaching. A detach that completes to zeroized would drain DNS by itself via the probe, but during the health-check detection window clients would receive no-route 404s from the zeroized member — and a detach that lands detached — unreachable would not drain DNS at all (its probe can stay green). Shifting DNS first makes the detach invisible and is the only drain that works in both terminals.

Step 2: Verify the drain through attribution (verify)

Request attribution is per data plane, not per gateway (see Per-data-plane request attribution). This verification is authoritative only when the member being detached has a dedicated data plane — one hosting no other serving gateway from any project. On a shared data plane the counts include every co-located gateway; fall back to the organisation's own edge metrics for the member's hostname.

  1. In the Gateways tab, start the detach on the member's row and read the confirmation dialog's traffic warning before confirming. It shows the request count for the member's data plane over the trailing hour — "gateway's data plane served N requests in the last hour" — and notes that counts are per data plane: gateways sharing a data plane share a count.

  2. Mind the window: the count covers a fixed trailing 60 minutes, so for up to an hour after the DNS shift it still includes pre-shift traffic. A non-zero count immediately after Step 1 does not mean the drain failed. Either wait out the window (TTL expiry plus the remainder of the hour) before treating the count as a verdict, or query the request-log stats surface with an explicit start_time after the TTL expired:

    $ curl -sS "https://management.example.com/v1/request-logs/stats?group_by=dataplane&metric=count&start_time=2026-07-22T14:30:00Z" \
    -H "Authorization: Bearer ${ADMIN_API_KEY}"

    Each entry is keyed by the reporting data plane's identity; add dataplane_id=<data-plane-id> to filter to the member's data plane. Requests recorded before data-plane attribution existed group under the empty key "" — treat that bucket as unattributable, never as the member's.

  3. The dialog count is best-effort. If it reads that recent traffic "couldn't be determined", the stats surface was unavailable — treat the drain as unverified and verify via the stats query above (or edge metrics) before detaching.

  4. A count that stays non-zero after the window has passed, on a dedicated data plane, means Step 1 is incomplete — stop and find who is still calling the member URL directly (stragglers with cached DNS or hardcoded member URLs show up here and nowhere else).

Step 3: Detach the member (act)

  1. In the Gateways tab, choose Detach on the member's row and confirm.
  2. The member leaves the ready set; a final, emptied configuration revision is made available to the gateway, and the management plane revokes the member's identity so it can never resync, rejoin, or fetch future configuration. Step 4 spells out exactly what that revocation does and does not stop.

Detach is idempotent: confirming a detach on a member that is already detached reports that it was already detached and changes nothing.

Step 4: Interpret the terminal state (verify)

The member lands in one of two terminal states, shown on its recently-detached row:

  • detached — zeroized — the gateway acknowledged the emptied configuration. Teardown is confirmed: routes, credentials, and catalogs are removed from the runtime; its probe is red; project traffic to it gets no-route 404s. Nothing further is required before decommissioning the runtime.
  • detached — unreachable — the gateway never acknowledged. The management plane has revoked the gateway's identity, so it can never resync, rejoin, or fetch future configuration — but its local state is unconfirmed, and revocation cannot reach a gateway that does not poll: client API keys are validated in-process on the gateway against its last-synced material, and the provider credentials embedded in that material remain usable until rotated at the provider. Assume a partitioned-but-reachable gateway is still serving its last-known configuration to anyone who can reach its URL directly — and still answering its membership probe green, so DNS health checks will not drain it automatically. Close the gap manually: confirm the member's DNS record is removed (Step 1 already did this if the runbook ran in order), verify through the hosting infrastructure that the runtime is actually down (or its last-known config torn down), and rotate the project's provider credentials if the host cannot be verified. Only then treat the member as decommissioned.

Documented limitation: rate limits are per-gateway

The identical-boundary guarantee covers authentication, authorization, guardrails, and catalogs. It does not cover rate state: token allowances are enforced per gateway, and counters are not shared between members.

Concretely: a client that fails over from gw-use1-payments to gw-usw2-payments arrives with fresh token-per-hour allowances on the new member. A workload near its allowance on one member effectively doubles its headroom by moving; during an active/active weighted setup, a project's aggregate consumption can approach the per-gateway allowance multiplied by the member count.

Budgets are different: they are evaluated at the management plane against fleet-aggregated request-log spend — accounting accrues as logs reach the management plane, not in real time on the request path — so membership neither fragments nor resets budget accounting.

What to do about it today:

  • Size per-gateway token allowances with the member count in mind if aggregate consumption is the concern.
  • Use per-data-plane attribution to watch aggregate consumption across members after a failover event (one data plane per member makes the buckets per-member).
  • Tell application teams that a failover resets their rate-limit window — relevant to any client-side backoff logic keyed to limit responses.

Treat token allowances as a per-member property when sizing and when reasoning about failover.

Behaviour during a management-plane outage

Member gateways are autonomous with respect to the management plane:

  • Members keep serving. Each member continues serving its last-known configuration while the management plane is unreachable. Data-path traffic does not depend on the management plane.
  • Probes stay green. The membership probe is answered by the gateway's own data plane with no management-plane round-trip, so DNS health checks — and therefore the failover pattern above — keep working through a full management-plane outage.
  • Detach effects are deferred — and do not drain via the probe. A detach issued while a member cannot reach the management plane takes effect on reconnect: the zeroized configuration lands and identity revocation bites when the gateway next contacts the management plane. Until then the member keeps serving its last-known configuration and its membership probe stays green — DNS health checks will not steer away from it. Shift DNS away from the member first, exactly as the runbook orders, and treat the member as detached — unreachable with the manual-verification posture until reconnect.

What to do next