# Gateway sets and DNS-level failover > Run a Tetrate Agent Router project on a set of like-configured gateways and fail traffic over between them with customer-managed DNS health checks. Enterprise Tier A project can attach more than one gateway, and every attached gateway serves the identical project configuration by construction. That turns a set of member gateways into interchangeable failover targets: point customer-managed DNS at the member URLs, health-check each member's membership probe endpoint, and a client that moves from one member to another encounters the same authentication, guardrails, and catalogs it left behind. This guide covers the membership model, the probe contract, a reference DNS failover pattern, and the runbook for detaching a member safely. :::info Availability Multi-gateway projects require an Agent Router Enterprise release that includes the multi-gateway projects capability, and the deployment itself must declare the Enterprise product line (`DEPLOYMENT_PRODUCT=enterprise`, derived from the installation charts; the gate fails closed, so a deployment that does not declare it behaves as Agent Router Service). On earlier releases a project maps to exactly one gateway and this guide does not apply. This surface does not exist on Agent Router Service (SaaS). ::: Agent Router provides the consistency guarantee, the membership lifecycle, per-member visibility, and per-data-plane request attribution. It ships **no traffic steering**: there is no built-in load balancer, GSLB, or failover executor in front of a project's gateways. Failover runs entirely under the operator's control, at the DNS (or equivalent routing) layer the operator already owns. This is deliberate — the DNS name clients call is the operator's artifact, and the failover policy (active/passive, weighted, geo) is an infrastructure decision Agent Router does not make. **Persona:** Platform operator working in the Admin Dashboard and the organisation's DNS provider (the examples use AWS Route 53). **Estimated time:** 30 to 45 minutes to set up the reference pattern for one project; 10 to 15 minutes to run the detach runbook. ## Outcomes By the end of this guide: - The membership model is understood: like-configured members, the four membership states, the zero-gateway state, and per-data-plane request attribution. - Customer-managed DNS failover is configured against the member URLs, health-checked on the membership probe endpoint. - The project's endpoint URL is set, so developers are given the client-facing name rather than a single member's URL. - A member has been detached safely: DNS shifted, drain verified through attribution, and the terminal state interpreted correctly. - The one documented limitation — per-gateway rate limits — is understood and communicated to application teams. ## Prerequisites - Administrator access to the Admin Dashboard, with project and data-plane operations permissions. - A project with at least one gateway attached and ready. See [Provision a gateway for a project](/agent-router-enterprise/guides/operate-and-govern/project-and-data-plane-management/provision-a-gateway/). - Control of the DNS zone that clients resolve, and a DNS provider that supports health-checked records (Route 53 failover or weighted records, or equivalent). ## Concepts: a project is a set of like-configured gateways ### Identical by construction A project is the single configuration authority; its attached gateways are interchangeable members. Every member receives the identical project configuration — authentication and authorization, guardrails, and the model and MCP catalogs — generated from the same project state. Per-member variance is limited to identity, URL, name, and labels. Divergence of *desired* configuration between members is not reconciled after the fact; it is unrepresentable — there is no per-member override surface at all. Delivery, though, is by data-plane-initiated polling: at any instant the guarantee is **identical desired configuration plus a bounded convergence window**, during which members can serve different revisions of the same project configuration. Each member's configuration version and sync state are surfaced on the **Gateways** tab. For security review this means the project, not the gateway, is the unit of review: no failover event can reach a boundary that was never part of the project's configuration, because there is no member on which a different rule set could exist. Reviewing the project's policy once covers every member it will ever have, including members attached after the review. The convergence window is the one nuance: after a security-*tightening* change, a client failing over to a member still catching up can observe the previous (looser) boundary until that member converges. Confirm every member reports the new configuration version before treating a tightened boundary as enforced fleet-wide — a check that is meaningful only for attached, reachable members with management-plane connectivity. Treat an empty configuration-version column as convergence **unconfirmed**, not as converged. ### Membership states Each member is in exactly one state, shown in the **Status** column of the project's **Gateways** tab: | State | Meaning | Operational significance | | :---- | :---- | :---- | | `syncing` | The member is converging on the project's current configuration version. | A freshly attached member that has never converged serves nothing and probes red — it cannot receive traffic prematurely. A previously ready member that is catching up to a config change keeps serving its last-known-good configuration and probes green. | | `ready` | The member has confirmed convergence on the project's current configuration version. | A valid traffic and failover target. Advertised in the project view with its configuration version and last-seen time. | | `detached — zeroized` | Detach completed and the gateway acknowledged the final, emptied configuration: routes, credentials, and catalogs removed. | Teardown is confirmed. The listener may still be up; the probe is red (a true no-route 404 — the probe path is exempt from gateway authorization). Client traffic on the drained hostname is rejected `403 unknown_host` if the data plane hosts any other serving gateway, and gets a no-route 404 only when it hosts none — expect 403s in client logs during a drain on a shared data plane. Nothing further is required. | | `detached — unreachable` | Detach was requested but the gateway never acknowledged the zeroized configuration; the management plane revoked its identity as the backstop. | The gateway can no longer fetch project configuration or authenticate to the management plane, but its local teardown is unconfirmed — a partitioned-but-reachable gateway may still be serving its last-known configuration and answering its probe green. Remove its DNS record manually and verify decommissioning (see the runbook below). | A member is promoted from `syncing` to `ready` only on confirmed convergence, so no client ever reaches a half-configured gateway: until a new member is ready, its probe stays red and health-checked DNS never routes to it. ### Zero gateways is a valid state A project with zero attached gateways is a fully valid, fully configured project — keys, models, guardrails, and members are all retained. The Gateways tab shows a "nothing serving" state, which is a description, not an error. Use it deliberately: - **Stage before capacity exists.** Configure a project completely — policy reviewed, keys issued — before any gateway is attached. Attaching the first gateway later materializes the already-reviewed configuration. - **Drain to zero safely.** Detaching the last member leaves the project intact and re-attachable; nothing about the project is torn down with its capacity. ### Per-data-plane request attribution Every request is attributed to the **data plane** that served it — the reporting data plane's identity, stamped server-side when its telemetry reaches the management plane — and the usage and request-log surfaces are filterable by data plane. Attribution is data-plane granularity, not per-gateway: a data plane can host gateways from several projects, and gateways co-located on one data plane share an attribution bucket. This is what makes failover auditable and drains verifiable: - An audit trail survives failover — the record shows which data plane served each request, not just which project. When each member runs on its own data plane, that resolves the record to the member. - An incident can be localized: filter usage or request logs by data plane and compare the members' data planes. - A drain can be verified empirically before a detach — this is the backbone of the runbook below — **provided the member's data plane hosts no other serving gateway, from any project**. A shared data plane's counts include every co-located gateway; give a member a dedicated data plane if drain verification matters. Two caveats. Attribution follows the data plane: migrating a gateway to a different data plane splits its request history across the two data-plane buckets. And requests recorded before data-plane attribution existed appear under an unattributed `""` bucket rather than any data plane's. ## The membership probe endpoint Every member gateway exposes a health endpoint whose meaning is *ready member of this project* — not process liveness. This is the endpoint DNS health checks must target. The endpoint, on the member's own hostname: ``` GET https:///healthz/membership ``` The scheme, host, and port are the member's URL exactly as configured on the gateway — the same listener your clients use. No separate port, no admin interface. **Standard HTTPS port (443) only:** hostname matching for the gateway's host-stamped routes (the probe included) is port-exact, and health checkers send `Host: host:port` for a non-standard port — so a member URL on a non-standard port is outside this contract. Front a non-443 listener with a standard-port endpoint before pointing DNS health checks at it. Green and red, exactly as specified: > - **Green (healthy):** HTTP `200` with body `{"status":"ready","gateway":""}` (`Content-Type: application/json`). The UUID identifies **which** member answered — useful when DNS fronts several members. > - **Red (unhealthy):** any non-2xx response — concretely HTTP `404` — or a connection failure/timeout. There is no "degraded" body; absence of the green answer **is** the signal. **Semantics: "ready member of this project" — not process liveness.** The probe is green if and only if the gateway currently holds the project's materialized, non-zeroized configuration: > | Membership state | Probe | Why | > |---|---|---| > | `ready` (config synced) | **200 green** | The probe route is materialized together with the member's project routes. | > | `syncing`, never synced (fresh attach) | **404 red** | Nothing materialized yet — the probe route does not exist until the first successful config rollout. | > | `syncing`, previously synced (catching up to new config) | **200 green** | The member still serves its last-known-good config; it remains a valid failover target. | > | detached — zeroized | **404 red** | Detach removes the member's config **and its probe route**; the listener stays up, so the probe goes red (the probe path is exempt from gateway authorization, so it is a true no-route 404) and DNS health checks drain the member. Client traffic on the drained hostname gets `403 unknown_host` when the data plane hosts any other serving gateway, and a no-route 404 only when it hosts none — expect 403s in client logs during the drain window on a shared data plane. | > | detached — unreachable (from the DP's side) | **red** once the zeroize lands or the gateway is down — but **may stay GREEN** while the gateway is partitioned from the management plane yet still client-reachable | The probe is data-plane-local configuration; a zeroize that never reaches the DP cannot turn it red. DNS will NOT drain this member automatically: remove the member's DNS record manually and verify decommissioning (see the detach runbook below). | > | Gateway/listener down | **red** (connection failure) | Handled by the health checker's own timeout. | The properties behind the table, summarised from the contract: - **Answered by the data plane itself.** The probe is pure gateway configuration (a native direct-response route) — no management-plane round-trip, no backend, no AI processing. It keeps answering during a full management-plane outage; a dashboard or management-plane failure can never fail your DNS health checks. The flip side is the detached-unreachable row above: because the probe is data-plane-local, a zeroize that never reaches the gateway cannot turn its probe red. - **Red by construction on delivered teardown.** The probe route exists only as part of the member's materialized project configuration and is removed with it when the zeroized configuration lands — there is no separate health state machine that can drift out of sync with what the gateway actually serves. - **Host-scoped to the member.** The probe answers only on the member's configured hostname — never on another member's hostname sharing the same data plane. A member with no URL configured has no probe (red; it cannot be in a DNS rotation anyway). On a hostname change, the probe on the **old** hostname goes red as soon as the new configuration rollout begins, and re-greens on the **new** hostname once that rollout is ready — a stalled rollout can never leave the old hostname green. - **Unauthenticated for `GET`/`HEAD` only.** Quoting the contract's security claim precisely: "DNS health checkers cannot send API keys, so the data plane exempts exactly `GET`/`HEAD` on this path from gateway API-key authentication. Any other method — and any other path — still requires an API key (`401` otherwise), so the probe cannot be used as an unauthenticated request surface." - **What the probe does not tell you:** provider/upstream health, spend or rate-limit state, or per-request data-path success. A ready member of a project with no assigned models still probes green — it is faithfully serving its (empty) configuration. The probe answers "is this a ready member", nothing more. - **The probe, not the console, is the DNS go-signal.** Probe-green can briefly trail the `ready` state shown on the Gateways tab (typically under a minute) — the console reports confirmed convergence while the probe route finishes materializing on the data plane. A member that shows `ready` with a still-red probe is converging, not broken; add it to a DNS rotation only once its probe answers green. Two consequences worth calling out: - **Zeroized members drain themselves; unreachable ones do not.** When a detach completes — the gateway acknowledges the zeroized configuration — the probe route is removed with the config, the health check fails, and any health-checked DNS record pointing at the member goes unhealthy automatically. But a member that lands in `detached — unreachable` while still client-reachable (partitioned from the management plane, gateway process up) keeps answering its probe green: DNS will **not** drain it. This is why the runbook below shifts DNS *first* — the DNS shift is the drain; the probe going red is the automatic backstop only for the zeroized terminal. - **A newly attached member cannot receive traffic early.** Its probe stays red until first convergence, so health-checked DNS never routes to it half-configured. Verify a member by hand at any time: ```bash $ curl -s https://ai-gw-usw2.payments.examplebank.com/healthz/membership {"status":"ready","gateway":"1c9f4a6e-8d21-4f0b-9b3a-5e7d2c840a17"} ``` ## Reference pattern: DNS-level failover The pattern is: one client-facing DNS name the operator owns, resolving to member gateway URLs, with per-member health checks on the membership probe. The examples use Route 53; any DNS provider with health-checked failover or weighted records works the same way. The running example is a project `payments-assist` with two ready members: | Gateway | URL | Data plane | | :---- | :---- | :---- | | `gw-use1-payments` | `https://ai-gw-use1.payments.examplebank.com` | `dp-prod-use1` | | `gw-usw2-payments` | `https://ai-gw-usw2.payments.examplebank.com` | `dp-prod-usw2` | Clients call one name: `ai.payments.examplebank.com`. That name is the operator's artifact: the operator creates it, owns it, and Agent Router never resolves or routes on it. Agent Router can be told the name so it shows developers the right URL — see [Step 3](#step-3-tell-developers-the-name) — but that is a label, not configuration. Whatever produces the name is the operator's choice. This guide's example is a health-checked failover record, which is the common case; a load balancer or global traffic manager fronting the members works identically as far as Agent Router is concerned, because Agent Router is not involved either way. ### Step 1: Create a health check per member For each member, create a health check against the membership probe on the member's own hostname: 1. In Route 53, create an **HTTPS** health check with domain `ai-gw-use1.payments.examplebank.com`, port `443`, and path `/healthz/membership`. Count **only 2xx** as healthy, and enable **string matching** on `"status":"ready"` — an edge-level redirect (3xx) must never be mistaken for green. 1. Repeat for `ai-gw-usw2.payments.examplebank.com`. 1. Route 53 knobs: **fast interval** (10s; standard is 30s) and **failure threshold 3**; the request timeout is fixed by Route 53, not configurable. Expect roughly **30 seconds** to mark a member unhealthy, plus the record TTL for clients to drain to the survivor. The health check must target the member's standard HTTPS port: the probe contract is 443-only (see above), so front any non-443 listener with a standard-port endpoint before health-checking it. Check the member's hostname directly, never the shared client-facing name — a health check on `ai.payments.examplebank.com` would follow the failover itself and observe nothing useful. On providers other than Route 53, use the tightest check interval offered, require 2xx (never 2xx/3xx), and enable response-body matching on `"status":"ready"` where supported. ### Step 2: Create the failover records Active/passive with Route 53 failover routing: 1. Create a record `ai.payments.examplebank.com` → `ai-gw-use1.payments.examplebank.com`, routing policy **Failover**, type **Primary**, associated with the `gw-use1-payments` health check. 1. Create a second record `ai.payments.examplebank.com` → `ai-gw-usw2.payments.examplebank.com`, routing policy **Failover**, type **Secondary**, associated with the `gw-usw2-payments` health check. 1. Keep the record TTL short (30 to 60 seconds): the TTL, not the health check, usually dominates total failover time for clients that respect it. Active/active is the same shape with **Weighted** records instead — one weighted, health-checked record per member — and degrades to the surviving members when a health check fails. TLS note: clients connect to `ai.payments.examplebank.com` but the record resolves to a member; the certificate presented on each member's listener must cover the name clients dial (a SAN for the client-facing name, or terminate TLS at an edge the operator controls). This is edge infrastructure the operator owns, like the DNS name itself. ### Step 3: Tell developers the name Nothing so far tells developers the name exists. Left alone, the developer console shows them one member's URL — because until now that is the only URL it has — and an application that hardcodes it is pinned to that member and will not fail over. Everything in Steps 1 and 2 is then bypassed by the client. Set the project's **endpoint URL** to the client-facing name: 1. In the admin dashboard, open the project, go to the **Gateways** tab, and use the **Project endpoint** card. 1. Enter the name clients call — `https://ai.payments.examplebank.com` — and save. The developer console then shows that name as the project's Base URL, in code samples, in API-key setup, and in generated client configuration, in place of any member URL. Clearing the field reverts to showing a member URL. Two things this field is **not**: - **It is not routing configuration.** Agent Router stores it and displays it. It does not resolve it, does not health-check it, and it never reaches a gateway: routes, authorization, and the membership probes are all still keyed on member hostnames, exactly as in Steps 1 and 2. Setting it changes nothing about how traffic is served. - **It is not verified.** It names infrastructure the operator owns and Agent Router cannot reach, so the only check applied is that the value is a well-formed `http://` or `https://` URL. If the record is wrong, Agent Router will faithfully show the wrong URL. The console labels it as administrator-provided for exactly this reason. Because Agent Router does not route on the name, the members must accept it themselves. **Members accept it already.** Each member gateway runs in its own cluster, as its own install, and a gateway that is the only one in its install serves its routes **without a hostname match** — it answers on whatever `Host` arrives. A client dialling `ai.payments.examplebank.com` reaches whichever member DNS resolved to and is served normally. Authorization is scoped the same way. This is what makes the whole pattern work: no edge rewriting, no per-name configuration on the Agent Router side, nothing to enable. That leaves one genuine operator-side requirement: **TLS**. Clients dial the client-facing name, so whatever terminates TLS in front of each member must present a certificate valid for it. Health checks dial the member's *own* hostname, so that name must be covered too — two names per member, from two different callers. How much work this is depends entirely on how the certificates are named: - **A wildcard on the shared domain** (`*.payments.examplebank.com`) covers every member hostname and the client-facing name at once. Adding a member later is DNS and an attach; **no certificate work on the existing members**. This is the shape to aim for — but it is only available when the client-facing name and the member hostnames live under the **same domain**. If the client-facing name is published from a different zone than the members (common when member hostnames sit in per-account infrastructure zones and the client-facing name sits in a corporate zone kept for certificate management), no single wildcard covers both and the multi-SAN shape below is the correct one, not a fallback. - **A multi-SAN certificate (explicit domain list)** carries each name explicitly: every member's certificate lists the member's own hostname *and* the client-facing name. Retrofitting the client-facing name onto existing members is a reissue per member — managed certificates are generally immutable, so a name cannot be added to an issued certificate, only reissued. Once every member's certificate carries the client-facing name, adding a member later costs exactly one new certificate (the new member's own); existing members are untouched. See the [worked example](#worked-example-acm-multi-san-certificates-on-albs) below. :::tip Decide the client-facing name before the second member exists Under an explicit domain list, adding the shared name to each member's certificate after the fact is a reissue per member, on your PKI's timeline, while the failover configuration is half-built. Getting the name onto every certificate up front makes it one coordinated change — and it has to be decided anyway, since [Step 3](#step-3-tell-developers-the-name) publishes it to developers. ::: Certificates are also typically **regional**: each member's load balancer needs one issued in its own region, so even a wildcard is requested once per region rather than shared across them. #### Worked example: ACM multi-SAN certificates on ALBs This example is the multi-SAN branch made concrete: TLS terminates at an AWS **Application Load Balancer** in front of each member (an `ingressClassName: alb` Ingress, as installed by the [AWS data-plane installation guide](/agent-router-enterprise/self-hosted-data-plane/data-plane-installation/installation-guide-aws/#appendix-a-tls-via-acm--https-listener)), certificates come from **AWS Certificate Manager (ACM)**, and the client-facing name lives in a *different* zone than the member hostnames, so a wildcard is not an option. The naming chain, extending the running example: ```none ai-payments.apps.examplebank.net client-facing FQDN, corporate zone (what clients dial) CNAME → ai.payments.examplebank.com Route 53 failover record (Step 2) → healthy member's hostname ai-gw-use1... (primary) / ai-gw-usw2... (secondary) CNAME → that member's regional ALB k8s-tarsdata-...us-east-1.elb.amazonaws.com ``` Which names each certificate must carry follows from who dials what. CNAME resolution never rewrites what the client sends: a client dialling `ai-payments.apps.examplebank.net` presents that name in SNI and `Host` no matter how many CNAME hops resolution took, so the ALB it lands on — **either** region's, depending on failover state — must present a certificate valid for it. Health checks dial the member's own hostname (Step 1), so that name must be on the member's certificate too. The intermediate Route 53 record name appears in neither SNI nor `Host`; it needs a SAN only if operators dial it directly. Per regional ALB, that is one ACM certificate with (at least) two SANs: | ALB | SANs required | | :---- | :---- | | us-east-1 (fronts `gw-use1-payments`) | `ai-gw-use1.payments.examplebank.com`, `ai-payments.apps.examplebank.net` | | us-west-2 (fronts `gw-usw2-payments`) | `ai-gw-usw2.payments.examplebank.com`, `ai-payments.apps.examplebank.net` | Request each in its own region — ACM certificates are regional, and an ALB can only use certificates from its region: ```shell aws acm request-certificate \ --region us-east-1 \ --domain-name ai-gw-use1.payments.examplebank.com \ --subject-alternative-names ai-payments.apps.examplebank.net \ --validation-method DNS ``` Repeat for `us-west-2` with the other member hostname. Every SAN must complete validation before ACM issues the certificate — including the shared client-facing name, in **each** region's request. Names served from a private-only zone cannot complete public DNS validation; publish the validation CNAMEs from a public zone, fall back to email validation, or import a certificate from your own PKI. This is a common reason the client-facing name is deliberately placed in a publicly validatable corporate zone while member hostnames stay in private infrastructure zones. Attach each certificate to its region's Ingress (`alb.ingress.kubernetes.io/certificate-arn`, per the [installation guide's TLS appendix](/agent-router-enterprise/self-hosted-data-plane/data-plane-installation/installation-guide-aws/#appendix-a-tls-via-acm--https-listener)). One more ALB-specific check: an ALB is layer 7 and routes by `Host`. A dedicated ALB whose Ingress has no `host:` rule forwards every name and needs no change, but if the Ingress uses host-based rules — typical when the ALB is shared — each member's Ingress must match **both** names, or failed-over clients get the ALB's 404 instead of the gateway: ```yaml rules: - host: ai-gw-usw2.payments.examplebank.com # health checks + operators http: { ... } - host: ai-payments.apps.examplebank.net # clients, whenever DNS points here http: { ... } ``` :::warning The failure hides until failover Route 53 health checks dial member hostnames and do **not** validate the certificate, and in steady state clients only ever reach the primary. A secondary whose certificate is missing the client-facing SAN therefore looks completely healthy — green checks, working member URL — until the first real failover, at which point every client gets a TLS hostname-mismatch error. Verify the SAN set on **every** member's certificate, secondary included, before the failover rehearsal: ```shell openssl s_client -connect :443 \ -servername ai-payments.apps.examplebank.net /dev/null \ | openssl x509 -noout -ext subjectAltName ``` Run it against each regional ALB hostname directly (this bypasses the failover record, so the secondary is reachable regardless of failover state) and confirm the output lists both the member hostname and the client-facing name. Then rehearse the failover itself, as Step 4 requires. ::: Adding a third member later is one new certificate — the new member's hostname plus the client-facing name, in the new member's region — and no reissue on existing members, provided the client-facing name went onto every certificate up front (the tip above). Where TLS terminates depends on how the data plane is exposed — for a self-hosted deployment on AWS, see [Gateway installation on EKS](/agent-router-enterprise/self-hosted-data-plane/gateway-installation/gateway-installation-guide-aws/). Health checks still target member hostnames, never the client-facing name — Step 1 already covers why. The membership probe is host-scoped to the member's own hostname even though routes are not, so probing a member always tells you about that member specifically. :::note One-gateway-per-cluster is the shape this assumes Members are clones of one another in separate clusters — that is what makes them failover targets, and it is what the managed platform installs. Two gateways sharing a single install is not a failover topology (a lost cluster takes both), and in that shape each gateway's routes become matched to its own hostname, so a request carrying the client-facing name would be rejected. Keep one gateway per cluster and the question does not arise. Running one gateway per cluster is also what makes drain verification authoritative — see [Per-data-plane request attribution](#per-data-plane-request-attribution). ::: ### Step 4: Confirm the failover semantics What this pattern gives, and what it deliberately does not: - **A newly attached member cannot receive traffic early.** Its probe is red until first convergence, so its record stays unhealthy — attach order and DNS order do not need coordinating. - **A member detached to `zeroized` drains automatically.** The delivered zeroize removes the probe route; the health check fails; DNS steers away. A member that lands in `detached — unreachable` does **not** self-drain — its probe can stay green while the gateway is partitioned from the management plane but still client-reachable — which is one more reason the runbook below shifts DNS before detaching. - **A member catching up on a config change keeps serving.** `syncing` after a previous sync probes green and serves last-known-good configuration — a routine config rollout never triggers failover. - **Failover is invisible to clients beyond the URL resolution.** API keys, model access, MCP access, and guardrail behaviour are identical on every converged member, by construction; a member catching up on a rollout serves the project's last-known-good revision until it converges (see the convergence note under [Identical by construction](#identical-by-construction)). See the rate-limit limitation below for the one per-member exception. - **No steering from Agent Router.** If the health checks, records, or TTLs are wrong, Agent Router will not compensate. Test the failover (take the primary's data plane offline in a rehearsal window and watch the secondary take traffic) before relying on it. Richer alarm-driven variants — for example a CloudWatch alarm on the health-check status feeding notifications, or a Lambda that shifts weights on custom conditions — layer on top of the same probe without any change on the Agent Router side. ## Runbook: detach a member gateway Detaching is how a member is decommissioned: the member is actively emptied — routes, credentials, and catalogs removed — so a decommissioned gateway cannot keep serving project traffic once the emptied configuration lands. The `detached — unreachable` terminal below is the exception that keeps verification on the operator. Detach is not delete: the gateway record survives and the project is untouched. Run the steps in this order; each verifies the one before it. ### Step 1: Shift DNS away from the member (act) 1. In the DNS provider, remove the member's record, zero its weight, or demote it from the failover set, so `ai.payments.examplebank.com` no longer resolves to it. 1. Wait out the record TTL so cached resolutions expire. Do this *before* detaching. A detach that completes to `zeroized` would drain DNS by itself via the probe, but during the health-check detection window clients would receive no-route 404s from the zeroized member — and a detach that lands `detached — unreachable` would not drain DNS at all (its probe can stay green). Shifting DNS first makes the detach invisible and is the only drain that works in both terminals. ### Step 2: Verify the drain through attribution (verify) Request attribution is per **data plane**, not per gateway (see [Per-data-plane request attribution](#per-data-plane-request-attribution)). This verification is authoritative only when the member being detached has a **dedicated data plane** — one hosting no other serving gateway from any project. On a shared data plane the counts include every co-located gateway; fall back to the organisation's own edge metrics for the member's hostname. 1. In the **Gateways** tab, start the detach on the member's row and read the confirmation dialog's traffic warning before confirming. It shows the request count for the member's data plane over the trailing hour — "*gateway*'s data plane served *N* requests in the last hour" — and notes that counts are per data plane: gateways sharing a data plane share a count. 1. Mind the window: the count covers a fixed trailing 60 minutes, so for up to an hour after the DNS shift it still includes pre-shift traffic. A non-zero count immediately after Step 1 does **not** mean the drain failed. Either wait out the window (TTL expiry plus the remainder of the hour) before treating the count as a verdict, or query the request-log stats surface with an explicit `start_time` after the TTL expired: ```bash $ curl -sS "https://management.example.com/v1/request-logs/stats?group_by=dataplane&metric=count&start_time=2026-07-22T14:30:00Z" \ -H "Authorization: Bearer ${ADMIN_API_KEY}" ``` Each entry is keyed by the reporting data plane's identity; add `dataplane_id=` to filter to the member's data plane. Requests recorded before data-plane attribution existed group under the empty key `""` — treat that bucket as unattributable, never as the member's. 1. The dialog count is best-effort. If it reads that recent traffic "couldn't be determined", the stats surface was unavailable — treat the drain as **unverified** and verify via the stats query above (or edge metrics) before detaching. 1. A count that stays non-zero after the window has passed, on a dedicated data plane, means Step 1 is incomplete — stop and find who is still calling the member URL directly (stragglers with cached DNS or hardcoded member URLs show up here and nowhere else). ### Step 3: Detach the member (act) 1. In the **Gateways** tab, choose **Detach** on the member's row and confirm. 1. The member leaves the ready set; a final, emptied configuration revision is made available to the gateway, and the management plane revokes the member's identity so it can never resync, rejoin, or fetch future configuration. Step 4 spells out exactly what that revocation does and does not stop. Detach is idempotent: confirming a detach on a member that is already detached reports that it was already detached and changes nothing. ### Step 4: Interpret the terminal state (verify) The member lands in one of two terminal states, shown on its recently-detached row: - **`detached — zeroized`** — the gateway acknowledged the emptied configuration. Teardown is **confirmed**: routes, credentials, and catalogs are removed from the runtime; its probe is red; project traffic to it gets no-route 404s. Nothing further is required before decommissioning the runtime. - **`detached — unreachable`** — the gateway never acknowledged. The management plane has revoked the gateway's identity, so it can never resync, rejoin, or fetch future configuration — but its local state is **unconfirmed**, and revocation cannot reach a gateway that does not poll: client API keys are validated in-process on the gateway against its last-synced material, and the provider credentials embedded in that material remain usable until rotated at the provider. Assume a partitioned-but-reachable gateway is still serving its last-known configuration to anyone who can reach its URL directly — **and still answering its membership probe green**, so DNS health checks will not drain it automatically. Close the gap manually: confirm the member's DNS record is removed (Step 1 already did this if the runbook ran in order), verify through the hosting infrastructure that the runtime is actually down (or its last-known config torn down), and rotate the project's provider credentials if the host cannot be verified. Only then treat the member as decommissioned. ## Documented limitation: rate limits are per-gateway The identical-boundary guarantee covers authentication, authorization, guardrails, and catalogs. It does **not** cover rate state: token allowances are enforced per gateway, and counters are not shared between members. Concretely: a client that fails over from `gw-use1-payments` to `gw-usw2-payments` arrives with **fresh token-per-hour allowances** on the new member. A workload near its allowance on one member effectively doubles its headroom by moving; during an active/active weighted setup, a project's aggregate consumption can approach the per-gateway allowance multiplied by the member count. Budgets are different: they are evaluated at the management plane against fleet-aggregated request-log spend — accounting accrues as logs reach the management plane, not in real time on the request path — so membership neither fragments nor resets budget accounting. What to do about it today: - Size per-gateway token allowances with the member count in mind if aggregate consumption is the concern. - Use per-data-plane attribution to watch aggregate consumption across members after a failover event (one data plane per member makes the buckets per-member). - Tell application teams that a failover resets their rate-limit window — relevant to any client-side backoff logic keyed to limit responses. Treat token allowances as a per-member property when sizing and when reasoning about failover. ## Behaviour during a management-plane outage Member gateways are autonomous with respect to the management plane: - **Members keep serving.** Each member continues serving its last-known configuration while the management plane is unreachable. Data-path traffic does not depend on the management plane. - **Probes stay green.** The membership probe is answered by the gateway's own data plane with no management-plane round-trip, so DNS health checks — and therefore the failover pattern above — keep working through a full management-plane outage. - **Detach effects are deferred — and do not drain via the probe.** A detach issued while a member cannot reach the management plane takes effect on reconnect: the zeroized configuration lands and identity revocation bites when the gateway next contacts the management plane. Until then the member keeps serving its last-known configuration and its membership probe **stays green** — DNS health checks will not steer away from it. Shift DNS away from the member first, exactly as the runbook orders, and treat the member as `detached — unreachable` with the manual-verification posture until reconnect. ## What to do next - **Provision a gateway for a project**: attach additional members, and the attach/detach mechanics, cap, and zero-gateway state. See [Provision a gateway for a project](/agent-router-enterprise/guides/operate-and-govern/project-and-data-plane-management/provision-a-gateway/). - **Migrate a gateway or upgrade its runtime**: move a member between data planes without changing its URL. See [Migrate a gateway or upgrade its runtime](/agent-router-enterprise/guides/operate-and-govern/project-and-data-plane-management/migrate-a-gateway/). - **Plan high availability and disaster recovery**: place gateway sets inside the broader HA/DR topology. See [Plan high availability and disaster recovery](/agent-router-enterprise/guides/operate-and-govern/plan-high-availability-and-disaster-recovery/). - **Load balance across regional deployments**: balancing *model backends* across regions, which composes with — and is distinct from — failover across *gateways*. See [Load balance across regional deployments](/agent-router-enterprise/guides/operate-and-govern/load-balance-across-regional-deployments/). Where to go next Provision a gateway for a project Attach members to a project, with the gateway cap and the zero-gateway state. Plan high availability and disaster recovery Place gateway sets inside the broader resilience topology.