Fail over between data planes

Enterprise Tier

From 0.7.0 two data planes in one organization can declare the same hostname, and a platform administrator can label both with the same project. Each data plane then serves the project's configuration on that name, and customer-managed DNS or a global server load balancer moves callers between them. Tetrate Agent Router ships no traffic steering: the DNS name, the health checks, and the failover policy stay the operator's.

---

Persona: Platform operator working in the Admin Console, the data planes' Helm values, and the organization's DNS provider (the examples use AWS Route 53).

Estimated time: 30 to 45 minutes for one pair, plus certificate issuance and DNS propagation.

Outcomes

By the end of this guide:

• Two data planes declare one client-facing hostname and each declares its own hostname for health checks.
• Both data planes carry the project's project label and report it verified.
• DNS fails callers over between the two data planes on the readiness probe, and developers are shown the client-facing name.

Prerequisites

• Two data planes on 0.7.0 or later, deployed and connected, typically in different regions or clusters. A data plane before 0.7.0 cannot declare hostnames.
• The project created, with its models granted.
• Control of the DNS zone that callers resolve, with health-checked failover or weighted records (Route 53 or an equivalent), and of the certificate each data plane's load balancer presents.

The running example uses project payments-assist, client-facing name ai.payments.examplebank.com, and two data planes:

Data plane	Own hostname	Region
dp-prod-use1	ai-use1.payments.examplebank.com	us-east-1
dp-prod-usw2	ai-usw2.payments.examplebank.com	us-west-2

Step 1: declare both names on each data plane

On each data plane, list the client-facing name first and the data plane's own name second. The console prints the first name wherever it shows a single address.

# dp-prod-use1 serve-helm values
global:
  gateway:
    hostnames:
      - ai.payments.examplebank.com
      - ai-use1.payments.examplebank.com

# dp-prod-usw2 serve-helm values
global:
  gateway:
    hostnames:
      - ai.payments.examplebank.com
      - ai-usw2.payments.examplebank.com

With the tetrate CLI, pass the same names as repeated --gateway-hostname flags to tetrate agentrouter dataplane upgrade. The management plane records the shared name for both data planes, and the Admin Console lists both under it. Another organization cannot claim the same name.

Each data plane's own name gives the health checks in Step 4 a target that reaches that data plane and no other. The full rules for the list are in Declare the hostnames a data plane serves.

Step 2: cover both names with each certificate

Callers dial the client-facing name and land on either data plane, and the health checks dial each data plane's own name. The certificate each load balancer presents must therefore cover two names:

Load balancer	Names the certificate must cover
In front of dp-prod-use1	ai.payments.examplebank.com, ai-use1.payments.examplebank.com
In front of dp-prod-usw2	ai.payments.examplebank.com, ai-usw2.payments.examplebank.com

A wildcard on *.payments.examplebank.com covers both. Otherwise request one certificate per region with both names as subject alternative names. Where the load balancer routes by host, add a rule for each name on each side, or a failed-over caller gets no route.

Route 53 health checks do not validate the certificate, and in steady state callers only reach the primary. A secondary missing the client-facing name looks healthy until the first failover. Check both sides before relying on the pair:

tetrate agentrouter doctor /path/to/dp-prod-usw2-credentials.json --only gateway

tetrate agentrouter doctor warns for a declared name the certificate does not cover. The Admin Console shows the same reading beside each name on the data plane's page.

Step 3: label both data planes with the project

1. In the Admin Console, open Directory → Data Planes and select the information icon at the far right of the dp-prod-use1 row.
2. Select Manage, toggle payments-assist on, confirm in the dialog, and select Done.
3. Repeat for dp-prod-usw2.

Each label shows three steps: label applied, config pulled (the revision the data plane holds and the one it picks up next), and verified (when the data plane first answered a request for that project). A data plane is a safe failover target once its label reads verified. The project's Data Planes tab shows the same progress per data plane.

The same labels can be applied with AddDataplaneProject (POST /admin/v1/customers/{customer_id}/dataplanes/{dataplane_id}/projects). Both data planes receive the identical project configuration: API keys, model access, Model Context Protocol (MCP) access, and guardrails behave the same on either side once both hold the current revision.

Step 4: health-check each data plane

Create one HTTPS health check per data plane, on its own name:

1. In Route 53, create an HTTPS health check with domain ai-use1.payments.examplebank.com, port 443, and path /healthz/membership.
2. Require a 2xx status, and turn on string matching for "status":"ready".
3. Use the fast interval (10 seconds) and a failure threshold of 3. Expect roughly 30 seconds to mark a data plane unhealthy, plus the record TTL for callers to move.
4. Repeat for ai-usw2.payments.examplebank.com.

/healthz/membership carries the readiness answer on every deployment. Bare /healthz carries the same answer only once global.publishHealthzContract is set on the data plane; before that it answers a host-blind liveness 200 with {"status":"ok"}, which the string match rejects. A green answer is:

curl -s https://ai-use1.payments.examplebank.com/healthz/membership

{"status":"ready","dataplane":"dp-prod-use1"}

Never health-check the client-facing name: the check would follow the failover itself and observe nothing about either data plane. The probe answers on port 443 only.

Step 5: create the failover records

1. Create a record ai.payments.examplebank.com pointing at dp-prod-use1's load balancer, routing policy Failover, type Primary, associated with the ai-use1 health check.
2. Create a second record ai.payments.examplebank.com pointing at dp-prod-usw2's load balancer, routing policy Failover, type Secondary, associated with the ai-usw2 health check.
3. Keep the TTL short, 30 to 60 seconds. The TTL usually decides how fast callers move.

Active/active is the same shape with Weighted records, one per data plane, each with its health check.

Step 6: show developers the client-facing name

The Developer Console lists every endpoint serving the project, so it lists ai.payments.examplebank.com and each data plane's own name. An application that hardcodes one data plane's own name does not fail over. Point developers at the shared name in one of two ways:

• A project owner stars ai.payments.examplebank.com on the project's Data Planes tab. The console then prints it as the Base URL on the getting-started card, beside a new key, and on the model catalog bar. This is the preferred endpoint.
• A platform administrator sets the project endpoint URL to https://ai.payments.examplebank.com through the admin API (public_endpoint_url on PATCH /admin/v1/customers/{customer_id}/projects/{project_id}). The console prints it in place of any star and offers no star while it is set. Agent Router displays this URL and never routes on it.

What the probe tells DNS on 0.7.0

On a data plane on 0.7.0 or later the readiness probe answers for the whole data plane on every hostname it declares, not for one project on it:

Data plane state	Probe
Declaring the name, carrying no projects	200 green
Carrying one or more projects	200 green
One project's label removed, others still served	200 green
Declaring an empty list	404 red
Gateway or listener down	Red (connection failure)
Partitioned from the management plane but reachable by callers	200 green

Removing the project's label from a data plane therefore does not take its address out of DNS. Drain first:

1. Remove the data plane's record from the failover set, or set its weight to zero, and wait out the TTL.
2. Check request logs filtered by that data plane until its count for the project falls to zero.
3. Remove the label from the data plane's more-info pop-out (Manage, toggle the project off, confirm). Removing the last label serving a project is refused unless the removal acknowledges that the project will be served nowhere.

To take a data plane out of rotation for every project at once, declare an empty list on it, which turns its probe red on the next reconcile.

Rate limits are per data plane

Each data plane counts only the traffic it sees. A project served from two data planes admits up to twice its configured token ceiling across the pair, and a caller that fails over meets a counter that has never seen it. Size per-key limits with the number of data planes in mind, and tell application teams that a failover resets their rate-limit window. Budgets are different: the management plane evaluates them against spend from every data plane. In an active/active weighted setup, a project served from N data planes can approach N times its configured allowance. Budget accounting accrues as request logs reach the management plane, not in real time on the request path, so membership neither fragments nor resets it. Per-data-plane attribution in usage and request logs shows aggregate consumption across the pair after a failover.

Data planes before 0.7.0

Before 0.7.0 a project is served through member gateways, each with its own hostname, and the steps below apply to those data planes. They need administrator access to the Admin Console with project and data-plane operations permissions, a project with at least one gateway attached and ready, and control of the DNS zone that clients resolve, with a DNS provider that supports health-checked records. Setting up the reference pattern for one project takes 30 to 45 minutes, and the detach runbook 10 to 15. The model they rely on, including the membership states, is in Projects, data planes, and hostnames.

Reference pattern: DNS-level failover

The pattern is: one client-facing DNS name the operator owns, resolving to member gateway URLs, with per-member health checks on the membership probe. The examples use Route 53; any DNS provider with health-checked failover or weighted records works the same way.

The running example is a project payments-assist with two ready members:

Gateway	URL	Data plane
gw-use1-payments	https://ai-gw-use1.payments.examplebank.com	dp-prod-use1
gw-usw2-payments	https://ai-gw-usw2.payments.examplebank.com	dp-prod-usw2

Clients call one name: ai.payments.examplebank.com. That name is the operator's artifact: the operator creates it, owns it, and Agent Router never resolves or routes on it. Agent Router can be told the name so it shows developers the right URL (see Step 3), but that is a label, not configuration.

Whatever produces the name is the operator's choice. This guide's example is a health-checked failover record, which is the common case; a load balancer or global traffic manager fronting the members works identically as far as Agent Router is concerned, because Agent Router is not involved either way.

Step 1: create a health check per member

For each member, create a health check against the membership probe on the member's own hostname:

1. In Route 53, create an HTTPS health check with domain ai-gw-use1.payments.examplebank.com, port 443, and path /healthz/membership, the path that carries the readiness answer on every release, whatever the deployment state. Switch the check to bare /healthz only once the health-check contract (global.publishHealthzContract, off by default) is enabled on that data plane; until then bare /healthz reports liveness, not readiness (see the note under On a data plane before 0.7.0). Count only 2xx as healthy, and enable string matching on "status":"ready": an edge-level redirect (3xx) must never be mistaken for green.
2. Repeat for ai-gw-usw2.payments.examplebank.com.
3. On a data plane on 0.7.0 or later, check a hostname that data plane alone declares, such as its regional name, so the check reports on that data plane rather than following failover. The body to match, "status":"ready", is the same on every release.
4. Route 53 knobs: fast interval (10s; standard is 30s) and failure threshold 3; the request timeout is fixed by Route 53, not configurable. Expect roughly 30 seconds to mark a member unhealthy, plus the record TTL for clients to drain to the survivor.

Check the member's hostname directly, never the shared client-facing name: a health check on ai.payments.examplebank.com would follow the failover itself and observe nothing useful.

On providers other than Route 53, use the tightest check interval offered, require 2xx (never 2xx/3xx), and enable response-body matching on "status":"ready" where supported.

Step 2: create the failover records

Active/passive with Route 53 failover routing:

1. Create a record ai.payments.examplebank.com → ai-gw-use1.payments.examplebank.com, routing policy Failover, type Primary, associated with the gw-use1-payments health check.
2. Create a second record ai.payments.examplebank.com → ai-gw-usw2.payments.examplebank.com, routing policy Failover, type Secondary, associated with the gw-usw2-payments health check.
3. Keep the record TTL short (30 to 60 seconds): the TTL, not the health check, usually dominates total failover time for clients that respect it.

Active/active is the same shape with Weighted records instead (one weighted, health-checked record per member) and degrades to the surviving members when a health check fails.

Clients connect to ai.payments.examplebank.com, but the record resolves to a member, so the certificate presented on each member's listener must cover the name clients dial. The TLS requirements are covered under Step 3.

Step 3: tell developers the name

Nothing so far tells developers the name exists. Left alone, the Developer Console shows them one member's URL (because until now that is the only URL it has), and an application that hardcodes it is pinned to that member and does not fail over. Everything in Steps 1 and 2 is then bypassed by the client.

Set the project endpoint URL to the client-facing name. The 0.7.0 Admin Console has no Project endpoint card, so the value is set over the admin API:

curl -sS -X PATCH "https://management.example.com/admin/v1/customers/${CUSTOMER_ID}/projects/payments-assist" \
  -H "Authorization: Bearer ${ADMIN_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{"public_endpoint_url": "https://ai.payments.examplebank.com"}'

On earlier releases the same value is set in the Admin Console: open the project, go to the Gateways tab, use the Project endpoint card, enter the name clients call, and save.

The Developer Console then shows that name as the project's Base URL, in code samples, in API-key setup, and in generated client configuration, in place of any member URL. Clearing the field reverts to the preferred endpoint a project owner starred, or else to the first address a serving data plane declares. On a data plane on 0.7.0 or later the client-facing name can be listed as a declared hostname as well, and then appears in the Developer Console's endpoint list without this field.

Two things this field is not:

• It is not routing. Agent Router stores it and displays it. It does not resolve it, does not health-check it, and it never reaches a gateway: routes, authorization, and the membership probes are all still keyed on member hostnames, exactly as in Steps 1 and 2. Setting it changes nothing about how traffic is served.
• It is not verified. It names infrastructure the operator owns and Agent Router cannot reach, so the only check applied is that the value is a well-formed http:// or https:// URL. If the record is wrong, Agent Router faithfully shows the wrong URL. The console labels it as administrator-provided for exactly this reason.

Because Agent Router does not route on the name, the members must accept it themselves.

Members accept it already. Each member gateway runs in its own cluster, as its own install, and a gateway that is the only one in its install serves its routes without a hostname match: it answers on whatever Host arrives. A client dialing ai.payments.examplebank.com reaches whichever member DNS resolved to and is served normally. Authorization is scoped the same way. This is what makes the whole pattern work: no edge rewriting, no per-name configuration on the Agent Router side, nothing to enable.

On data planes on 0.7.0 or later, list the client-facing name under global.gateway.hostnames on every member's data plane as well. Each data plane then answers its readiness probe on that name and reads the certificate serving it, and the console lists the name among the project's endpoints. See the steps above.

That leaves one operator-side requirement: TLS.

Clients dial the client-facing name, so whatever terminates TLS in front of each member must present a certificate valid for it. Health checks dial the member's own hostname, so that name must be covered too, two names per member, from two different callers. How much work this is depends entirely on how the certificates are named:

• A wildcard on the shared domain (*.payments.examplebank.com) covers every member hostname and the client-facing name at once. Adding a member later is DNS and an attach; no certificate work on the existing members. This is the shape to aim for, but it is only available when the client-facing name and the member hostnames live under the same domain. If the client-facing name is published from a different zone than the members (common when member hostnames sit in per-account infrastructure zones and the client-facing name sits in a corporate zone kept for certificate management), no single wildcard covers both and the multi-SAN shape below is the correct one, not a fallback.
• A multi-SAN certificate (explicit domain list) carries each name explicitly: every member's certificate lists the member's own hostname and the client-facing name. Retrofitting the client-facing name onto existing members is a reissue per member: managed certificates are generally immutable, so a name cannot be added to an issued certificate, only reissued. Once every member's certificate carries the client-facing name, adding a member later costs exactly one new certificate (the new member's own); existing members are untouched. See the worked example below.

Decide the client-facing name before the second member exists

Under an explicit domain list, adding the shared name to each member's certificate after the fact is a reissue per member, on the PKI's timeline, while the failover configuration is half-built. Getting the name onto every certificate up front makes it one coordinated change, and it has to be decided anyway, since Step 3 publishes it to developers.

Certificates are also typically regional: each member's load balancer needs one issued in its own region, so even a wildcard is requested once per region rather than shared across them.

Worked example: ACM multi-SAN certificates on ALBs

This example is the multi-SAN branch made concrete: TLS terminates at an AWS Application Load Balancer in front of each member (an ingressClassName: alb Ingress, as installed by the AWS data-plane installation guide), certificates come from AWS Certificate Manager (ACM), and the client-facing name lives in a different zone than the member hostnames, so a wildcard is not an option.

The naming chain, extending the running example:

ai-payments.apps.examplebank.net              client-facing FQDN, corporate zone (what clients dial)
  CNAME → ai.payments.examplebank.com         Route 53 failover record (Step 2)
    → healthy member's hostname               ai-gw-use1... (primary) / ai-gw-usw2... (secondary)
      CNAME → that member's regional ALB      k8s-tarsdata-...us-east-1.elb.amazonaws.com

Which names each certificate must carry follows from who dials what. CNAME resolution never rewrites what the client sends: a client dialing ai-payments.apps.examplebank.net presents that name in SNI and Host no matter how many CNAME hops resolution took, so the ALB it lands on (either region's, depending on failover state) must present a certificate valid for it. Health checks dial the member's own hostname (Step 1), so that name must be on the member's certificate too. The intermediate Route 53 record name appears in neither SNI nor Host; it needs a SAN only if operators dial it directly.

Per regional ALB, that is one ACM certificate with (at least) two SANs:

ALB	SANs required
us-east-1 (fronts gw-use1-payments)	ai-gw-use1.payments.examplebank.com, ai-payments.apps.examplebank.net
us-west-2 (fronts gw-usw2-payments)	ai-gw-usw2.payments.examplebank.com, ai-payments.apps.examplebank.net

Request each in its own region: ACM certificates are regional, and an ALB can only use certificates from its region:

aws acm request-certificate \
  --region us-east-1 \
  --domain-name ai-gw-use1.payments.examplebank.com \
  --subject-alternative-names ai-payments.apps.examplebank.net \
  --validation-method DNS

Repeat for us-west-2 with the other member hostname. Every SAN must complete validation before ACM issues the certificate, including the shared client-facing name, in each region's request. Names served from a private-only zone cannot complete public DNS validation; publish the validation CNAMEs from a public zone, fall back to email validation, or import a certificate from an internal PKI. This is a common reason the client-facing name is deliberately placed in a publicly validatable corporate zone while member hostnames stay in private infrastructure zones.

Attach each certificate to its region's Ingress (alb.ingress.kubernetes.io/certificate-arn, per the installation guide's TLS appendix). One more ALB-specific check: an ALB is layer 7 and routes by Host. A dedicated ALB whose Ingress has no host: rule forwards every name and needs no change, but if the Ingress uses host-based rules (typical when the ALB is shared), each member's Ingress must match both names, or failed-over clients get the ALB's 404 instead of the gateway:

rules:
  - host: ai-gw-usw2.payments.examplebank.com   # health checks + operators
    http: { ... }
  - host: ai-payments.apps.examplebank.net      # clients, whenever DNS points here
    http: { ... }

The failure hides until failover

Route 53 health checks dial member hostnames and do not validate the certificate, and in steady state clients only ever reach the primary. A secondary whose certificate is missing the client-facing SAN therefore looks completely healthy (green checks, working member URL) until the first real failover, at which point every client gets a TLS hostname-mismatch error. Verify the SAN set on every member's certificate, secondary included, before the failover rehearsal:

openssl s_client -connect <regional-alb-hostname>:443 \
  -servername ai-payments.apps.examplebank.net </dev/null 2>/dev/null \
  | openssl x509 -noout -ext subjectAltName

Run it against each regional ALB hostname directly (this bypasses the failover record, so the secondary is reachable regardless of failover state) and confirm the output lists both the member hostname and the client-facing name. Then rehearse the failover itself, as Step 4 requires.

Adding a third member later is one new certificate (the new member's hostname plus the client-facing name, in the new member's region) and no reissue on existing members, provided the client-facing name went onto every certificate up front (the tip above).

Where TLS terminates depends on how the data plane is exposed: for a self-hosted deployment on AWS, see Gateway installation on EKS.

The membership probe is host-scoped to the member's own hostname even though routes are not, so probing a member always reports on that member specifically.

One-gateway-per-cluster is the shape this assumes

Members are clones of one another in separate clusters; that is what makes them failover targets, and it is what the managed platform installs. Two gateways sharing a single install is not a failover topology (a lost cluster takes both), and in that shape each gateway's routes become matched to its own hostname, so a request carrying the client-facing name would be rejected. Keep one gateway per cluster and the question does not arise.

On data planes on 0.7.0 or later the same constraint is met without it: each data plane declares the client-facing name itself, and several projects share that name on one data plane through project labels rather than through separate gateways.

Running one gateway per cluster is also what makes drain verification authoritative; see Per-data-plane request attribution.

Step 4: confirm the failover semantics

What this pattern gives, and what it deliberately does not:

• A newly attached member cannot receive traffic early. Its probe is red until first convergence, so its record stays unhealthy: attach order and DNS order do not need coordinating.
• On a data plane before 0.7.0, a member detached to zeroized drains automatically. The delivered zeroize removes the probe route; the health check fails; DNS steers away. On a data plane on 0.7.0 or later the probe belongs to the data plane and stays green after a detach, so nothing drains until DNS is shifted. A member that lands in detached (unreachable) does not self-drain: its probe can stay green while the gateway is partitioned from the management plane but still client-reachable, which is one more reason the runbook shifts DNS before detaching.
• A member catching up on a config change keeps serving. syncing after a previous sync probes green and serves last-known-good configuration: a routine config rollout never triggers failover.
• Failover is invisible to clients beyond the URL resolution. API keys, model access, MCP access, and guardrail behavior are identical on every converged member, by construction; a member catching up on a rollout serves the project's last-known-good revision until it converges (see the convergence note under Identical by construction). See Rate limits are per data plane for the one per-member exception.
• No steering from Agent Router. If the health checks, records, or TTLs are wrong, Agent Router does not compensate. Test the failover (take the primary's data plane offline in a rehearsal window and watch the secondary take traffic) before relying on it.

Richer alarm-driven variants (for example a CloudWatch alarm on the health-check status feeding notifications, or a Lambda that shifts weights on custom conditions) layer on top of the same probe without any change on the Agent Router side.

Runbook: detach a member gateway

Detaching is how a member is decommissioned: the member is actively emptied (routes, credentials, and catalogs removed) so a decommissioned gateway cannot keep serving project traffic once the emptied configuration lands. The detached (unreachable) terminal below is the exception that keeps verification on the operator. Detach is not delete: the gateway record survives and the project is untouched. Run the steps in this order; each verifies the one before it.

Step 1: shift DNS away from the member (act)

1. In the DNS provider, remove the member's record, zero its weight, or demote it from the failover set, so ai.payments.examplebank.com no longer resolves to it.
2. Wait out the record TTL so cached resolutions expire.

Do this before detaching. On a data plane on 0.7.0 or later this step is the only drain: the readiness probe answers for the data plane and stays green after the detach, and clients still resolving to it receive 403 for the detached project. On an earlier data plane, a detach that completes to zeroized would drain DNS by itself via the probe, but during the health-check detection window clients would receive no-route 404s from the zeroized member, and a detach that lands detached (unreachable) would not drain DNS at all (its probe can stay green). Shifting DNS first makes the detach invisible and is the only drain that works in both terminals.

Step 2: verify the drain through attribution (verify)

Request attribution is per data plane, not per gateway (see Per-data-plane request attribution). This verification is authoritative only when the member being detached has a dedicated data plane, one hosting no other serving gateway from any project. On a shared data plane the counts include every co-located gateway; fall back to the organization's own edge metrics for the member's hostname.

1. In the Gateways tab, start the detach on the member's row and read the confirmation dialog's traffic warning before confirming. It shows the request count for the member's data plane over the trailing hour ("gateway's data plane served N requests in the last hour") and notes that counts are per data plane: gateways sharing a data plane share a count.

2. Mind the window: the count covers a fixed trailing 60 minutes, so for up to an hour after the DNS shift it still includes pre-shift traffic. A non-zero count immediately after Step 1 does not mean the drain failed. Either wait out the window (TTL expiry plus the remainder of the hour) before treating the count as a verdict, or query the request-log stats surface with an explicit start_time after the TTL expired:

   $ curl -sS "https://management.example.com/v1/request-logs/stats?group_by=dataplane&metric=count&start_time=2026-07-22T14:30:00Z" \
       -H "Authorization: Bearer ${ADMIN_API_KEY}"

   Each entry is keyed by the reporting data plane's identity; add dataplane_id=<data-plane-id> to filter to the member's data plane. Requests recorded before data-plane attribution existed group under the empty key "". Treat that bucket as unattributable, never as the member's.

3. The dialog count is best-effort. If it reads that recent traffic "couldn't be determined", the stats surface was unavailable. Treat the drain as unverified and verify via the stats query above (or edge metrics) before detaching.

4. A count that stays non-zero after the window has passed, on a dedicated data plane, means Step 1 is incomplete. Stop and find who is still calling the member URL directly (stragglers with cached DNS or hardcoded member URLs show up here and nowhere else).

Step 3: detach the member (act)

1. In the Gateways tab, choose Detach on the member's row and confirm.
2. The member leaves the ready set; a final, emptied configuration revision is made available to the gateway, and the management plane revokes the member's identity so it can never resync, rejoin, or fetch future configuration. Step 4 spells out exactly what that revocation does and does not stop.

Detach is idempotent: confirming a detach on a member that is already detached reports that it was already detached and changes nothing.

Step 4: interpret the terminal state (verify)

The member lands in one of two terminal states, shown on its recently-detached row:

• detached (zeroized): the gateway acknowledged the emptied configuration. Teardown is confirmed: routes, credentials, and catalogs are removed from the runtime; on a data plane before 0.7.0 its probe is red, while on 0.7.0 or later the data plane's probe stays green for its other projects; project traffic to it is refused. Nothing further is required before decommissioning the runtime.
• detached (unreachable): the gateway never acknowledged. The management plane has revoked the gateway's identity, so it can never resync, rejoin, or fetch future configuration, but its local state is unconfirmed, and revocation cannot reach a gateway that does not poll: client API keys are validated in-process on the gateway against its last-synced material, and the provider credentials embedded in that material remain usable until rotated at the provider. Assume a partitioned-but-reachable gateway is still serving its last-known configuration to anyone who can reach its URL directly, and is still answering its membership probe green, so DNS health checks do not drain it automatically. Close the gap manually: confirm the member's DNS record is removed (Step 1 already did this if the runbook ran in order), verify through the hosting infrastructure that the runtime is actually down (or its last-known config torn down), and rotate the project's provider credentials if the host cannot be verified. Only then treat the member as decommissioned.

Behavior during a management-plane outage

Member gateways are autonomous with respect to the management plane:

• Members keep serving. Each member continues serving its last-known configuration while the management plane is unreachable. Data-path traffic does not depend on the management plane.
• Probes stay green. The membership probe is answered by the gateway's own data plane with no management-plane round-trip, so DNS health checks (and therefore the failover pattern above) keep working through a full management-plane outage.
• Detach effects are deferred, and do not drain via the probe. A detach issued while a member cannot reach the management plane takes effect on reconnect: the zeroized configuration lands and identity revocation bites when the gateway next contacts the management plane. Until then the member keeps serving its last-known configuration and its membership probe stays green: DNS health checks do not steer away from it. Shift DNS away from the member first, exactly as the runbook orders, and treat the member as detached (unreachable) with the manual-verification posture until reconnect.

---

Where to go next

Projects, data planes, and hostnames

How a project, its data planes, and their hostnames fit together.

Readiness probe and the /healthz contract

What the probe answers on each release, and how to publish the readiness contract.

Declare the hostnames a data plane serves

The list, DNS, certificates, routes, and the certificate probe.

Plan high availability and disaster recovery

Place the pair inside the broader resilience topology.

Load balance across regional deployments

Balancing model backends across regions, which composes with (and is distinct from) failover across data planes.

Set rate limits on projects and keys

Size limits for a project served from several data planes.