Readiness probe and the /healthz contract

By default a self-hosted gateway answers /healthz with an unconditional controller-liveness 200. Enabling the presence-coupled readiness contract switches /healthz to a readiness answer. On a data plane on 0.7.0 or later that answer is about the data plane, on every hostname it declares; on an earlier data plane it is about one member gateway, so a member holding no live config drains from rotation. One ordering makes the switch fail-safe.

Order is load-bearing

Repoint the load balancer health check before flipping the flag. Flipping first false-drains the whole fleet. The steps below are in the safe order; do not reorder them.

Persona: Platform operator with access to the data plane's Helm values and to the cloud load balancer (LB) health-check configuration in front of the gateway.

Estimated time: 15 to 30 minutes per data plane.

Outcomes

By the end of this guide:

• The cloud load balancer health check targets the isolated liveness listener on port 10081 and stays green through the flip.
• Bare /healthz answers the readiness contract on each declared hostname (on a data plane on 0.7.0 or later) or on each member's hostname (on an earlier data plane).
• A host the data plane does not serve answers 404 instead of a false 200.

What this changes

HTTPS GET /healthz is the published customer health-check contract: 2xx only, body containing "status":"ready", polled by DNS health checks to decide regional failover.

	Bare /healthz before	Bare /healthz after
Answered by	Controller liveness ({"status":"ok"})	Data plane readiness ({"status":"ready","dataplane":"<data-plane-id>"}) on 0.7.0 or later; membership readiness ({"status":"ready","gateway":"<uuid>"}) before 0.7.0
Host match	None: any host, any member	The hostnames the data plane declares on 0.7.0 or later; the member's own hostname before 0.7.0
A host the data plane does not declare	Still returns 200	Returns 404
A member with no live config (before 0.7.0)	Still returns 200	Returns 404 (drains)

Before the flip, the catch-all liveness route answers 200 for any member whose Controller is up, including a member holding no materialized config. A DNS health check reading /healthz therefore keeps a dead member in rotation. That inversion is exactly what this contract removes. On a data plane on 0.7.0 or later the contract answers for the data plane instead: it stays green while the data plane serves any declared hostname, whatever its projects are doing, and goes red for a hostname the data plane no longer declares.

/healthz/membership is unchanged and not deprecated: the readiness answer is additive on /healthz.

The two probers

The switch changes what one prober reads and requires repointing the other:

• The cloud LB or target-group health check probes the gateway pods by IP, usually with no Host header. Today it gets its 200 from the host-blind catch-all on the public listener. The flip removes that catch-all, so this probe would start getting 404 and the LB would pull every pod. It must move to the liveness listener (below), which keeps the unconditional liveness 200.
• The DNS health check (for example Route 53) probes the public hostname on 443 with a real Host header and string-matches "status":"ready". After the flip this reads the readiness answer on /healthz, which is the intended behavior, provided the hostname checked is one the data plane declares; no change is needed beyond optionally moving it off /healthz/membership onto /healthz.

The liveness listener

Enabling the contract adds an isolated liveness listener to every egress gateway (it renders in both flag states, so it is already present after any v0.3.0 upgrade, inert until the LB targets it):

• Port 10081, HTTP, path Exact /healthz → controller liveness ({"status":"ok"}, host-blind, unconditional).
• Route-isolated (allowedRoutes: from: Same) so it can never admit serving AI routes without the auth module. Do not widen it; do not place other routes on it.

Prerequisites

• A self-hosted data plane on v0.3.0 or later, already installed and healthy.
• Access to the cloud load balancer or target-group health-check configuration in front of the gateway; its port changes in Step 1.
• Port 10081 on the gateway pods reachable from the LB's health checker. If the Service or NLB target group only exposes 443, add 10081 (HTTP) to the target group first: the listener answers, but the LB must be able to reach it.

Step 1: repoint the LB health check to the liveness listener

Point the cloud LB or target-group health check at:

• Port: 10081
• Protocol: HTTP
• Path: /healthz
• Healthy when: 200

Confirm it goes healthy on 10081 before proceeding. From a pod or a host that can reach the gateway directly:

curl -sS -o /dev/null -w '%{http_code}\n' http://<gateway-pod-ip>:10081/healthz
# expect: 200

Do not continue until the LB reports every member healthy against 10081.

Step 2: flip the flag

The single knob is global.publishHealthzContract (default false). It drives both subcharts in lockstep: the egress chart removes the public catch-all and the controller re-materializes the /healthz readiness match.

Option A: tetrate agentrouter dataplane install (recommended)

Put the flag in a values file:

# publish-healthz.yaml
global:
  publishHealthzContract: true

Re-run install with --helm-values. tetrate pipes the generated values first (-f -) and appends the supplied file after it, so helm's last-wins merge carries the flag through to the chart:

tetrate agentrouter dataplane install /path/to/data-plane-credentials.json \
  --helm-values publish-healthz.yaml

Option B: raw Helm

When the data plane is installed with Helm directly, add the flag to the same values.yaml the install guide already uses and re-run helm upgrade -f values.yaml (avoid --reuse-values --set, a known cross-version Helm pitfall):

# values.yaml — add to your existing data-plane values
global:
  publishHealthzContract: true

helm upgrade tars "oci://${PRIVATE_IMAGE_REGISTRY}/serve-helm" \
  -f values.yaml \
  # ...your existing install flags (--skip-crds, --set-string global.serveUrl=, etc.)

A plain helm upgrade applies the flip in the safe order by construction: the chart deletes the catch-all immediately, while the controller env change forces a pod rollout, so the readiness match re-materializes only after the catch-all is gone. GitOps deployments: see the caveat below. A GitOps engine's prune ordering is not guaranteed to be safe.

Step 3: verify

# 1. Public /healthz now speaks readiness on a healthy member.
curl -sS https://proxy.acme.com/healthz
# expect 2xx. Body on a data plane on 0.7.0 or later: {"status":"ready","dataplane":"<data-plane-id>"}
# Body on an earlier data plane: {"status":"ready","gateway":"<uuid>"}

# 2. Liveness still answers on the isolated listener (what the LB probes).
curl -sS -o /dev/null -w '%{http_code}\n' http://<gateway-pod-ip>:10081/healthz
# expect: 200

# 3. A member holding no live config drains (returns 404) instead of a false 200.
#    Confirm via your LB/DNS health dashboard that a red member leaves rotation.

Negative check (security invariant: the liveness listener must never serve inference): an authenticated inference request sent to port 10081 must return a no-route 404, never a completion. Verified live by healthz-contract-smoke.sh (the [C] leg: inference → 404 no body, /metrics → 404, liveness route match Exact); its lane home is fraser#8285.

Move DNS health checks onto /healthz (optional)

DNS health checks can now string-match "status":"ready" on /healthz directly instead of /healthz/membership. Both work; /healthz is the published contract going forward. Count only 2xx as healthy and keep the "status":"ready" string match so an edge redirect (3xx) is never read as green.

Rollback

Set the flag back and the public catch-all returns:

# tetrate
#   publish-healthz.yaml -> global.publishHealthzContract: false, re-run tetrate agentrouter dataplane install
# helm
helm upgrade tars "oci://${PRIVATE_IMAGE_REGISTRY}/serve-helm" \
  --reuse-values --set global.publishHealthzContract=false

The liveness listener on 10081 stays put across the toggle, so the LB health check can stay on 10081; no second LB change is needed to roll back.

GitOps caveat (ArgoCD / Flux)

The safe ordering under helm upgrade is incidental timing, not a guarantee. A GitOps engine that prunes removed resources in a late sync wave can let the new readiness match appear while the old catch-all still exists. During that overlap a red member falls through to the catch-all 200, the inversion this contract exists to prevent. When deploying via GitOps, enforce the order explicitly: apply the chart change and confirm the tars-controller-health catch-all HTTPRoute is deleted before the controller reconciles the readiness match.

The controller also self-heals late-prune races: with the flag on, it deletes a stale tars-controller-health route after cache sync and fails startup if it cannot. Treat that as a backstop, not a substitute for the ordered flip.

Troubleshooting

Symptom	Cause	Fix
LB drains all members right after the flip	LB health check still on the public listener (443 / catch-all gone)	Complete Step 1: move it to 10081
/healthz returns 404 on a member expected green	On a data plane on 0.7.0 or later: the hostname is not in global.gateway.hostnames, or the list is empty. Before 0.7.0: the member holds no materialized config. On either: Step 2 hit a mis-ordered GitOps sync	Check the declared list with kubectl -n tars-system get configmap tars-gateway-declaration -o jsonpath='{.data.gateway-hostnames}', or the member's config sync before 0.7.0; for GitOps, confirm the catch-all was deleted before the readiness match published
/healthz still returns {"status":"ok"} for any host	Flag not actually applied to the chart	Re-check the merged values (--helm-values file present / --set reached the release); confirm the controller pod rolled
Inference request to 10081 returns a completion	Isolation invariant broken: a route leaked onto the liveness listener	Stop; the liveness listener must be from: Same with only the controller-liveness route. Escalate: this is an auth bypass

What the readiness probe answers

DNS health checks target a readiness endpoint on each data plane's public HTTPS listener. What a green answer means depends on the data plane's release.

On a data plane on 0.7.0 or later

The probe answers for the data plane as a whole, on every hostname it declares:

GET https://<declared-hostname>/healthz/membership
GET https://<declared-hostname>/healthz

• Green: HTTP 200 with {"status":"ready","dataplane":"<data-plane-id>"}. The id names which data plane answered, which helps when DNS fronts several. A data plane that cannot name itself answers {"status":"ready"}.
• Red: any non-2xx response (concretely 404) or a connection failure.

Data plane state	Probe
Installed, declaring a hostname, serving no project	200 green
Serving one or more projects	200 green, however many
One project detached or zeroized, others still served	200 green
Declaring an empty hostname list	404 red. An empty list is how a data plane is drained deliberately
Request whose Host is not a declared hostname, including a load balancer probing by IP	404 red. Point load balancer liveness checks at the isolated liveness listener on port 10081
Partitioned from the management plane but reachable by clients	200 green. The probe is local configuration

A green answer says the data plane is up and reachable on that address. It does not say that any given project is served there or holds its current revision; that is read from each project label's progress on the data plane's page. Because detaching a project no longer turns the address red, shift DNS away from a data plane before removing a project from it.

A data plane whose chart declares no hostname answers on the address stored for it on the management plane, on its project gateway URLs, and on the hostnames its serving routes carry. The /healthz/membership path answers on every deployment; bare /healthz answers the same body only once the health-check contract is enabled, as the note below describes.

On a data plane before 0.7.0

Every member gateway exposes a health endpoint whose meaning is ready member of this project, not process liveness. This is the endpoint DNS health checks must target.

The endpoint, on the member's own hostname:

GET https://<gateway-hostname>/healthz

Which path carries the readiness answer

/healthz/membership carries the presence-coupled readiness answer described here on every release, independent of deployment state: 200 with {"status":"ready",...} for a ready member, and a no-route 404 for a member that holds no materialized configuration. It is the only deployment-independent path and is not deprecated. Bare /healthz carries that same answer only once the operator enables the health-check contract (the global.publishHealthzContract Helm value, off by default) on the data plane; before that flip, bare /healthz answers a host-blind controller-liveness 200 ({"status":"ok"}) for any member, whether or not it holds the project's configuration, so it reports liveness, not readiness. Target /healthz/membership until the data plane is on the contract; DNS health checks already pointing at it keep working unchanged.

This readiness answer is served on the member's public HTTPS listener, the same listener clients use, on standard port 443. It is distinct from the internal liveness port a cloud load balancer targets in a self-hosted install (see the gateway installation guides). DNS failover health checks target this readiness path on 443, never the liveness port, because only the readiness path goes red when a member stops holding the project's configuration.

The scheme, host, and port are the member's URL exactly as configured on the gateway; there is no separate port and no admin interface. Standard HTTPS port (443) only: hostname matching for the gateway's host-stamped routes (the probe included) is port-exact, and health checkers send Host: host:port for a non-standard port, so a member URL on a non-standard port is outside this contract. Front a non-443 listener with a standard-port endpoint before pointing DNS health checks at it.

Green and red, exactly as specified:

• Green (healthy): HTTP 200 with body {"status":"ready","gateway":"<gateway-uuid>"} (Content-Type: application/json). The UUID identifies which member answered, useful when DNS fronts several members.
• Red (unhealthy): any non-2xx response (concretely HTTP 404) or a connection failure/timeout. There is no "degraded" body; absence of the green answer is the signal.

Semantics: "ready member of this project", not process liveness. The probe is green if and only if the gateway currently holds the project's materialized, non-zeroized configuration:

Membership state	Probe	Why
ready (config synced)	200 green	The probe route is materialized together with the member's project routes.
syncing, never synced (fresh attach)	404 red	Nothing materialized yet. The probe route does not exist until the first successful config rollout.
syncing, previously synced (catching up to new config)	200 green	The member still serves its last-known-good config; it remains a valid failover target.
detached (zeroized)	404 red	Detach removes the member's config and its probe route; the listener stays up, so the probe goes red (the probe path is exempt from gateway authorization, so it is a true no-route 404) and DNS health checks drain the member. Client traffic on the drained hostname gets 403 unknown_host when the data plane hosts any other serving gateway, and a no-route 404 only when it hosts none. Expect 403s in client logs during the drain window on a shared data plane.
detached (unreachable) (from the data plane's side)	red once the zeroize lands or the gateway is down, but may stay GREEN while the gateway is partitioned from the management plane yet still client-reachable	The probe is data-plane-local configuration; a zeroize that never reaches the data plane cannot turn it red. DNS does NOT drain this member automatically: remove the member's DNS record manually and verify decommissioning (see the detach runbook).
Gateway/listener down	red (connection failure)	Handled by the health checker's own timeout.

The properties behind the table, summarized from the contract:

• Answered by the data plane itself. The probe is pure gateway configuration (a native direct-response route), with no management-plane round-trip, no backend, and no AI processing. It keeps answering during a full management-plane outage; an Admin Console or management-plane failure can never fail DNS health checks.
• Red by construction on delivered teardown. The probe route exists only as part of the member's materialized project configuration and is removed with it when the zeroized configuration lands. There is no separate health state machine that can drift out of sync with what the gateway actually serves.
• Host-scoped to the member. The probe answers only on the member's configured hostname, never on another member's hostname sharing the same data plane. A member with no URL configured has no probe (red; it cannot be in a DNS rotation anyway). On a hostname change, the probe on the old hostname goes red as soon as the new configuration rollout begins, and re-greens on the new hostname once that rollout is ready. A stalled rollout can never leave the old hostname green.
• Unauthenticated for GET/HEAD only. Quoting the contract's security claim precisely: "DNS health checkers cannot send API keys, so the data plane exempts exactly GET/HEAD on this path from gateway API-key authentication. Any other method (and any other path) still requires an API key (401 otherwise), so the probe cannot be used as an unauthenticated request surface."
• What the probe does not report: provider/upstream health, spend or rate-limit state, or per-request data-path success. A ready member of a project with no assigned models still probes green. It is faithfully serving its (empty) configuration. The probe answers "is this a ready member", nothing more.
• The probe, not the console, is the DNS go-signal. Probe-green can briefly trail the ready state shown on the Gateways tab (typically under a minute): the console reports confirmed convergence while the probe route finishes materializing on the data plane. A member that shows ready with a still-red probe is converging, not broken; add it to a DNS rotation only once its probe answers green.

Two consequences follow on a data plane before 0.7.0:

• Zeroized members drain themselves; unreachable ones do not. When a detach completes (the gateway acknowledges the zeroized configuration), the probe route is removed with the config, the health check fails, and any health-checked DNS record pointing at the member goes unhealthy automatically. But a member that lands in detached (unreachable) while still client-reachable (partitioned from the management plane, gateway process up) keeps answering its probe green: DNS does not drain it. This is why the detach runbook shifts DNS first. The DNS shift is the drain; the probe going red is the automatic backstop only for the zeroized terminal.
• A newly attached member cannot receive traffic early. Its probe stays red until first convergence, so health-checked DNS never routes to it half-configured.

Verify a member by hand at any time:

$ curl -s https://ai-gw-usw2.payments.examplebank.com/healthz/membership
{"status":"ready","gateway":"1c9f4a6e-8d21-4f0b-9b3a-5e7d2c840a17"}

On a data plane on 0.7.0 or later the same request returns the data plane's id instead:

{"status":"ready","dataplane":"dp-prod-usw2"}