Publish the /healthz readiness contract

By default a self-hosted gateway answers /healthz with an unconditional controller-liveness 200. Enabling the presence-coupled readiness contract switches /healthz to a readiness answer. On a data plane on 0.7.0 or later that answer is about the data plane, on every hostname it declares; on an earlier data plane it is about one member gateway, so a member holding no live config drains from rotation. One ordering makes the switch fail-safe.

Order is load-bearing

Repoint the load balancer health check before flipping the flag. Flipping first false-drains the whole fleet. The steps below are in the safe order; do not reorder them.

Persona: Platform operator with access to the data plane's Helm values and to the cloud load balancer (LB) health-check configuration in front of the gateway.

Estimated time: 15 to 30 minutes per data plane.

Outcomes

By the end of this guide:

• The cloud load balancer health check targets the isolated liveness listener on port 10081 and stays green through the flip.
• Bare /healthz answers the readiness contract on each declared hostname (on a data plane on 0.7.0 or later) or on each member's hostname (on an earlier data plane).
• A host the data plane does not serve answers 404 instead of a false 200.

What this changes

HTTPS GET /healthz is the published customer health-check contract: 2xx only, body containing "status":"ready", polled by DNS health checks to decide regional failover.

	Bare /healthz before	Bare /healthz after
Answered by	Controller liveness ({"status":"ok"})	Data plane readiness ({"status":"ready","dataplane":"<data-plane-id>"}) on 0.7.0 or later; membership readiness ({"status":"ready","gateway":"<uuid>"}) before 0.7.0
Host match	None: any host, any member	The hostnames the data plane declares on 0.7.0 or later; the member's own hostname before 0.7.0
A host the data plane does not declare	Still returns 200	Returns 404
A member with no live config (before 0.7.0)	Still returns 200	Returns 404 (drains)

Before the flip, the catch-all liveness route answers 200 for any member whose Controller is up, including a member holding no materialized config. A DNS health check reading /healthz therefore keeps a dead member in rotation. That inversion is exactly what this contract removes. On a data plane on 0.7.0 or later the contract answers for the data plane instead: it stays green while the data plane serves any declared hostname, whatever its projects are doing, and goes red for a hostname the data plane no longer declares.

/healthz/membership is unchanged and not deprecated: the readiness answer is additive on /healthz.

The two probers

The switch changes what one prober reads and requires repointing the other:

• The cloud LB or target-group health check probes the gateway pods by IP, usually with no Host header. Today it gets its 200 from the host-blind catch-all on the public listener. The flip removes that catch-all, so this probe would start getting 404 and the LB would pull every pod. It must move to the liveness listener (below), which keeps the unconditional liveness 200.
• The DNS health check (for example Route 53) probes the public hostname on 443 with a real Host header and string-matches "status":"ready". After the flip this reads the readiness answer on /healthz, which is the intended behavior, provided the hostname checked is one the data plane declares; no change is needed beyond optionally moving it off /healthz/membership onto /healthz.

The liveness listener

Enabling the contract adds an isolated liveness listener to every egress gateway (it renders in both flag states, so it is already present after any v0.3.0 upgrade, inert until the LB targets it):

• Port 10081, HTTP, path Exact /healthz → controller liveness ({"status":"ok"}, host-blind, unconditional).
• Route-isolated (allowedRoutes: from: Same) so it can never admit serving AI routes without the auth module. Do not widen it; do not place other routes on it.

Prerequisites

• A self-hosted data plane on v0.3.0 or later, already installed and healthy.
• Access to the cloud load balancer or target-group health-check configuration in front of the gateway; its port changes in Step 1.
• Port 10081 on the gateway pods reachable from the LB's health checker. If the Service or NLB target group only exposes 443, add 10081 (HTTP) to the target group first: the listener answers, but the LB must be able to reach it.

Step 1: repoint the LB health check to the liveness listener

Point the cloud LB or target-group health check at:

• Port: 10081
• Protocol: HTTP
• Path: /healthz
• Healthy when: 200

Confirm it goes healthy on 10081 before proceeding. From a pod or a host that can reach the gateway directly:

curl -sS -o /dev/null -w '%{http_code}\n' http://<gateway-pod-ip>:10081/healthz
# expect: 200

Do not continue until the LB reports every member healthy against 10081.

Step 2: flip the flag

The single knob is global.publishHealthzContract (default false). It drives both subcharts in lockstep: the egress chart removes the public catch-all and the controller re-materializes the /healthz readiness match.

Option A: tetrate agentrouter dataplane install (recommended)

Put the flag in a values file:

# publish-healthz.yaml
global:
  publishHealthzContract: true

Re-run install with --helm-values. tetrate pipes the generated values first (-f -) and appends the supplied file after it, so helm's last-wins merge carries the flag through to the chart:

tetrate agentrouter dataplane install /path/to/data-plane-credentials.json \
  --helm-values publish-healthz.yaml

Option B: raw Helm

When the data plane is installed with Helm directly, add the flag to the same values.yaml the install guide already uses and re-run helm upgrade -f values.yaml (avoid --reuse-values --set, a known cross-version Helm pitfall):

# values.yaml — add to your existing data-plane values
global:
  publishHealthzContract: true

helm upgrade tars "oci://${PRIVATE_IMAGE_REGISTRY}/serve-helm" \
  -f values.yaml \
  # ...your existing install flags (--skip-crds, --set-string global.serveUrl=, etc.)

A plain helm upgrade applies the flip in the safe order by construction: the chart deletes the catch-all immediately, while the controller env change forces a pod rollout, so the readiness match re-materializes only after the catch-all is gone. GitOps deployments: see the caveat below. A GitOps engine's prune ordering is not guaranteed to be safe.

Step 3: verify

# 1. Public /healthz now speaks readiness on a healthy member.
curl -sS https://proxy.acme.com/healthz
# expect 2xx. Body on a data plane on 0.7.0 or later: {"status":"ready","dataplane":"<data-plane-id>"}
# Body on an earlier data plane: {"status":"ready","gateway":"<uuid>"}

# 2. Liveness still answers on the isolated listener (what the LB probes).
curl -sS -o /dev/null -w '%{http_code}\n' http://<gateway-pod-ip>:10081/healthz
# expect: 200

# 3. A member holding no live config drains (returns 404) instead of a false 200.
#    Confirm via your LB/DNS health dashboard that a red member leaves rotation.

Negative check (security invariant: the liveness listener must never serve inference): an authenticated inference request sent to port 10081 must return a no-route 404, never a completion. Verified live by healthz-contract-smoke.sh (the [C] leg: inference → 404 no body, /metrics → 404, liveness route match Exact); its lane home is fraser#8285.

Move DNS health checks onto /healthz (optional)

DNS health checks can now string-match "status":"ready" on /healthz directly instead of /healthz/membership. Both work; /healthz is the published contract going forward. Count only 2xx as healthy and keep the "status":"ready" string match so an edge redirect (3xx) is never read as green.

Rollback

Set the flag back and the public catch-all returns:

# tetrate
#   publish-healthz.yaml -> global.publishHealthzContract: false, re-run tetrate agentrouter dataplane install
# helm
helm upgrade tars "oci://${PRIVATE_IMAGE_REGISTRY}/serve-helm" \
  --reuse-values --set global.publishHealthzContract=false

The liveness listener on 10081 stays put across the toggle, so the LB health check can stay on 10081; no second LB change is needed to roll back.

GitOps caveat (ArgoCD / Flux)

The safe ordering under helm upgrade is incidental timing, not a guarantee. A GitOps engine that prunes removed resources in a late sync wave can let the new readiness match appear while the old catch-all still exists. During that overlap a red member falls through to the catch-all 200, the inversion this contract exists to prevent. When deploying via GitOps, enforce the order explicitly: apply the chart change and confirm the tars-controller-health catch-all HTTPRoute is deleted before the controller reconciles the readiness match.

The controller also self-heals late-prune races: with the flag on, it deletes a stale tars-controller-health route after cache sync and fails startup if it cannot. Treat that as a backstop, not a substitute for the ordered flip.

Troubleshooting

Symptom	Cause	Fix
LB drains all members right after the flip	LB health check still on the public listener (443 / catch-all gone)	Complete Step 1: move it to 10081
/healthz returns 404 on a member expected green	On a data plane on 0.7.0 or later: the hostname is not in global.gateway.hostnames, or the list is empty. Before 0.7.0: the member holds no materialized config. On either: Step 2 hit a mis-ordered GitOps sync	Check the declared list with kubectl -n tars-system get configmap tars-gateway-declaration -o jsonpath='{.data.gateway-hostnames}', or the member's config sync before 0.7.0; for GitOps, confirm the catch-all was deleted before the readiness match published
/healthz still returns {"status":"ok"} for any host	Flag not actually applied to the chart	Re-check the merged values (--helm-values file present / --set reached the release); confirm the controller pod rolled
Inference request to 10081 returns a completion	Isolation invariant broken: a route leaked onto the liveness listener	Stop; the liveness listener must be from: Same with only the controller-liveness route. Escalate: this is an auth bypass