Publish the /healthz readiness contract

Draft — validated, but do not publish until v0.3.0 ships

Every value here was verified live end-to-end on a real cluster (dev/healthz-contract-smoke.sh green — green member /healthz→200 ready, red→404, host-blind→404 on the public listener but 200 on the liveness listener, and inference//metrics→404 on the liveness listener). But the feature is still on an unmerged branch (fraser ADR 072/073, PR #8688 draft, flag default-off), so customers cannot use it yet. Keep this DRAFT and gate merge to master on: (a) #8688 merged, (b) v0.3.0 shipped, (c) values re-checked against the released chart.

By default a self-hosted gateway answers /healthz with an unconditional controller-liveness 200. This guide switches /healthz to the presence-coupled readiness contract so a member holding no live config drains from rotation — and shows the one ordering that makes the switch fail-safe.

Order is load-bearing

Repoint your load balancer health check before you flip the flag. Flipping first false-drains the whole fleet. The steps below are in the safe order — do not reorder them.

What this changes

HTTPS GET /healthz is the published customer health-check contract: 2xx only, body containing "status":"ready", polled by your DNS health checks to decide regional failover.

	Bare /healthz before	Bare /healthz after
Answered by	Controller liveness ({"status":"ok"})	Membership readiness ({"status":"ready","gateway":"<uuid>"})
Host match	None — any host, any member	Host-scoped, per member
A member with no live config	Still returns 200	Returns 404 (drains)

Before the flip, the catch-all liveness route answers 200 for any member whose Envoy control plane is up — including a member holding no materialized config. A DNS health check reading /healthz therefore keeps a dead member in rotation. That inversion is exactly what this contract removes.

/healthz/membership is unchanged and not deprecated — the readiness answer is additive on /healthz.

The two probers (this is the whole trick)

The switch moves one thing and repoints one other thing. Keep them straight:

• Your cloud LB / target-group health check — probes the gateway pods by IP, usually with no Host header. Today it gets its 200 from the host-blind catch-all on the public listener. The flip removes that catch-all, so this probe would start getting 404 and the LB would pull every pod. → It must move to the liveness listener (below), which keeps the unconditional liveness 200.
• Your DNS health check (e.g. Route 53) — probes the public hostname on 443 with a real Host header and string-matches "status":"ready". After the flip this reads the presence-coupled readiness answer on /healthz. This is the behavior you want — no change needed beyond optionally moving it off /healthz/membership onto /healthz.

The liveness listener

Enabling the contract adds an isolated liveness listener to every egress gateway (it renders in both flag states, so it is already present after any v0.3.0 upgrade — inert until your LB targets it):

• Port 10081, HTTP, path Exact /healthz → controller liveness ({"status":"ok"}, host-blind, unconditional).
• Route-isolated (allowedRoutes: from: Same) so it can never admit serving AI routes without the auth module. Do not widen it; do not place other routes on it.

Prerequisites

• A self-hosted data plane on v0.3.0 or later, already installed and healthy.
• Access to the cloud load balancer / target-group health-check configuration in front of the gateway (you will change its port).
• Confirm port 10081 on the gateway pods is reachable from your LB's health checker. If your Service / NLB target group only exposes 443, add 10081 (HTTP) to the target group first — the listener answers, but the LB must be able to reach it.

Step 1 — Repoint the LB health check to the liveness listener

Point your cloud LB / target-group health check at:

• Port: 10081
• Protocol: HTTP
• Path: /healthz
• Healthy when: 200

Confirm it goes healthy on 10081 before proceeding. From a pod or a host that can reach the gateway directly:

curl -sS -o /dev/null -w '%{http_code}\n' http://<gateway-pod-ip>:10081/healthz
# expect: 200

Do not continue until the LB reports every member healthy against 10081.

Step 2 — Flip the flag

The single knob is global.publishHealthzContract (default false). It drives both subcharts in lockstep — the egress chart removes the public catch-all and the controller re-materializes the /healthz readiness match.

Option A — tare install (recommended)

Put the flag in a values file:

# publish-healthz.yaml
global:
  publishHealthzContract: true

Re-run install with --helm-values. tare pipes the generated values first (-f -) and appends your file after it, so helm's last-wins merge carries the flag through to the chart:

tare install /path/to/data-plane-credentials.json \
  --serve-url https://proxy.acme.com \
  --helm-values publish-healthz.yaml

Option B — raw Helm

If you install the data plane with Helm directly, add the flag to the same values.yaml your install guide already uses and re-run helm upgrade -f values.yaml (matching house style — avoid --reuse-values --set, a known cross-version Helm footgun):

# values.yaml — add to your existing data-plane values
global:
  publishHealthzContract: true

helm upgrade tars "oci://${PRIVATE_IMAGE_REGISTRY}/serve-helm" \
  -f values.yaml \
  # ...your existing install flags (--skip-crds, --set-string global.serveUrl=, etc.)

A plain helm upgrade applies the flip in the safe order by construction: the chart deletes the catch-all immediately, while the controller env change forces a pod rollout, so the readiness match re-materializes only after the catch-all is gone. GitOps users: see the caveat below — your engine's prune ordering is not guaranteed to be safe.

Step 3 — Verify

# 1. Public /healthz now speaks readiness on a healthy member.
curl -sS https://proxy.acme.com/healthz
# expect 2xx, body: {"status":"ready","gateway":"<uuid>"}

# 2. Liveness still answers on the isolated listener (what the LB probes).
curl -sS -o /dev/null -w '%{http_code}\n' http://<gateway-pod-ip>:10081/healthz
# expect: 200

# 3. A member holding no live config drains (returns 404) instead of a false 200.
#    Confirm via your LB/DNS health dashboard that a red member leaves rotation.

Negative check (security invariant — the liveness listener must never serve inference): an authenticated inference request sent to port 10081 must return a no-route 404, never a completion. Verified live by healthz-contract-smoke.sh (the [C] leg: inference → 404 no body, /metrics → 404, liveness route match Exact); its lane home is fraser#8285.

Move DNS health checks onto /healthz (optional)

Your DNS health checks can now string-match "status":"ready" on /healthz directly instead of /healthz/membership. Both work; /healthz is the published contract going forward. Count only 2xx as healthy and keep the "status":"ready" string match so an edge redirect (3xx) is never read as green.

Rollback

Set the flag back and the public catch-all returns:

# tare
#   publish-healthz.yaml -> global.publishHealthzContract: false, re-run tare install
# helm
helm upgrade tars "oci://${PRIVATE_IMAGE_REGISTRY}/serve-helm" \
  --reuse-values --set global.publishHealthzContract=false

The liveness listener on 10081 stays put across the toggle, so you can leave the LB health check on 10081 — no second LB change is needed to roll back.

GitOps caveat (ArgoCD / Flux)

The safe ordering under helm upgrade is incidental timing, not a guarantee. A GitOps engine that prunes removed resources in a late sync wave can let the new readiness match appear while the old catch-all still exists. During that overlap a red member falls through to the catch-all 200 — the inversion this contract exists to prevent. If you deploy via GitOps, enforce the order explicitly: apply the chart change and confirm the tars-controller-health catch-all HTTPRoute is deleted before the controller reconciles the readiness match.

The controller also self-heals late-prune races: with the flag on, it deletes a stale tars-controller-health route after cache sync and fails startup if it cannot. Treat that as a backstop, not a substitute for the ordered flip.

Troubleshooting

Symptom	Cause	Fix
LB drains all members right after the flip	LB health check still on the public listener (443 / catch-all gone)	Complete Step 1 — move it to 10081
/healthz returns 404 on a member you expect green	Member holds no materialized config, or Step 2 hit a mis-ordered GitOps sync	Check the member's config sync; for GitOps, confirm the catch-all was deleted before the readiness match published
/healthz still returns {"status":"ok"} for any host	Flag not actually applied to the chart	Re-check the merged values (--helm-values file present / --set reached the release); confirm the controller pod rolled
Inference request to 10081 returns a completion	Isolation invariant broken — a route leaked onto the liveness listener	Stop; the liveness listener must be from: Same with only the controller-liveness route. Escalate — this is an auth bypass