Reading tetrate agentrouter doctor output
tetrate agentrouter doctor inspects a self-hosted data plane and prints a single health verdict, a per-category checklist, and the findings behind them. For the flag list, see the generated command reference.
Running it
tetrate agentrouter doctor /path/to/data-plane-credentials.json --verbose
The identity file is optional. Supplying it lets the command upload a diagnostic bundle to the management plane, which is what a support engagement needs; omitting it runs the same checks locally and prints the report only.
Two other invocations matter when reading output:
-o jsonemits the same report as a machine-readable envelope, which is what the Admin Console and thetare-doctorCronJob consume. From 0.8.0 the envelope also carries the schedule the in-cluster CronJob runs on, underdata.doctor_schedule, and the management plane stores it with each data plane's status. The field is absent when the CronJob could not be read.--only <check>runs one named check instead of the full pipeline. The checklist is suppressed in this mode, deliberately: categories that never ran would otherwise be printed asOK.
The shape of a report
Sections appear in a fixed order. Sections with nothing to report are omitted.
CLUSTER HEALTH: DEGRADED
SUMMARY: 3 issues (0 errors, 1 warnings, 2 advisories)
CHECKLIST:
- Doctor access: OK
- Namespaces: OK
- CRDs: OK
...
- Routing: WARNING
- Pod health: OK
RESOURCES:
...
FEATURE CONFIGURATION:
...
WARNINGS:
- AIGatewayRoute tars-dataplane/route-openai: ...
ADVISORIES (informational; do not affect status):
- ...
| Section | What it holds |
|---|---|
CLUSTER HEALTH | The single roll-up verdict, described below |
SUMMARY | Issue counts by severity. The advisories count is present only when advisories were raised, and from 0.8.0 the doctor-self count only when the doctor's own access has gaps |
CHECKLIST | One row per category, each carrying the worst severity found in it. Replaced by a single CHECK: line on an --only run |
RESOURCES | A kubectl-style inventory of what was found, with the status column each resource reports |
FEATURE CONFIGURATION | Which optional capabilities are configured, read from live cluster objects. It carries no verdict |
CHECKS PERFORMED | The full list of checks in the pipeline. Printed only with --verbose |
ERRORS, WARNINGS, ADVISORIES | The findings themselves, grouped by severity |
DOCTOR SELF | From 0.8.0, gaps in the doctor's own access, described under Doctor access gaps. They do not affect status |
Status and severity
The status line is derived from the finding counts, not judged separately:
| Status | Condition | Meaning |
|---|---|---|
HEALTHY | No errors and no warnings | Nothing needs attention. Advisories may still be listed |
DEGRADED | At least one warning, no errors | The data plane is serving, but something is inconsistent or incomplete |
BROKEN | At least one error | Something the data plane needs is missing or rejected |
Advisories are the third severity and behave differently from the other two. They are counted in SUMMARY, printed in their own section, and deliberately excluded from the roll-up, so a report can read HEALTHY while listing several. They flag things worth knowing rather than things to fix, and a dev cluster routinely raises them.
Doctor access gaps
From 0.8.0 a gap in what the doctor itself is permitted to read is kept out of the status. When an optional read is refused, typically because the installed tare-doctor chart predates a grant a newer doctor uses, the finding is marked doctor_self. It keeps its severity and its checklist row, is printed under DOCTOR SELF rather than with the data plane's findings, carries doctor_self in the JSON report, and is left out of the roll-up, so a serving data plane no longer reads DEGRADED because of the doctor's own configuration. Upgrading the tare-doctor chart clears these findings. A required grant that is refused is still an ERROR, because without it the run cannot vouch for the data plane.
Exit codes
The exit code follows the --fail-on threshold, not the status line:
| Exit code | When |
|---|---|
0 | No finding at or above the threshold. With the default threshold, error, a DEGRADED report with warnings only exits 0, as does a report with advisories only |
1 | At least one finding at or above the threshold: any error by default, or any error or warning with --fail-on warning |
--fail-on none never fails on findings. Advisories and doctor-self findings never count toward the threshold. The in-cluster tare-doctor CronJob runs with --fail-on none unless tareDoctor.failOn sets another threshold, so its Jobs do not fail on findings; the report it uploads carries the status. A run cut short by its own time budget on a reachable data plane with no genuine error exits 0 (see Timeouts on a large data plane).
A failure of the command itself, such as an unreachable cluster, also exits 1. Under tetrate agentrouter doctor, an invalid invocation exits 2, as described in Exit codes.
Checklist categories
Each row rolls up every finding whose resource falls into that category. A row reads OK when nothing was found, which for a category whose resources are absent entirely means absence was not itself a problem.
| Category | Covers |
|---|---|
| Doctor access | Whether the doctor's own service account could read what it needed. From 0.8.0 a refused optional read is a doctor-self finding that does not affect status (see Doctor access gaps) |
| Namespaces | tars-system and tars-dataplane presence and state |
| CRDs | Custom resource definitions, their versions, and drift from the expected contract |
| Controllers | The controller Deployments and, from 0.8.0, whether they and the egress proxy run their desired replica count (see Replicas below the desired count) |
| Helm releases | The installed release and its revision |
| Management plane | Reachability of the management plane. Present only when the probe ran and found something |
| Gateway | GatewayClass, Gateway, EnvoyProxy, and EnvoyPatchPolicy objects, and since 0.7.0 the certificate serving each declared hostname (see The serving certificate) |
| Identity config | The Secrets and ConfigMaps carrying data plane identity and configuration |
| AI backends | AIServiceBackend objects, one per provider-model backend |
| Backends | The underlying Backend objects those resolve to |
| Backend traffic policies | BackendTrafficPolicy objects |
| Backend security policies | BackendSecurityPolicy objects, which carry provider credential references |
| Backend TLS policies | BackendTLSPolicy objects |
| Client traffic policies | ClientTrafficPolicy objects, including the connection buffer limit |
| HTTPRoute filters | HTTPRouteFilter objects |
| Reference grants | Cross-namespace ReferenceGrant objects |
| Routing | AIGatewayRoute, HTTPRoute, and RouteDeployment objects, and whether they are accepted |
| Route consistency | Whether the route set agrees with what the management plane expects |
| MCP routes | MCPRoute objects and their churn rate |
| Proxy health | The tars-dataplane proxy workload |
| Pod health | Pod-level state across both namespaces |
| Nodes | From 0.8.0, the nodes that host a data plane pod: pressure conditions, readiness, and memory use (see Nodes under pressure). Present only when a node hosts a data plane pod or the node read produced a finding |
| Semantic router | The optional SemanticRouter guardrail provider. Present only when it is deployed |
Two further rows appear with --verbose only: EnvoyPatchPolicy existence and Envoy Tars image. Both can read UNAVAILABLE, which means the check could not determine an answer rather than that it failed.
Common findings
Routing or AI backends empty on a fresh install
A report showing no AIGatewayRoute objects, or routes naming a backend that does not exist, is most often a provider credential problem rather than a routing problem. A provider with no working credential produces no backend, and the routes that reference it cannot be accepted.
Check, in order:
- That the provider has a credential set and shows as active in the Admin Console. Until it does, no model behind it is callable. See Add providers and models to projects.
- That the change has had time to land. Routes and the backends they point at are written as separate steps, so a snapshot taken between them looks like a permanent failure. See Configuration propagation.
- That the models are assigned to the project, not merely enabled in the organization catalog. Enabling alone does not make a model reachable.
Rotating the provider credential is a reliable way to force the backend and its route to be written, because it re-runs the step that creates both.
The serving certificate
Since 0.7.0 the gateway check reads the TLS certificate serving each hostname the data plane declares under global.gateway.hostnames. It reads the certificate from the Secret a Gateway listener references where one exists. Where none does, which is every data plane whose TLS terminates on a load balancer in front of the cluster, it dials each declared address over HTTPS and reads the certificate it is handed. Run the check on its own with:
tetrate agentrouter doctor identity.json --only gateway
The findings it reports:
| Severity | Finding | What to do |
|---|---|---|
ERROR | The certificate has expired, or is not valid yet. The message carries the time validity starts. | A not-yet-valid certificate points at the cluster's clock or at the issuing certificate authority. Callers' TLS stacks refuse a certificate on either side of its validity window. |
WARNING | The certificate does not cover a declared hostname. | Add the name to the certificate, or remove it from the declaration. |
WARNING | The certificate expires within 30 days. The message carries the expiry date. | Plan the renewal. |
INFO, listed under ADVISORIES | The address could not be reached. The finding carries the openssl s_client command to read the certificate by hand. | Nothing, if the load balancer is deliberately unreachable from the cluster. The data plane stays healthy. |
The check honors global.gateway.certificateProbe in the serve-helm values, the same key the data plane uses for its own reading, so the report and the Admin Console read the certificate from the same place. A data plane whose chart sets certificateProbe.enabled to false is not dialed, and one naming an address there is read at that address. The key is described in Declare the hostnames a data plane serves.
Identity config errors
Findings against Secrets or ConfigMaps in tars-system usually point at the data plane credential rather than at a provider. The distinguishing symptom on the traffic side is a 503 with service_unavailable on every request, including requests with a valid client key. See Data plane credentials.
ArgoCD-managed clusters
The doctor reads applications.argoproj.io to record deployment history for GitOps-driven rollouts, because ArgoCD applies charts without leaving the Helm release secrets the normal path reads.
That read is skipped silently on clusters where the ArgoCD CRD is not registered, so a plain Helm install needs no configuration. On a cluster that does run ArgoCD, the chart must be told about it:
argocd:
enabled: true
namespace: argocd
argocd.enabled governs only whether the Role and RoleBinding granting that read are rendered. It defaults to false because rendering them into a namespace that does not exist would fail the install. Left at false on a cluster that does run ArgoCD, the read is refused rather than skipped, and the deployment history in the Admin Console stays empty for ArgoCD-driven syncs.
An autoscaler that cannot scale
Since 0.3.0 the report covers the data plane's autoscalers. A HorizontalPodAutoscaler that cannot compute its metric reports ScalingActive=False and leaves its workload frozen on the replicas it already has, which earlier releases read as Healthy because no autoscaler was read at all. A data plane in that state is now Degraded, and the finding names the autoscaler, the workload it scales, the reason Kubernetes gave, and the replica count it is stuck on alongside the configured minReplicas and maxReplicas. An autoscaler that has genuinely run out of headroom at maxReplicas is reported too, as is one the doctor is not permitted to read. Data planes with autoscaling switched off are unaffected, because they have no autoscaler to report.
Nodes under pressure
From 0.8.0 the report covers the nodes that run a data plane pod. A node on a shared cluster that hosts no data plane pod is not judged.
| Severity | Finding |
|---|---|
ERROR | The node reports MemoryPressure, DiskPressure, or PIDPressure, or is not Ready. Data plane pods on it are at risk of eviction or are not running |
WARNING | The node's memory working set is at 90% or more of its allocatable memory. The finding names the node and the percentage. Checked only when metrics-server is installed |
A node read the doctor is not permitted to make is a doctor access gap, not a node finding.
Replicas below the desired count
From 0.8.0 the report flags a data plane Controller or the egress proxy that runs below its desired replica count, not only one with no replicas left. Below half the desired count is an ERROR and sets the GatewayReady condition to False; any smaller shortfall is a WARNING. Pods still starting after a scale-out or during a rollout are not counted against the Deployment, and a rollout in progress is reported as a WARNING at most.
Routing errors after the upgrade to 0.8.0
From 0.8.0 the route consistency check compares the routes of API keys that have finished rolling out against the routing actually serving them. Earlier releases silently skipped every such key, so a key whose routing was never fully created looked healthy. The first run after the upgrade can therefore report route consistency errors that were present before and went unreported. Each key named is checked before the report is treated as a regression.
A key left stuck deploying by a routing change that needed fewer routes than before recovers on its next routing change; until then, the report lists its leftover routes as orphan routes.
Port 443 listeners
From 0.8.0, on a data plane that publishes declared hostnames on port 443 (see Terminate TLS on the data plane), the report checks the HTTPS listeners. A hostname whose listener is missing authentication, so that it answers 403 to every request, is reported as an ERROR, as is a listener serving inference with no authentication at all. The ResourceNotFound status the TLS patch policies show when two or more hostnames are published is the healthy state and is not reported. The listener checks read the Envoy configuration dump; without it, an advisory says the 443 listeners were not checked.
Timeouts on a large data plane
The check pipeline makes dozens of throttled list calls across two namespaces, and on a data plane of realistic size it runs well past the 30-second mark. The command's own default allows for this. A run cancelled by an externally imposed deadline reports no health at all rather than partial health, so a wrapper enforcing its own timeout should allow several minutes.
From 0.8.0 a run that reaches its own time budget no longer marks the data plane unhealthy on that basis alone. When the liveness checks finished and no genuine fault was found, the report carries a doctor-self finding that names the checks that did not finish and suggests raising the timeout (--timeout, or tareDoctor.timeout for the CronJob). When even the liveness checks were cut short, the run is still an ERROR. Two checks whose cost grew with every API key now cost the same regardless of how many keys the data plane carries, so a data plane with hundreds of keys finishes inside the budget.
From 0.8.0 the doctor's CronJob requests 256Mi of memory and 50m of CPU, with a memory limit of 512Mi, where before it requested nothing. On a cluster with little headroom its Job can wait for a node that fits it. The values are under tareDoctor.resources in the data plane's values.
Related
tetrate agentrouter doctorcommand reference: the generated flag list- Configuration propagation: why a route can exist before its backend does
- Gateway behavior: the error codes a client sees while the data plane is in this state
- Data plane credentials: rotating and recovering the credential