Upgrade a self-hosted data plane

The order and the rules for keeping a self-hosted data plane current: what upgrades first, what keeps working while the data plane lags a release, how to read the release notes before upgrading, and when a patch is worth taking off-schedule. The command itself is covered by the tetrate agentrouter dataplane upgrade reference.

The order: the management plane leads, the data plane follows

The Tetrate-hosted management plane is always on the current release; Tetrate upgrades it as part of the weekly cut. A data plane is therefore never newer than its management plane, and the operator's half of every upgrade is the data plane and the tare CLI, whose version matches the release tag.

Because the management plane is always current, "upgrade the management plane first" is not a step an operator performs: it has already happened by the time a release appears on the release notes page. The operator's decision is only when the data plane follows.

Nothing forces an immediate upgrade

Each release is supported for six months from the day it ships, and a supported data plane keeps working against the current management plane for that whole window. The end-of-support date for every release is on its release notes entry and on Release compatibility.

Since 0.6.0 the Admin Console also warns in place: a data plane whose release is within 30 days of its end of support, or past it, is marked on the data plane list under Directory → Data Planes, and its detail page shows a notice. The warning means the data plane's release is leaving the support window, so its upgrade should be planned; it reads the data plane's version against the release archive, and a release whose end date is not yet published shows nothing. From 0.8.0 that version is the one the list's Version column shows: the release read from the deployed chart on a data plane on 0.8.0 or later, otherwise the release of the last successful install or upgrade, and the version of the tare CLI that ran doctor only where the column shows none, as for a data plane managed by ArgoCD or Flux. From 0.8.0 the list also marks a data plane that lacks a high or critical security fix shipped in a newer release, and its detail page names the release to upgrade to; see Where the running version is read.

Fixes are not backported: getting a fix means upgrading to a release that contains it. That, rather than compatibility, is what usually decides the timing.

What lags while the data plane is behind

A management plane feature can need a newer data plane. The feature stays off for that data plane until it is upgraded; nothing breaks, and the rest of the release keeps working. Each such feature names its version in the release notes ("available once the data plane is on X or later"), and the Minimum data plane column on Release compatibility carries the highest such version per release: a data plane at or above it has every feature of that release.

Skipping releases: an upgrade goes direct

A data plane several releases behind upgrades in one move, not release by release. The upgrade re-renders the deployment at the target tag, whichever path installed it, so there is no chain of intermediate upgrades to walk. No release to date has restricted which release an upgrade may start from or required a stepping-stone release in between; the release contract reserves fields for both, so a release that ever introduces such a requirement carries it in its Upgrade and rollback block, which is the thing to read before any jump.

What a multi-release jump does change is the reading list. Everything the skipped releases shipped arrives at once: their behavior changes, their deprecations, and any per-release required action all land in the same upgrade, so the notes of every release between the current one and the target are part of the pre-upgrade read, not just the target's. The release notes index lists them in order.

Read the release before upgrading

Every entry on the release notes page carries an Upgrade and rollback block. Four things in it decide what the upgrade needs:

1. The breaking-change line. When operator action is required before upgrading, the entry carries a warning naming the action. A release can also carry a breaking change that requires no action; the line says which.
2. The traffic line. Most upgrades roll through without dropping traffic. When a release does not, the line says so, and a maintenance window is needed.
3. The rollback line. Rollback to the previous release is unconditional unless the entry carries a rollback condition; a conditional rollback names exactly what stops working after rolling back.
4. The Kubernetes range. The range is fixed per release and enforced at install and upgrade time, and managed Kubernetes cannot be downgraded in place, so a cluster below a new release's floor must be recreated before the data plane can follow. The range per release is on Release compatibility.

Deprecations is the fifth check, on a slower clock: anything deprecated keeps working for twelve months, and the running list shows what is on the clock and its replacement.

When to take a patch off-schedule

Releases are cut every Wednesday, and a patch ships when it is needed rather than waiting for the next cut. A patch that carries only security updates, the shape v0.2.1 had, changes no source, API, configuration, or behavior: it rolls through without dropping traffic and rolls back unconditionally, so the cost of taking it immediately is low and the reason it shipped early is the reason not to wait.

Who can run an upgrade

Since 0.5.0, the upgrade operations sit behind their own authorization boundary: they require the dataplane_operator scope on the platform credential, and an organization admin credential can no longer run them. On a Tetrate-hosted management plane the grant is prepared by Tetrate; on a deployment that runs its own management plane, the upgrade prepares it, and the release is deployed in a fixed order. The boundary, the order, and what fails when a step is skipped are on Data plane upgrade authorization.

Upgrading to 0.5.0

The 0.5.0 upgrade carries two required actions of its own. Both are checked up front, so missing one fails the upgrade rather than half-applying it:

1. Set RBAC_AUTH_DB_MODE in the auth values. The setting declares whether the deployment owns its auth database or shares a hosted one. An installation that never set it was running as shared, so keeping shared preserves the behavior it has today; the render fails when the setting is absent.
2. Make sure the platform credential gains the dataplane_operator scope. The upgrade's bootstrapper job adds it with a database migration; when the bootstrapper is disabled, the migration is run by hand first, per Data plane upgrade authorization. A deployment upgraded without it keeps serving traffic but loses the ability to upgrade its data plane, and the resulting permission_denied does not name the cause.

Upgrading to 0.6.0

Before 0.6.0, a proxy setting changed on the management plane did not reach a data plane that serves project gateways: the change was stored and shown as applied, while those gateways kept the values rendered at their last gateway change, often from when the gateway was created. From 0.6.0 the change is delivered. The consequence for the upgrade is that any proxy setting changed while running an earlier release may apply for the first time once the data plane is on 0.6.0: cost and fee handling, overrun thresholds, cache sizing, and the request-log mode set from the Admin Console are all carried in those settings. Cost figures a project-gateway data plane recorded before the upgrade reflect the older values. Check each such setting after upgrading and confirm the value in effect is the intended one.

Upgrading to 0.7.0

0.7.0 moves the data plane to Envoy Gateway v1.9.1, which raises two cluster requirements. Meet both before the data plane upgrade starts. tare checks both before anything is applied, so a cluster that misses either is refused rather than half-upgraded. The Helm and GitOps paths check only the Gateway API CRDs, so on those paths the Kubernetes version is confirmed by hand:

1. Kubernetes 1.33.0 to 1.36.x. The range was 1.32.0 to 1.35.x. A cluster on 1.32 is upgraded first. Kubernetes 1.37 is outside the range.
2. Gateway API CRDs at bundle v1.6.0 or later. The data plane is tested against v1.6.1. v1.6.0 has the same CRD schemas and is accepted.

The cluster's Kubernetes version, Gateway API bundle, and release channel:

kubectl version
kubectl get crd gateways.gateway.networking.k8s.io \
  -o go-template='{{ index .metadata.annotations "gateway.networking.k8s.io/bundle-version" }} {{ index .metadata.annotations "gateway.networking.k8s.io/channel" }}{{ "\n" }}'

Who moves the Gateway API CRDs depends on who installed them. The data plane never takes them over from a managed add-on:

Cluster	Owner of the Gateway API CRDs	Action before upgrading the data plane
GKE with the Gateway API add-on (--gateway-api=standard, as in the GCP installation guide)	GKE	Upgrade the cluster to GKE 1.36.4-gke.1391000 or a later 1.36 patch, available on the Regular channel, where the add-on installs v1.6.0. Then hold it on 1.36.
CRDs installed by tare install (channel experimental, or no channel shown)	tare	None beforehand: run tare upgrade with --force-gateway-api-crds. On the Helm path, pass the same flag to tare install --crds-only.
Standard-channel CRDs installed by the operator (EKS, AKS as in the installation guides, or any other cluster)	The operator	Apply the v1.6.1 standard-channel manifest, then upgrade.
GitOps, with dataplane-crds-helm set to crds.gatewayAPI: true	The CRDs Application	Sync the CRDs Application at 0.7.0, then the workload Application.

The standard-channel manifest for the third row:

kubectl apply --server-side \
  -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.6.1/standard-install.yaml

--force-gateway-api-crds is refused on a standard-channel cluster: it applies the experimental CRDs the data plane bundles, which a managed add-on or the Gateway API admission policy rejects.

What else changes in the 0.7.0 upgrade

Three things happen during or after the upgrade that an operator plans around, none of which stops it:

• Upgrade the management plane before the data planes. On a Tetrate-hosted management plane this has already happened. On a deployment that runs its own, the order matters from 0.7.0 on: the data plane now removes the identity headers only the gateway sets, and every x-ai-eg-* and x-internal-traces-* header, from every request on arrival. A data plane upgraded ahead of its management plane removes x-tars-idp-token before the gateway can verify it, on any gateway that already has a gateway auth policy. Callers that send one of those headers for a model provider or Model Context Protocol (MCP) server to read need to move the value to a header the data plane forwards before the upgrade, because the upstream no longer receives it. The full list is under Headers removed from every request.
• Every gateway pod restarts once. The data plane moves to Envoy AI Gateway v1.1.0, and the guardrails filter is now built with the rest of the data plane, so the gateway image tag changes and the upgrade rolls every gateway proxy. The rollout keeps traffic flowing under the pod disruption budget, as any gateway image change does.
• The router_* metrics on the Envoy Prometheus endpoint are renamed. On :19001/stats/prometheus each one now carries an envoy_dynamicmodulescustom_ prefix: envoy_dynamicmodulescustom_router_requests_total where router_requests_total was exposed before. Anyone who scrapes that endpoint directly updates the dashboards and alerts that query router_* names. Metrics sent to the management plane, metrics shipped through an OpenTelemetry Collector that strips the dynamicmodulescustom. scope, and tare doctor are unaffected. See the data plane metrics.

Hold GKE on 1.36

GKE 1.37 is outside the range 0.7.0 supports, and a cluster on a release channel moves to it automatically when the channel does. A cluster cannot be downgraded in place, so stop minor upgrades before that happens. A no_minor_upgrades maintenance exclusion keeps the cluster on 1.36 and still takes 1.36 patches:

gcloud container clusters update "${CLUSTER_NAME}" --zone "${ZONE}" \
  --add-maintenance-exclusion-name=hold-gke-1-36 \
  --add-maintenance-exclusion-scope=no_minor_upgrades \
  --add-maintenance-exclusion-until-end-of-support

Remove the exclusion with --remove-maintenance-exclusion=hold-gke-1-36 once the data plane runs a release that supports 1.37.

AKS Managed Gateway API is not covered

A cluster that runs AKS Managed Gateway API (--enable-gateway-api) is outside the Azure installation guide, which does not enable it. On AKS 1.36 that feature installs Gateway API v1.5.1, below the requirement, and blocks installing the CRDs by hand. Contact Tetrate support before upgrading such a cluster to 0.7.0.

Upgrading to 0.8.0

0.8.0 has no cluster step: the Kubernetes range, the Gateway API CRDs, and Envoy Gateway are the same as for 0.7.0. What needs attention is the CLI. A single tetrate command replaces tare, tars, and admin, and a script that drives the CLI stops or fails at upgrade time unless it is updated first. The release notes mark no breaking change, so the upgrade banner in the Admin Console does not list these steps.

Change scripts before upgrading

Each of these stops an unattended run, or changes what a script reads, on the 0.8 CLI:

1. Pass --yes to commands that change a cluster. Install, upgrade, manage install, manage gateway install, and mcp install now ask for confirmation and name the cluster first. Without a terminal to prompt on, they stop instead of acting. This applies to the tare and tars spellings too.

2. Install tare explicitly where a script runs it. The install script now installs only tetrate by default, into ~/.tetrate/bin. A script that installs the CLI and then calls tare asks for it:

   curl -fsSL https://tare.tetrate.ai/tools/install.sh | bash -s -- --cli tare

   --cli all installs every tool. Every binary the script installs is signature-verified.

3. Drop --proxy-url. tars serve install --proxy-url is removed, and the command fails naming the removal. The management plane endpoint comes from the identity file.

4. Repoint an identity deliberately. Installing an identity that names a different data plane from the one already installed is refused unless --accept-identity-change is passed. An expired identity is refused with no override: a current one is downloaded instead.

5. Spell out --verify-checksum. -v now means --verbose on every command. On tars sa check it used to mean --verify-checksum, which has no short form any more.

6. Read the new exit codes. A mistyped subcommand now fails instead of exiting 0: with 2 under tetrate, and with 1 under the old names. Under tetrate, a command that refuses to act exits 3, and one that changed the cluster before failing exits 4, where both exited 1 before; the old names keep exiting 1 for every failure. A pipeline that treats any non-zero exit as a failure needs no change. One that retries does not retry on 3, and treats 4 as a cluster that was partly changed.

Recommended, not required

• Rename the environment variables. The CLI reads TETRATE_ names: TARS_API_KEY becomes TETRATE_API_KEY (the inference key), and API_KEY becomes TETRATE_ADMIN_API_KEY (the admin credential). The old names still work and warn once.
• Add ~/.tetrate/bin to PATH. The install script prints the line to add. Per-user state moves to ~/.tetrate; the old locations are still read, and the bundled helm and kubectl download once more on first use. Commands that change a cluster log the operations they ran under ~/.tetrate/logs.
• Move scripts to the tetrate names. Tetrate Agent Router's commands live under tetrate agentrouter, so tare doctor becomes tetrate agentrouter doctor. The old names keep working until at least 1 February 2027, and each prints its replacement on stderr, so output parsed from stdout is unchanged. See Deprecations.

After the upgrade

• Every user signs in again once. Sign-in session cookies are now limited to the environment's own domain, where they were sent to every tetrate.ai host before.
• Rate limits count once per key. On a data plane on 0.8.0, one limit on a key covers every model at once, where each model route kept its own counter before, and the per-key counters start at zero. A project-wide rate limit saved before the upgrade is enforced on keys with no settings of their own only once it is next saved. See Set rate limits on projects and keys.
• tare doctor reports more.
  • Missing optional doctor permissions are listed as doctor_self under "Doctor access" and no longer mark the data plane Degraded; upgrading the tare-doctor chart clears them.
  • The first run may report API keys whose routing was never fully created. Those errors were present before and went unreported, so each key named is checked before the report is treated as a regression.
  • The scheduled doctor job now requests 256Mi of memory and 50m of CPU, capped at 512Mi. On a cluster with little headroom it may wait for a node that fits; the values are under tareDoctor.resources in the data plane's values.
  • Data planes deployed with the GCP Terraform module run doctor every 5 minutes instead of 15. Setting tare_doctor.schedule = "*/15 * * * *" keeps the old cadence.
• MCP access changes.
  • Listing, adding, and removing a project's MCP servers needs Project Owner or Super Admin; the MCP Admin role no longer covers it.
  • An OAuth catalog server whose users sign in from their MCP profile needs requires_profile: true.
  • A user whose MCP tool calls ask them to authorize again signs in again from the project they make the calls in.
• Routing previews need a project. A tag preview of routing policy returns the project's policy only when project_id is sent. Organization and tag routing policies are no longer resolved.
• Old Developer Console pages are gone. /config, /config/<id>, and /dashboard-legacy are removed; an API key's configuration is at /api-keys/<id>/config.
• A data plane ID shared with another organization. When URL lookups or Prompt fail for a data plane after the upgrade, another organization uses the same data plane ID: listing the data plane's hostnames under global.gateway.hostnames restores them, as in Declare a data plane's hostnames.

Upgrading to 0.9.0

The release notes mark no breaking change, so the upgrade banner in the Admin Console does not list these steps.

After the upgrade

• Audit records name the person. A change made through either console names the signed-in user and their email, and the console's key ID moves to metadata.via_key, so a SIEM rule or audit-log filter that matches a console's key ID on the user field moves there. Key rotation entries change resourceType from api_key to client_key, with the client ID as resourceId. Project telemetry changes made in the Developer Console are recorded as CREATE_TELEMETRY_CONFIG and UPDATE_TELEMETRY_CONFIG instead of telemetry.project_config.set and telemetry.project_config.disable. See Changes in 0.9.0.

Run the upgrade

• CLI path: tetrate agentrouter dataplane upgrade with the data plane's identity file. The command upgrades atomically and rolls back on failure; flags, the doctor rows, and the drain behavior are in the tetrate agentrouter dataplane upgrade reference. Since 0.5.0 the command also stops when the cluster's Gateway API CRDs are older than the release requires; when the CLI owns the CRDs, --force-gateway-api-crds (or TETRATE_FORCE_GATEWAY_API_CRDS=1) proceeds anyway.
• Helm and GitOps paths: the upgrade is a re-render at the new tag through the same path that installed: pull the new chart version for a Helm install, or re-render and commit the manifests for a GitOps install, CRDs first where they sync as their own Application. Since 0.6.0 these paths get the same Gateway API CRD check as the CLI, run inside the cluster: an install job stops the rollout when the cluster's CRDs are older than the release requires, and the job's log names the version to move to. Move the CRDs there first, through the cloud's managed add-on where one owns them, or by applying the upstream Gateway API release manifest.

Afterward, tetrate agentrouter doctor confirms component state, and the data plane's release appears in the Admin Console data plane list's Version column. On a data plane on 0.8.0 or later the column shows the release read from the deployed chart, so it reflects the upgrade once doctor next runs; a data plane managed by ArgoCD or Flux shows no version. See Check the management plane version and release notes.

Rolling back

Rollback to the previous release is unconditional unless that release's notes carry a rollback condition. A conditional rollback never blocks the rollback itself: the condition names what stops working until the data plane is rolled forward again, so the decision can be made with the blast radius known.

A rollback reverts the data plane workloads, not the CRDs. Every path installs the CRDs outside the workload release (the CLI CRD phase, tetrate agentrouter dataplane install --crds-only, or the CRDs Application), so the CRDs stay at the newer release. Re-applying the previous release's CRDs is not a rollback step: CRDs move forward only.

Rolling back from 0.7.0

0.7.0 carries a rollback condition: the workloads go back to 0.6.x, and the Gateway API, Envoy Gateway, and Envoy AI Gateway CRDs stay at 0.7.0's versions. 0.6.x runs on them unchanged: Gateway API v1.6 still serves every API version 0.6.x reads, and the Envoy Gateway and Envoy AI Gateway CRDs only gained fields, which 0.6.x ignores.

The CRDs cannot follow the workloads back. After the upgrade, TCPRoute, UDPRoute, and TLSRoute are stored at v1, which the previous Gateway API bundle does not define, so the API server refuses an older CRD that drops it. Where the data plane installed the Gateway API CRDs, the safe-upgrades.gateway.networking.k8s.io admission policy also refuses any bundle older than v1.5.

Path	Roll back with	Do not
tare CLI	helm rollback tars -n tars-system	Run a 0.6.x tare upgrade or tare install to go back.
Helm	helm rollback tars -n tars-system	Re-run tare install --crds-only with a 0.6.x tare.
GitOps	Point the workload Application back at the 0.6.x render.	Move the CRDs Application back; keep it at 0.7.0.
GKE with the Gateway API add-on	Roll back the data plane as above.	Downgrade the cluster; it stays on GKE 1.36.

A 0.6.x tare re-applies 0.6.x's CRDs: the Gateway API ones, where it applies them, are refused, and the Envoy Gateway and Envoy AI Gateway ones would drop the fields 0.7.0 added. helm rollback restarts the gateway pods once more, onto the 0.6.x images.

helm history tars -n tars-system
helm rollback tars -n tars-system --wait --timeout 10m

---

Where to go next

tetrate agentrouter dataplane upgrade

Flags, atomic rollback, drain behavior, and the doctor rows.

Release notes

What changed in each release, with upgrade and rollback notes.

Release compatibility

Kubernetes range, Envoy Gateway, and minimum data plane per release.

Deprecations

The removal policy and what is on the clock today.