Skip to main content

tetrate agentrouter dataplane upgrade

Upgrade an installed Tetrate Agent Router data plane to the version embedded in the current tetrate binary, with zero downtime when the data plane is HA.


Synopsis​

tetrate agentrouter dataplane upgrade <identity-file> [flags]

Description​

tetrate agentrouter dataplane upgrade performs the operator workflow for moving an existing Agent Router data plane release to a newer version. Unlike tetrate agentrouter dataplane install, it does not fresh-install and does not re-prompt for configuration the operator already supplied at install time. Instead it reads the live release's Helm values via helm get values and carries the operator-supplied config forward.

Phases:

  1. Preflight (Helm 3+, kubectl, cluster connectivity; warns when metrics-server is absent and HA is on)
  2. Discover existing release in the target namespace (refuses to run if none is found: that path is tetrate agentrouter dataplane install)
  3. Read existing release values and carry forward registry, customer, environment, deployment mode, serve URL, image pull secret name, and the OTel collector enable + endpoint + auth-header config
  4. HA preflight: read the EnvoyProxy CRD's envoyHpa.minReplicas; refuse to upgrade if it is < 2 unless --allow-downtime is set
  5. Apply CRDs from the embedded chart (Helm's crds/ directory is install-only, so an upgrade has to upsert CRDs explicitly)
  6. helm upgrade --install --atomic --wait against the embedded chart

helm and kubectl are downloaded automatically on first run into ~/.tetrate/tools/ (~/.tare/tools/ before 0.8.0) and reused on subsequent runs.

The new tetrate binary's embedded chart supplies the new chart defaults: notably the HA-by-default flip and the EnvoyProxy.spec.shutdown block. Existing operator-supplied values override chart defaults where they overlap.

Image sync is out of scope

tetrate agentrouter dataplane upgrade does not pull new images into the operator's registry. Sync first, then upgrade:

tetrate agentrouter dataplane install identity.json --image-sync <REG> --sync-only
tetrate agentrouter dataplane upgrade identity.json

Usage​

Standard upgrade​

After installing a newer tetrate binary with curl -fsSL https://tare.tetrate.ai/tools/install.sh | bash, which places it in ~/.tetrate/bin/:

tetrate agentrouter dataplane upgrade identity.json

Single-replica install (lab / non-production)​

If the existing install is single-replica (an install from a release that predates the HA default, or tetrate agentrouter dataplane install --ha=false), tetrate agentrouter dataplane upgrade refuses to proceed because rolling the single data-plane Envoy pod would drop in-flight LLM streams:

data-plane Envoy is single-replica (envoyHpa.minReplicas=1).
Upgrading now will RST in-flight LLM streams.

Fix HA first (idempotent, takes ~1m):
tetrate agentrouter dataplane install --ha identity.json

Or force this upgrade with downtime:
tetrate agentrouter dataplane upgrade identity.json --allow-downtime

The recommended path is to migrate to HA first; see Migration: single replica to HA.

Override carried-forward values​

CLI flags take precedence over carryover. Override only what has to change, e.g., to move to a new image registry as part of the upgrade:

# 1. Sync to the new registry
tetrate agentrouter dataplane install identity.json --image-sync newregistry.example.com --sync-only

# 2. Upgrade using the new registry
tetrate agentrouter dataplane upgrade identity.json --image-registry newregistry.example.com

Declare or change the hostnames​

tetrate agentrouter dataplane upgrade carries global.gateway.hostnames forward from the installed release, so an upgrade with no hostname flag keeps the list the data plane already declares. A release installed before 0.7.0 has no such key, and the upgrade leaves it out, so the data plane keeps the address the management plane stores for it.

Pass --gateway-hostname to replace the list. The flag is repeatable, and the first entry is the address the consoles print:

tetrate agentrouter dataplane upgrade identity.json --gateway-hostname=eu.acme.example.com

The first name declared turns on a check on the management plane: from then on a project gateway URL can only name a hostname this data plane declares. List every hostname the data plane's existing project gateways already serve before upgrading with the flag. --gateway-hostname="" on its own declares that the data plane serves no hostname: the management plane retires the names it recorded, and the readiness probe stops answering. See Move a data plane to declared hostnames.

Disable atomic rollback​

When debugging a stuck release, run without --atomic to leave the half-deployed release in place:

tetrate agentrouter dataplane upgrade identity.json --no-atomic

The default --atomic is the safer behavior for production. Use this flag only to inspect a failed rollout.

Flags​

Main​

FlagDefaultDescription
--gateway-hostname <host[:port]>carry-forwardSince 0.7.0. Replace the hostnames this data plane serves (global.gateway.hostnames). Repeatable. Without it the installed list is kept; --gateway-hostname="" alone declares no hostname.
--allow-downtimefalseProceed even when the data plane is single-replica. In-flight LLM streams are dropped.
--hatrueHA-safe defaults for the data-plane Envoy proxy (HPA min 2, PDB min 1). Pass --ha=false to keep single-replica.
--drain-timeout-seconds300EnvoyProxy.spec.shutdown.drainTimeout (seconds). Maximum time Envoy waits for in-flight requests (long LLM streams) to finish before SIGKILL.
--no-atomicfalseDisable helm --atomic (no auto-rollback on failure). Leaves stuck releases in place for debugging.
--timeout10mhelm upgrade --wait timeout. Should exceed drainTimeout × replicas to allow serial drain.

Telemetry​

FlagDefaultDescription
--enable-otel-collectorcarry-forwardOverride the OTel collector enable state. Default carries the existing release's value.
--otel-collector-endpoint <URL>carry-forwardOverride OTel OTLP endpoint.
--otel-exporter-auth-headers <H>carry-forwardOverride OTel Authorization header value.

Advanced / hidden​

Hidden from --help but accepted for advanced use. A carry-forward default keeps the installed release's value unless overridden.

FlagDefaultDescription
--release-nametarsHelm release name.
--system-namespacetars-systemKubernetes system namespace.
--dataplane-namespacetars-dataplaneKubernetes dataplane namespace.
--chart-pathembeddedPath or OCI/HTTP reference to chart.
--chart-versionnoneChart version (for remote/OCI charts).
--image-registrycarry-forwardImage registry override.
--image-tagembedded manifestImage tag override.
--deployment-modecarry-forwardenterprise or saas.
--customercarry-forwardCustomer label override.
--environmentcarry-forwardEnvironment label override.
--serve-urlignoredDeprecated since 0.3.0: accepted, prints a notice, and is ignored. Declare the hostnames with --gateway-hostname.
--redis-typeclustercluster or hosted.
--no-rate-limitingfalseDisable rate limiting.
--skip-preflightfalseSkip Helm/kubectl/cluster preflight (also TETRATE_SKIP_PREFLIGHT=1).

Carryover behavior​

tetrate agentrouter dataplane upgrade reads the existing release's user-supplied values (helm get values returns only what was passed via -f or --set at install time, not chart defaults). The following fields are carried forward into the regenerated values:

Helm values pathCLI flag override
global.imageRegistry--image-registry
global.customer--customer
global.environment--environment
global.deploymentMode--deployment-mode
global.serveUrl(no override; --serve-url is deprecated and ignored)
global.gateway.hostnames--gateway-hostname (since 0.7.0; left out when the installed release has no such key)
global.imagePullSecretName(no override; lives in operator-supplied pull secret)
otelCollector.enabled--enable-otel-collector
otelCollector.exporters.otlp.endpoint--otel-collector-endpoint
otelCollector.exporters.otlp.headers.Authorization--otel-exporter-auth-headers

Fields not in the carryover list fall back to embedded chart defaults (from the new tetrate binary). Pass the corresponding flag to override.

Single-replica preflight​

tetrate agentrouter dataplane upgrade reads spec.provider.kubernetes.envoyHpa.minReplicas from the EnvoyProxy CRD in the system namespace. The check has three outcomes:

  • minReplicas ≥ 2: proceed.
  • minReplicas < 2 and --allow-downtime not set: abort with the migration message.
  • EnvoyProxy resource not found, or kubectl call fails: print a warning and skip the preflight (operator's responsibility to verify HA).

The preflight does not inspect live Deployment.spec.replicas: the EnvoyProxy CRD HPA configuration is the source of truth for how the data plane scales over time.

Atomic rollback​

The default helm upgrade --atomic makes the upgrade transactional: if any pod fails its readiness probe within --timeout, Helm rolls the release back to the previous revision and exits non-zero. The data plane returns to its previous state without operator intervention.

--no-atomic disables this: useful for inspecting a stuck release mid-rollout, at the cost of leaving traffic potentially affected.

Output​

tetrate agentrouter dataplane upgrade writes phase-prefixed status lines ([i/N] <phase>) to stderr. Final lines include the rollback hint and inspection commands:

[1/5] Preflight (Helm / kubectl / cluster)
[2/5] Discover existing release
release tars found in tars-system
carried forward: registry="gcr.io/acme/tars" customer="acme" env="" otel=true
[3/5] HA preflight
EnvoyProxy minReplicas=2 (HA)
Using embedded chart
[4/5] Apply / update CRDs
3 created, 12 updated, 4 skipped
[5/5] Helm upgrade
...
Upgrade completed.

Rollback (if needed):
helm rollback tars -n tars-system
Inspect:
kubectl get pods -n tars-system
kubectl get pods -n tars-dataplane

Verification​

# Pods in both namespaces should be Running with the new image tag
kubectl get pods -n tars-system
kubectl get pods -n tars-dataplane

# EnvoyProxy HA configuration
kubectl get envoyproxy -n tars-system -o yaml | yq .spec.provider.kubernetes.envoyHpa

# Active rollout status
kubectl get deploy -n tars-dataplane -o wide

Management-plane event reporting​

tetrate agentrouter dataplane upgrade reports the upgrade as an event to the management plane via the same in-cluster ConfigMap outbox that tetrate agentrouter dataplane install uses. After a successful helm upgrade, the CLI writes a ConfigMap labeled tars.io/component=install-event to tars-system carrying operation=upgrade and the new Helm revision. The tare-doctor CronJob forwards it on its next tick (default every 5 minutes); the Admin Console's deployment-history timeline shows the row at helm_revision = N+1 with from_version and to_version populated when chart versions differ.

The event POST never blocks the upgrade and never changes the exit code.

Failed atomic-rollback upgrades (helm upgrade --atomic that triggers a rollback) produce two Admin Console rows:

  • One for the failed upgrade attempt: operation: upgrade, status: failed, error_summary populated.
  • One for the auto-rollback: operation: rollback, status: success, on the new helm revision the rollback created. This is the tare-doctor reconciler recognizing helm's "Rollback to N" description and labeling the artifact correctly, instead of attributing the system reversion to the operator who triggered the upgrade.

Rollback​

tetrate agentrouter dataplane upgrade defaults to --atomic, so failed upgrades roll back automatically. To roll back a successful but problematic upgrade:

# View revision history
helm history tars -n tars-system

# Roll back to the previous revision
helm rollback tars -n tars-system

# Or to a specific revision number
helm rollback tars 5 -n tars-system
note

helm rollback does not roll back CRDs. Any CRD changes applied during the upgrade remain in place. CRDs move forward only: an upgrade can change a CRD's stored version (0.7.0 stores Gateway API TCPRoute, UDPRoute and TLSRoute at v1), after which the API server refuses the older CRD. Do not re-apply the previous release's CRDs to complete a rollback. See Rolling back.

Migration: single replica to HA​

Customers whose tetrate agentrouter dataplane install ran before the HA default flip are on single-replica installs. The migration path is one idempotent install call that bumps the EnvoyProxy CRD's HPA minReplicas from 1 to 2:

tetrate agentrouter dataplane install --ha identity.json

This calls helm upgrade --install, so it does not require uninstall and does not change any other configuration. The Envoy Gateway controller scales the data-plane Deployment up to 2 pods (a one-time brief disruption while the second pod comes up).

After this migration, tetrate agentrouter dataplane upgrade is zero-downtime.