Agent Router data plane installation using Helm

The data plane is installed with plain Helm from a private container registry (Azure Container Registry, Harbor, JFrog Artifactory, Nexus, or Google Artifact Registry), for platform teams that own the Helm release lifecycle.

---

In this guide:

• tare syncs Agent Router release artifacts into the private registry.
• tare applies CRDs outside Helm.
• tare renders Helm values.
• Helm installs or upgrades the data plane workload from the private registry.

Plan for 30 to 60 minutes, plus the time to mirror artifacts into the private registry.

info

This procedure installs the data plane only: it does not create a public gateway (Kubernetes Gateway API resource), ingress, DNS record, or TLS certificate. Exposure is covered in Step 9.

Architecture overview

Agent Router Enterprise uses a split-plane model. The management plane, hosted by Tetrate, holds configuration and serves the Console and the Admin Console. The data plane runs in customer-managed cluster infrastructure and handles all AI traffic. The two planes communicate over a single outbound HTTPS connection initiated by the data plane; no inbound connections from the internet are required.

Overview

• Prepare: obtain the credential and set install variables
  • Prerequisites
  • Step 1: Obtain the data plane credential
  • Step 2: Set variables
• Mirror: copy Agent Router artifacts to the private registry
  • Step 3: Sync release artifacts
• Install: apply CRDs and deploy with Helm
  • Step 4: Apply CRDs
  • Step 5: Render Helm values, with the optional egress NetworkPolicy
  • Step 6: Create namespaces and pull secrets
  • Step 7: Install or upgrade with Helm
• Verify and expose: confirm health, expose the service, and smoke test
  • Step 8: Verify
  • Step 9: Expose the data plane
  • Step 10: Validation and smoke tests
• Operate
  • Upgrade and rollback
  • If artifacts are already mirrored
  • Troubleshooting

Prerequisites

Cluster requirements

Already running Envoy Gateway cluster-wide?

The data plane installs a GatewayClass named tars-egress whose controllerName is the Envoy Gateway default, gateway.envoyproxy.io/gatewayclass-controller. An Envoy Gateway that already watches the whole cluster claims tars-egress too, and the two controllers contend over it. There is no override: install the data plane in a cluster of its own. See A cluster already running Envoy Gateway.

Required access and tools

• An Agent Router data plane credential from the Admin Console, saved as data-plane-credentials.json.
• tare, helm, and kubectl installed locally.
• Access to the target Kubernetes cluster.
• Push access to the private registry mirror.
• Pull access from the target cluster to the same private registry.

Conventions

Later steps reuse the shell variables defined in Step 2: Set variables, such as DP_CREDENTIAL and PRIVATE_IMAGE_REGISTRY. The steps assume a single shell session. If the session changes, re-export the variables before continuing.

Step 1: obtain the data plane credential

In the Admin Console, go to Directory → Data Planes.

If this data plane is not registered yet:

1. Click Register Data plane.
2. Set Data plane ID to a stable ID such as acme-prod.
3. Optionally set Name, Description, and Labels.
4. Click Register Data plane.

Then generate the credential from the data plane row:

1. Click the row's edit action to open the Data plane drawer.
2. In the Credential section, click Generate credential.
3. Save the downloaded file as data-plane-credentials.json on the machine the install runs from.

Each data plane uses its own credential. A credential can be rotated or revoked from the same Credential section in the data plane edit drawer.

Step 2: set variables

For Azure Container Registry:

DP_CREDENTIAL=./data-plane-credentials.json
PRIVATE_REGISTRY_HOST=${ACR_NAME}.azurecr.io
PRIVATE_IMAGE_REGISTRY=${ACR_NAME}.azurecr.io/tare
PULL_SECRET=tare-registry-pull

For another private registry:

DP_CREDENTIAL=./data-plane-credentials.json
PRIVATE_REGISTRY_HOST=registry.acme.example.com
PRIVATE_IMAGE_REGISTRY=registry.acme.example.com/tare
PULL_SECRET=tare-registry-pull
REGISTRY_USERNAME=<registry-user>
REGISTRY_PASSWORD=<registry-password-or-token>

Step 3: sync release artifacts

Log in locally with credentials that can push to the destination registry.

For ACR:

az acr login --name "${ACR_NAME}"

For a generic private registry:

printf '%s' "${REGISTRY_PASSWORD}" | \
  docker login "${PRIVATE_REGISTRY_HOST}" \
    --username "${REGISTRY_USERNAME}" \
    --password-stdin

Copy the pinned Agent Router images and the serve-helm OCI chart:

tare install "${DP_CREDENTIAL}" \
  --image-sync "${PRIVATE_IMAGE_REGISTRY}" \
  --sync-only \
  --parallel 2 \
  --stall-threshold 5m

tare uses the data plane credential to pull from Tetrate's source registry. The local registry login is used only to push to the private registry.

Step 4: apply CRDs

Apply CRDs outside Helm:

tare install "${DP_CREDENTIAL}" \
  --crds-only \
  --image-registry "${PRIVATE_IMAGE_REGISTRY}"

CRDs are intentionally managed outside the Helm release so helm uninstall does not delete cluster-scoped APIs and cascade-delete custom resources.

Step 5: render Helm values

Render values for the Helm release:

tare install "${DP_CREDENTIAL}" \
  --image-registry "${PRIVATE_IMAGE_REGISTRY}" \
  --gateway-hostname=ai-gateway.acme.com \
  --print-helm-values > values.yaml

--gateway-hostname declares a hostname client applications call, and the rendered values carry it under global.gateway.hostnames. The flag is repeatable; put the name callers should use first. The hostname does not need to resolve yet. When it is not known yet, leave the flag out and add the list to values.yaml later, as described in Step 9. Every later render that regenerates values.yaml needs the same flags, or the list is lost.

Treat values.yaml as sensitive because it contains the data plane identity secret. Do not commit it as plaintext. Use SOPS, Sealed Secrets, External Secrets, or the organization's approved secret workflow.

Make sure the rendered values do not set the legacy serveUrl to the management-plane API URL. Keep these fields empty:

global:
  serveUrl: ""
controller:
  config:
    proxyUrl: ""

If needed, override them during Helm install:

--set-string global.serveUrl= \
--set-string controller.config.proxyUrl=

Optional: restrict the gateway pods' egress with a NetworkPolicy

From 0.7.0 the chart can install a NetworkPolicy for the gateway proxy pods. It is off by default. When enabled, the pods can reach DNS, every pod in the cluster, and any off-cluster address outside the internal and reserved ranges (cloud metadata, RFC 1918, carrier-grade NAT, link-local, loopback, multicast, and the other reserved IPv4 and IPv6 ranges). Every other connection is refused, which stops the gateway fetching a gateway auth policy's jwks_uri from an internal address. The policy takes effect only on a CNI that enforces NetworkPolicy, such as Calico or Cilium, and does not match an internal IPv4 address reached through NAT64.

Value	Default	What it does
egressResources.egress.networkPolicy.enabled	false	Renders the policy
egressResources.egress.networkPolicy.extraEgress	[]	Egress rules appended as written, for each off-cluster private destination the pods still need
egressResources.egress.networkPolicy.inClusterNamespaceSelector	{}	Narrows which namespaces' pods stay reachable; empty keeps every namespace, which the gateway needs for its external processor, guardrails filter, OTLP receiver, and any in-cluster model service
egressResources.egress.networkPolicy.allowIPv6	true	Allows off-cluster IPv6 outside the internal ranges; false allows no off-cluster IPv6 at all
egressResources.egress.networkPolicy.extraExceptCIDRs, extraExceptCIDRsV6	[]	More off-cluster ranges to refuse, such as an on-premises management subnet

Before setting enabled: true, list in extraEgress every off-cluster private destination the gateway still reaches. The ones commonly missed:

• identity providers and their key endpoints
• outbound proxies
• the management plane, where it is reached over a private network
• telemetry collectors
• model endpoints on a private network or a private link
• a node-local DNS cache on a link-local address, such as 169.254.20.10

egressResources:
  egress:
    networkPolicy:
      enabled: true
      extraEgress:
        - to:
            - ipBlock:
                cidr: 10.20.0.0/16
          ports:
            - protocol: TCP
              port: 443

Add the block to values.yaml after rendering it in Step 5, and keep it there for every later helm upgrade.

Step 6: create namespaces and pull secrets

Create the data plane namespaces:

kubectl create namespace tars-system --dry-run=client -o yaml | kubectl apply -f -
kubectl create namespace tars-dataplane --dry-run=client -o yaml | kubectl apply -f -

For AKS attached to ACR, no image pull secret is usually needed after running:

az aks update \
  --resource-group "${RESOURCE_GROUP}" \
  --name "${AKS_CLUSTER_NAME}" \
  --attach-acr "${ACR_NAME}"

For a private registry that requires Kubernetes pull credentials, create the same secret in both namespaces:

kubectl create secret docker-registry "${PULL_SECRET}" \
  --docker-server="${PRIVATE_REGISTRY_HOST}" \
  --docker-username="${REGISTRY_USERNAME}" \
  --docker-password="${REGISTRY_PASSWORD}" \
  --namespace tars-system \
  --dry-run=client -o yaml | kubectl apply -f -

kubectl create secret docker-registry "${PULL_SECRET}" \
  --docker-server="${PRIVATE_REGISTRY_HOST}" \
  --docker-username="${REGISTRY_USERNAME}" \
  --docker-password="${REGISTRY_PASSWORD}" \
  --namespace tars-dataplane \
  --dry-run=client -o yaml | kubectl apply -f -

When using a pull secret, render values with:

tare install "${DP_CREDENTIAL}" \
  --image-registry "${PRIVATE_IMAGE_REGISTRY}" \
  --image-pull-secret-name "${PULL_SECRET}" \
  --gateway-hostname=ai-gateway.acme.com \
  --print-helm-values > values.yaml

Step 7: install or upgrade with Helm

Use the chart version for the Agent Router release:

CHART_VERSION="0.7.0"

Install or upgrade the release:

helm upgrade --install tars "oci://${PRIVATE_IMAGE_REGISTRY}/serve-helm" \
  --version "${CHART_VERSION}" \
  -f values.yaml \
  -n tars-system \
  --skip-crds \
  --set ai-gateway-crds.enabled=false \
  --set-string global.serveUrl= \
  --set-string controller.config.proxyUrl=

Pass the same values.yaml on every later helm upgrade. An upgrade without the file that carries global.gateway.hostnames drops the key, and the data plane falls back to the address stored for it.

Use --skip-crds and --set ai-gateway-crds.enabled=false together. The first skips chart crds/ entries; the second disables AI Gateway CRDs rendered as normal Helm templates.

A release chart carries the release version, such as 0.7.0, and since 0.7.0 a prerelease chart reads 0.7.0-rc.1.gabc1234, one string that is valid as both a SemVer version and an OCI tag. A chart from before 0.7.0 carried its commit as + build metadata, such as 0.6.0-rc.3+8d71fcc: OCI stores that tag with _, but helm --version still takes the + form.

Step 8: verify

Check the Helm release:

helm list -n tars-system
helm status tars -n tars-system

Wait for workloads:

kubectl rollout status deployment/controller -n tars-system --timeout=300s
kubectl rollout status deployment/egress -n tars-dataplane --timeout=300s

Confirm images come from the private registry:

kubectl get deploy egress -n tars-dataplane \
  -o jsonpath='{range .spec.template.spec.containers[*]}{.name}={.image}{"\n"}{end}'

Run doctor:

tare doctor "${DP_CREDENTIAL}" --verbose

A fresh data plane with no routes can report warnings for missing RouteDeployments, AIGatewayRoutes, HTTPRoutes, or EnvoyPatchPolicies. The core install is healthy when pods roll out and images pull from the private registry.

Step 9: expose the data plane

This Helm workflow installs the data plane but does not create a public gateway (Kubernetes Gateway API resource), ingress, DNS record, or TLS certificate.

For gateway exposure, follow the published cloud install guide for the target platform, or use the organization's preferred ingress or Gateway stack.

The data plane serves traffic through the in-cluster egress service:

• Service: egress
• Namespace: tars-dataplane
• Port: 10080
• Paths: /v1/*, /mcp/*, and /.well-known/*

The hostname client applications call is declared in the chart, not in the Admin Console. If values.yaml does not carry it yet, add the list and re-run the helm upgrade from Step 7:

global:
  gateway:
    hostnames:
      - ai-gateway.acme.com   # first: the address the consoles print

Confirm the chart rendered it. The Admin Console lists the same name on the data plane's page under Directory → Data Planes after the next configuration poll:

kubectl -n tars-system get configmap tars-gateway-declaration -o jsonpath='{.data.gateway-hostnames}'

Declaring the hostname creates no DNS record, certificate, or route; the gateway stack above provides those. A data plane older than 0.7.0 cannot declare a hostname, and its address is set through the deprecated SetDataplaneURL admin API. The rules for the list and the certificate settings are in Declare the hostnames a data plane serves.

Step 10: validation and smoke tests

A healthy install can still read Degraded in the Console

Enabling a provider or model in the organization catalog makes it available to grant, not routable. Nothing routes through this data plane until a model is granted to a project attached to it, and until then the cluster reads healthy while the Console shows Degraded and the gateway's model list stays empty. That state is configuration, not a failed install: see Add providers and models to projects.

Validate the Helm install first:

helm status tars -n tars-system
kubectl rollout status deployment/controller -n tars-system --timeout=300s
kubectl rollout status deployment/egress -n tars-dataplane --timeout=300s
kubectl get deploy egress -n tars-dataplane \
  -o jsonpath='{range .spec.template.spec.containers[*]}{.name}={.image}{"\n"}{end}'
tare doctor "${DP_CREDENTIAL}" --verbose

Then run traffic smoke tests after onboarding has configured provider keys and routes. Until the gateway is exposed, use a local port-forward to the in-cluster egress service:

kubectl port-forward -n tars-dataplane svc/egress 18080:10080

In another terminal, set the local endpoint and an API key from the Developer Console:

export DP_SCHEME=http
export DP_HOST=127.0.0.1:18080
export TARS_API_KEY=<your-api-key-from-router-app>

At minimum, verify model listing with and without authentication:

curl -s -o /tmp/tare-models.json -w "%{http_code}\n" \
  "${DP_SCHEME}://${DP_HOST}/v1/models" \
  -H "Authorization: Bearer ${TARS_API_KEY}"

curl -s -o /tmp/tare-models-no-auth.json -w "%{http_code}\n" \
  "${DP_SCHEME}://${DP_HOST}/v1/models"

Expected result: the authenticated request returns 200; the unauthenticated request returns 401.

Also smoke test every API shape enabled during onboarding, such as /v1/chat/completions, /v1/responses, /v1/messages, /v1/embeddings, /v1/images/generations, /v1/audio/speech, and MCP profile paths.

Upgrade and rollback

Use the same split workflow for upgrades: update CRDs first, then upgrade the Helm workload release.

If the new Agent Router release includes CRD changes, re-run CRD apply:

tare install "${DP_CREDENTIAL}" \
  --crds-only \
  --image-registry "${PRIVATE_IMAGE_REGISTRY}"

Run it with the tare of the release being upgraded to. When that release raises the Gateway API requirement, as 0.7.0 does, the command stops on a cluster below it: move the CRDs first through the managed add-on that owns them, or, where tare installed them, add --force-gateway-api-crds. See Upgrading to 0.7.0.

Render values for the new release:

tare install "${DP_CREDENTIAL}" \
  --image-registry "${PRIVATE_IMAGE_REGISTRY}" \
  --gateway-hostname=ai-gateway.acme.com \
  --print-helm-values > values.yaml

If a pull secret is used, include it when rendering values:

tare install "${DP_CREDENTIAL}" \
  --image-registry "${PRIVATE_IMAGE_REGISTRY}" \
  --image-pull-secret-name "${PULL_SECRET}" \
  --gateway-hostname=ai-gateway.acme.com \
  --print-helm-values > values.yaml

Upgrade the workload release:

CHART_VERSION="<new-chart-version>"

helm upgrade tars "oci://${PRIVATE_IMAGE_REGISTRY}/serve-helm" \
  --version "${CHART_VERSION}" \
  -f values.yaml \
  -n tars-system \
  --skip-crds \
  --set ai-gateway-crds.enabled=false \
  --set-string global.serveUrl= \
  --set-string controller.config.proxyUrl=

Rollback uses Helm's normal rollback flow:

helm rollback tars -n tars-system

CRDs are not affected by rollback because they are not part of the Helm release manifest. Existing custom resources remain in the cluster. Leave the CRDs at the newer release: do not re-run tare install --crds-only with the previous tare. CRDs move forward only, and from 0.7.0 on the cluster refuses the previous Gateway API CRDs. See Rolling back.

If artifacts are already mirrored

If the Agent Router images and chart are already mirrored, skip the --image-sync step and start at CRD apply:

tare install "${DP_CREDENTIAL}" \
  --crds-only \
  --image-registry "${PRIVATE_IMAGE_REGISTRY}"

Then render values and install with Helm as above.

Troubleshooting

Destination registry reports repository not found

Create the destination repository before syncing. For example, with Google Artifact Registry:

gcloud artifacts repositories create tare \
  --repository-format docker \
  --location <region>

ACR and many enterprise registries create repositories on first push, but some registries require pre-creation.

Helm reports invalid ownership metadata

This usually means Helm is trying to adopt CRDs already applied by tare --crds-only.

Example:

CustomResourceDefinition "aigatewayroutes.aigateway.envoyproxy.io" exists and cannot be imported into the current release: invalid ownership metadata

Install or upgrade with both:

--skip-crds \
--set ai-gateway-crds.enabled=false

Helm reports improper constraint

helm --version expects a SemVer constraint, not a bare commit SHA. Use the chart's full SemVer, such as 0.7.0 for a release or 0.7.0-rc.1.gabc1234 for a prerelease (a chart from before 0.7.0 reads 0.6.0-rc.3+8d71fcc).

Bare commit SHAs are rejected before Helm contacts the registry. The registry stores SemVer build metadata with _, but helm --version should use the SemVer form with +.

Pods are stuck in ImagePullBackOff

Check that:

• the private registry contains every image tag for the Agent Router release
• the cluster identity can pull from the private registry
• pull secrets exist in both tars-system and tars-dataplane when required
• the pull secret is referenced from values.yaml; if not, re-render values with --image-pull-secret-name
• values.yaml points global.imageRegistry at the intended registry

Helm cannot pull the chart from the mirror

If helm install or helm upgrade fails locally with pull access denied, log in to the OCI registry from the operator machine:

helm registry login "${PRIVATE_REGISTRY_HOST}"

This is separate from Kubernetes image pull access. Helm pulls the chart artifact from the local machine; the cluster pulls workload images using node identity or image pull secrets.

---

Where to go next

Gateway installation

Install the data plane gateway components that manage inbound access and configure request routing.

Self-Hosted Data Plane

See the full install sequence: gateway, SSO, then developer handoff.