Skip to main content

Agent Router data plane installation for AWS

This guide installs the Tetrate Agent Router data plane on AWS Elastic Kubernetes Service (EKS).

Overview​

The data plane runs in the target AWS account and handles all AI traffic. It connects to the Tetrate-hosted management plane over one outbound HTTPS connection; no inbound internet connections are required. See Planes and core components.

By the end of this guide:

  • An EKS cluster with the EBS CSI driver installed
  • An ECR registry mirroring the Agent Router images
  • The Agent Router data plane running in the tars-system and tars-dataplane namespaces
  • An Application Load Balancer fronting the data plane at a DNS name the organization owns
  • An OpenTelemetry pipeline forwarding metrics to the observability stack

Plan for 30 to 45 minutes of installation time, plus DNS propagation.

Table of contents​

Prerequisites​

Cluster requirements​

Already running Envoy Gateway cluster-wide?

The data plane installs a GatewayClass named tars-egress whose controllerName is the Envoy Gateway default, gateway.envoyproxy.io/gatewayclass-controller. An Envoy Gateway that already watches the whole cluster claims tars-egress too, and the two controllers contend over it. There is no override: install the data plane in a cluster of its own. See A cluster already running Envoy Gateway.

Admin Console and Developer Console access​

Both URLs are provided during onboarding:

  • The Admin Console (admin): https://dashboard.<your-tenant>.tetrate.ai. Used in Step 1 and Step 9.
  • The Developer Console (end-user): https://router.<your-tenant>.tetrate.ai. Used in Step 11 to create API keys for callers.

Tools​

Required on the host that runs the guide:

ToolInstall
aws CLI v2https://docs.aws.amazon.com/cli/latest/userguide/install.html
eksctl ≥ 0.220https://eksctl.io/installation/ (or brew install eksctl). Older versions do not recognize current EKS K8s versions.
kubectlhttps://kubernetes.io/docs/tasks/tools/
helm (3+)https://helm.sh/docs/intro/install/
dockerhttps://docs.docker.com/get-docker/
curlpreinstalled on macOS and most Linux distros
tetrate CLIcovered in Step 2

A data-plane-credentials.json file is also required; see Step 1.

Infrastructure​

A dedicated workload cluster must be provisioned before starting the installation. The cluster should consist of at least three nodes. See Cluster sizing for more details.

warning

Tetrate support does not cover client-side infrastructure provisioning or Kubernetes issues. The instructions for creating clusters and related infrastructure components are provided as a courtesy and should be carefully evaluated before executing them.

AWS permissions​

The operator needs permission to manage these AWS services on the target account:

ServiceUsed for
EKS (AmazonEKSClusterPolicy, AmazonEKSWorkerNodePolicy)Cluster creation and node groups
EC2 + VPCCluster networking, EBS volumes, ELB provisioning
IAMOIDC provider, IRSA roles for EBS CSI driver and AWS Load Balancer Controller
CloudFormationeksctl deploys everything via CloudFormation stacks
ECRCreate repositories, push/pull images
ELB (Elastic Load Balancing)ALB created by the AWS Load Balancer Controller

The simplest setup is an admin role on a sandbox account. For a constrained role, AWS publishes the minimum policy eksctl needs at https://eksctl.io/usage/minimum-iam-policies/.

Cluster sizing​

The chart's default install runs multiple always-on components, including the AI Gateway proxy (Envoy-based; referred to as egress envoy in manifests and logs) with a minimum of 2 replicas, AI gateway controller and ext_proc, controller / worker, Redis / rate-limit.

The egress envoy is the dominant resource consumer. CPU and memory grow with the configuration size the proxy holds in memory: the number of AIGatewayRoute and AIServiceBackend resources, header-mutation rules, and other per-route features. The AI gateway team's scaling benchmark shows roughly linear growth as routes are added. Capacity should allow for routes to grow as providers, models, and projects are added. General-purpose EC2 families (m5.* or m6i.*) are the default choice.

SizeUse caseRecommended node poolApprox allocatable target
Smalldev / test / low traffic3 × m5.large≥ 6 vCPU, ≥ 20 GiB RAM
Mediumstaging / light production3 × m5.xlarge≥ 12 vCPU, ≥ 40 GiB RAM
Highproduction with burst headroom3 × m5.2xlarge (or split into system + data plane node groups)≥ 24 vCPU, ≥ 80 GiB RAM

Practical floor: 3 nodes minimum to survive a node drain or upgrade. Demo installs can start at Small; production should start at Medium.

Conventions​

Environment variables (AWS_REGION, EKS_CLUSTER_NAME, AWS_ACCOUNT, ECR_HOST, CREDENTIAL_FILE, SERVE_URL, and others) are exported as they become needed, and each later step assumes they are still exported in the current shell. After starting a new shell mid-install, re-export them.

Step 1: obtain the data plane credential​

If the management plane was just onboarded, the credential was already downloaded in Step 5 of the wizard, and this step can be skipped.

To register a new data plane, go to Directory → Data Planes in the Admin Console and select + Register Data plane at the top of the screen.

At a minimum the Data plane ID is required to save the configuration. The rest can be configured later.

Save the downloaded file as data-plane-credentials.json on the host where the install runs. This file is the long-lived identity the data plane uses to authenticate to the management plane.

note

Some parts of the product still use older "service account" naming for this file. CLI output and in-cluster paths may reference it as a service account; it is the same file. The Admin Console is standardizing on "data plane credential".

Each data plane uses its own credential. Revoke a credential from the Admin Console or generate additional ones (for example, one per environment) at any time.

Step 2: install the tetrate CLI​

tetrate is the Agent Router installer and lifecycle tool. It mirrors the container images (Step 5), installs the Helm chart (Step 6), and reports component health (Step 10). It runs on the install host, so it can be installed before EKS exists.

The version of tetrate decides the version of the data plane, so the CLI is installed at the management plane release, not the latest release. That release is shown at the bottom of the popup that opens from the profile icon in the Admin Console. Set TARE_VERSION to it, keeping the leading v:

TARE_VERSION=v<management-plane-release>
curl -fsSL https://tare.tetrate.ai/tools/install.sh | bash -s -- --version "${TARE_VERSION}"

The installer verifies the signed release manifest, installs tetrate into ~/.tetrate/bin, and prints the PATH line to add when that directory is not on PATH yet. Installers before 0.8.0 used ~/.tare/bin, which now holds a compatibility link kept for the deprecation window; ~/.tetrate/bin is the directory to put on PATH.

Add the install directory to PATH and verify the version:

export PATH="$PATH:$HOME/.tetrate/bin"
echo 'export PATH="$PATH:$HOME/.tetrate/bin"' >> ~/.zshrc   # or ~/.bashrc
$ tetrate --version
tetrate version v<management-plane-release>

The reported version has to match TARE_VERSION.

Step 3: provision the EKS cluster​

This step produces an empty EKS cluster reachable through kubectl, plus the AWS_REGION, EKS_CLUSTER_NAME, and AWS_ACCOUNT variables that every later step depends on. The cluster is sized for the Medium tier in Cluster sizing, with an OIDC provider for binding IAM roles to Kubernetes service accounts (IRSA) for the EBS CSI driver and the AWS Load Balancer Controller.

Step 3.1: confirm the AWS identity​

Every command in this guide acts on the account and role that the shell's AWS credentials resolve to. Confirm both first: a wrong profile surfaces as a mid-creation failure or as resources built in the wrong account.

aws sts get-caller-identity

Account must be the target account, and Arn must be the role holding the permissions listed in AWS permissions. If the command fails, credentials are not configured yet: run aws configure, or aws sso login --profile <profile> for SSO, then retry.

Step 3.2: set environment variables​

These variables identify the target account, region, and cluster, and nearly every later command uses them. K8S_VERSION is queried rather than hardcoded so the cluster starts on the version EKS currently defaults to in the region, which keeps it inside the supported window for as long as possible.

export AWS_REGION=<region>                 # e.g. us-east-1
export EKS_CLUSTER_NAME=<cluster-name>
export AWS_ACCOUNT=$(aws sts get-caller-identity --query Account --output text)

# Pick the regional default Kubernetes version so the version doesn't
# go out of support unexpectedly.
export K8S_VERSION=$(aws eks describe-cluster-versions \
  --region "${AWS_REGION}" \
  --query 'clusterVersions[?defaultVersion==`true`].clusterVersion' \
  --output text)

# Optional resource tags: adapt or drop these for your organization.
export TAGS="Owner=<your-name>,Team=<your-team>,Purpose=development"
warning

The supported Kubernetes range is fixed per tetrate release and enforced when the data plane is installed, not when the cluster is created. A cluster outside it fails in Step 6 with Kubernetes 1.36.2-eks-... is outside the supported range 1.30.0-1.35.x for this tare release, which names the range the release accepts. That failure arrives after the cluster has been built and the images synced, and EKS cannot be downgraded in place, so the cluster has to be rebuilt. The range for every supported release is listed on Release compatibility.

tip

Confirm eksctl version reports ≥ 0.220 before continuing. Older binaries reject the K8s version EKS now defaults to and the cluster create call fails with invalid version, supported values: 1.23, …, 1.31.

Step 3.3: create the EKS cluster​

This creates the EKS control plane and a fixed three-node managed node group in a new VPC, all as CloudFormation stacks. The command blocks until the cluster is ready and adds its context to the local kubeconfig.

eksctl create cluster \
  --name "${EKS_CLUSTER_NAME}" \
  --region "${AWS_REGION}" \
  --version "${K8S_VERSION}" \
  --nodes 3 \
  --nodes-min 3 \
  --nodes-max 3 \
  --node-type m5.xlarge \
  --with-oidc \
  --tags "${TAGS}"

Provisioning takes ~15 minutes. The --nodes 3 and --node-type m5.xlarge values match the Medium tier in Cluster sizing; adjust for the target environment.

Step 3.4: verify​

All three nodes must register, or the data plane cannot schedule all of its components in Step 6.

kubectl get nodes

Expected:

NAME                             STATUS   ROLES    AGE    VERSION
ip-192-168-14-164.ec2.internal   Ready    <none>   2m     v1.35.x-eks-...
ip-192-168-25-245.ec2.internal   Ready    <none>   2m     v1.35.x-eks-...
ip-192-168-40-57.ec2.internal    Ready    <none>   2m     v1.35.x-eks-...

Step 4: install the EBS CSI driver addon​

The data plane's Redis instance needs a persistent volume, and EKS (unlike AKS and GKE) ships without a CSI driver to provision EBS volumes. Without the addon the volume claim stays Pending and dependent pods never start.

The first command creates an IAM role letting the driver manage EBS volumes; the second installs the addon and binds it to that role through IRSA.

# IAM role for the addon (one-shot, keeps the role even after cluster recreation)
eksctl create iamserviceaccount \
  --name ebs-csi-controller-sa \
  --namespace kube-system \
  --cluster "${EKS_CLUSTER_NAME}" \
  --region "${AWS_REGION}" \
  --attach-policy-arn arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy \
  --approve --role-only \
  --role-name "${EKS_CLUSTER_NAME}-AmazonEKS_EBS_CSI_DriverRole"
# Install the addon itself
eksctl create addon \
  --cluster "${EKS_CLUSTER_NAME}" \
  --name aws-ebs-csi-driver \
  --region "${AWS_REGION}" \
  --service-account-role-arn "arn:aws:iam::${AWS_ACCOUNT}:role/${EKS_CLUSTER_NAME}-AmazonEKS_EBS_CSI_DriverRole" \
  --force

Wait for it to become ACTIVE (~1 min):

aws eks describe-addon --cluster-name "${EKS_CLUSTER_NAME}" \
  --addon-name aws-ebs-csi-driver --region "${AWS_REGION}" \
  --query 'addon.status' --output text

Step 5: mirror Agent Router images to ECR​

This step copies the images into an ECR registry in the target account and gives the cluster credentials to pull from it.

ECR does not create a repository on first push, and tetrate agentrouter dataplane install --image-sync does not create them either. A single missing repository aborts the whole sync, so the repositories are created first (Step 5.3) and the images synced afterwards (Step 5.4).

Step 5.1: set the registry variables​

These name the registry and the path images are pushed under.

export ECR_HOST="${AWS_ACCOUNT}.dkr.ecr.${AWS_REGION}.amazonaws.com"
export ECR_PREFIX="${EKS_CLUSTER_NAME}"   # repos are created as ${EKS_CLUSTER_NAME}/<image>

Step 5.2: authenticate Docker to ECR​

ECR uses short-lived tokens, minted here from the current AWS identity.

aws ecr get-login-password --region "${AWS_REGION}" | docker login --username AWS --password-stdin "${ECR_HOST}"

ECR tokens expire after 12 hours. Re-run this command if subsequent steps fail with unauthorized: authentication required.

Step 5.3: create the ECR repositories​

The image list changes across tetrate releases, so the repository list is read from the CLI. Point CREDENTIAL_FILE at the saved data-plane-credentials.json:

export CREDENTIAL_FILE=/path/to/data-plane-credentials.json

Ask tetrate which images it will sync:

tetrate agentrouter dataplane install "${CREDENTIAL_FILE}" --print-images

Each line is a fully qualified source reference, such as registry.tetrate.ai/tare/gateway:v1.2.3. The destination keeps the full path after the registry host, so that image is pushed to ${ECR_HOST}/${ECR_PREFIX}/tare/gateway and the repository to create is ${ECR_PREFIX}/tare/gateway.

Create one repository per image. Stripping the registry host and the tag from each line leaves the repository path; repositories that already exist are left alone, so the loop is safe to re-run:

tetrate agentrouter dataplane install "${CREDENTIAL_FILE}" --print-images \
  | sed -E 's#^[^/]+/##; s#:[^:/]+$##' \
  | sort -u \
  | while read -r IMAGE; do
      REPO="${ECR_PREFIX}/${IMAGE}"
      aws ecr describe-repositories --repository-names "${REPO}" \
        --region "${AWS_REGION}" >/dev/null 2>&1 && continue
      aws ecr create-repository --repository-name "${REPO}" \
        --region "${AWS_REGION}" --query 'repository.repositoryName' --output text
    done

Confirm what now exists:

aws ecr describe-repositories --region "${AWS_REGION}" \
  --query "repositories[?starts_with(repositoryName, '${ECR_PREFIX}/')].repositoryName" \
  --output json

Step 5.4: sync the images​

tetrate agentrouter dataplane install "${CREDENTIAL_FILE}" \
  --image-sync "${ECR_HOST}/${ECR_PREFIX}" \
  --sync-only

Expect ~3 to 5 minutes on the first run. On success it prints:

✓ Image sync done in 2m22s

--sync-only pushes the images and exits without deploying.

The sync resumes from where it stopped, so re-running after a failure is safe. Two failures are worth expecting:

  • NAME_UNKNOWN: The repository with name '...' does not exist. A destination repository that Step 5.3 did not predict. Create exactly the name the error prints, which already carries the ${ECR_PREFIX} prefix, then re-run the sync:

    aws ecr create-repository --repository-name "<name-from-the-error>" \
      --region "${AWS_REGION}"

  • Transient registry errors, reported as i/o timeout, context deadline exceeded, EOF, connection reset, or TLS handshake. Wait a few seconds and re-run the sync. If the failure is unauthorized: authentication required instead, the ECR token has expired; re-run Step 5.2 first.

Step 5.5: create the image-pull secret​

This creates the two data plane namespaces and puts a pull secret in each, which the deployments reference in the next step.

kubectl create ns tars-system --dry-run=client -o yaml | kubectl apply -f -
kubectl create ns tars-dataplane --dry-run=client -o yaml | kubectl apply -f -

ECR_TOKEN=$(aws ecr get-login-password --region "${AWS_REGION}")

for NS in tars-system tars-dataplane; do
  kubectl create secret docker-registry registry-secret \
    --docker-username=AWS \
    --docker-password="${ECR_TOKEN}" \
    --docker-email=[email protected] \
    -n "${NS}" \
    --dry-run=client -o yaml | kubectl apply -f -
done

ECR tokens expire every 12 hours, so this secret stops working after that window. For production deployments, use IAM Roles for Service Accounts (IRSA) or EKS Pod Identity to mint tokens on demand instead of a static secret; see AWS docs.

Step 6: install the data plane​

Each hostname client applications call is passed with the repeatable --gateway-hostname flag, the name callers should use first. The hostname does not need to resolve yet; DNS is wired in Step 9. When the hostname is not known yet, leave the flag out and add it later with tetrate agentrouter dataplane upgrade --gateway-hostname (Step 9.2). The --serve-url flag is deprecated and ignored.

tetrate agentrouter dataplane install "${CREDENTIAL_FILE}" \
  --image-sync "${ECR_HOST}/${ECR_PREFIX}" \
  --image-pull-secret-name registry-secret \
  --gateway-hostname=proxy.example.com

The command:

  • Re-syncs any images (idempotent after the previous step).
  • Installs the Helm chart into tars-system and tars-dataplane.

Watch the pods come up:

kubectl get pods -n tars-system
kubectl get pods -n tars-dataplane

Expected output (all Running, 1/1 or 3/3):

NAME                                    READY   STATUS    RESTARTS   AGE
ai-gateway-controller-...               1/1     Running   0          2m
controller-...                          1/1     Running   0          2m
controller-worker-...                   1/1     Running   0          2m
envoy-gateway-...                       1/1     Running   0          2m
envoy-ratelimit-...                     1/1     Running   0          1m
tars-redis-master-...                   1/1     Running   0          2m
tars-tare-doctor-...                    1/1     Running   0          1m

NAME                      READY   STATUS    RESTARTS   AGE
egress-...                3/3     Running   0          1m

The install command also prints a Dataplane unreachable warning at this point, which is expected: DNS does not resolve the data-plane hostname until Step 9.

Step 7: install the AWS Load Balancer Controller​

The AWS Load Balancer Controller provisions the ALB that forwards inbound HTTP to the egress Service in tars-dataplane.

Step 7.1: IAM policy​

The latest controller version needs permissions beyond the v2.7.1 reference policy, so the current policy is fetched from main:

curl -sSL -o /tmp/iam-policy.json https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/main/docs/install/iam_policy.json

aws iam create-policy \
  --policy-name AWSLoadBalancerControllerIAMPolicy \
  --policy-document file:///tmp/iam-policy.json

If the policy already exists from a prior run, update it instead:

POLICY_ARN=arn:aws:iam::${AWS_ACCOUNT}:policy/AWSLoadBalancerControllerIAMPolicy
aws iam create-policy-version \
  --policy-arn "${POLICY_ARN}" \
  --policy-document file:///tmp/iam-policy.json \
  --set-as-default

Step 7.2: IAM service account​

This binds the policy from Step 7.1 to an aws-load-balancer-controller service account in kube-system through IRSA. eksctl creates the IAM role, the service account, and the trust relationship in one command.

eksctl create iamserviceaccount \
  --cluster="${EKS_CLUSTER_NAME}" \
  --region="${AWS_REGION}" \
  --namespace=kube-system \
  --name=aws-load-balancer-controller \
  --attach-policy-arn="arn:aws:iam::${AWS_ACCOUNT}:policy/AWSLoadBalancerControllerIAMPolicy" \
  --approve --override-existing-serviceaccounts

Step 7.3: Helm install​

serviceAccount.create=false points the deployment at the account created in Step 7.2 rather than a new one without the IAM role, and vpcId is read from the cluster because the chart cannot discover it.

helm repo add eks https://aws.github.io/eks-charts
helm repo update

VPC_ID=$(aws eks describe-cluster --name "${EKS_CLUSTER_NAME}" \
  --region "${AWS_REGION}" \
  --query "cluster.resourcesVpcConfig.vpcId" --output text)

helm install aws-load-balancer-controller eks/aws-load-balancer-controller \
  -n kube-system \
  --set clusterName="${EKS_CLUSTER_NAME}" \
  --set serviceAccount.create=false \
  --set serviceAccount.name=aws-load-balancer-controller \
  --set region="${AWS_REGION}" \
  --set vpcId="${VPC_ID}"

Wait for the controller pods to roll out:

kubectl rollout status -n kube-system deployment/aws-load-balancer-controller --timeout=180s

Step 8: create the ALB Ingress​

A single Ingress serves all data-plane traffic (/v1/*, /mcp/*, /.well-known/*) through the egress Service in tars-dataplane on port 10080.

cat <<'EOF' | kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: tars-ingress
  namespace: tars-dataplane
  annotations:
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}]'
spec:
  ingressClassName: alb
  rules:
  - http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: egress
            port:
              number: 10080
EOF

Wait for the ALB to provision (~1 to 2 min) and capture its hostname:

kubectl get ingress tars-ingress -n tars-dataplane

The ADDRESS column populates with something like k8s-tarsdata-tarsingr-xxxxxxxxxx-yyyyyyyyy.us-east-1.elb.amazonaws.com.

If ADDRESS stays empty for more than a few minutes, check the controller for FailedDeployModel events:

kubectl get events -n tars-dataplane --sort-by=.lastTimestamp | tail -5
kubectl logs -n kube-system deployment/aws-load-balancer-controller --tail=30 | grep -iE "error|fail"

The most common cause is an outdated IAM policy missing a permission such as elasticloadbalancing:DescribeListenerAttributes. Re-run the IAM policy update in Step 7.1 and restart the controller.

The Ingress persists across tetrate agentrouter dataplane install reinstalls; re-applying it during upgrades is not required.

Step 9: wire DNS and confirm the hostname​

Step 9.1: DNS​

Point the data-plane hostname at the ALB via a CNAME record at the DNS provider:

proxy.example.com  CNAME  k8s-tarsdata-tarsingr-xxxxxxxxxx-yyyyyyyyy.us-east-1.elb.amazonaws.com

Verify it resolves (propagation may take a couple of minutes):

dig +short proxy.example.com

Step 9.2: confirm the data plane declares the hostname​

The hostname passed with --gateway-hostname in Step 6 is the address the data plane reports. Confirm the chart rendered it:

kubectl -n tars-system get configmap tars-gateway-declaration -o jsonpath='{.data.gateway-hostnames}'

The Admin Console lists the same name on the data plane's page under Directory → Data Planes after the data plane's next configuration poll. If the hostname was not passed at install, or a further name is needed, declare the full list with tetrate agentrouter dataplane upgrade; the flag replaces the installed list, so every name goes on the command line:

tetrate agentrouter dataplane upgrade "${CREDENTIAL_FILE}" --gateway-hostname=proxy.example.com

No console field sets the hostname. A data plane older than 0.7.0 cannot declare one, and its address is set through the deprecated SetDataplaneURL admin API instead. More names, certificate checks, and the rules for the list are covered in Declare the hostnames a data plane serves.

Step 10: verify the install​

Run tetrate agentrouter doctor to check pod, CRD, and policy state:

tetrate agentrouter doctor "${CREDENTIAL_FILE}"

A healthy install reports its components in Accepted state.

A data plane that has just been installed has no routes delivered to it yet, so the report also carries warnings for EnvoyPatchPolicy, RouteDeployment and AIGatewayRoute ("none found"). Those clear once the management plane delivers configuration for a project assigned to this data plane.

Read the component states rather than the exit code: tetrate agentrouter doctor exits non-zero on any error, and on a freshly installed data plane it can report forward/check failed (apiserver pod-proxy) for the egress pod even when the Kubernetes API server can reach it. Check that case before escalating, by requesting the pod's health endpoint over the same path the doctor uses:

POD=$(kubectl get pods -n tars-dataplane -l app=egress -o jsonpath='{.items[0].metadata.name}')
kubectl get --raw "/api/v1/namespaces/tars-dataplane/pods/${POD}:10080/proxy/healthz"

{"status":"ok"} means the pod-proxy path works and the report is not describing a network fault. If any component is not Accepted, or the health endpoint does not answer, contact Tetrate Support.

Step 11: smoke tests​

Once tetrate agentrouter doctor reports clean, validate end-to-end traffic.

Create an API key from the Developer Console (Router → API Keys → + Create):

export DP_HOST=proxy.example.com
export TETRATE_API_KEY=<your-api-key>

Save the suite as smoke.sh. Each endpoint is called twice, once with the key and once without, and a 200 on an unauthenticated call is reported as an auth bypass:

cat > smoke.sh <<'SCRIPT'
#!/usr/bin/env bash
# Smoke test the Agent Router data plane. Every endpoint is called twice: once
# with an API key (expect 200) and once without one (expect 401). A 200 on an
# unauthenticated call is an auth bypass and fails the run.
#
# Requires: curl, and DP_HOST and TETRATE_API_KEY exported in the environment.
# Set DP_SCHEME=https once TLS is terminated at the load balancer.
#
# The model names are the defaults from onboarding. Override any of them to
# match the models enabled in the project.

set -uo pipefail

: "${DP_HOST:?set DP_HOST}"
: "${TETRATE_API_KEY:?set TETRATE_API_KEY}"

BASE="${DP_SCHEME:-http}://${DP_HOST}"
CHAT_MODEL="${CHAT_MODEL:-gpt-5-mini}"
ANTHROPIC_MODEL="${ANTHROPIC_MODEL:-claude-haiku-4-5}"
EMBEDDING_MODEL="${EMBEDDING_MODEL:-text-embedding-3-small}"
IMAGE_MODEL="${IMAGE_MODEL:-gpt-image-1.5}"

JSON="Content-Type: application/json"
AUTH="Authorization: Bearer ${TETRATE_API_KEY}"
PASS=0; FAIL=0; BYPASS=0

call() {  # call <label> <expected-status> <curl args...>
  local label=$1 expected=$2; shift 2
  local body status mark
  body=$(mktemp)
  status=$(curl -s -o "${body}" -w '%{http_code}' "$@")
  if [[ "${status}" == "${expected}" ]]; then
    mark=ok; PASS=$((PASS+1))
  elif [[ "${expected}" == "401" && "${status}" == "200" ]]; then
    mark="AUTH BYPASS"; BYPASS=$((BYPASS+1))
  else
    mark=fail; FAIL=$((FAIL+1))
  fi
  printf '%-12s %-34s http=%-4s expected=%s\n' "${mark}" "${label}" "${status}" "${expected}"
  [[ "${mark}" == ok ]] || printf '             body: %s\n' "$(head -c 200 "${body}" | tr '\n' ' ')"
  rm -f "${body}"
}

echo "==== list models (GET /v1/models) ===="
call "models auth"      200 "${BASE}/v1/models" -H "${AUTH}"
call "models no-auth"   401 "${BASE}/v1/models"

echo
echo "==== chat completions (POST /v1/chat/completions) ===="
DATA="{\"model\":\"${CHAT_MODEL}\",\"messages\":[{\"role\":\"user\",\"content\":\"reply with one word\"}]}"
call "chat auth"        200 "${BASE}/v1/chat/completions" -X POST -H "${JSON}" -H "${AUTH}" -d "${DATA}"
call "chat no-auth"     401 "${BASE}/v1/chat/completions" -X POST -H "${JSON}" -d "${DATA}"

echo
echo "==== responses (POST /v1/responses) ===="
DATA="{\"model\":\"${CHAT_MODEL}\",\"input\":\"reply with one word\"}"
call "responses auth"    200 "${BASE}/v1/responses" -X POST -H "${JSON}" -H "${AUTH}" -d "${DATA}"
call "responses no-auth" 401 "${BASE}/v1/responses" -X POST -H "${JSON}" -d "${DATA}"

echo
echo "==== anthropic messages (POST /v1/messages) ===="
DATA="{\"model\":\"${ANTHROPIC_MODEL}\",\"max_tokens\":32,\"messages\":[{\"role\":\"user\",\"content\":\"reply with one word\"}]}"
AV="anthropic-version: 2023-06-01"
call "messages auth"     200 "${BASE}/v1/messages" -X POST -H "${JSON}" -H "${AV}" -H "${AUTH}" -d "${DATA}"
call "messages no-auth"  401 "${BASE}/v1/messages" -X POST -H "${JSON}" -H "${AV}" -d "${DATA}"

echo
echo "==== embeddings (POST /v1/embeddings) ===="
DATA="{\"model\":\"${EMBEDDING_MODEL}\",\"input\":\"hello world\"}"
call "embeddings auth"    200 "${BASE}/v1/embeddings" -X POST -H "${JSON}" -H "${AUTH}" -d "${DATA}"
call "embeddings no-auth" 401 "${BASE}/v1/embeddings" -X POST -H "${JSON}" -d "${DATA}"

echo
echo "==== image generation (POST /v1/images/generations) ===="
DATA="{\"model\":\"${IMAGE_MODEL}\",\"prompt\":\"a red apple on a white background\",\"size\":\"1024x1024\",\"n\":1}"
call "images auth"        200 "${BASE}/v1/images/generations" -X POST -H "${JSON}" -H "${AUTH}" -d "${DATA}"
call "images no-auth"     401 "${BASE}/v1/images/generations" -X POST -H "${JSON}" -d "${DATA}"

echo
echo "----"
printf 'pass: %d   fail: %d   auth-bypass: %d\n' "${PASS}" "${FAIL}" "${BYPASS}"
[[ "${FAIL}" -eq 0 && "${BYPASS}" -eq 0 ]]
SCRIPT
chmod +x smoke.sh

Run it:

bash smoke.sh

Expected output:

==== list models (GET /v1/models) ====
ok           models auth                        http=200  expected=200
ok           models no-auth                     http=401  expected=401

==== chat completions (POST /v1/chat/completions) ====
ok           chat auth                          http=200  expected=200
ok           chat no-auth                       http=401  expected=401

==== responses (POST /v1/responses) ====
ok           responses auth                     http=200  expected=200
ok           responses no-auth                  http=401  expected=401

==== anthropic messages (POST /v1/messages) ====
ok           messages auth                      http=200  expected=200
ok           messages no-auth                   http=401  expected=401

==== embeddings (POST /v1/embeddings) ====
ok           embeddings auth                    http=200  expected=200
ok           embeddings no-auth                 http=401  expected=401

==== image generation (POST /v1/images/generations) ====
ok           images auth                        http=200  expected=200
ok           images no-auth                     http=401  expected=401

----
pass: 12   fail: 0   auth-bypass: 0

If any row fails, and in particular if a no-auth row returns 200 instead of 401, contact Tetrate Support.

Once TLS is terminated at the load balancer (Appendix A), re-run the suite with DP_SCHEME=https.

Upgrading​

An upgrade re-runs the install command. The Helm chart, CRDs, and config in tars-config are upgraded; persistent state (Redis-backed rate-limit counters, accumulated audit data) survives. The ALB Ingress persists, the IAM policy does not change, and the EBS CSI addon remains installed.

tetrate agentrouter dataplane install "${CREDENTIAL_FILE}" \
  --image-sync "${ECR_HOST}/${ECR_PREFIX}" \
  --image-pull-secret-name registry-secret

Pass the same --gateway-hostname flags again: tetrate agentrouter dataplane install renders the values from its flags, and a re-run without them leaves the declaration out, so the data plane falls back to the address stored for it. tetrate agentrouter dataplane upgrade reads the installed list from the release and carries it forward without the flags.

note

Some runtime patches (notably observability env vars on the egress deployment) are reset on every install. Re-apply the patches from Appendix B after each upgrade.

Cleanup​

To remove everything created by this guide:

# 1. Ingress (releases the ALB)
kubectl delete ingress tars-ingress -n tars-dataplane --ignore-not-found
# 2. AWS Load Balancer Controller
helm uninstall aws-load-balancer-controller -n kube-system 2>/dev/null || true
eksctl delete iamserviceaccount \
  --name aws-load-balancer-controller \
  --namespace kube-system \
  --cluster "${EKS_CLUSTER_NAME}" \
  --region "${AWS_REGION}" 2>/dev/null || true
ALB_POLICY_ARN=$(aws iam list-policies \
  --query "Policies[?PolicyName=='AWSLoadBalancerControllerIAMPolicy'].Arn" \
  --output text 2>/dev/null)
if [ -n "${ALB_POLICY_ARN}" ] && [ "${ALB_POLICY_ARN}" != "None" ]; then
  # Delete non-default versions first (IAM rule)
  for V in $(aws iam list-policy-versions --policy-arn "${ALB_POLICY_ARN}" \
              --query 'Versions[?IsDefaultVersion==`false`].VersionId' --output text); do
    aws iam delete-policy-version --policy-arn "${ALB_POLICY_ARN}" --version-id "${V}"
  done
  aws iam delete-policy --policy-arn "${ALB_POLICY_ARN}"
fi
# 3. ECR repositories (the cluster prefix matches what was used at create time)
for REPO in $(aws ecr describe-repositories --region "${AWS_REGION}" \
                --query "repositories[?starts_with(repositoryName, '${EKS_CLUSTER_NAME}/')].repositoryName" \
                --output text); do
  aws ecr delete-repository --repository-name "${REPO}" \
    --region "${AWS_REGION}" --force
done
# 4. EKS cluster (also tears down the EBS CSI addon and its IAM role via
#    CloudFormation, takes ~10 min)
eksctl delete cluster --name "${EKS_CLUSTER_NAME}" --region "${AWS_REGION}"
# 5. Local kubeconfig file
rm -f "${HOME}/kubeconfig-${EKS_CLUSTER_NAME}"
note

DNS records (CNAMEs at the DNS provider) must be removed manually.

Troubleshooting​

SymptomLikely causeFix
eksctl create cluster errors with invalid version, supported values: 1.23, …, 1.31eksctl is older than the K8s version EKS currently defaults tobrew upgrade eksctl (or re-download), then retry
tetrate agentrouter dataplane install fails at preflight with Kubernetes ... is outside the supported rangeThe cluster was created on a version newer than the tetrate release supports, commonly because the EKS regional default moved aheadRecreate the cluster on a version inside the range (Step 3.2); an EKS cluster cannot be downgraded in place
tetrate agentrouter dataplane install fails at Register data plane URL with permission_denied ... requires scopes [dataplane_operator]The install was run with the deprecated --serve-url flag on an older CLI. Nothing has been deployedRe-run without --serve-url (or upgrade tetrate, which ignores it), and declare the hostname with --gateway-hostname as in Step 6. See the error reference
tetrate agentrouter dataplane install --sync-only errors with NAME_UNKNOWN: The repository with name '...' does not existECR repo for that image was never createdRe-run the repository-creation loop in Step 5.3, then re-run the sync
Ingress ADDRESS stays empty for more than 2 minLB Controller IAM policy is missing newer permissionsUpdate the IAM policy from main, restart the controller
Pods stuck ImagePullBackOff with no basic auth credentialsECR token in registry-secret expired (12h lifetime)Re-run the image-pull-secret creation step (Step 5.5)
egress pods restart loop with connection refused to redisEBS CSI driver missing, so Redis PV never bindsInstall the EBS CSI addon (Step 4)
The first requests after a few idle minutes fail with 503 "connection termination"A NAT Gateway in front of the cluster drops idle connections to the model providers before the gateway noticesSet egressResources.egress.upstreamIdleTimeout in the data plane values below the NAT's idle timeout, for example 300s on AWS. See Requests fail after the gateway has been idle

Requests fail after the gateway has been idle​

A data plane behind a cloud NAT with a short idle timeout, such as AWS NAT Gateway, can stop failing the first requests after a few idle minutes with 503 "connection termination". Two values in the data plane's Helm values address it, and both are off by default:

  • egressResources.egress.upstreamIdleTimeout is set below the NAT's idle timeout, for example 300s on AWS, so the gateway closes an idle connection before the NAT drops it.
  • egressResources.egress.upstreamTcpKeepalive keeps the connection of a long-running request open through the NAT.
egressResources:
  egress:
    upstreamIdleTimeout: 300s
    upstreamTcpKeepalive:
      idleTime: 60s
      interval: 30s
      probes: 3

The idle timeout closes a connection that has no request on it and does not cut a long stream. A connection that carries a request but moves no bytes for longer than the NAT's idle timeout, such as a long non-streaming completion, is covered by the keepalive probes. AWS NAT Gateway drops an idle connection after 350 seconds, so 300s is below it.

Set these only on a data plane behind such a NAT. Adding, changing, or removing either one makes the running data plane rebuild its connections to every provider at once, which briefly raises its memory use, so the egress proxy is given memory headroom first on a large data plane. tetrate agentrouter dataplane upgrade keeps both values from the installed data plane.

Appendix A: TLS via ACM + HTTPS listener​

The default flow uses HTTP on port 80. For production, terminate TLS at the ALB using an AWS Certificate Manager (ACM) certificate.

  1. Request the certificate in ACM for the data-plane hostname. Validate via DNS (CNAME) or email per the domain control method. The certificate must cover every hostname the data plane declares. If this data plane serves a project behind DNS failover, it must also carry the client-facing name as a SAN. A name missing here stays invisible until the first failover. See Fail over between data planes and the multi-SAN worked example.

  2. Get the certificate ARN:

    CERT_ARN=$(aws acm list-certificates --region "${AWS_REGION}" \
      --query "CertificateSummaryList[?DomainName=='proxy.example.com'].CertificateArn" \
      --output text)

  3. Update the Ingress to listen on HTTPS:443 and redirect HTTP to HTTPS:

    metadata:
      annotations:
        alb.ingress.kubernetes.io/scheme: internet-facing
        alb.ingress.kubernetes.io/target-type: ip
        alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS": 443}]'
        alb.ingress.kubernetes.io/ssl-redirect: '443'
        alb.ingress.kubernetes.io/certificate-arn: <CERT_ARN>
        alb.ingress.kubernetes.io/ssl-policy: ELBSecurityPolicy-TLS13-1-2-2021-06

  4. Nothing changes on the management plane. A declared hostname is reached over HTTPS, so the data plane's address needs no update. Once the listener serves HTTPS, tetrate agentrouter doctor --only gateway reads the certificate on each declared hostname and warns for a name it does not cover. A data plane older than 0.7.0 still stores an http:// address set through SetDataplaneURL; update that address to https://... through the same API.

The ALB picks up the annotation change without redeploying. Verify with:

curl -I https://proxy.example.com/v1/models -H "Authorization: Bearer ${TETRATE_API_KEY}"

Appendix B: observability with OpenTelemetry​

The data plane emits OTLP metrics (router_* family). To forward them to an observability backend, deploy an OpenTelemetry Collector in-cluster. The collector's transform/strip_scope processor below removes Envoy's dynamicmodulescustom. scope prefix, so the names it exports are the bare router_* ones on every data plane release. A scraper that reads the Envoy admin endpoint (:19001/stats/prometheus) directly instead sees envoy_dynamicmodulescustom_router_* names on a data plane running 0.7.0 or later; see the metric list in the Azure guide.

cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Namespace
metadata:
  name: otel-system

apiVersion: v1
kind: ConfigMap
metadata:
  name: otel-collector-config
  namespace: otel-system
data:
  config.yaml: |
    receivers:
      otlp:
        protocols:
          http:
            endpoint: 0.0.0.0:4318
          grpc:
            endpoint: 0.0.0.0:4317
    processors:
      batch:
        timeout: 5s
      # Strip Envoy's internal dynamic-modules scope prefix so router_* metrics
      # ship with their canonical names (router_requests_total, etc.) instead
      # of dynamicmodulescustom.router_requests_total.
      transform/strip_scope:
        metric_statements:
          - context: metric
            statements:
              - replace_pattern(name, "^dynamicmodulescustom\\.", "")
    exporters:
      debug:
        verbosity: detailed
      # Replace this with your real backend exporter (Datadog, Grafana Cloud,
      # SigNoz, CloudWatch via the awsemf exporter, etc.).
    service:
      pipelines:
        metrics:
          receivers: [otlp]
          processors: [batch, transform/strip_scope]
          exporters: [debug]

apiVersion: apps/v1
kind: Deployment
metadata:
  name: otel-collector
  namespace: otel-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: otel-collector
  template:
    metadata:
      labels:
        app: otel-collector
    spec:
      containers:
      - name: collector
        image: otel/opentelemetry-collector-contrib:0.98.0
        ports:
        - containerPort: 4317
        - containerPort: 4318
        volumeMounts:
        - name: config
          mountPath: /etc/otelcol-contrib
      volumes:
      - name: config
        configMap:
          name: otel-collector-config

apiVersion: v1
kind: Service
metadata:
  name: otel-collector
  namespace: otel-system
spec:
  selector:
    app: otel-collector
  ports:
  - name: otlp-http
    port: 4318
    targetPort: 4318
  - name: otlp-grpc
    port: 4317
    targetPort: 4317
EOF

Point the AI Gateway proxy at the collector:

kubectl set env deploy/egress -n tars-dataplane \
  OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector.otel-system.svc.cluster.local:4318 \
  OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf

Generate traffic via the smoke tests, then verify metrics are flowing:

kubectl logs -n otel-system deployment/otel-collector --tail=50 | grep "Name:"

Expected metrics include router_requests_total, router_auth_attempts_total, and similar.

note

The kubectl set env patch is reset by tetrate agentrouter dataplane install re-runs. Re-apply after every upgrade until the chart accepts these settings via Helm values.

Where to go next

Gateway installation

Install the data plane gateway components that manage inbound access and configure request routing.

Self-Hosted Data Plane

See the full install sequence: gateway, SSO, then developer handoff.