Skip to main content

Agent Router data plane installation for AWS

This guide installs the Agent Router data plane on AWS Elastic Kubernetes Service (EKS).

Overview​

Agent Router uses a split-plane model. A Management Plane hosted by Tetrate holds configuration and exposes the web applications. A Data Plane runs in the target AWS account and handles all AI traffic. The two planes communicate over one outbound HTTPS connection initiated by the data plane; no inbound internet connections are required.

By the end of this guide:

  • An EKS cluster with the EBS CSI driver installed
  • An ECR registry mirroring the Agent Router images
  • The Agent Router data plane running in the tars-system and tars-dataplane namespaces
  • An Application Load Balancer fronting the data plane at a DNS name the organization owns
  • An OpenTelemetry pipeline forwarding metrics to the observability stack

Plan for 30 to 45 minutes of installation time, plus DNS propagation.

Table of contents​

Prerequisites​

Cluster requirements​

Already running Envoy Gateway cluster-wide?

The data plane installs a GatewayClass named tars-egress whose controllerName is the Envoy Gateway default, gateway.envoyproxy.io/gatewayclass-controller. An Envoy Gateway that already watches the whole cluster claims tars-egress too, and the two controllers contend over it. There is no override: install the data plane in a cluster of its own. See A cluster already running Envoy Gateway.

Dashboard and Router app access​

Agent Router has two web surfaces. Both URLs are provided during onboarding:

  • Dashboard (admin): https://dashboard.<your-tenant>.tetrate.ai. Used in Step 1 and Step 12.
  • Router app (end-user): https://router.<your-tenant>.tetrate.ai. Used in Step 14 to create API keys for callers.

Tools​

Install the following on the host that will run the guide:

ToolInstall
aws CLI v2https://docs.aws.amazon.com/cli/latest/userguide/install.html
eksctl ≥ 0.220https://eksctl.io/installation/ (or brew install eksctl). Older versions do not recognize current EKS K8s versions.
kubectlhttps://kubernetes.io/docs/tasks/tools/
helm (3+)https://helm.sh/docs/intro/install/
dockerhttps://docs.docker.com/get-docker/
curlpreinstalled on macOS and most Linux distros
tare CLIcovered in Step 2

A data-plane-credentials.json file is also required; see Step 1.

Infrastructure​

A dedicated workload cluster must be provisioned before starting the installation. The cluster should consist of at least three (3) nodes. See Cluster sizing for more details.

warning

Tetrate support does not cover client-side infrastructure provisioning or Kubernetes issues. The instructions for creating clusters and related infrastructure components are provided as a courtesy and should be carefully evaluated before executing them.

AWS permissions​

The operator needs permission to manage these AWS services on the target account:

ServiceUsed for
EKS (AmazonEKSClusterPolicy, AmazonEKSWorkerNodePolicy)Cluster creation and node groups
EC2 + VPCCluster networking, EBS volumes, ELB provisioning
IAMOIDC provider, IRSA roles for EBS CSI driver and AWS Load Balancer Controller
CloudFormationeksctl deploys everything via CloudFormation stacks
ECRCreate repositories, push/pull images
ELB (Elastic Load Balancing)ALB created by the AWS Load Balancer Controller

The simplest setup is an admin role on a sandbox account. For a constrained role, AWS publishes the minimum policy eksctl needs at https://eksctl.io/usage/minimum-iam-policies/.

Cluster sizing​

The chart's default install runs multiple always-on components, including the Agent Router egress proxy (Envoy-based; referred to as egress envoy in manifests and logs) with a minimum of 2 replicas, AI gateway controller and ext_proc, controller / worker, Redis / rate-limit. This is not a single-node footprint.

The egress envoy is the dominant resource consumer. Both CPU and memory usage grow with the configuration size the proxy holds in memory: the number of AIGatewayRoute and AIServiceBackend resources, header-mutation rules, and other per-route features. The AI gateway team's control-plane scaling benchmark shows roughly linear CPU and memory growth from adding routes, with memory staying elevated to keep the xDS state available to serve traffic. Plan for routes to scale up as the data plane adds providers, models, and projects. General-purpose EC2 families (m5.* or m6i.*, balanced CPU/RAM) are the right default.

SizeUse caseRecommended node poolApprox allocatable target
Smalldev / test / low traffic3 × m5.large≥ 6 vCPU, ≥ 20 GiB RAM
Mediumstaging / light production3 × m5.xlarge≥ 12 vCPU, ≥ 40 GiB RAM
Highproduction with burst headroom3 × m5.2xlarge (or split into system + data plane node groups)≥ 24 vCPU, ≥ 80 GiB RAM

Practical floor: 3 nodes minimum to survive a node drain or upgrade. Demo installs can start at Small; production should start at Medium.

Conventions​

The steps below export environment variables (AWS_REGION, EKS_CLUSTER_NAME, AWS_ACCOUNT, ECR_HOST, CREDENTIAL_FILE, SERVE_URL, and others) as they become needed. Each later step assumes the variables defined earlier are still exported in the current shell. If a new shell is started mid-install, re-export them before continuing.

Step 1: obtain the data plane credential​

In the dashboard, go to System → Settings → Data plane credentials → + Generate Data plane credential.

Save the downloaded file as data-plane-credentials.json on the host where the install runs. This file is the long-lived identity the data plane uses to authenticate to the management plane.

note

Some parts of the product still use older "service account" naming for this file. CLI output and in-cluster paths may reference it as a service account; it is the same file. The dashboard is in the process of standardizing on "data plane credential".

Each data plane uses its own credential. Revoke a credential from the dashboard or generate additional ones (for example, one per environment) at any time.

Step 2: install the tare CLI​

curl -sSL https://tare.tetrate.ai/tools/install.sh | bash

Output:

==> tare installer
==> channel: stable

==> Detected platform: darwin-arm64

==> Installing tare for darwin-arm64...
==> Downloading from: https://tare.tetrate.ai/tools/tags/v0.1.0-beta.2/tare-darwin-arm64.tar.gz
ok Installed tare to /Users/johndoe/.tare/bin/tare

==> tare version: tare version v0.1.0-beta.2

ok Installation directory is already in your PATH

==> Get started:
    tare install identity.json --serve-url https://proxy.acme.com
    tare install --help

The installer prints the install path (typically ~/.tare/bin/tare). Add it to PATH and verify the version:

export PATH="$PATH:$HOME/.tare/bin"
echo 'export PATH="$PATH:$HOME/.tare/bin"' >> ~/.zshrc   # or ~/.bashrc
$ tare --version
tare version v0.1.0-beta.2

Step 3: provision the eks cluster​

Step 3.1: set environment variables​

export AWS_REGION=<region>          # e.g. us-east-1
export EKS_CLUSTER_NAME=<cluster-name>
export AWS_ACCOUNT=$(aws sts get-caller-identity --query Account --output text)

# Each tare release supports a fixed Kubernetes range and rejects anything
# outside it. The EKS regional default runs ahead of that range: it is 1.36 in
# us-east-1 today, which current releases reject. Pin the version instead.
export K8S_VERSION=1.35

# Optional resource tags: adapt or drop these for your organization.
TAGS="Owner=<your-name>,Team=<your-team>,Purpose=development"
warning

The supported Kubernetes range is fixed per tare release and enforced when the data plane is installed, not when the cluster is created. A cluster outside it fails in Step 9 with Kubernetes 1.36.2-eks-... is outside the supported range 1.30.0-1.35.x for this tare release, which names the range the release accepts. That failure arrives after the cluster has been built and the images synced, and EKS cannot be downgraded in place, so the cluster has to be rebuilt. The range for every supported release is listed on Release compatibility.

tip

Confirm eksctl version reports ≥ 0.220 before continuing. Older binaries reject the K8s version EKS now defaults to and the cluster create call fails with invalid version, supported values: 1.23, …, 1.31.

Step 3.2: log in to AWS​

aws sts get-caller-identity

Step 3.3: create the eks cluster​

eksctl create cluster \
  --name "${EKS_CLUSTER_NAME}" \
  --region "${AWS_REGION}" \
  --version "${K8S_VERSION}" \
  --nodes 3 \
  --nodes-min 3 \
  --nodes-max 3 \
  --node-type m5.xlarge \
  --with-oidc \
  --tags "${TAGS}"

Provisioning takes ~15 minutes. --with-oidc enables IAM Roles for Service Accounts (IRSA), which the EBS CSI driver and AWS Load Balancer Controller depend on.

The --nodes 3 and --node-type m5.xlarge values above match the Medium tier in Cluster sizing. Adjust for the target environment.

Step 3.4: verify​

kubectl get nodes

Expected:

NAME                             STATUS   ROLES    AGE    VERSION
ip-192-168-14-164.ec2.internal   Ready    <none>   2m     v1.35.x-eks-...
ip-192-168-25-245.ec2.internal   Ready    <none>   2m     v1.35.x-eks-...
ip-192-168-40-57.ec2.internal    Ready    <none>   2m     v1.35.x-eks-...
Already have an EKS cluster?

Reuse it after confirming that:

  • OIDC provider is associated: aws eks describe-cluster --name <name> --region <region> --query "cluster.identity.oidc.issuer"
  • EBS CSI driver is installed: aws eks list-addons --cluster-name <name> --region <region> --query 'addons' | grep aws-ebs-csi-driver
  • K8s version is supported (aws eks describe-cluster-versions --region <region> shows the cluster's version is not deprecated)

Step 4: install the ebs csi driver addon​

The Agent Router egress proxy and Redis use persistent volumes, and EKS does not install a CSI driver by default.

# IAM role for the addon (one-shot, keeps the role even after cluster recreation)
eksctl create iamserviceaccount \
  --name ebs-csi-controller-sa \
  --namespace kube-system \
  --cluster "${EKS_CLUSTER_NAME}" \
  --region "${AWS_REGION}" \
  --attach-policy-arn arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy \
  --approve --role-only \
  --role-name "${EKS_CLUSTER_NAME}-AmazonEKS_EBS_CSI_DriverRole"
# Install the addon itself
eksctl create addon \
  --cluster "${EKS_CLUSTER_NAME}" \
  --name aws-ebs-csi-driver \
  --region "${AWS_REGION}" \
  --service-account-role-arn "arn:aws:iam::${AWS_ACCOUNT}:role/${EKS_CLUSTER_NAME}-AmazonEKS_EBS_CSI_DriverRole" \
  --force

Wait for it to become ACTIVE (~1 min):

aws eks describe-addon --cluster-name "${EKS_CLUSTER_NAME}" \
  --addon-name aws-ebs-csi-driver --region "${AWS_REGION}" \
  --query 'addon.status' --output text

Step 5: create ecr repositories​

Agent Router images must live in the target registry. Unlike most cloud registries, ECR requires each repository to be pre-created; tare install --image-sync does not auto-create them, and a single missing repository aborts the entire sync rather than skipping one image.

The set of repositories changes across tare releases. Rather than hand-maintaining a list, this guide uses a helper that asks tare which images it will sync, creates a repository for each, and then runs the sync. A repository name keeps the full path after the registry host, so an image published as otel/opentelemetry-collector-contrib needs <prefix>/otel/opentelemetry-collector-contrib and not <prefix>/opentelemetry-collector-contrib. Anything the image list does not predict is created from the error tare reports, and the sync is retried.

export ECR_HOST="${AWS_ACCOUNT}.dkr.ecr.${AWS_REGION}.amazonaws.com"
export ECR_PREFIX="${EKS_CLUSTER_NAME}"   # repos are created as ${EKS_CLUSTER_NAME}/<image>

Save the helper as sync-images.sh in the current directory. It is run in Step 7, once Docker is authenticated to ECR.

cat > sync-images.sh <<'SCRIPT'
#!/usr/bin/env bash
# Sync Agent Router images to ECR, creating the repositories they need.
#
# tare install --image-sync requires every destination repository to exist
# before it pushes, and one missing repository aborts the whole sync. So this
# script works in two phases:
#   1. ask tare which images it will sync and create a repository for each.
#      The repository name keeps the full path after the registry host, so
#      registry.tetrate.ai/otel/opentelemetry-collector-contrib:<tag> becomes
#      <prefix>/otel/opentelemetry-collector-contrib
#   2. run the sync. If tare still reports NAME_UNKNOWN for a repository the
#      list did not predict, create that one and retry; tare resumes from
#      where it stopped. The same retry covers transient registry errors.
#
# Requires: aws, tare, and AWS_REGION, ECR_HOST, ECR_PREFIX, CREDENTIAL_FILE
# exported in the environment.

set -uo pipefail

: "${AWS_REGION:?set AWS_REGION}"
: "${ECR_HOST:?set ECR_HOST}"
: "${ECR_PREFIX:?set ECR_PREFIX}"
: "${CREDENTIAL_FILE:?set CREDENTIAL_FILE}"

OUT="${OUT:-$(mktemp -t tare-sync.XXXXXX)}"
MAX_ITER="${MAX_ITER:-20}"

create_repo() {
  if aws ecr describe-repositories --repository-names "$1" \
       --region "${AWS_REGION}" >/dev/null 2>&1; then
    echo "    exists:  $1"
    return 0
  fi
  aws ecr create-repository --repository-name "$1" --region "${AWS_REGION}" \
    --query 'repository.repositoryName' --output text | sed 's/^/    created: /'
}

echo "=== creating a repository for every image tare will sync ==="
IMAGES=$(tare install "${CREDENTIAL_FILE}" --print-images) || {
  echo "=== could not list images; check the credential file ==="
  exit 1
}
echo "${IMAGES}" | sed -E 's#^[^/]+/##; s#:[^:/]+$##' | sort -u | while read -r img; do
  [[ -n "${img}" ]] && create_repo "${ECR_PREFIX}/${img}"
done

for i in $(seq 1 "${MAX_ITER}"); do
  echo "=== sync attempt ${i}/${MAX_ITER} ==="
  tare install "${CREDENTIAL_FILE}" \
    --image-sync "${ECR_HOST}/${ECR_PREFIX}" \
    --sync-only 2>&1 | tee "${OUT}"
  RC=${PIPESTATUS[0]}

  if [[ ${RC} -eq 0 ]]; then
    echo "=== sync complete ==="
    exit 0
  fi

  # A repository the image list did not predict:
  # "with name 'my-cluster/envoy-tars' does not exist"
  MISSING=$(grep -oE "name '[^']+' does not exist" "${OUT}" | head -1 \
            | sed -E "s/name '([^']+)' does not exist/\1/")

  if [[ -n "${MISSING}" ]]; then
    echo "=== creating missing repository: ${MISSING} ==="
    create_repo "${MISSING}"
    continue
  fi

  # Transient registry errors: back off and retry.
  if grep -qE "i/o timeout|context deadline exceeded|EOF|connection reset|TLS handshake" "${OUT}"; then
    echo "=== transient registry error, retrying in 15s ==="
    sleep 15
    continue
  fi

  echo "=== unrecoverable failure (rc=${RC}), see ${OUT} ==="
  exit "${RC}"
done

echo "=== gave up after ${MAX_ITER} attempts ==="
exit 1
SCRIPT
chmod +x sync-images.sh

Step 6: authenticate Docker to ecr​

aws ecr get-login-password --region "${AWS_REGION}" | docker login --username AWS --password-stdin "${ECR_HOST}"

ECR tokens expire after 12 hours. Re-run this command if subsequent steps fail with unauthorized: authentication required.

Step 7: sync Agent Router images to ecr​

Save data-plane-credentials.json to a known path, then run:

export CREDENTIAL_FILE=/path/to/data-plane-credentials.json

bash sync-images.sh

The helper creates any missing ECR repos as tare reports them. Expect ~3 to 5 minutes on the first run. On success it prints:

✓ Image sync done in 2m22s
=== sync complete ===

Step 8: create the image-pull secret​

kubectl create ns tars-system --dry-run=client -o yaml | kubectl apply -f -
kubectl create ns tars-dataplane --dry-run=client -o yaml | kubectl apply -f -

ECR_TOKEN=$(aws ecr get-login-password --region "${AWS_REGION}")

for NS in tars-system tars-dataplane; do
  kubectl create secret docker-registry registry-secret \
    --docker-server="${ECR_HOST}" \
    --docker-username=AWS \
    --docker-password="${ECR_TOKEN}" \
    --docker-email=[email protected] \
    -n "${NS}" \
    --dry-run=client -o yaml | kubectl apply -f -
done

ECR tokens expire every 12 hours, so this secret stops working after that window. For production deployments, use IAM Roles for Service Accounts (IRSA) or EKS Pod Identity to mint tokens on demand instead of a static secret; see AWS docs.

Step 9: install the data plane​

Step 9.1: apply the CRDs​

The chart references Envoy Gateway custom resources, so the CRDs are applied first. On a cluster that has never run Agent Router this pass is required; without it the install stops during preflight with no matches for kind "BackendTrafficPolicy" in version "gateway.envoyproxy.io/v1alpha1".

tare install "${CREDENTIAL_FILE}" --crds-only

This applies 28 CRDs and takes around four minutes. It creates no namespaces, no pull secret, and no workloads.

Step 9.2: install​

tare install "${CREDENTIAL_FILE}" \
  --image-sync "${ECR_HOST}/${ECR_PREFIX}" \
  --image-pull-secret-name registry-secret

The gateway URL is not part of the install: it is registered on the management plane in Step 12.2, once the hostname is known. Since 0.3.0 the management plane owns that value and pushes it to the running data plane; the old --serve-url flag is deprecated and ignored.

The command:

  • Re-syncs any images (idempotent after the previous step).
  • Installs the helm chart into tars-system and tars-dataplane.

Watch the pods come up:

kubectl get pods -n tars-system
kubectl get pods -n tars-dataplane

Expected output (all Running, 1/1 or 3/3):

NAME                                    READY   STATUS    RESTARTS   AGE
ai-gateway-controller-...               1/1     Running   0          2m
controller-...                          1/1     Running   0          2m
controller-worker-...                   1/1     Running   0          2m
envoy-gateway-...                       1/1     Running   0          2m
envoy-ratelimit-...                     1/1     Running   0          1m
tars-redis-master-...                   1/1     Running   0          2m
tars-tare-doctor-...                    1/1     Running   0          1m

NAME                      READY   STATUS    RESTARTS   AGE
egress-...                3/3     Running   0          1m

The install command also prints a Dataplane unreachable warning at this point because DNS does not resolve the data-plane hostname yet. This is expected; DNS is wired in Step 12 below.

Step 10: install the AWS load balancer controller​

The data plane needs an L7 load balancer to terminate inbound HTTP and forward to the egress Service in tars-dataplane. The AWS Load Balancer Controller provisions an ALB from Kubernetes Ingress resources.

Step 10.1: iam policy​

The latest controller version requires permissions beyond what the v2.7.1 reference policy covered. Fetch the current policy from main:

curl -sSL -o /tmp/iam-policy.json https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/main/docs/install/iam_policy.json

aws iam create-policy \
  --policy-name AWSLoadBalancerControllerIAMPolicy \
  --policy-document file:///tmp/iam-policy.json

If the policy already exists from a prior run, update it instead:

POLICY_ARN=arn:aws:iam::${AWS_ACCOUNT}:policy/AWSLoadBalancerControllerIAMPolicy
aws iam create-policy-version \
  --policy-arn "${POLICY_ARN}" \
  --policy-document file:///tmp/iam-policy.json \
  --set-as-default

Step 10.2: iam service account​

eksctl create iamserviceaccount \
  --cluster="${EKS_CLUSTER_NAME}" \
  --region="${AWS_REGION}" \
  --namespace=kube-system \
  --name=aws-load-balancer-controller \
  --attach-policy-arn="arn:aws:iam::${AWS_ACCOUNT}:policy/AWSLoadBalancerControllerIAMPolicy" \
  --approve --override-existing-serviceaccounts

Step 10.3: Helm install​

helm repo add eks https://aws.github.io/eks-charts
helm repo update

VPC_ID=$(aws eks describe-cluster --name "${EKS_CLUSTER_NAME}" \
  --region "${AWS_REGION}" \
  --query "cluster.resourcesVpcConfig.vpcId" --output text)

helm install aws-load-balancer-controller eks/aws-load-balancer-controller \
  -n kube-system \
  --set clusterName="${EKS_CLUSTER_NAME}" \
  --set serviceAccount.create=false \
  --set serviceAccount.name=aws-load-balancer-controller \
  --set region="${AWS_REGION}" \
  --set vpcId="${VPC_ID}"

Wait for the controller pods to roll out:

kubectl rollout status -n kube-system deployment/aws-load-balancer-controller --timeout=180s

Step 11: create the alb ingress​

All data-plane traffic (/v1/*, /mcp/*, /.well-known/*) is served by the egress Service in tars-dataplane on port 10080. A single Ingress is enough.

cat <<'EOF' | kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: tars-ingress
  namespace: tars-dataplane
  annotations:
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}]'
spec:
  ingressClassName: alb
  rules:
  - http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: egress
            port:
              number: 10080
EOF

Wait for the ALB to provision (~1 to 2 min) and capture its hostname:

kubectl get ingress tars-ingress -n tars-dataplane

The ADDRESS column populates with something like k8s-tarsdata-tarsingr-xxxxxxxxxx-yyyyyyyyy.us-east-1.elb.amazonaws.com.

If ADDRESS stays empty for more than a few minutes, check the controller for FailedDeployModel events:

kubectl get events -n tars-dataplane --sort-by=.lastTimestamp | tail -5
kubectl logs -n kube-system deployment/aws-load-balancer-controller --tail=30 | grep -iE "error|fail"

The most common cause is an outdated IAM policy missing a permission such as elasticloadbalancing:DescribeListenerAttributes. Re-run the IAM policy update above and restart the controller.

The Ingress persists across tare install reinstalls; re-applying it during upgrades is not required.

Step 12: wire DNS and register the URL​

Step 12.1: DNS​

Point the data-plane hostname at the ALB via a CNAME record at the DNS provider:

proxy.example.com  CNAME  k8s-tarsdata-tarsingr-xxxxxxxxxx-yyyyyyyyy.us-east-1.elb.amazonaws.com

Verify it resolves (propagation may take a couple of minutes):

dig +short proxy.example.com

Step 12.2: register the URL on the management plane​

In the Admin Console: System → Settings → Data planes → edit the data plane → URL. Set it to the hostname just wired in DNS. The management plane pushes the value to the running data plane; no re-install is needed.

Step 13: verify the install​

Run tare doctor to check pod, CRD, and policy state:

tare doctor "${CREDENTIAL_FILE}"

A healthy install reports its components in Accepted state.

A data plane that has just been installed has no routes delivered to it yet, so the report also carries warnings for EnvoyPatchPolicy, RouteDeployment and AIGatewayRoute ("none found"). Those clear once the management plane delivers configuration for a project assigned to this data plane.

Read the component states rather than the exit code: tare doctor exits non-zero on any error, and on a freshly installed data plane it can report forward/check failed (apiserver pod-proxy) for the egress pod even when the control plane can reach it. Check that case before escalating, by requesting the pod's health endpoint over the same path the doctor uses:

POD=$(kubectl get pods -n tars-dataplane -l app=egress -o jsonpath='{.items[0].metadata.name}')
kubectl get --raw "/api/v1/namespaces/tars-dataplane/pods/${POD}:10080/proxy/healthz"

{"status":"ok"} means the pod-proxy path works and the report is not describing a network fault. If any component is not Accepted, or the health endpoint does not answer, contact Tetrate Support.

Step 14: smoke tests​

Once tare doctor reports clean, validate end-to-end traffic.

Create an API key from the router app (Router → API Keys → + Create):

export DP_HOST=proxy.example.com
export TARS_API_KEY=<your-api-key>

Save the suite as smoke.sh. Each endpoint is called twice, once with the key and once without, and a 200 on an unauthenticated call is reported as an auth bypass:

cat > smoke.sh <<'SCRIPT'
#!/usr/bin/env bash
# Smoke test the Agent Router data plane. Every endpoint is called twice: once
# with an API key (expect 200) and once without one (expect 401). A 200 on an
# unauthenticated call is an auth bypass and fails the run.
#
# Requires: curl, and DP_HOST and TARS_API_KEY exported in the environment.
# Set DP_SCHEME=https once TLS is terminated at the load balancer.
#
# The model names are the defaults from onboarding. Override any of them to
# match the models enabled in the project.

set -uo pipefail

: "${DP_HOST:?set DP_HOST}"
: "${TARS_API_KEY:?set TARS_API_KEY}"

BASE="${DP_SCHEME:-http}://${DP_HOST}"
CHAT_MODEL="${CHAT_MODEL:-gpt-5-mini}"
ANTHROPIC_MODEL="${ANTHROPIC_MODEL:-claude-haiku-4-5}"
EMBEDDING_MODEL="${EMBEDDING_MODEL:-text-embedding-3-small}"
IMAGE_MODEL="${IMAGE_MODEL:-gpt-image-1.5}"

JSON="Content-Type: application/json"
AUTH="Authorization: Bearer ${TARS_API_KEY}"
PASS=0; FAIL=0; BYPASS=0

call() {  # call <label> <expected-status> <curl args...>
  local label=$1 expected=$2; shift 2
  local body status mark
  body=$(mktemp)
  status=$(curl -s -o "${body}" -w '%{http_code}' "$@")
  if [[ "${status}" == "${expected}" ]]; then
    mark=ok; PASS=$((PASS+1))
  elif [[ "${expected}" == "401" && "${status}" == "200" ]]; then
    mark="AUTH BYPASS"; BYPASS=$((BYPASS+1))
  else
    mark=fail; FAIL=$((FAIL+1))
  fi
  printf '%-12s %-34s http=%-4s expected=%s\n' "${mark}" "${label}" "${status}" "${expected}"
  [[ "${mark}" == ok ]] || printf '             body: %s\n' "$(head -c 200 "${body}" | tr '\n' ' ')"
  rm -f "${body}"
}

echo "==== list models (GET /v1/models) ===="
call "models auth"      200 "${BASE}/v1/models" -H "${AUTH}"
call "models no-auth"   401 "${BASE}/v1/models"

echo
echo "==== chat completions (POST /v1/chat/completions) ===="
DATA="{\"model\":\"${CHAT_MODEL}\",\"messages\":[{\"role\":\"user\",\"content\":\"reply with one word\"}]}"
call "chat auth"        200 "${BASE}/v1/chat/completions" -X POST -H "${JSON}" -H "${AUTH}" -d "${DATA}"
call "chat no-auth"     401 "${BASE}/v1/chat/completions" -X POST -H "${JSON}" -d "${DATA}"

echo
echo "==== responses (POST /v1/responses) ===="
DATA="{\"model\":\"${CHAT_MODEL}\",\"input\":\"reply with one word\"}"
call "responses auth"    200 "${BASE}/v1/responses" -X POST -H "${JSON}" -H "${AUTH}" -d "${DATA}"
call "responses no-auth" 401 "${BASE}/v1/responses" -X POST -H "${JSON}" -d "${DATA}"

echo
echo "==== anthropic messages (POST /v1/messages) ===="
DATA="{\"model\":\"${ANTHROPIC_MODEL}\",\"max_tokens\":32,\"messages\":[{\"role\":\"user\",\"content\":\"reply with one word\"}]}"
AV="anthropic-version: 2023-06-01"
call "messages auth"     200 "${BASE}/v1/messages" -X POST -H "${JSON}" -H "${AV}" -H "${AUTH}" -d "${DATA}"
call "messages no-auth"  401 "${BASE}/v1/messages" -X POST -H "${JSON}" -H "${AV}" -d "${DATA}"

echo
echo "==== embeddings (POST /v1/embeddings) ===="
DATA="{\"model\":\"${EMBEDDING_MODEL}\",\"input\":\"hello world\"}"
call "embeddings auth"    200 "${BASE}/v1/embeddings" -X POST -H "${JSON}" -H "${AUTH}" -d "${DATA}"
call "embeddings no-auth" 401 "${BASE}/v1/embeddings" -X POST -H "${JSON}" -d "${DATA}"

echo
echo "==== image generation (POST /v1/images/generations) ===="
DATA="{\"model\":\"${IMAGE_MODEL}\",\"prompt\":\"a red apple on a white background\",\"size\":\"1024x1024\",\"n\":1}"
call "images auth"        200 "${BASE}/v1/images/generations" -X POST -H "${JSON}" -H "${AUTH}" -d "${DATA}"
call "images no-auth"     401 "${BASE}/v1/images/generations" -X POST -H "${JSON}" -d "${DATA}"

echo
echo "----"
printf 'pass: %d   fail: %d   auth-bypass: %d\n' "${PASS}" "${FAIL}" "${BYPASS}"
[[ "${FAIL}" -eq 0 && "${BYPASS}" -eq 0 ]]
SCRIPT
chmod +x smoke.sh

Run it:

bash smoke.sh

Expected output:

==== list models (GET /v1/models) ====
ok           models auth                        http=200  expected=200
ok           models no-auth                     http=401  expected=401

==== chat completions (POST /v1/chat/completions) ====
ok           chat auth                          http=200  expected=200
ok           chat no-auth                       http=401  expected=401

==== responses (POST /v1/responses) ====
ok           responses auth                     http=200  expected=200
ok           responses no-auth                  http=401  expected=401

==== anthropic messages (POST /v1/messages) ====
ok           messages auth                      http=200  expected=200
ok           messages no-auth                   http=401  expected=401

==== embeddings (POST /v1/embeddings) ====
ok           embeddings auth                    http=200  expected=200
ok           embeddings no-auth                 http=401  expected=401

==== image generation (POST /v1/images/generations) ====
ok           images auth                        http=200  expected=200
ok           images no-auth                     http=401  expected=401

----
pass: 12   fail: 0   auth-bypass: 0

If any row fails, and in particular if a no-auth row returns 200 instead of 401, contact Tetrate Support.

Once TLS is terminated at the load balancer (Appendix A), re-run the suite with DP_SCHEME=https.

Upgrading​

An upgrade re-runs the install command. The helm chart, CRDs, and config in tars-config are upgraded; persistent state (Redis-backed rate-limit counters, accumulated audit data) survives. The ALB Ingress persists, the IAM policy does not change, and the EBS CSI addon remains installed.

tare install "${CREDENTIAL_FILE}" \
  --image-sync "${ECR_HOST}/${ECR_PREFIX}" \
  --image-pull-secret-name registry-secret

The gateway URL is already registered on the management plane and does not need to be passed again.

note

Some runtime patches (notably observability env vars on the egress deployment) are reset on every install. Re-apply the patches from Appendix B after each upgrade.

Cleanup​

To remove everything created by this guide:

# 1. Ingress (releases the ALB)
kubectl delete ingress tars-ingress -n tars-dataplane --ignore-not-found
# 2. AWS Load Balancer Controller
helm uninstall aws-load-balancer-controller -n kube-system 2>/dev/null || true
eksctl delete iamserviceaccount \
  --name aws-load-balancer-controller \
  --namespace kube-system \
  --cluster "${EKS_CLUSTER_NAME}" \
  --region "${AWS_REGION}" 2>/dev/null || true
ALB_POLICY_ARN=$(aws iam list-policies \
  --query "Policies[?PolicyName=='AWSLoadBalancerControllerIAMPolicy'].Arn" \
  --output text 2>/dev/null)
if [ -n "${ALB_POLICY_ARN}" ] && [ "${ALB_POLICY_ARN}" != "None" ]; then
  # Delete non-default versions first (IAM rule)
  for V in $(aws iam list-policy-versions --policy-arn "${ALB_POLICY_ARN}" \
              --query 'Versions[?IsDefaultVersion==`false`].VersionId' --output text); do
    aws iam delete-policy-version --policy-arn "${ALB_POLICY_ARN}" --version-id "${V}"
  done
  aws iam delete-policy --policy-arn "${ALB_POLICY_ARN}"
fi
# 3. ECR repositories (the cluster prefix matches what was used at create time)
for REPO in $(aws ecr describe-repositories --region "${AWS_REGION}" \
                --query "repositories[?starts_with(repositoryName, '${EKS_CLUSTER_NAME}/')].repositoryName" \
                --output text); do
  aws ecr delete-repository --repository-name "${REPO}" \
    --region "${AWS_REGION}" --force
done
# 4. EKS cluster (also tears down the EBS CSI addon and its IAM role via
#    CloudFormation, takes ~10 min)
eksctl delete cluster --name "${EKS_CLUSTER_NAME}" --region "${AWS_REGION}"
# 5. Local kubeconfig file
rm -f "${HOME}/kubeconfig-${EKS_CLUSTER_NAME}"
note

DNS records (CNAMEs at the DNS provider) must be removed manually.

Troubleshooting​

SymptomLikely causeFix
eksctl create cluster errors with invalid version, supported values: 1.23, …, 1.31eksctl is older than the K8s version EKS currently defaults tobrew upgrade eksctl (or re-download), then retry
tare install fails at preflight with Kubernetes ... is outside the supported rangeThe cluster was created on a version newer than the tare release supports, commonly because the EKS regional default moved aheadRecreate the cluster on a version inside the range (Step 3.1); an EKS cluster cannot be downgraded in place
tare install fails at Register data plane URL with permission_denied ... requires scopes [dataplane_operator]The install was run with the deprecated --serve-url flag on an older CLI. Nothing has been deployedRe-run without --serve-url (or upgrade tare, which ignores it), and register the URL in Step 12.2. See the error reference
tare install --sync-only errors with NAME_UNKNOWN: The repository with name '...' does not existECR repo for that image was never createdThe sync-images.sh helper from Step 5 handles this; re-run it
Ingress ADDRESS stays empty for more than 2 minLB Controller IAM policy is missing newer permissionsUpdate the IAM policy from main, restart the controller
Pods stuck ImagePullBackOff with no basic auth credentialsECR token in registry-secret expired (12h lifetime)Re-run the image-pull-secret creation step (Step 8)
egress pods restart loop with connection refused to redisEBS CSI driver missing, so Redis PV never bindsInstall the EBS CSI addon (Step 4)

Appendix a: TLS via acm + HTTPS listener​

The default flow uses HTTP on port 80. For production, terminate TLS at the ALB using an AWS Certificate Manager (ACM) certificate.

  1. Request the certificate in ACM for the data-plane hostname. Validate via DNS (CNAME) or email per the domain control method. If this gateway serves a multi-gateway project fronted by DNS failover, the certificate must also carry the project's client-facing name as a SAN. A name missing here stays invisible until the first failover. See the multi-SAN worked example in the failover guide.

  2. Get the certificate ARN:

    CERT_ARN=$(aws acm list-certificates --region "${AWS_REGION}" \
      --query "CertificateSummaryList[?DomainName=='proxy.example.com'].CertificateArn" \
      --output text)

  3. Update the Ingress to listen on HTTPS:443 and redirect HTTP → HTTPS:

    metadata:
      annotations:
        alb.ingress.kubernetes.io/scheme: internet-facing
        alb.ingress.kubernetes.io/target-type: ip
        alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS": 443}]'
        alb.ingress.kubernetes.io/ssl-redirect: '443'
        alb.ingress.kubernetes.io/certificate-arn: <CERT_ARN>
        alb.ingress.kubernetes.io/ssl-policy: ELBSecurityPolicy-TLS13-1-2-2021-06

  4. Update the gateway URL on the management plane to https://..., in the Admin Console under System → Settings → Data planes → edit the data plane → URL.

The ALB picks up the annotation change without redeploying. Verify with:

curl -I https://proxy.example.com/v1/models -H "Authorization: Bearer ${TARS_API_KEY}"

Appendix b: observability with OpenTelemetry​

The data plane emits OTLP metrics (router_* family). To forward them to an observability backend, deploy an OpenTelemetry Collector in-cluster.

cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Namespace
metadata:
  name: otel-system

apiVersion: v1
kind: ConfigMap
metadata:
  name: otel-collector-config
  namespace: otel-system
data:
  config.yaml: |
    receivers:
      otlp:
        protocols:
          http:
            endpoint: 0.0.0.0:4318
          grpc:
            endpoint: 0.0.0.0:4317
    processors:
      batch:
        timeout: 5s
      transform/strip_scope:
        metric_statements:
          - context: metric
            statements:
              - replace_pattern(name, "^dynamicmodulescustom\\.", "router_")
    exporters:
      debug:
        verbosity: detailed
      # Replace this with your real backend exporter (Datadog, Grafana Cloud,
      # SigNoz, CloudWatch via the awsemf exporter, etc.).
    service:
      pipelines:
        metrics:
          receivers: [otlp]
          processors: [batch, transform/strip_scope]
          exporters: [debug]

apiVersion: apps/v1
kind: Deployment
metadata:
  name: otel-collector
  namespace: otel-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: otel-collector
  template:
    metadata:
      labels:
        app: otel-collector
    spec:
      containers:
      - name: collector
        image: otel/opentelemetry-collector-contrib:0.98.0
        ports:
        - containerPort: 4317
        - containerPort: 4318
        volumeMounts:
        - name: config
          mountPath: /etc/otelcol-contrib
      volumes:
      - name: config
        configMap:
          name: otel-collector-config

apiVersion: v1
kind: Service
metadata:
  name: otel-collector
  namespace: otel-system
spec:
  selector:
    app: otel-collector
  ports:
  - name: otlp-http
    port: 4318
    targetPort: 4318
  - name: otlp-grpc
    port: 4317
    targetPort: 4317
EOF

Point the Agent Router egress proxy at the collector:

kubectl set env deploy/egress -n tars-dataplane \
  OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector.otel-system.svc.cluster.local:4318 \
  OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf

Generate traffic via the smoke tests, then verify metrics are flowing:

kubectl logs -n otel-system deployment/otel-collector --tail=50 | grep "Name:"

Expected metrics include router_requests_total, router_auth_attempts_total, and similar.

note

The kubectl set env patch is reset by tare install re-runs. Re-apply after every upgrade until the chart accepts these settings via Helm values.

Where to go next

Gateway installation

Install the data plane gateway components that manage inbound access and configure request routing.

Self-Hosted Data Plane

See the full install sequence: gateway, SSO, then developer handoff.