Terminate TLS on the data plane

From 0.8.0 a self-hosted data plane terminates TLS itself. Each declared hostname listed with a certificate is published on port 443, and egress-lb, a layer 4 load balancer that passes TLS through to the data plane, is the only object exposed to the network. No layer 7 load balancer or cloud certificate service sits in front.

Persona: Platform operator with access to the data plane's Helm values, kubectl on the cluster, the certificates for each hostname, and the DNS zone.

Estimated time: 15 to 30 minutes, plus the cloud's load balancer provisioning and DNS propagation.

Outcomes

By the end of this guide:

Each hostname to publish serves HTTPS on port 443 with its own certificate, terminated at the data plane, and authenticates requests exactly as the data plane's existing listener does.
egress-lb, an internal layer 4 load balancer, publishes port 443 and nothing else.
Each published hostname resolves to egress-lb.

Architecture

            callers
               │  TLS (SNI: a.example.com, b.example.com)
               ▼
┌──────────────────────────────┐
│ egress-lb (Service)          │  ← layer 4, port 443 only
│ cloud load balancer          │    passes TCP through, no TLS here
└──────────────┬───────────────┘
               │  443 → 10443 on the gateway Pods
               ▼
┌──────────────────────────────┐
│ data plane gateway (Envoy)   │  ← terminates TLS per hostname,
│  443: a.example.com, b...    │    then authenticates and routes
│  10080: in-cluster only      │
└──────────────────────────────┘

The data plane gets one HTTPS listener per published hostname, each presenting the certificate from a Kubernetes Secret. egress-lb is a Service of type LoadBalancer that publishes port 443 to those listeners and nothing else, so the data plane's in-cluster port 10080 is never on the load balancer. egress-lb is installed in its own Helm release, separate from the data plane's.

The listeners are on port 443 in the Gateway. Envoy Gateway shifts a port below 1024 up by 10000, so the gateway Pods bind them on 10443, which is the port egress-lb targets. 10080 is above 1024 and is not shifted.

Publication is per hostname. The hostnames listed for TLS are a subset of the declared hostnames; any other declared hostname gets no listener on 443 and is not reachable through egress-lb. A data plane labelled with a project serves the project on every hostname it declares (see Serve a project from a data plane), so publishing a hostname publishes it for every project on that data plane. One data plane cannot publish a name for one project and keep it internal for another.

info

This path is an alternative to the cloud-specific fronts in the AWS, AKS, and GCP gateway guides, which remain supported. A data plane already exposed through one of them keeps serving on it until DNS is repointed, and the previous front is removed only after that. See Move an exposed data plane.

Prerequisites

A self-hosted data plane on 0.8.0 or later.
Every hostname to publish already declared under global.gateway.hostnames. See Declare the hostnames a data plane serves. A hostname listed for TLS that is not declared fails the install.
A certificate and private key for each hostname, issued by a CA that callers trust. Issuance and renewal stay with the operator (cert-manager, a corporate CA, or any other source).
kubectl access to the cluster and permission to create Secrets in the tars-dataplane namespace.
Control of the DNS zone for each hostname.

Conventions

The commands below assume these environment variables:

export CREDENTIAL_FILE=/path/to/data-plane-credentials.json
export CLOUD=aws          # aws, azure, or gcp

Step 1: create a certificate Secret for each hostname

Each hostname needs a Secret of type kubernetes.io/tls in the tars-dataplane namespace, with the full chain (leaf followed by intermediates) in tls.crt. The certificate must cover its hostname.

To have certificates renewed automatically, issue them with cert-manager: a Certificate per hostname whose secretName is the Secret named in Step 2. cert-manager renews by replacing the Secret's data in place, which the gateway picks up without a restart. Create Certificate resources directly; do not annotate the data plane's Gateway, whose listeners the chart owns. With the default internal load balancer, an ACME issuer needs the DNS-01 challenge.

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: cert-a
  namespace: tars-dataplane
spec:
  secretName: cert-a
  dnsNames:
    - a.example.com
  issuerRef:
    kind: ClusterIssuer
    name: <issuer>

With a certificate issued elsewhere, create the Secret from its files:

kubectl -n tars-dataplane create secret tls cert-a \
  --cert=a.example.com.crt --key=a.example.com.key
kubectl -n tars-dataplane create secret tls cert-b \
  --cert=b.example.com.crt --key=b.example.com.key

Step 2: list the hostnames to publish

Add httpsListeners to the data plane's values, one entry per hostname, each naming its Secret. This example publishes two of the three declared hostnames; internal.example.com stays reachable only inside the cluster:

# tls-values.yaml
egressResources:
  egress:
    gateway:
      httpsListeners:
        - hostname: a.example.com
          tls:
            certificateRefs:
              - name: cert-a
        - hostname: b.example.com
          tls:
            certificateRefs:
              - name: cert-b

Apply the values by re-running the install with the values file, or with helm upgrade of the data plane release:

tetrate agentrouter dataplane install "${CREDENTIAL_FILE}" \
  --gateway-hostname=a.example.com \
  --gateway-hostname=b.example.com \
  --gateway-hostname=internal.example.com \
  --helm-values=tls-values.yaml

Each entry takes the Gateway API listener's own field names: hostname (required) and tls.certificateRefs (required, more than one is allowed, for example an RSA and an ECDSA certificate). The listener name, port, protocol, and TLS mode are fixed and fail the install if set.

A plain helm upgrade can drop the listeners

tetrate agentrouter dataplane upgrade keeps httpsListeners and httpsListenerDefaults from the installed release. A helm upgrade without this file and without --reuse-values drops them, and every published hostname stops serving on 443. Keep the file in the values applied on every helm upgrade.

Nothing is reachable from outside the cluster yet, unless the data plane is already exposed with tetrate agentrouter dataplane ingress install --type aws. That path makes the data plane's own Service a Network Load Balancer, which publishes every listener port: from this step on it also publishes 443, next to the plaintext 10080. See Move a data plane exposed on AWS.

note

Until the data plane has put authentication on a newly published hostname, that hostname answers 403 to every request, valid key included. It never serves an unauthenticated response. The 403 clears on its own, usually within seconds.

Step 3: confirm the data plane serves the hostnames

kubectl -n tars-dataplane get gateway egress \
  -o jsonpath='{range .status.listeners[*]}{.name} {.conditions[?(@.type=="Programmed")].status}{"\n"}{end}'
# https-0 True
# https-1 True

tetrate agentrouter doctor

Each listener https-<n> reads Programmed True, and tetrate agentrouter doctor reports no error under the gateway's 443 listeners.

With two or more published hostnames, the EnvoyPatchPolicies tars-tls-default-deny and tars-proxy-tls (and tars-guardrails-tls when guardrails are configured) report the reason ResourceNotFound on their Programmed condition:

kubectl -n tars-dataplane get envoypatchpolicy \
  -o custom-columns=NAME:.metadata.name,REASON:'.status.ancestors[0].conditions[?(@.type=="Programmed")].reason'
# NAME                    REASON
# tars-proxy              Programmed
# tars-proxy-tls          ResourceNotFound
# tars-tls-default-deny   ResourceNotFound

That is the healthy state: the policies carry an entry for every listener name the gateway can use, and only the one in use resolves. tetrate agentrouter doctor reads the listeners themselves and reports any real fault.

Step 4: install egress-lb

tetrate agentrouter dataplane ingress install "${CREDENTIAL_FILE}" \
  --type "${CLOUD}" --lb-tls-passthrough --wait

--type selects the cloud's defaults. Without other values, every cloud gets an internal load balancer:

`--type`	Load balancer	Default annotations
`aws`	Internal Network Load Balancer, in-tree EKS cloud provider	`service.beta.kubernetes.io/aws-load-balancer-type: nlb`, `service.beta.kubernetes.io/aws-load-balancer-internal: "true"`
`azure`	Internal Azure Load Balancer	`service.beta.kubernetes.io/azure-load-balancer-internal: "true"`
`gcp`	Internal passthrough Network Load Balancer	`networking.gke.io/load-balancer-type: "Internal"`

Every cloud also gets type: LoadBalancer and externalTrafficPolicy: Local. To use another load balancer controller or EKS Auto Mode, see Customize egress-lb.

--wait returns once the load balancer reports an address and prints the DNS record to create and a check to run against it. The command installs egress-lb only: the data plane's existing exposure (an AWS NLB, an AKS Application Gateway, or a GKE Gateway) is left as it is.

Step 5: verify against the load balancer address

Before changing DNS, send requests to the new address directly. With an internal load balancer, run them from inside the network, for example from a Pod in the cluster (with node targets, from a node that runs no data plane gateway Pod):

LB=<address printed by Step 4>
curl -s -o /dev/null -w '%{http_code}\n' \
  --connect-to a.example.com:443:"${LB}":443 \
  https://a.example.com/v1/models
# 401

curl -s -o /dev/null -w '%{http_code}\n' \
  --connect-to a.example.com:443:"${LB}":443 \
  -H "Authorization: Bearer ${API_KEY}" \
  https://a.example.com/v1/models
# 200

The TLS handshake presents the certificate of the hostname named in the request, a request without a valid key gets 401, and one with a valid key succeeds. --connect-to accepts the load balancer's IP address or hostname. Repeat for each published hostname.

Step 6: point DNS at egress-lb

Create a record for each published hostname: an A record when the address is an IPv4 address (Azure, GCP), AAAA for an IPv6 address, or a CNAME when it is a hostname (AWS). The address of egress-lb stays the same when hostnames are added or removed, so this step is needed once per hostname.

Move an exposed data plane

A data plane that already serves callers through one of the cloud-specific fronts moves to TLS on the data plane without downtime. The steps above are the same; what changes is the order around them. egress-lb gets an address of its own, so the move happens in DNS: both fronts serve the published hostnames until DNS points at egress-lb, and the previous front is removed only after that.

	Action	Where
1	Create the certificate Secrets and add `httpsListeners`	Step 1 to Step 3
2	Install `egress-lb`	Step 4
3	Verify against the `egress-lb` address	Step 5
4	Lower the TTL of each hostname's DNS record, and wait for the old TTL to pass	DNS
5	Point DNS at `egress-lb`, and wait for the TTL to pass	Step 6
6	Remove the previous front	Below

The TTL can be lowered at any point up to step 4. Lowering it first lets the wait for the old TTL run alongside steps 1 to 3; lowering it at step 4 leaves DNS untouched until the new path is verified.

Until step 6 the previous front keeps serving exactly as before, and pointing DNS back at it undoes the move. At step 6, once DNS has moved, remove the previous front and the cloud resources it created for the data plane. On AWS it is the data plane's own Service, which needs more care: see Move a data plane exposed on AWS.

Before starting, check three things that differ from the previous front:

The certificates move into the cluster. The previous front held the certificate in the cloud (an ACM certificate on AWS, the Application Gateway on AKS, a Certificate Manager map on GKE). egress-lb passes TLS through, so each published hostname needs its certificate and key as a Secret in the cluster (Step 1).
egress-lb is internal by default. A previous front that faced the internet (the default on AKS and GKE) needs egress-lb to face the internet too, or callers lose the hostname at step 5. Set the provider's external settings before step 2: see Customize egress-lb.
Callers' clients see a different load balancer. An allow list of addresses on the caller's side, or a pinned IP, has to include the egress-lb address before step 5.

Move a data plane exposed on AWS

tetrate agentrouter dataplane ingress install --type aws exposes the data plane by patching its EnvoyProxy so that the gateway's own Service, egress in tars-dataplane, is a Network Load Balancer. That Service carries every listener port. Two consequences shape the move on AWS:

From step 1, the existing NLB publishes 443 as soon as the HTTPS listeners exist, next to the plaintext 10080 it already publishes. Both answer exactly as they do through egress-lb.
Step 6 is what takes 10080 off the load balancer. Until it runs, the existing NLB keeps publishing 10080 in plain HTTP next to the TLS hostnames on 443, so a caller can still reach the data plane without TLS. Run it once DNS has moved.

Steps 1 to 4. Follow the table above. Two checks are specific to AWS.

After step 1, the existing NLB has gained a 443 listener:

OLD=$(kubectl -n tars-dataplane get service egress -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')
aws elbv2 describe-listeners \
  --load-balancer-arn "$(aws elbv2 describe-load-balancers --query "LoadBalancers[?DNSName=='${OLD}'].LoadBalancerArn" --output text)" \
  --query 'Listeners[].Port'
# [443, 10080, 10081]

After step 2, there are two load balancers with different addresses: the existing NLB ($OLD) and egress-lb. Installing egress-lb does not change the existing NLB.

LB=$(kubectl -n tars-dataplane get service egress-lb -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')
echo "existing: ${OLD}"
echo "egress-lb: ${LB}"

	Existing NLB	`egress-lb`
Created by	`--type aws`, through the EnvoyProxy	the `tars-egress-lb` Helm release
Ports	443, `10080`, `10081`	443
A published hostname on 443	`401` without a key, `200` with one	the same

Step 5. Change each hostname's CNAME from $OLD to $LB, and wait for the TTL to pass. In between, a caller reaches either load balancer and gets the same answer.

Step 6: remove the existing NLB. Patch the EnvoyProxy's Service type back to ClusterIP. Change only type, and leave the load balancer annotations as they are: the cloud provider reads them to find the NLB to delete.

kubectl -n tars-system patch envoyproxy tars-egress-proxy --type merge \
  -p '{"spec":{"provider":{"kubernetes":{"envoyService":{"type":"ClusterIP"}}}}}'

A values change does not remove it. --type aws set the NLB on the live EnvoyProxy, and egressResources.envoyProxy.service.type in the data plane's values reads ClusterIP throughout, so setting it and upgrading changes nothing.

Confirm the NLB is gone:

kubectl -n tars-dataplane get events --field-selector involvedObject.name=egress
# DeletingLoadBalancer, then DeletedLoadBalancer
kubectl -n tars-dataplane get service egress -o jsonpath='{.spec.type}{"\n"}'
# ClusterIP
aws elbv2 describe-load-balancers --query "LoadBalancers[?DNSName=='${OLD}']"
# []

Change the published hostnames

tetrate agentrouter dataplane upgrade takes no values file, so it keeps the published hostnames as they are. To add, remove, or change one, edit tls-values.yaml and apply it as in Step 2: re-run tetrate agentrouter dataplane install with --helm-values=tls-values.yaml, or helm upgrade with the file. Pass --gateway-hostname for every declared hostname on each re-run, as in Step 2: unlike tetrate agentrouter dataplane upgrade, a re-run of the install does not keep the declared list, and a published hostname that is no longer declared fails the install.

httpsListeners is replaced whole, never merged: the file lists every hostname to publish, not only the new one. A hostname to add must be declared first, and a hostname left out of the file stops serving on 443. httpsListeners: [] stops publishing on 443 altogether.

As in Step 2, a newly published hostname answers 403 to every request, a valid key included, until the data plane has added authentication to its new listener. If the 403 persists, see Every request on a hostname answers 403.

egress-lb serves every published hostname at one address, which adding a hostname does not change. Add a DNS record for the new hostname pointing at that address, unless an existing wildcard record already covers it.

Customize egress-lb

The defaults are kept in the chart. Values passed with --helm-values replace a default; they are never merged with it. Setting service.annotations replaces every default annotation, and service.annotations: {} renders none. What the operator sets is the operator's to own: whether the load balancer is internal, which controller provisions it, its target type, and its health check.

Value	Default	Effect
`service.annotations`	The cloud's internal annotations above	Replaces all of them
`service.type`	`LoadBalancer`	Replaces it
`service.externalTrafficPolicy`	`Local`	Replaces it
`service.loadBalancerClass`	Unset (the cloud's in-tree provider)	Hands the Service to another controller
`service.allocateLoadBalancerNodePorts`	Unset	`false` only for load balancers that target Pods directly
`service.loadBalancerSourceRanges`	Unset	Restricts client CIDRs
`service.loadBalancerIP`	Unset	A reserved static address, where the provider takes it this way

The published port (443) and the selector are fixed by the chart and fail the install if set.

tetrate agentrouter dataplane ingress install "${CREDENTIAL_FILE}" \
  --type "${CLOUD}" --lb-tls-passthrough --helm-values=egress-lb.yaml --wait

Health checks

The load balancer's health check depends on what it targets:

Node targets (the default on every cloud): with externalTrafficPolicy: Local, the provider's default health check on the Service's health check node port answers only on a node running a ready data plane gateway Pod. Nothing needs to be set.
Pod targets (the AWS Load Balancer Controller in IP mode, EKS Auto Mode): point the health check at the gateway's readiness endpoint, HTTP on port 19003, path /ready, through the provider's annotations.

The gateway's liveness port is never a load balancer health check.

EKS Auto Mode

EKS Auto Mode does not run the in-tree load balancer provider, so the AWS default does not apply. Hand the Service to Auto Mode's controller:

service:
  loadBalancerClass: eks.amazonaws.com/nlb
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-scheme: internal
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-protocol: HTTP
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-port: "19003"
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-path: /ready
  allocateLoadBalancerNodePorts: false

Auto Mode registers Pod IPs as targets.

loadBalancerClass cannot change on an existing Service. Set it on the first install, or uninstall the tars-egress-lb release and install again.

Install with Helm or Argo CD

egress-lb is the egress-lb-helm chart. A Helm or Argo CD install sets cloud (aws, azure, or gcp, required) and the service values above, and gets the same Service as the CLI.

Certificates

Rotation. Replace the Secret's data in place, under the same name. The gateway picks it up without a restart:
```
kubectl -n tars-dataplane create secret tls cert-a \
  --cert=a.example.com.crt --key=a.example.com.key \
  --dry-run=client -o yaml | kubectl apply -f -
```
A certificate issued by cert-manager (see Step 1) is rotated this way on its own. Deleting the Secret and creating it again is not a rotation: the hostname stops serving in between.
Renaming a Secret is a values change: update certificateRefs and apply the file as in Change the published hostnames.
A Secret in another namespace needs a ReferenceGrant in that namespace. Add namespace to the certificateRefs entry, and either create the grant, or set egressResources.egress.gateway.httpsListenerDefaults.createReferenceGrants: true to have the chart render it.
A missing, malformed, or expired certificate stops only its own hostname. The listener reads ResolvedRefs=False with reason InvalidCertificateRef (or RefNotPermitted for a cross-namespace Secret without a grant), the handshake on that hostname fails, and the other hostnames keep serving.

TLS parameters

With no settings the gateway serves TLS 1.2 and 1.3. To raise the minimum version, restrict ciphers, set ALPN, or require client certificates, set the Envoy Gateway ClientTrafficPolicy TLS fields under egressResources.egress.gateway.httpsListenerDefaults.clientTrafficPolicy.tls. They apply to every published hostname.

egressResources:
  egress:
    gateway:
      httpsListenerDefaults:
        clientTrafficPolicy:
          tls:
            minVersion: "1.3"

Troubleshooting

egress-lb stays pending

The cloud controller explains why in the Service's events:

kubectl -n tars-dataplane describe service egress-lb | sed -n '/^Events:/,$p'

On EKS Auto Mode or with the AWS Load Balancer Controller, FailedBuildModel ... unable to resolve at least one subnet means no subnet carries the tag the controller looks for (kubernetes.io/role/internal-elb for an internal load balancer). Tag the subnets, or name them in service.beta.kubernetes.io/aws-load-balancer-subnets.

Every request on a hostname answers 403

The data plane has not yet put authentication on that hostname. Wait a few seconds after an upgrade. If it persists, run tetrate agentrouter doctor: it names the hostname and the missing policy.

The handshake fails or the connection resets

The hostname in the request has no listener: it is missing from httpsListeners, or its certificate does not resolve. Check the listener status in Step 3 and the certificate conditions under Certificates.

Where to go next

Declare the hostnames a data plane serves

Add or retire a hostname before publishing it.

Serve a project from a data plane

Label the data plane with a project so it is callable on the published hostnames.

Outcomes​

Architecture​

Prerequisites​

Conventions​

Step 1: create a certificate Secret for each hostname​

Step 2: list the hostnames to publish​

Step 3: confirm the data plane serves the hostnames​

Step 4: install egress-lb​

Step 5: verify against the load balancer address​

Step 6: point DNS at egress-lb​

Move an exposed data plane​

Move a data plane exposed on AWS​

Change the published hostnames​

Customize egress-lb​

Health checks​

EKS Auto Mode​

Install with Helm or Argo CD​

Certificates​

TLS parameters​

Troubleshooting​

egress-lb stays pending​

Every request on a hostname answers 403​

The handshake fails or the connection resets​