Data Plane Credentials

Agent Router Enterprise · Self-Hosted Data Plane

The data plane authenticates to the management plane with the service account credential issued during onboarding. Rotation and revocation take effect at the management plane immediately, while the data plane keeps using the credential installed in the cluster. Closing that gap is a manual step.

Rotating the credential can break inference

Rotation revokes the credential currently in use as soon as the action is confirmed, and the replacement is downloaded once in the browser. Nothing writes that replacement into the cluster. Until the new file is installed and the release is upgraded, the data plane presents a revoked credential to the management plane, and gateway replicas that observe the rejection stop serving: inference requests return 503, telemetry export stops, and configuration changes no longer arrive. Rotate only inside a maintenance window, following Rotate the credential.

What the credential is

The credential is the service-account.json file produced in Step 5 of management plane onboarding and consumed by tare install. It is the data plane's identity, not an inference key, and it is not the same as the API keys developers use.

Four properties matter for day 2 operations:

One credential is active at a time. The identity is fixed per deployment, so a replacement never coexists with the credential it replaces. Issuing a new credential revokes the old one.
The private key is presented once. It is returned in the download and is not stored anywhere retrievable afterward. A lost file cannot be recovered, only replaced.
The default validity is 365 days. A credential that is never rotated expires a year after it is issued.
The data plane uses it continuously. Every call the data plane makes to the management plane carries it: configuration sync, request log and usage export, and the API key revocation list. A validity check runs every 30 seconds.

What each action does

Action	Effect at the management plane	Effect on the running data plane
Rotate	Issues a new credential and revokes the active one in the same operation. The new file downloads once.	None until the new file is installed. The data plane keeps presenting the revoked credential and is rejected.
Revoke	Marks the credential revoked and withdraws its signing key.	Every call to the management plane is rejected. Replicas fail closed and stop serving.
Restore	Reactivates a revoked credential and revokes the currently active one in the same transaction, so there is never a window with two active credentials.	Service resumes once the change reaches the data plane, without reinstalling or restarting.

Rotate and revoke are available in the Admin Console under System | Settings, on the Data planes tab, through Edit on the data plane. Rotation is performed there with Generate credential. Restore is available through the management API only, as described in Recover a data plane holding a revoked credential.

How quickly a change is noticed

The data plane revalidates its own credential every 30 seconds. A rejection at that check causes the replica to fail closed and reject every API key until a later check succeeds.
The API key revocation list refreshes every 5 minutes by default.
Public signing keys are cached for 5 minutes, which bounds how long a restore takes to become effective.

Rotate the credential

Rotation is a maintenance operation with a service interruption between the moment the old credential is revoked and the moment the new one is running in the cluster. Keep that window short.

Schedule a maintenance window and confirm that a workstation with the tare CLI and cluster access is ready before starting.
In the Admin Console, go to System | Settings, open the Data planes tab, select Edit on the data plane, and click Generate credential. Save the downloaded file immediately, for example as /path/to/data-plane-credentials.json. The download cannot be repeated.

Install the new credential in the cluster. For a CLI installation, rerun the install with the new file:

tare install /path/to/data-plane-credentials.json \
  --serve-url "https://proxy.example.com"

For a Helm installation, re-render the values and upgrade the release:

tare install /path/to/data-plane-credentials.json \
  --image-registry "${PRIVATE_IMAGE_REGISTRY}" \
  --print-helm-values > values.yaml

helm upgrade --install tars "oci://${PRIVATE_IMAGE_REGISTRY}/serve-helm" \
  --version "${CHART_VERSION}" \
  -f values.yaml \
  -n tars-system \
  --skip-crds \
  --set ai-gateway-crds.enabled=false

values.yaml contains the new identity secret. Treat it as sensitive and store it through the approved secret workflow, as described in the Helm installation guide.

Confirm the workloads picked up the new identity, and restart them if they did not:

kubectl rollout restart deployment/controller -n tars-system
kubectl rollout restart deployment/egress -n tars-dataplane

Verify the auth chain and serve a test request:
```
tare doctor /path/to/data-plane-credentials.json --verbose
```
The run reports the data plane credentials as revoked while any replica still holds the old file.

GitOps and secret managers do not close the gap

When the identity is delivered by External Secrets, Argo CD, or Flux, a rotation performed in the Admin Console does not reach the source those tools read. They continue to sync the previous version and continue to report success, so nothing appears unhealthy while the data plane is rejected on every call. Update the source of the identity secret as part of the rotation, not afterward.

Revoke the credential

Revocation withdraws the data plane's identity and is appropriate in two cases:

The credential file is believed to be exposed, and access must be cut immediately.
The data plane is being decommissioned and should no longer reach the management plane.

Revocation is not a way to pause traffic and is not a prerequisite for rotation, because rotation already revokes the credential it replaces. A revoked data plane stays revoked until the credential is restored or a new one is installed.

What a revoked data plane does

Inference requests return HTTP 503 with a retry-after header and a service_unavailable error code, rather than a 401. The status reflects a gateway that cannot validate credentials, not a bad client key.
Request logs and usage records are rejected and dropped, so traffic disappears from Request Logs and analytics for the affected period.
The API key revocation list stops refreshing, so an API key revoked in the Console can keep working at the gateway.
Configuration changes made in the Console or the Admin Console no longer reach the data plane.

Recover a data plane holding a revoked credential

If the replacement file was lost, never installed, or installed only partially, the fastest recovery is to restore the credential the data plane still holds rather than to reinstall.

List the service accounts for the deployment and identify the one the data plane is using. Its identifier also appears in the data plane logs, in the message no public key found for kid: <id>.
```
curl -sS "https://api.<tenant>.tetrate.ai/admin/v1/service-accounts" \
  -H "Authorization: Bearer ${ADMIN_API_KEY}"
```

Restore it, recording a reason for the audit trail:

curl -sS -X POST "https://api.<tenant>.tetrate.ai/admin/v1/service-accounts/${SERVICE_ACCOUNT_ID}:restore" \
  -H "Authorization: Bearer ${ADMIN_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{"reason":"data plane still holds this credential"}'

Wait up to 5 minutes for the signing key to become visible again, then confirm recovery:
```
tare doctor /path/to/data-plane-credentials.json --verbose
```

Restore revokes whatever credential is active at that moment, so any newer credential issued by the rotation stops working. Restart the data plane workloads if a replica has not resumed serving after two validity checks, roughly one minute.

Restoring is a stopgap. The credential still expires on its original date, and a further rotation reproduces the outage unless the new file is installed in the cluster.

Diagnose a suspected credential problem

Signal	Where to look
`DataplaneCredentialsRevoked` under the auth chain condition	`tare doctor <credential> --verbose`
`Data plane license revoked by management plane; rejecting all tokens`	Controller and gateway logs
`no public key found for kid: <id>` on every revocation cache refresh	Gateway (`egress`) logs in the `tars-dataplane` namespace
`503` with error code `service_unavailable` on every request	Client responses and Request Logs
Revocation reasons such as `Auto-revoked for credential rotation`	Audit logs, filtered on the `service_account` resource type (Audit log events)

Fully Managed deployments

These actions are not part of Fully Managed operations. Rotating or revoking the credential of a Tetrate-operated data plane breaks it, and the replacement cannot be installed by the customer. Contact Tetrate instead of rotating.

Management plane onboarding

Where the credential is first issued, in Step 5 of the wizard.

Data plane installation

The install and upgrade paths a rotation has to be applied through.

Gateway behavior

Status codes, retry semantics, and the error format clients receive.

Audit log events

The recorded actions and resource types, including service accounts.

What the credential is​

What each action does​

How quickly a change is noticed​

Rotate the credential​

Revoke the credential​

What a revoked data plane does​

Recover a data plane holding a revoked credential​

Diagnose a suspected credential problem​