# Custom observability backends

> Fan data-plane telemetry out to your own OTLP backends per signal — metrics, traces, logs, access logs, request logs, and guardrail triggers — from the data-plane collector.

Enterprise Tier

  The data plane runs its own OpenTelemetry collector, and everything the management plane learns about traffic passes through infrastructure Tetrate operates. Many organisations need a second copy — the same telemetry, delivered to systems they run themselves: a SIEM that must see every guardrail firing, a logging stack that must hold full request content for compliance, a metrics platform that already drives the on-call rotation. The <code>global.observability.backends.custom[]</code> Helm values list is the single mechanism for all of it. Each entry attaches one OTLP destination to the data-plane collector and subscribes it to a chosen subset of signals; the collector then fans each signal out to every subscribed backend, independently of what does or does not travel to the management plane. Configuration is deliberately GitOps-only — adding a backend is a reviewed infrastructure change by the cluster owner, not a dashboard click — because two of the signals carry full prompt, response, and matched-content data.

**Persona:** Platform operator who owns the data-plane Helm values (usually through a GitOps flow), typically in coordination with security or compliance stakeholders for the content-bearing signals.

**Estimated time:** 15 to 30 minutes to add and verify a backend; longer if the receiving side (network reachability, credentials) still has to be arranged.

## When this guide applies

| Situation | What a custom backend adds |
| :---- | :---- |
| A SIEM or logging stack must receive every guardrail trigger with full matched content | Subscribe a backend to `guardrailtrigger`; the copy is always full content, regardless of the dashboard content-mode setting |
| Full request/response bodies must land in a self-managed store, possibly while the management plane receives less | Subscribe a backend to `requestlog` |
| Data-plane metrics, traces, or logs should also feed an existing observability platform | Subscribe a backend to any of the four legacy signals |
| One signal must reach several destinations | Add several entries; every subscribed backend receives its own copy |

Two adjacent surfaces are **not** this mechanism: trace export configured from the Developer Console (see [Export telemetry to observability](/agent-router-enterprise/guides/observability-and-analytics/export-telemetry-to-an-observability-stack)) and the management-plane request-log modes set in the Admin Dashboard (see [Control request log data](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/configuring-request-logs)). Custom backends are configured in Helm values only; there is no dashboard or API surface for them.

## Outcomes

By the end of this guide:

- Every `backends.custom[]` field is understood, including the explicit-subscription rule for the content-bearing signals.
- A backend is subscribed to `requestlog` and/or `guardrailtrigger` and verified receiving records.
- The delivery guarantees — sized in-memory queues, drop-on-overflow, bounded retry — are understood well enough to size the queue and pick a protocol deliberately.
- The relationship between the always-full collector copy of guardrail triggers and the dashboard's management-plane content-mode setting is clear.

## How the fan-out works

The data-plane cluster runs a local OpenTelemetry collector (`otel-gateway`, a stateless two-replica Deployment in the system namespace). Data-plane components push their telemetry to this collector, and the collector forwards it: management-plane-bound copies travel on their own pipelines, and each `backends.custom[]` entry adds one more OTLP exporter that joins the pipelines of the signals it subscribes to. The management-plane copies are unchanged by any backend you add.

Six signals exist, in two groups with different subscription semantics:

| Signal | Content | Subscription |
| :---- | :---- | :---- |
| `metrics` | Data-plane metrics | Default set |
| `traces` | Request traces | Default set |
| `logs` | Application logs of the data-plane components | Default set |
| `accesslog` | Envoy access logs (requires the access-log lane to be enabled with an endpoint: `global.observability.accessLog`) | Default set |
| `requestlog` | **Full request-log records, including request and response bodies** | **Explicit only** |
| `guardrailtrigger` | **Full guardrail-trigger records, including matched and sanitized message content** | **Explicit only** |

**The explicit-subscription rule:** a backend whose `signals` list is omitted or empty receives the four legacy signals — `metrics`, `traces`, `logs`, `accesslog` — and **never** `requestlog` or `guardrailtrigger`. The content-bearing signals only flow to a backend that names them, for example `signals: [requestlog]`. This is a deliberate privacy guardrail: upgrading a chart, or adding a catch-all backend, can never silently start shipping full prompt and response content somewhere new.

For the two content-bearing signals, the chart wires the entire path from one predicate — `global.observability.enabled` AND `signals.<signal>.enabled` AND at least one subscribed backend. When it holds, the collector renders a dedicated receiver and pipeline for the signal **and** the emitting component is configured (via chart-injected environment variables on the egress pod) to push to it; when it does not hold, both disappear together and nothing is emitted. There is nothing to enable by hand on the emitting side — subscribing a backend is the switch.

Each signal that a backend does not subscribe to, and each signal whose `signals.<s>.enabled` toggle is off, simply does not flow to it; a backend subscribed to a disabled signal keeps its exporter but receives nothing until the signal is enabled.

## The `backends.custom[]` fields

A complete entry, with every field:

```yaml
global:
  observability:
    backends:
      custom:
        - name: siem                    # required
          endpoint: otel.security.example.com:4317   # required
          protocol: otlp                # otlp (default) | otlphttp
          signals: [guardrailtrigger]   # omitted/empty = metrics, traces, logs, accesslog
          compression: none             # omitted = gzip
          tls:
            insecure: false
            insecureSkipVerify: false
          auth:
            headers:
              Authorization: { secretKey: SIEM_AUTH_HEADER }   # from the credentials Secret
              X-Scope-OrgID: { value: acme }                   # non-secret literal
          sendingQueue: { queueSize: 64, numConsumers: 4 }
```

### `name` (required)

Lowercase letters, digits, and hyphens (`[a-z0-9-]`). The name becomes part of the collector exporter's identity (`otlp_grpc/custom-<name>` or `otlphttp/custom-<name>`), which is how the backend appears in the collector's own logs and metrics — pick something that will make sense in a 3 a.m. dashboard.

### `endpoint` (required)

Where the backend listens. The expected shape depends on `protocol`:

- `protocol: otlp` (OTLP/gRPC): host and port, for example `otel-collector.observability.svc.cluster.local:4317`.
- `protocol: otlphttp` (OTLP/HTTP): a full `http(s)://` URL.

### `protocol`

`otlp` (OTLP/gRPC, the default) or `otlphttp` (OTLP/HTTP). Choose `otlphttp` in two situations:

1. **Oversized single records.** gRPC receivers commonly cap messages at 4 MiB by default, and a single record larger than the receiving side's cap can *never* be delivered over gRPC — it is retried for the full retry budget and then dropped. Request-log records for large prompts and responses can reach low-single-digit MiB, so a `requestlog` subscriber whose records may approach its receiver's cap should either raise the cap on its receiver or subscribe over `otlphttp`, which has no 4 MiB framing default.
1. **Intermediate infrastructure.** Load balancers and proxies between the data plane and the backend that do not pass gRPC through cleanly. OTLP/HTTP traverses ordinary HTTP infrastructure.

### `signals`

The subset of signals to ship, from the six in the table above. Omitted or empty means the four legacy signals and never the content-bearing two — see the explicit-subscription rule.

### `compression`

Omit for the collector default (gzip). Set `compression: none` for backends that do not accept gzip-compressed OTLP.

### `tls`

`tls.insecure: true` disables TLS entirely (plaintext — in-cluster hops only). `tls.insecureSkipVerify: true` keeps TLS but skips certificate verification. Leave both unset for full TLS verification against system CAs.

### `auth.headers`

Headers attached to every export request. Each header takes one of two value forms:

- `{ value: <literal> }` — a non-secret literal, rendered into the collector config as-is.
- `{ secretKey: <KEY> }` — the value is read from the key `<KEY>` of the well-known Kubernetes Secret **`otel-gateway-backend-credentials`** in the collector's (release) namespace. Keys must be valid environment-variable names (`[A-Za-z_][A-Za-z0-9_]*`), because the Secret is loaded onto the collector pod as environment variables. Create the Secret manually, with External Secrets Operator, sealed-secrets, or any equivalent — the Secret is optional, so the collector starts fine when no backend needs credentials.

```yaml
apiVersion: v1
kind: Secret
metadata:
  name: otel-gateway-backend-credentials
  namespace: <collector-namespace>
stringData:
  SIEM_AUTH_HEADER: "Bearer <token>"
```

### `sendingQueue`

Per-backend sizing of the exporter's in-memory queue: `{ queueSize, numConsumers }`. **`queueSize` counts batches, not bytes or records**, so its memory bound is `queueSize × the pipeline's batch cap`. The collector default (1000) is fine for the low-volume signals, but on the `requestlog` pipeline — where a batch can reach ~4 MiB — the default would bound roughly 4 GiB of queue memory, past the collector pod's 2 Gi limit. **Size the queue down on every `requestlog` subscriber**; `queueSize: 64` bounds about 256 MiB. The collector's memory limiter (75 % of the pod limit) is the backstop, but a deliberately sized queue fails cleanly (dropping the oldest data for one backend) instead of pressuring the whole collector.

## Worked example: request logs to a logging stack

The scenario from [Control request log data](/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/configuring-request-logs): full request-log records, bodies included, must land in a self-managed logging stack.

```yaml
global:
  observability:
    enabled: true
    signals:
      requestlog:
        enabled: true          # chart default
    backends:
      custom:
        - name: request-log-sink
          endpoint: otel-collector.logging.svc.cluster.local:4317
          signals: [requestlog]          # explicit — never implied
          tls: { insecure: true }        # in-cluster plaintext hop
          sendingQueue: { queueSize: 64 }
```

Applying this makes the chart render the collector's dedicated request-log lane (an OTLP/gRPC receiver on port 3501, raised message-size cap, plus an OTLP/HTTP receiver on 3502) and configure the gateway's data-plane module to push request logs to it over gRPC. Verify end to end:

1. Confirm the emission wiring and the lane exist:

   ```shell
   kubectl get envoyproxy -n <egress-namespace> -o yaml | grep -A1 OTEL_
   kubectl get cm otel-gateway-config -n <system-namespace> -o yaml | grep -A5 requestlog
   ```

1. Send a prompt through the gateway (the MANGO smoke test from the request-logs guide works well).
1. At the backend, expect one record per request: the body is the readable request-log JSON (`request_id`, model, token and cost fields, request and response bodies inline), record attributes include `event.type: request_log` plus `gen_ai.*` and `tars.*` dimensions, and the resource carries `log.type: request_log`.

Remove the `signals: [requestlog]` subscription and the whole lane — receiver, pipeline, and module emission — disappears together on the next upgrade.

## Worked example: guardrail triggers to a SIEM

Security wants every guardrail firing, full matched content included, in the corporate SIEM — while the management plane keeps only metadata (see the content-mode section below for why those two settings are independent).

```yaml
global:
  observability:
    enabled: true
    signals:
      guardrailtrigger:
        enabled: true          # chart default
    backends:
      custom:
        - name: siem
          protocol: otlphttp                       # SIEM sits behind an HTTP load balancer
          endpoint: https://otel.security.example.com/otlp
          signals: [guardrailtrigger]
          auth:
            headers:
              Authorization: { secretKey: SIEM_AUTH_HEADER }
```

Create the `otel-gateway-backend-credentials` Secret with the `SIEM_AUTH_HEADER` key as shown above, apply the values, then fire a deterministic rule — for example a `regexp` guardrail matching `\d{3}-\d{2}-\d{4}` (SSN-shaped) in enforce mode — and send a request through the gateway containing a matching string. One record per rule evaluation arrives at the SIEM; the record shape is specified in the next section.

## Guardrail-trigger destinations

### The collector copy is always full content

Guardrail triggers exist in two copies with different owners:

- **The management-plane copy** — what the dashboard's trigger views show — is scoped by the dashboard content-mode setting described below.
- **The collector fan-out copy** — what subscribed custom backends receive — is **always full content**. The content-mode setting does not touch it. `Off` stops the management-plane copy entirely; a subscribed backend still receives every trigger, matched messages included.

This is deliberate. The fan-out exists precisely so an organisation can hold full trigger content in its own systems while independently throttling what leaves for the management plane; coupling the two would make the strictest privacy setting silently break the customer's own security pipeline. The consequence for auditing is equally deliberate: **`backends.custom[]` subscriptions, not the dashboard content modes, are the authority on where full content goes.** The only ways to stop full triggers reaching a backend are to remove its `guardrailtrigger` subscription or disable the signal.

### The dashboard setting scopes only the management-plane copy

In the Admin Dashboard, **Settings → Guardrails**, the **Guardrail triggers** card sets "What guardrail trigger data gets sent to the management plane":

| Mode | Management-plane copy | Collector fan-out copy |
| :---- | :---- | :---- |
| **Full** | Matched prompt and output text, including sanitized messages, is stored on the management plane. | Full content |
| **Metadata only** | Scores, check types, actions, durations and ids are stored; matched message content is not sent to the management plane. | Full content |
| **Off** | No trigger records reach the management plane; dashboard trigger views are empty. | Full content |

The card itself says so: "Additional observability destinations are configured by your platform team via helm values" — that sentence points here.

### The record consumers see

Each rule evaluation produces **one OTLP log record**. A trigger in which three rules evaluated emits three records that share the trigger's `id` in the body, so a backend can treat each record as a self-contained per-rule event (the natural unit for filtering and alerting) or re-correlate the whole trigger on `id` without unpacking arrays.

**Body** — a camelCase JSON object:

- The trigger's common fields at the top level: `id`, `triggeredAt`, `guardrailId`, `consumerId`, `resourceId`, `executionDurationMs`, `metadata`, `customerId`, `projectId`.
- `requestId` — the originating request's `x-request-id` (omitted when the request carried none).
- A single `ruleEvaluation` object for this record's evaluation: `guardrailRuleId`, `score`, `action`, `mode`, `executedOn`, `checkType`, `providerId`, `executeOnContentType`, `executionDurationMs`, and the `inputResults` / `outputResults` arrays with `message` / `sanitized_message` content.

**Record attributes** — for filtering and aggregation without parsing the body:

| Attribute | Value |
| :---- | :---- |
| `event.type` | `guardrail_trigger` |
| `tars.request.id` | The request's `x-request-id` — the **same key stamped on request-log records**, so triggers and request logs join on one attribute across backends |
| `tars.customer.id`, `tars.project.id`, `tars.id`, `tars.user.id`, `tars.gateway.id`, `tars.dataplane.id` | The TARS identity context, using the same keys as the platform's guardrail metrics, so logs and metrics slice on the same dimensions (empty values omitted) |
| `tars.guardrail.id`, `tars.guardrail.rule.id`, `tars.consumer.id` | Guardrail, rule, and consumer identity |
| `gen_ai.provider.name`, `gen_ai.request.model` | Provider and model, per GenAI semantic conventions |
| `tars.guardrail.rule.action`, `tars.guardrail.rule.mode` | The rule's configured action and mode, as enum names (e.g. `RULE_ACTION_REDACT`) |
| `tars.guardrail.input.redacted`, `tars.guardrail.output.redacted` | Booleans, **derived rather than parroted from config**: true only when the action is REDACT, the mode actually enforced it, and the sanitized message is a real rewrite of the original |

**Severity** is `ERROR` exactly when the evaluation actually blocked the request or response (a BLOCK action in an enforcing mode); everything else — including a BLOCK rule in monitor mode — is `INFO`. Alert on severity, not on the action attribute, to page only on requests that were actually stopped.

**Resource attributes**: `log.type: guardrail_trigger`, `service.name: aidiscovery`. The record timestamp is the trigger's `triggeredAt`.

On the emitting side, trigger export rides a bounded in-memory queue (2 048 records, oldest dropped on overflow, batches of up to 512 flushed every second, 30 s export timeout): a slow or unreachable collector drops trigger copies after bounded retry and can never stall guardrail evaluation or management-plane reporting.

## Delivery semantics: what is and is not guaranteed

Fan-out delivery is engineered to protect the traffic path and the sibling backends, not to guarantee delivery to any single backend. Concretely, per hop:

| Hop | Guarantee | Where data can be lost |
| :---- | :---- | :---- |
| Emitting component → collector | OTLP request/response; the sender sees failures and retries within its own bounded buffer | Records emitted during a full collector outage are lost to the fan-out path (management-plane copies travel separately and are unaffected) |
| Collector receiver → per-backend queue | Non-blocking enqueue into a **sized, in-memory** queue | **Overflow drops** (counted in the collector's own metrics) rather than blocking the pipeline — a slow backend never backpressures ingest or the sibling backends; a collector crash loses whatever was queued |
| Per-backend exporter → backend | Retry with backoff: 5 s initial, 30 s max interval, up to 300 s total per batch | After the 300 s budget the batch is **dropped**. A single record larger than the receiver's message cap can never succeed and is dropped after the same budget — the fix is `protocol: otlphttp` (or raising the receiver's cap), not more retries |

The queues are in-memory by design: the collector is a stateless Deployment, and pretending to offer restart durability would be a promise the architecture cannot keep. If a backend must not miss records across collector restarts, put a collector or queue you operate (with its own persistence) close to the data plane as the subscribed backend, and let it forward onward.

Batching on the request-log pipeline is byte-aware: small batches (up to 8 records, ~4 MiB typical, flushed every 2 s) so that assembled batches stay under common gRPC frame caps on the receiving side.

## Frequently asked questions

### Can the dashboard's request-log or trigger content modes reduce what a custom backend receives?

No. Both content modes scope only the management-plane copy. The collector fan-out copies of `requestlog` and `guardrailtrigger` are always full content for every subscribed backend — that is the contract, and subscriptions are therefore the thing to review when auditing data flows.

### Why does my backend receive metrics and traces but no request logs?

Almost certainly the explicit-subscription rule: an omitted `signals` list means the four legacy signals only. Add `signals: [requestlog]` (plus whatever else the backend should receive — an explicit list replaces the default set, so name the legacy signals too if they should keep flowing).

### Does adding a backend change what the management plane receives?

No. Management-plane copies travel on their own pipelines and transports; `backends.custom[]` entries only add destinations.

### The collector logs show queue-overflow drops for one backend. What do I do?

That backend is slower than the signal's ingest rate. Raise its `sendingQueue.queueSize` (respecting the batch-size arithmetic above), raise `numConsumers`, or fix the backend's throughput. The drops are confined to that backend; siblings and the management-plane copies are unaffected.

Where to go next

  <Link to="/agent-router-enterprise/guides/operate-and-govern/cost-and-quota-management/configuring-request-logs" className="tare-nav-card">
    Control request log data
    The management-plane content modes, and the collector-routed path for shipping full request logs to your own backend.
  </Link>
  <Link to="/agent-router-enterprise/guides/operate-and-govern/safety-and-data-protection/protect-requests-with-guardrails" className="tare-nav-card">
    Protect requests with guardrails
    The guardrail model behind the trigger records this guide ships to your backends.
  </Link>