Custom observability backends

Enterprise Tier

The data plane runs its own OpenTelemetry collector, and everything the management plane learns about traffic passes through infrastructure Tetrate operates. Many organisations need a second copy — the same telemetry, delivered to systems they run themselves: a SIEM that must see every guardrail firing, a logging stack that must hold full request content for compliance, a metrics platform that already drives the on-call rotation. The global.observability.backends.custom[] Helm values list is the single mechanism for all of it. Each entry attaches one OTLP destination to the data-plane collector and subscribes it to a chosen subset of signals; the collector then fans each signal out to every subscribed backend, independently of what does or does not travel to the management plane. Configuration is deliberately GitOps-only — adding a backend is a reviewed infrastructure change by the cluster owner, not a dashboard click — because two of the signals carry full prompt, response, and matched-content data.

---

Persona: Platform operator who owns the data-plane Helm values (usually through a GitOps flow), typically in coordination with security or compliance stakeholders for the content-bearing signals.

Estimated time: 15--30 minutes to add and verify a backend; longer if the receiving side (network reachability, credentials) still has to be arranged.

When this guide applies

Situation	What a custom backend adds
A SIEM or logging stack must receive every guardrail trigger with full matched content	Subscribe a backend to guardrailtrigger; the copy is always full content, regardless of the dashboard content-mode setting
Full request/response bodies must land in a self-managed store, possibly while the management plane receives less	Subscribe a backend to requestlog
Data-plane metrics, traces, or logs should also feed an existing observability platform	Subscribe a backend to any of the four legacy signals
One signal must reach several destinations	Add several entries; every subscribed backend receives its own copy

Two adjacent surfaces are not this mechanism: trace export configured from the Developer Console (see Export telemetry to observability) and the management-plane request-log modes set in the Admin Dashboard (see Control request log data). Custom backends are configured in Helm values only; there is no dashboard or API surface for them.

Outcomes

By the end of this guide:

• Every backends.custom[] field is understood, including the explicit-subscription rule for the content-bearing signals.
• A backend is subscribed to requestlog and/or guardrailtrigger and verified receiving records.
• The delivery guarantees — sized in-memory queues, drop-on-overflow, bounded retry — are understood well enough to size the queue and pick a protocol deliberately.
• The relationship between the always-full collector copy of guardrail triggers and the dashboard's management-plane content-mode setting is clear.

How the fan-out works

The data-plane cluster runs a local OpenTelemetry collector (otel-gateway, a stateless two-replica Deployment in the system namespace). Data-plane components push their telemetry to this collector, and the collector forwards it: management-plane-bound copies travel on their own pipelines, and each backends.custom[] entry adds one more OTLP exporter that joins the pipelines of the signals it subscribes to. The management-plane copies are unchanged by any backend you add.

Six signals exist, in two groups with different subscription semantics:

Signal	Content	Subscription
metrics	Data-plane metrics	Default set
traces	Request traces	Default set
logs	Application logs of the data-plane components	Default set
accesslog	Envoy access logs (requires the access-log lane to be enabled with an endpoint: global.observability.accessLog)	Default set
requestlog	Full request-log records, including request and response bodies	Explicit only
guardrailtrigger	Full guardrail-trigger records, including matched and sanitized message content	Explicit only

The explicit-subscription rule: a backend whose signals list is omitted or empty receives the four legacy signals — metrics, traces, logs, accesslog — and never requestlog or guardrailtrigger. The content-bearing signals only flow to a backend that names them, for example signals: [requestlog]. This is a deliberate privacy guardrail: upgrading a chart, or adding a catch-all backend, can never silently start shipping full prompt and response content somewhere new.

For the two content-bearing signals, the chart wires the entire path from one predicate — global.observability.enabled AND signals.<signal>.enabled AND at least one subscribed backend. When it holds, the collector renders a dedicated receiver and pipeline for the signal and the emitting component is configured (via chart-injected environment variables on the egress pod) to push to it; when it does not hold, both disappear together and nothing is emitted. There is nothing to enable by hand on the emitting side — subscribing a backend is the switch.

Each signal that a backend does not subscribe to, and each signal whose signals.<s>.enabled toggle is off, simply does not flow to it; a backend subscribed to a disabled signal keeps its exporter but receives nothing until the signal is enabled.

The backends.custom[] fields

A complete entry, with every field:

global:
  observability:
    backends:
      custom:
        - name: siem                    # required
          endpoint: otel.security.example.com:4317   # required
          protocol: otlp                # otlp (default) | otlphttp
          signals: [guardrailtrigger]   # omitted/empty = metrics, traces, logs, accesslog
          compression: none             # omitted = gzip
          tls:
            insecure: false
            insecureSkipVerify: false
          auth:
            headers:
              Authorization: { secretKey: SIEM_AUTH_HEADER }   # from the credentials Secret
              X-Scope-OrgID: { value: acme }                   # non-secret literal
          sendingQueue: { queueSize: 64, numConsumers: 4 }

name (required)

Lowercase letters, digits, and hyphens ([a-z0-9-]). The name becomes part of the collector exporter's identity (otlp_grpc/custom-<name> or otlphttp/custom-<name>), which is how the backend appears in the collector's own logs and metrics — pick something that will make sense in a 3 a.m. dashboard.

endpoint (required)

Where the backend listens. The expected shape depends on protocol:

• protocol: otlp (OTLP/gRPC): host and port, for example otel-collector.observability.svc.cluster.local:4317.
• protocol: otlphttp (OTLP/HTTP): a full http(s):// URL.

protocol

otlp (OTLP/gRPC, the default) or otlphttp (OTLP/HTTP). Choose otlphttp in two situations:

1. Oversized single records. gRPC receivers commonly cap messages at 4 MiB by default, and a single record larger than the receiving side's cap can never be delivered over gRPC — it is retried for the full retry budget and then dropped. Request-log records for large prompts and responses can reach low-single-digit MiB, so a requestlog subscriber whose records may approach its receiver's cap should either raise the cap on its receiver or subscribe over otlphttp, which has no 4 MiB framing default.
2. Intermediate infrastructure. Load balancers and proxies between the data plane and the backend that do not pass gRPC through cleanly. OTLP/HTTP traverses ordinary HTTP infrastructure.

signals

The subset of signals to ship, from the six in the table above. Omitted or empty means the four legacy signals and never the content-bearing two — see the explicit-subscription rule.

compression

Omit for the collector default (gzip). Set compression: none for backends that do not accept gzip-compressed OTLP.

tls

tls.insecure: true disables TLS entirely (plaintext — in-cluster hops only). tls.insecureSkipVerify: true keeps TLS but skips certificate verification. Leave both unset for full TLS verification against system CAs.

auth.headers

Headers attached to every export request. Each header takes one of two value forms:

• { value: <literal> } — a non-secret literal, rendered into the collector config as-is.
• { secretKey: <KEY> } — the value is read from the key <KEY> of the well-known Kubernetes Secret otel-gateway-backend-credentials in the collector's (release) namespace. Keys must be valid environment-variable names ([A-Za-z_][A-Za-z0-9_]*), because the Secret is loaded onto the collector pod as environment variables. Create the Secret manually, with External Secrets Operator, sealed-secrets, or any equivalent — the Secret is optional, so the collector starts fine when no backend needs credentials.

apiVersion: v1
kind: Secret
metadata:
  name: otel-gateway-backend-credentials
  namespace: <collector-namespace>
stringData:
  SIEM_AUTH_HEADER: "Bearer <token>"

sendingQueue

Per-backend sizing of the exporter's in-memory queue: { queueSize, numConsumers }. queueSize counts batches, not bytes or records, so its memory bound is queueSize × the pipeline's batch cap. The collector default (1000) is fine for the low-volume signals, but on the requestlog pipeline — where a batch can reach ~4 MiB — the default would bound roughly 4 GiB of queue memory, past the collector pod's 2 Gi limit. Size the queue down on every requestlog subscriber; queueSize: 64 bounds about 256 MiB. The collector's memory limiter (75 % of the pod limit) is the backstop, but a deliberately sized queue fails cleanly (dropping the oldest data for one backend) instead of pressuring the whole collector.

Worked example: request logs to a logging stack

The scenario from Control request log data: full request-log records, bodies included, must land in a self-managed logging stack.

global:
  observability:
    enabled: true
    signals:
      requestlog:
        enabled: true          # chart default
    backends:
      custom:
        - name: request-log-sink
          endpoint: otel-collector.logging.svc.cluster.local:4317
          signals: [requestlog]          # explicit — never implied
          tls: { insecure: true }        # in-cluster plaintext hop
          sendingQueue: { queueSize: 64 }

Applying this makes the chart render the collector's dedicated request-log lane (an OTLP/gRPC receiver on port 3501, raised message-size cap, plus an OTLP/HTTP receiver on 3502) and configure the gateway's data-plane module to push request logs to it over gRPC. Verify end to end:

1. Confirm the emission wiring and the lane exist:

   kubectl get envoyproxy -n <egress-namespace> -o yaml | grep -A1 OTEL_
   kubectl get cm otel-gateway-config -n <system-namespace> -o yaml | grep -A5 requestlog

2. Send a prompt through the gateway (the MANGO smoke test from the request-logs guide works well).

3. At the backend, expect one record per request: the body is the readable request-log JSON (request_id, model, token and cost fields, request and response bodies inline), record attributes include event.type: request_log plus gen_ai.* and tars.* dimensions, and the resource carries log.type: request_log.

Remove the signals: [requestlog] subscription and the whole lane — receiver, pipeline, and module emission — disappears together on the next upgrade.

Worked example: guardrail triggers to a SIEM

Security wants every guardrail firing, full matched content included, in the corporate SIEM — while the management plane keeps only metadata (see the content-mode section below for why those two settings are independent).

global:
  observability:
    enabled: true
    signals:
      guardrailtrigger:
        enabled: true          # chart default
    backends:
      custom:
        - name: siem
          protocol: otlphttp                       # SIEM sits behind an HTTP load balancer
          endpoint: https://otel.security.example.com/otlp
          signals: [guardrailtrigger]
          auth:
            headers:
              Authorization: { secretKey: SIEM_AUTH_HEADER }

Create the otel-gateway-backend-credentials Secret with the SIEM_AUTH_HEADER key as shown above, apply the values, then fire a deterministic rule — for example a regexp guardrail matching \d{3}-\d{2}-\d{4} (SSN-shaped) in enforce mode — and send a request through the gateway containing a matching string. One record per rule evaluation arrives at the SIEM; the record shape is specified in the next section.

Guardrail-trigger destinations

The collector copy is always full content

Guardrail triggers exist in two copies with different owners:

• The management-plane copy — what the dashboard's trigger views show — is scoped by the dashboard content-mode setting described below.
• The collector fan-out copy — what subscribed custom backends receive — is always full content. The content-mode setting does not touch it. Off stops the management-plane copy entirely; a subscribed backend still receives every trigger, matched messages included.

This is deliberate. The fan-out exists precisely so an organisation can hold full trigger content in its own systems while independently throttling what leaves for the management plane; coupling the two would make the strictest privacy setting silently break the customer's own security pipeline. The consequence for auditing is equally deliberate: backends.custom[] subscriptions, not the dashboard content modes, are the authority on where full content goes. The only ways to stop full triggers reaching a backend are to remove its guardrailtrigger subscription or disable the signal.

The dashboard setting scopes only the management-plane copy

In the Admin Dashboard, Settings → Guardrails, the Guardrail triggers card sets "What guardrail trigger data gets sent to the management plane":

Mode	Management-plane copy	Collector fan-out copy
Full	Matched prompt and output text, including sanitized messages, is stored on the management plane.	Full content
Metadata only	Scores, check types, actions, durations and ids are stored; matched message content is not sent to the management plane.	Full content
Off	No trigger records reach the management plane; dashboard trigger views are empty.	Full content

The card itself says so: "Additional observability destinations are configured by your platform team via helm values" — that sentence points here.

The record consumers see

Each rule evaluation produces one OTLP log record. A trigger in which three rules evaluated emits three records that share the trigger's id in the body, so a backend can treat each record as a self-contained per-rule event (the natural unit for filtering and alerting) or re-correlate the whole trigger on id without unpacking arrays.

Body — a camelCase JSON object:

• The trigger's common fields at the top level: id, triggeredAt, guardrailId, consumerId, resourceId, executionDurationMs, metadata, customerId, projectId.
• requestId — the originating request's x-request-id (omitted when the request carried none).
• A single ruleEvaluation object for this record's evaluation: guardrailRuleId, score, action, mode, executedOn, checkType, providerId, executeOnContentType, executionDurationMs, and the inputResults / outputResults arrays with message / sanitized_message content.

Record attributes — for filtering and aggregation without parsing the body:

Attribute	Value
event.type	guardrail_trigger
tars.request.id	The request's x-request-id — the same key stamped on request-log records, so triggers and request logs join on one attribute across backends
tars.customer.id, tars.project.id, tars.id, tars.user.id, tars.gateway.id, tars.dataplane.id	The TARS identity context, using the same keys as the platform's guardrail metrics, so logs and metrics slice on the same dimensions (empty values omitted)
tars.guardrail.id, tars.guardrail.rule.id, tars.consumer.id	Guardrail, rule, and consumer identity
gen_ai.provider.name, gen_ai.request.model	Provider and model, per GenAI semantic conventions
tars.guardrail.rule.action, tars.guardrail.rule.mode	The rule's configured action and mode, as enum names (e.g. RULE_ACTION_REDACT)
tars.guardrail.input.redacted, tars.guardrail.output.redacted	Booleans, derived rather than parroted from config: true only when the action is REDACT, the mode actually enforced it, and the sanitized message is a real rewrite of the original

Severity is ERROR exactly when the evaluation actually blocked the request or response (a BLOCK action in an enforcing mode); everything else — including a BLOCK rule in monitor mode — is INFO. Alert on severity, not on the action attribute, to page only on requests that were actually stopped.

Resource attributes: log.type: guardrail_trigger, service.name: aidiscovery. The record timestamp is the trigger's triggeredAt.

On the emitting side, trigger export rides a bounded in-memory queue (2 048 records, oldest dropped on overflow, batches of up to 512 flushed every second, 30 s export timeout): a slow or unreachable collector drops trigger copies after bounded retry and can never stall guardrail evaluation or management-plane reporting.

Delivery semantics: what is and is not guaranteed

Fan-out delivery is engineered to protect the traffic path and the sibling backends, not to guarantee delivery to any single backend. Concretely, per hop:

Hop	Guarantee	Where data can be lost
Emitting component → collector	OTLP request/response; the sender sees failures and retries within its own bounded buffer	Records emitted during a full collector outage are lost to the fan-out path (management-plane copies travel separately and are unaffected)
Collector receiver → per-backend queue	Non-blocking enqueue into a sized, in-memory queue	Overflow drops (counted in the collector's own metrics) rather than blocking the pipeline — a slow backend never backpressures ingest or the sibling backends; a collector crash loses whatever was queued
Per-backend exporter → backend	Retry with backoff: 5 s initial, 30 s max interval, up to 300 s total per batch	After the 300 s budget the batch is dropped. A single record larger than the receiver's message cap can never succeed and is dropped after the same budget — the fix is protocol: otlphttp (or raising the receiver's cap), not more retries

The queues are in-memory by design: the collector is a stateless Deployment, and pretending to offer restart durability would be a promise the architecture cannot keep. If a backend must not miss records across collector restarts, put a collector or queue you operate (with its own persistence) close to the data plane as the subscribed backend, and let it forward onward.

Batching on the request-log pipeline is byte-aware: small batches (up to 8 records, ~4 MiB typical, flushed every 2 s) so that assembled batches stay under common gRPC frame caps on the receiving side.

Frequently asked questions

Can the dashboard's request-log or trigger content modes reduce what a custom backend receives?

No. Both content modes scope only the management-plane copy. The collector fan-out copies of requestlog and guardrailtrigger are always full content for every subscribed backend — that is the contract, and subscriptions are therefore the thing to review when auditing data flows.

Why does my backend receive metrics and traces but no request logs?

Almost certainly the explicit-subscription rule: an omitted signals list means the four legacy signals only. Add signals: [requestlog] (plus whatever else the backend should receive — an explicit list replaces the default set, so name the legacy signals too if they should keep flowing).

Does adding a backend change what the management plane receives?

No. Management-plane copies travel on their own pipelines and transports; backends.custom[] entries only add destinations.

The collector logs show queue-overflow drops for one backend. What do I do?

That backend is slower than the signal's ingest rate. Raise its sendingQueue.queueSize (respecting the batch-size arithmetic above), raise numConsumers, or fix the backend's throughput. The drops are confined to that backend; siblings and the management-plane copies are unaffected.

---

Where to go next

Control request log data

The management-plane content modes, and the collector-routed path for shipping full request logs to your own backend.

Protect requests with guardrails

The guardrail model behind the trigger records this guide ships to your backends.