Observability and Audit
Feeds category C (usage observability and analytics) and A3 (audit trail).
Starting state from the fast track: several days of real traffic exist to search, from the evaluator and the onboarded teammates.
Grade each scenario Pass (2 points), Pass with gaps (1), or Fail (0) against the grading rubric, and carry the subtotal into the summary scorecard.
Scoresheet document
For your convenience, this page can be downloaded as PDF or printed directly. Site navigation is omitted, the scoring table keeps its rows intact with the header repeated on every page, and the Grade and Notes columns leave room for handwriting.
| ID | Scenario | Guide | Grade | Notes |
|---|---|---|---|---|
| 5.1 | Monitor traffic and usage in the Console: request logs, per-key usage, latency, and cost. Verify: a request made in the First requests section can be found with its provider, tokens, and latency. | Monitor traffic and usage | ||
| 5.2 | Export telemetry to the existing observability stack over OpenTelemetry. Verify: traces and metrics arrive in the backend with LLM attributes. | Export telemetry to an observability stack | ||
| 5.3 | Configure request logging modes and confirm what is captured per mode. Verify: switching modes changes the stored payload as documented. | Configuring request logs | ||
| 5.4 | Audit platform activity: locate the audit events for the admin changes made so far. Verify: model, provider, key, and SSO changes appear with actor and timestamp. | Audit platform activity | ||
| 5.5 | Export audit and policy decisions to a SIEM. Verify: a guardrail decision from the Safety and data protection section is queryable in the SIEM with its correlation ID. | Export audit and policy decisions to a SIEM | ||
| Subtotal | /10 |