Hands-On evaluation scenarios
An Agent Router evaluation expressed as a sequence of hands-on scenarios. Each scenario follows one or more documentation guides, ends with a verifiable outcome, and receives a grade, so the evaluation produces evidence rather than impressions.
How to use these pages:
- Run the fast track first: it builds the organization, SSO, keys, traffic, teams, and budgets the graded sections assume, and pre-completes several scenarios (see the carry-over table).
- Then work through the graded sections in order; each section is a separate page and notes which evaluation categories it feeds.
- For each scenario, follow the linked guide, run the scenario, and check the verification.
- Grade the scenario Pass, Pass with gaps, or Fail using the rubric below, and note anything surprising (gaps, workarounds, documentation problems).
- Total each section and carry the results into the scorecard. The Installation and onboarding section applies to Self-Hosted Data Plane evaluations only; fully hosted evaluations (Service or Enterprise Fully Managed) start at First requests and developer onboarding. Scenarios marked (optional) depend on the evaluation scope.
For your convenience, this page can be downloaded as PDF or printed directly. Site navigation is omitted, the scoring table keeps its rows intact with the header repeated on every page, and the Grade and Notes columns leave room for handwriting.
Grading rubric
| Grade | Points | Definition |
|---|---|---|
| Pass | 2 | Scenario completed as documented; at most minor polish items. |
| Pass with gaps | 1 | Scenario completed, but the documentation was unclear or workarounds not in the documentation were needed. |
| Fail | 0 | Scenario could not be completed, or only partially; capability missing or non-functional. |
On a printed scoresheet, mark the Grade column P, G, or F; the points are tallied in each section's subtotal.
Prerequisites checklist
Before starting, confirm the following:
- Console access: sign-in works for the Agent Router Console of the evaluation environment
- Admin Dashboard access: sign-in works with the initial admin credentials
- At least one AI provider credential on hand (for example, an OpenAI API key)
- Terminal or HTTP client available (curl, Python, Postman)
- OpenID Connect (OIDC) identity provider (IdP) details: issuer URL, client ID, client secret, and an IdP administrator reachable during setup
For self-hosted data plane configurations:
- A Kubernetes cluster meeting the prerequisites, plus cloud CLI access
Optionally, depending on the evaluation scenario:
- Model Context Protocol (MCP) client installed (Claude Code, Cursor, or VS Code)
- OpenTelemetry-capable observability backend for telemetry export testing
Evaluation sections
The ungraded Fast-Track Evaluation runs first and covers the end-to-end first week, from first access to an onboarded team. The graded sections follow:
- Installation and onboarding (Self-Hosted Data Plane only)
- First requests and developer onboarding
- Routing, resilience, and provider abstraction
- MCP gateway
- Observability and audit
- Cost control and budgets
- Safety and data protection
- Identity and developer governance
- Models, providers, and projects
- Enterprise operations and compliance
Fast-track carry-over
Work done in the fast track counts here. Grades for fully covered scenarios can be claimed from fast-track evidence; partial rows list what remains.
| Fast-track job | Pre-completes | Coverage |
|---|---|---|
| D0.1 Onboarding wizard | 9.1 Add a provider and enable models | Partial: verify the disabled-model error separately |
| D0.2 Configure SSO | 8.1 Configure single sign-on | Full |
| D1.2 Key and agent | 2.1 Make a first API call; 2.5 Connect a coding agent | Full |
| D1.3 Analytics onboarding | 5.1 Monitor traffic and usage | Partial: locate a specific fast-track request with its provider, tokens, and latency |
| D1.6 Hello-world budget | 6.4 Exhaust a test budget | Partial: convert the warn-mode budget to a hard cap and exhaust it |
| W1.1 Invite users | 8.3 Onboard a developer and issue keys | Partial: key revocation still to test |
| W1.3 Teams and budgets | 6.3 Create a team budget | Full |
Summary scoring
Count only the scenarios in scope for the evaluation (skip Installation and onboarding for fully hosted paths; skip scenarios marked optional that were not attempted) and compute the percentage of the applicable maximum.
| Section | Feeds categories | Scenarios | Max | Score |
|---|---|---|---|---|
| Installation and onboarding | F, G | 6 | 12 | |
| First requests and developer onboarding | G2 | 5 | 10 | |
| Routing, resilience, and provider abstraction | D | 7 | 14 | |
| MCP gateway | E | 4 | 8 | |
| Observability and audit | C, A3 | 5 | 10 | |
| Cost control and budgets | D2, D3 | 9 | 18 | |
| Safety and data protection | A, B | 6 | 12 | |
| Identity and developer governance | F1 | 3 | 6 | |
| Models, providers, and projects | D1, G1 | 11 | 22 | |
| Enterprise operations and compliance | F2, F3, G1 | 7 | 14 | |
| Total | 63 | 126 |
| Percentage | Fit | Typical profile |
|---|---|---|
| 90--100% | Excellent fit | Nearly all scenarios pass clean; none fail |
| 70--89% | Strong fit | Most scenarios pass; some documentation gaps |
| 50--69% | Acceptable fit | Scenarios complete, but with gaps throughout or a few failures |
| below 50% | Not recommended | Failures on core scenarios |
A Fail on any non-optional scenario caps the outcome at Acceptable fit, regardless of the percentage.
Reporting issues
When a scenario fails or a guide does not match product behavior, record:
- Scenario ID and the guide followed
- Expected behavior per the documentation
- Actual behavior, with the exact error message or screenshot
- Environment: deployment path (Service, Fully Managed, Self-Hosted), browser, and OS
Share the record with the Tetrate contact for resolution, and grade the scenario on what was achieved, not on the promise.
Where to go next
Where to begin
Set up the evaluation environment: the role and path router for all three deployment paths.
Compliance
Residency, retention, and the due-diligence answers behind the Enterprise operations and compliance section.
Sizing and scale
Throughput, resource sizing, and the deployment footprint for load planning.