Audit a Coval evaluation

Inspect what the customer actually ran and what decision those results support. Stay read-only unless they separately request a concrete repair or test.

Collect the smallest useful evidence

Confirm organization, workspace, API environment, agent, time window and decision. Use explicit resource IDs where available. Start with a recent relevant run, its agent, persona, cases and metrics; sample completed, failed and unscored conversations. Don't download the whole account or launch a run to fill a gap.

Discover installed commands with coval --agent agent manifest and resource context. Common reads:

coval --agent runs get <run-id>
coval --agent agents get <agent-id>
coval --agent personas get <persona-id>
coval --agent test-sets get <test-set-id>
coval --agent test-cases list --test-set-id <test-set-id> --page-size 100
coval --agent simulated-conversations list --run-id <run-id> --page-size 100
coval --agent simulated-conversations get <simulation-id>
coval --agent simulated-conversations metrics <simulation-id>
coval --agent metrics get <metric-id>

Check ok, errors and pagination. Some CLI versions return only the first page without its token. For complete coverage, use the published /v1/openapi schemas and public API pagination (page_token from next_page_token), retaining the original filter and workspace header. Never infer missing resources from a failed read, a truncated page or an unsearched workspace. If evidence is bounded, state the inspected sample and leave completeness unknown.

Inspect six decision risks

Read actual transcript excerpts and metric explanations for consequential findings. For audio claims, listen to the relevant recording segments. For tool or retrieval claims, inspect correlated traces and returned evidence; missing instrumentation is an observability gap, not proof a tool never ran.

Numerical latency, counts and calibrated categorical metrics are legitimate. Do not force everything into binary judging. Prefer deterministic checks for exact constraints; require an explicit decision threshold when the result is used as a release gate. Safety-critical requirements can justify tests before any naturally occurring failure is observed.

Report

For each material finding, provide severity relative to the customer's decision, observed evidence and resource IDs, impact, smallest fix, and how to verify it. Mark each inspected area supported, problem observed, or unknown. Keep hypotheses separate from confirmed findings. Do not invent a numeric readiness score.

Finish with:

• What was inspected (counts, scope, window, versions and missing evidence).
• Whether the evidence supports the stated decision, and why.
• At most three next actions, ordered by expected learning per cost.

Use app URLs verified from the user's org/workspace context or returned by Coval. Do not expose credentials, signed recording URLs, private metadata or full customer transcripts in a shareable report. No resource mutations, notifications, scheduled monitoring or paid evaluations are part of this audit.