Build a useful Coval test suite

Create tests that change a product decision. A small set covering distinct failure mechanisms is more useful than many paraphrases of a happy path.

Understand and reuse

Confirm organization/workspace, agent and scope. Inspect the selected agent, existing cases, relevant recordings and the customer's source-of-truth policy. Use coval --agent agents get <id>, test-sets get, and test-cases list --test-set-id <id>. Check command help, errors and pagination. Do not replace an existing suite or derive requirements solely from a buggy agent prompt.

For a large corpus, distill-test-set can help shortlist source rows; inspect its resulting input/expectation pairs. For unexplored conversations, use coval-discover-failures. Neither is mandatory when the requirement is known.

Design the coverage before prose

Make a compact matrix with: requirement or observed failure, source, meaningful variation, scenario, expected behaviors, evidence needed, and risk/priority. Distinguish observed failure, policy requirement and synthetic hypothesis.

Choose axes that could alter behavior: ambiguous request, changed information, missing tool result, unsupported request, caller correction, or a relevant audio condition. Avoid an automatic Cartesian product. Generate combinations first, then realistic caller scenarios; discard impossible combinations and semantic duplicates. Preserve a few untouched cases for later regression checks.

For voice, separate scenario from persona and audio condition. Start with normal audio; vary accent/noise/interruption only for a stated hypothesis. Don't put sensitive real customer details in synthetic cases. Keep examples fictional and realistic for the customer's supported language and workflow.

Example for a clinic whose published hours are Monday–Friday, 8am–5pm:

This verifies information, not that an appointment was persisted.

Make expectations evaluable

Write each mandatory behavior as a separate array item. State the relevant precondition and observable outcome. Avoid “helpful,” arbitrary numeric scores, or requiring a particular phrase when a correct paraphrase is acceptable.

Don't leak the answer through input_str or persona instructions. A scenario sets the caller's goal, not the agent's response. Choose the input mode from the current test-case spec: SCENARIO for flexible interaction; SCRIPT only when exact caller turns matter. Do not call generated dialogue real production evidence.

Use transcript evidence for conversation behavior, audio for acoustic behavior, and correlated tool results for actual side effects. If the evidence cannot support a criterion, revise it or flag the instrumentation gap before scoring.

Before configuration writes, inspect the repository for Terraform ownership. If these resources are managed as code, use the installed coval-evals-as-code skill or the repository's declared workflow to edit and plan the change; do not bypass it with the direct CLI creates/updates below. Read-only inspection and design guidance still apply. If the skill is absent, continue from the existing Terraform configuration and public provider docs rather than assuming a sibling skill path.

Create and read back

Prepare the exact cases locally before asking for any missing write authority. Respect creation already authorized in the conversation. Use structured JSON rather than interpolating customer prose into shell strings:

coval --agent test-sets create --input-json @test-set.json
coval --agent test-cases create --input-json @case.json

test-set.json uses the public API shape:

{"display_name":"Clinic information checks","test_set_type":"SCENARIO"}

After obtaining the actual test-set ID, case.json is:

{
  "test_set_id":"<returned-id>",
  "input_str":"Ask when the clinic closes on Friday. After the answer, thank them and end the conversation.",
  "expected_behaviors":["The assistant states that the clinic closes at 5pm on Friday."],
  "description":"Friday closing time; source: office policy",
  "input_type":"SCENARIO"
}

Check coval test-cases create --help and the current test-cases schema before creation. A CLI lacking structured input can use the documented public API. Check every response before creating the next item; on ambiguous timeouts inspect existing resources before retrying to avoid duplicates. Read every created case back and verify its input and individual expected behaviors. Confirm the count using pagination when necessary.

Handoff

Show the coverage matrix, IDs and untested gaps. Keep source distribution separate from intentionally overweighted failure cases. Creating a suite does not authorize running it. Propose an explicit small subset for quick-eval, with counts including iterations, personas and base + mutations. Use configure-metrics to turn appropriate criteria into a provisional measure.