Agent Data & Evals

Two jobs, tightly linked: define the data the product stores (including the traces that make the agent observable) and build the eval set that defines what "good" means. For an agent product this is the step that most determines whether you can improve the product at all — without traces you can't debug, and without evals you can't tell if a change helped or silently broke something.

The eval set does not start from scratch here. The PRD (§5) already carries an acceptance eval for every core requirement — that is the seed. This step expands that seed into the full runnable harness. If the PRD's acceptance evals and the set you build here have drifted apart, that's a signal something changed silently between PRD and build; reconcile it.

Conventions

Follow _conventions.md (read-first order, docs/ vs artifacts/, don't-clobber guard).

Before you start

Read PROJECT.md, artifacts/prd.md, artifacts/prd-review.md, artifacts/system-design.md, and artifacts/tech-spec.md. The PRD's §5 acceptance evals and §6 eval strategy, the system design's observability layer, and any findings the review gate flagged (especially overridden ones — those are known risks that deserve eval coverage) are the direct inputs here.

Part A — Data & event logging

Define:

1. Entities — what the product stores, derived from the tech-spec data model. What must be stored vs. what can be derived (don't store what you can compute).
2. Events to capture from day one — this is the thing founders most regret skipping: you can't analyze data you never logged. List the events that later become your analytics and your eval material (runs started/completed, tool calls, failures, user corrections, escape-hatch activations).
3. The trace record — the concrete shape of one agent run's trace (from the system design's observability layer): run_id, inputs, each step's reasoning + tool call + result, final output, tokens, latency, cost, outcome.
4. Retention & privacy — per PROJECT.md: PII handling, how long traces live, anything regulated.

Part B — The eval set (expand the PRD's acceptance evals)

The eval set is the product's test suite. Start from the PRD's §5 acceptance evals — lift each one directly — then grow the set until it's a real, runnable starter (even 20–50 hand-written cases catches most regressions).

Build in this order so nothing is invented that the PRD already specified:

1. Lift every acceptance eval from PRD §5. Each core requirement's case becomes a case here, in runnable form. If any requirement lost its eval, that's a gap — flag it.
2. Add coverage for the review gate's findings. Every overridden finding in artifacts/prd-review.md is a known, accepted risk — write an eval that will catch it if it bites. This is how the override trail feeds back into quality.
3. Add the unacceptable-failure cases. There should be an eval for each hard line from PRD §4 the agent must never cross, especially for state-changing actions.
4. Fill remaining gaps — common inputs, edge cases, the boring-but-frequent paths.

For each eval case define:

• The input (a real task the agent should handle).
• The expected outcome OR a grading rubric.
• The check type:
  • Deterministic — did it call the right tool? return valid output? stay under budget? These are cheap and should cover as much as possible.
  • Judgment-based — was the reasoning sound? the tone right? the action appropriate? These need an LLM-as-judge or human grading. Note which.
• Whether it's an offline eval (run before shipping a change) or an online one (sampled from real traffic).
• The source: which PRD requirement / gate finding / failure mode it traces back to.

Use assets/eval-cases.template.jsonl as the starting format.

The growth rule (state this to the user)

Every time production surfaces a new failure, it becomes a new eval case so the same mistake can't silently return. The eval set is a permanent, growing asset — treat it like code. It began in the PRD, expands here, and keeps growing from production forever. The mechanism for that growth is a dedicated step: agent-failure-to-eval is what you run each time a real failure shows up, turning it into a case here plus (if it's a new failure mode) a guardrail update. This step builds the starter set; the failure-to-eval step grows it for the life of the product.

Output

• Write artifacts/data-model.md (entities, events, trace record, retention) — apply guard.
• Write artifacts/evals/eval-cases.jsonl (the starter set) — apply guard.
• Write artifacts/evals/README.md explaining how to run/grow them.

End by stating how many eval cases you created, how many were lifted from the PRD vs. newly added, which unacceptable failure modes and overridden gate findings are covered, and — honestly — which important behaviors are NOT yet covered and should be added.