Failure → Eval (the growth loop)

Steps 00–12 run roughly once per project. This one runs every week for the life of the product. It is the engine behind the single most important sentence in the framework — "every production failure becomes an eval case so the same mistake can't silently return" — and for a solo founder it's the highest-ROI skill in the set, because it's the one you'll actually run over and over.

The job: take one thing the agent got wrong and convert it into a permanent guard, so a future change can't quietly reintroduce it. Three outputs, in priority order: (1) a new eval case, (2) an optional guardrail update if it's a new kind of failure, and (3) a ledger line so the loop itself is auditable.

Conventions

Follow _conventions.md (read-first order, docs/ vs artifacts/, don't-clobber guard, the decisions ledger, and — especially here — the append-only rule for ledgers).

Before you start

Read PROJECT.md (esp. §4 success definition and the unacceptable-failure list), artifacts/evals/eval-cases.jsonl (the case format you'll append to), and artifacts/guardrails.md (the failure taxonomy and handling policy — the modes you'll classify against). The input is one concrete failure: a bad trace/run, a user complaint, or a near-miss the user describes. If there's no trace, reconstruct the minimal input that triggered it.

Steps

1. Triage — what actually went wrong, in one line. Name the failure precisely (not "it was bad" — "it recommended an action with no supporting evidence and the user almost approved it"). Distinguish the symptom from the cause where you can.
2. Classify against the failure taxonomy. Match it to an existing mode in artifacts/guardrails.md. Two outcomes:
   • Known mode → this is a new instance of a failure you already anticipated. Good — the guardrail exists; you just need the regression case (step 3).
   • New mode → the taxonomy didn't predict this. Flag it for a guardrail update (step 4). A new mode is the more valuable find; say so.
3. Reduce to a minimal eval case and append it. Strip the incident to the smallest input that reproduces the failure, then write a case in the same shape as evals/eval-cases.jsonl:
   • input — the concrete task/trace that triggered it (minimal, real).
   • expected — the correct behavior, OR a grading rubric.
   • check — deterministic (right tool? valid output? under budget? refused when it should?) or judgment-based (LLM-judge / human) — prefer deterministic where the failure allows it.
   • mode — offline (runs before every release) — a caught production failure almost always becomes an offline regression guard.
   • source — "incident:<id/date>" so the case traces back to why it exists. Append to artifacts/evals/eval-cases.jsonl (never rewrite the file). If it maps to an unacceptable-failure line in PRD §4, note that — those deserve deterministic checks.
4. If it's a new mode — propose a guardrail update (don't silently apply). Draft the row it adds to artifacts/guardrails.md: the failure mode + the handling policy (retry / fallback / escalate / refuse). Present it to the user; update guardrails.md only on their nod, then record the change in the decisions ledger (artifacts/decisions.md).
5. Log the loop. Append one line to artifacts/evals/failure-ledger.jsonl so the growth loop is itself auditable and you can see the failure rate trend over time:

   {"incident": "<id/date>", "trace": "<ref or 'described'>", "failure": "<one line>",
    "mode_status": "known | new", "eval_case_id": "<id>", "guardrail_changed": false,
    "timestamp": "<ISO-8601>"}

Feeds

This step is where agent-guardrails and agent-data-and-evals keep growing after launch. It closes the loop agent-monitoring opens: monitoring finds the bad runs, this step converts them into guards. The eval set that began in the PRD (§5 acceptance evals) and expanded in the data-and-evals step grows here, one production failure at a time, forever.

Output

• Append one case to artifacts/evals/eval-cases.jsonl (apply guard: append, never overwrite).
• Optionally, a proposed/approved row in artifacts/guardrails.md (+ a decisions.md line).
• Append one line to artifacts/evals/failure-ledger.jsonl.

End by stating: the one-line failure, whether it was a known or new mode (new modes are the ones worth celebrating — the taxonomy just got smarter), the id of the eval case you added, and whether a guardrail changed. If the same mode is showing up repeatedly in the ledger, say so — a recurring mode means the guardrail isn't holding and needs rethinking, not another eval case.