📮 dead-letter — Failed Task Triage & Capture

Capture a failed or blocked task before context clears. Categorizes failure modes into a 9-part taxonomy, distinguishes transient from permanent failures, verifies actual state before any retry, preserves partial work and orphaned resources, and generates either a bounded autonomous retry prompt or an escalation decision point. Recovery follows a strict ordered sequence — classify → decide → precondition → fix → verify-against-baseline → close — tracked step-by-step in the record so a retry resumes exactly where recovery stopped. At close-out, deterministic failures can be converted into a repro test pack: the capture that documents a failure becomes the regression test that prevents its recurrence. The status sweep clusters open records by root cause — repeated failures across different tasks surface as one systemic finding with one fix, not a storm of individual retries.

Modes

Every invocation resolves to exactly one mode. Load only the matched reference:

When to Use

• An agent returns "I can't proceed", "blocked on", "error:", or a partial result.
• After a long-running subagent batch or background command fails.
• Before reporting an unresolvable failure to the user or orchestrator.
• When an in-progress task must be abandoned and resumable later.
• When sweeping open dead-letter records (dead-letter status) — past-SLA or high-retry records need triage.
• When a bug or blocker is worth handing to Dev/QA as a reproduction case (repro pack).
• On a scheduled sweep (nightly failure triage) — open records get clustered by root cause so systemic issues surface once, not per-task.

Quick Reference

Procedure

Step 1: Ingest Failure Signals

Read full error logs, terminal stderr, or subagent exit messages.

Step 2: Classify Failure Mode

Assign exact code from the 9-part taxonomy.

Step 3: Recovery Decision & Circuit Breaker Check

Before authoring any retry packet, verify against references/circuit-breaker.md:

1. Circuit Breaker Status: If the same (code, root-cause) failed $\ge 2$ times on this task or blast radius $\ge 3$, the circuit is TRIPPED. Immediately freeze execution, generate an incident packet at .agents/artifacts/incident-<task-id>-<timestamp>.md, and escalate. Do not author a retry packet.
2. Transient or permanent? Transient (crash, network blip, rate limit): retry is legitimate. Permanent (defective logic, missing capability): a retry packet is forbidden — escalate or re-scope.
3. Autonomy level: auto (mechanical fix — diagnose → fix → re-run up to 3 verification rounds without asking), confirm (recovery path obvious but non-mechanical — one approval question), escalate (ambiguous root cause, tripped circuit, or repeated failure — specific decision question, never open-ended).

Step 4: State Verification & Precondition Delta Gate

Never retry blindly. A "failed" state can be mid-backoff, mid-poll, or already partially applied.

1. Emit one Precondition Check — a command or query confirming the system is in the retryable state.
2. Require an explicit Precondition Delta: What concretely changed in the code or environment? (e.g., file edited, env var exported). If Precondition Delta is empty/none, RETRY IS STRICTLY FORBIDDEN — escalate immediately to prevent thrashing.

Step 5: Write Dead-Letter Record (Atomic & Sanitized)

Save record to .agents/dead-letter-<timestamp>.md using the canonical template in references/record-schema.md:

• Zero-Credential Sanitization: Pass all captured stderr, stdout, and error dumps through the redaction filter (Bearer [REDACTED], sk-[REDACTED], postgres://...). Never write raw secrets to disk.
• Atomic Persistence: Write initially to .agents/artifacts/.tmp-dl-* and atomically rename to prevent corruption under concurrent subagent crashes.
• The record carries a Recovery Sequence — the ordered checklist (classify → decide → precondition → fix → verify-against-baseline → close) is the single resume point for any retry: completed steps are marked, and a retry resumes from the first unchecked step, never from an arbitrary point. Load the schema reference only when authoring a record.

Step 6: Emit Inline Summary

Output failure code and immediate next step to terminal.

Status Sweep & Cluster Triage (dead-letter status)

On request (or on a scheduled sweep — nightly failure triage), run the three-pass sweep (inventory → cluster by root cause → verdict per cluster) per references/sweep-protocol.md. Load that reference only when sweeping — capture and retry do not need it. Invariant: clusters form on root cause, not error-string similarity; verdicts are systemic / coincidental / cascade / escalate-cluster; a sweep that ends with retry authorization for a systemic cluster has failed. Recurring sweeps consult the findings ledger (.agents/artifacts/dead-letter-ledger.md) to deduplicate verdicts and measure convergence.

Repro Test Pack (close-out conversion)

A dead letter is high-signal evidence; a closed deterministic failure shouldn't just be archived — it should be converted into the regression test that prevents recurrence. At close-out (Recovery Sequence step 6), when the failure is deterministic (permanent-class: FAILED-LOGIC, FAILED-TOOL, or a BLOCKED-* with a stable local trigger) and the environment permits writing test files, generate a repro pack.

When NOT to generate (skip the pack, close the record plainly):

• Transient or environment-dependent failures (network blips, rate limits) — a flaky repro is worse than none.
• The fix is already verified against the Baseline Reference and shipped — the test suite already covers it.
• The project has no test runner or the failure needs external systems (real APIs, prod data) — record the manual steps instead.

Where: .agents/artifacts/repro-<slug>-<timestamp>/ (working-artifacts rule: never the tracked tree; the record links the pack path).

What the pack contains (REPRO.md + one failing test when feasible):

The pack is a handoff artifact: Dev/QA (or a fresh agent) runs the test red → fixes → runs green, without re-deriving the failure from logs. The dead-letter record links the pack; on recurrence of the same root cause, the pack is the parent record's evidence attachment — the regression test makes the parent record's retry packet verifiable.

Pitfalls

• Generic Error Codes: Never classify as "Unknown Error"; pinpoint the exact blocker code.
• Discarding Partial Work: Always list and preserve partial code files created before failure.
• Open-Ended Escalations: Every escalation must include a clear, binary or multiple-choice decision question.
• Blind Retries: Retrying without the precondition check — a mid-backoff or half-applied state turns a clean failure into a corrupt one.
• Permanent-Failure Retry Loops: Permanent-class failures (logic, capability gaps) get no retry packet; the third occurrence of the same root cause is always an escalation.
• Leaked Resources: Closing a record without enumerating orphaned PRs, branches, or processes — cleanup commands go in the record, not in memory.
• Improvised Recovery Order: Resuming a failed recovery from an arbitrary step — the record's Recovery Sequence is the single resume point; skip nothing, reorder nothing.
• Clean-Exit Regression: Declaring success on a green exit code without comparing against the Baseline Reference — a fix that reintroduces old breakage must fail the verification round.
• Unseen Red: Shipping a repro pack whose test was never observed failing — a red you have not seen is a claim; run the test against the un-fixed state and record the observed failure output in the pack.
• String-Match Clustering: Grouping records by similar error text instead of root cause — identical stack traces can hide different causes, and different messages can share one. Read What-Was-Learned for the causal evidence before clustering.
• Retry Storm on Systemic Clusters: Authorizing per-record retries for records sharing one root cause — the sweep's purpose is one fix covering all members; retrying members individually is the failure the sweep exists to prevent.

Verification

• Confirm .agents/dead-letter-<timestamp>.md exists.
• Confirm either a concrete retry prompt OR a routed escalation question is fully specified.
• Confirm the Recovery Decision is filled (seen-before count, transient/permanent, autonomy level, circuit breaker status).
• Confirm the Precondition Check ran and its observed result is recorded.
• Confirm Precondition Delta is documented; if empty/none, confirm retry was forbidden.
• Confirm Zero-Credential Sanitization passed (no raw tokens, keys, or passwords in stack dumps).
• Confirm Orphaned Resources is enumerated (or "None.") — no silent leaks.
• Confirm the Recovery Sequence tracks completed steps and a retry resumes from the first unchecked one.
• Confirm a Baseline Reference exists when last-known-good state is available, and the verification loop compares against it — not just exit codes.
• For deterministic failures closed with a pack: confirm REPRO.md exists at the linked path, the test was observed red (output recorded), and the spec note states which document is the source of truth.
• For a status sweep: confirm clusters were derived from root-cause evidence (What-Was-Learned, precondition results) — not error-string similarity — and the report ends with a sweep action line; systemic clusters carry one fix/escalation, not per-record retries.