Incident RCA & Postmortem

"Human error" is where bad investigations stop and good ones start. A person could make that mistake because the system permitted it; the root cause is the missing guardrail, not the hand on the keyboard. Incidents are almost always a chain, never a single cause.

Reconstruct the timeline from evidence

1. Build a precise, sourced timeline: what happened, when, observed via what signal. Separate when it broke from when it was detected from when it was mitigated — the gaps between these are their own findings.
2. Trigger vs cause: the trigger (the deploy, the traffic spike) is rarely the root cause; it's the thing that exposed a latent weakness that was already there. Find the latent condition.
3. Contributing factors, not a single root: most incidents need multiple things to go wrong at once (Swiss-cheese model). Enumerate every layer of defense that should have caught it and didn't.

Dig past the first answer

• Use iterative "why" but branch, don't just descend: each "why" can have several answers; a linear five-whys finds one thread and misses the others.
• For each contributor ask: why did the system allow this, why wasn't it caught earlier, and why was the impact this large? Three independent axes: prevention, detection, blast radius.
• Distinguish proximate (the immediate mechanism) from systemic (the process/design/incentive that made it likely). Fixing only the proximate cause guarantees a similar incident later.

Adversarial validation

• The counterfactual test: for each proposed root cause, would fixing it actually have prevented this incident? If not, it's a contributing factor, not the root.
• Blame-detection: if the action item is "be more careful" or "add a reminder", the investigation stopped too early — those don't change the system and won't hold.
• Detection gap: why did it take so long to notice? An incident that took hours to detect has a monitoring root cause independent of the technical one.
• Recurrence check: has a similar incident happened before? If yes, the previous postmortem's action items failed — investigate that failure too.
• Hindsight bias guard: the responders acted on the information they had then, not what's obvious now. Judge decisions by what was knowable at the time.

Non-obvious risks

• Action items that are vague, unowned, or unprioritized never get done; the postmortem becomes theater and the incident recurs.
• Fixing the specific trigger (rollback the bad deploy) without the systemic cause (no canary, no automated rollback) just delays the next one.
• Over-correcting (heavyweight process after one incident) can cause more harm than the incident; match the fix to the actual risk.

Output structure

Deliver: the evidence-based timeline (broke / detected / mitigated), the trigger distinguished from the latent root cause, the contributing factors across prevention/detection/blast-radius, the counterfactual-validated root causes, and owned, prioritized, specific action items (each one a system change, not a plea for vigilance) — written blamelessly.