Postmortem Analysis

Turn an incident into changes that prevent the next one, without turning it into a search for someone to blame.

Purpose

Use this when:

• An incident is resolved and needs a write-up.
• Reviewing someone's postmortem draft.
• A similar incident has recurred — which means the previous postmortem did not work.
• A near miss occurred. These are the cheapest learning opportunities available and the most commonly skipped.

Do not use this when:

• The incident is ongoing — use incident-triage. Writing up an unresolved incident produces a fiction.
• The cause is not yet understood — use production-debugging first. A postmortem with a guessed cause generates action items that fix nothing.

What this skill assumes: the incident timeline, the telemetry, and access to the people involved. A postmortem written from logs alone misses the decision-making, which is usually where the useful findings are.

Operating Procedure

1. Write the timeline first, from evidence. Timestamps, what happened, what was observed, what was done. No interpretation yet.
2. Separate what was known at the time from what is known now. This is the discipline that makes a postmortem blameless in substance rather than in tone.
3. Identify contributing factors, plural. A single root cause is nearly always a simplification that hides the interesting failures.
4. Analyse detection and response separately from cause. How long to detect, to engage, to mitigate? Those are independently improvable and often the bigger win.
5. Write action items that are specific, owned, dated and small enough to actually complete.
6. Review with the people involved before publishing.
7. Track the actions. An untracked action item is a note.

Everything in the timeline is Observed with a source. Interpretation is clearly marked as interpretation. "We believe" and "the logs show" are different claims.

Investigation

Step 1 - the timeline

| Time (UTC) | Event | Source |
| 09:14 | checkout-service v2.5.0 deployed to production | ArgoCD |
| 09:22 | Database CPU rises 20% -> 85% | Grafana |
| 09:31 | First customer report | Support ticket #4821 |
| 09:34 | p99 checkout latency crosses 10s | Prometheus |
| 09:41 | Burn-rate alert fires, on-call paged | Alertmanager |
| 09:44 | On-call acknowledges, begins investigating | PagerDuty |
| 09:58 | Deploy identified as the likely cause | Incident channel |
| 10:02 | Rollback to v2.4.1 initiated | ArgoCD |
| 10:09 | Error rate normal | Prometheus |

Include every event, including the ones that are uncomfortable: the alert that did not fire, the 27 minutes between the first customer report and the page, the investigation that went the wrong way for ten minutes. Those gaps are the findings.

Derive the durations, because they are what you improve against:

"Impact to detection" of 19 minutes with a customer report arriving first is, in this example, a bigger finding than the missing index — it will apply to every future incident of any kind.

Step 2 - contributing factors, not a root cause

Asking "what was the root cause?" produces one answer and stops. Real incidents need several conditions to line up.

For the example above:

Remove any one of these and the incident is smaller or does not happen. That is what makes them contributing factors — and it gives you six candidate interventions instead of one.

Use "five whys" carefully. It produces a single chain, and single chains lead to conclusions like "the engineer should have been more careful", which is not a fixable property of a system.

Step 3 - counterfactual discipline

The core rule of a blameless postmortem: describe what happened, not what should have happened.

This is not politeness. Counterfactual statements ("if only X had...") describe a world that did not happen and generate no fixable action. The people involved acted reasonably given what they knew at the time; if that is not true, the system made the unreasonable action easy, which is the finding.

A useful test: replace every name with "an engineer". If a sentence stops making sense, it was about a person rather than about the system.

Step 4 - detection and response

Answer these explicitly:

• Did an alert fire? If not, why not — and is the fix a new alert or a better SLI?
• Did the right alert fire first? A page for a symptom beats a page for a cause.
• Was the alert actionable? Did it say what was wrong and where to look?
• Did the responder have what they needed? Access, runbook, dashboard, permission to roll back.
• What slowed the response? Missing access is a common and cheap fix.
• Was the mitigation reversible? If not, why was an irreversible action necessary?

A monitoring gap is worth more than a code fix: the code fix prevents one incident, and better detection shortens every future incident of every kind.

Step 5 - action items that work

Rules:

• Every item has a single named owner — not a team. Teams do not do things; people do.
• Every item has a date, and it is weeks away, not quarters.
• Every item is small enough to finish. "Redesign the deployment pipeline" will not happen; split it.
• Prioritise by which factor, if removed, most reduces the next incident — often detection.
• Mark each as prevent, detect, or mitigate. A list that is all "prevent" means you have not improved your ability to handle the next surprise, and there will be one.

Most importantly: put them in the same tracker as normal work, with the same visibility. Action items in a document are notes.

Decision Tree

Incident resolved
├── Is the cause understood? → No → production-debugging first
├── Was there user impact or a near miss? → Yes → write it up
├── Building the analysis
│   ├── Timeline from evidence, with sources
│   ├── Separate known-then from known-now
│   ├── List contributing factors (expect 3-6)
│   └── Analyse detection and response separately
├── Any statement blaming a person?
│   ├── Yes → restate as a system property
│   └── No  → continue
├── Action items
│   ├── Owner, date, small, tracked → good
│   └── Vague or unowned → rewrite
└── Has this incident happened before?
    └── Yes → the previous action items are the subject of this postmortem

Common Causes

Patterns that recur across postmortems, worth checking against every incident:

Safety

Read-only

Writing a postmortem is analysis. Reading telemetry, logs, deploy history and chat transcripts changes nothing.

Two cautions specific to this work:

• Do not put credentials, personal data or customer identifiers in the document. Postmortems circulate widely, get pasted into tickets, and outlive their access controls. Redact as you write, not later.
• Preserve evidence before it expires. Kubernetes events last an hour, some log retention is seven days, and dashboards move. Capture screenshots and query results while they exist — a postmortem written three weeks later frequently cannot reconstruct the timeline at all.

Potentially disruptive

The postmortem process itself carries a risk that is worth naming: a postmortem that assigns blame, even implicitly, makes the next incident worse. People who expect to be blamed report later, share less, and hedge in the timeline. That is a direct, measurable reliability cost. If a draft names individuals, fix it before it circulates.

Publishing externally is a separate decision with legal and commercial implications. That is not the incident owner's call to make alone.

Destructive

This skill proposes no destructive operations.

Verification

A postmortem is verified by what changes, not by whether it was written.

Before: 3 similar checkout incidents in 6 months; mean time to detect 22 min
After:  0 recurrences in 6 months; mean time to detect 4 min; 5 of 6 action items completed

Check at 30 and 90 days:

1. Were the action items completed? A completion rate below about half means the process is producing documents, not change — and that is a finding about the process.
2. Did the incident recur? If yes, the analysis identified the wrong factor.
3. Did detection improve? Measurable in the next incident of any kind.
4. Do people reference it? An unread postmortem had no effect regardless of its quality.

The honest test: when a similar incident occurs, does the response go faster because of what was written? That is the only outcome the document exists for.

Anti-Patterns

Bad: Root cause: human error. Better: Human error is where the analysis starts, not where it ends. Ask why the system made the error easy, why nothing caught it, and why it had this blast radius.

Bad: Action item: be more careful. Better: Care is not an engineering control. Add a check, a guard rail, a confirmation, or a test — something that works when the person is tired.

Bad: One root cause. Better: Incidents in production systems need several conditions to coincide. Listing one gives you one intervention and leaves the rest of the alignment in place.

Bad: Skip the postmortem because the incident was short. Better: Short incidents are the cheapest possible learning. And a short incident with a 20 minute detection gap is a long incident waiting for a worse trigger.

Bad: Write it alone, from the logs. Better: The logs show what happened, not why someone decided what they did. The decision points are where the process findings are.

Bad: Fifteen action items. Better: Fifteen items means nothing gets done. Pick the three or four that most reduce the next incident, and be explicit that you are deliberately not doing the rest.

Bad: Judge the response by what you know now. Better: Evaluate decisions against the information available at the time. Hindsight makes every incident look obvious and teaches nothing transferable.

Composition

• incident-triage — the response this postmortem analyses.
• production-debugging — establishing the cause, which must happen first.
• slo-design — error budget spend and the detection gaps this analysis exposes.

References

• examples/postmortem-template.md — a template with the sections filled in as prompts.
• Google SRE Book: Postmortem Culture
• Google SRE Book: example postmortem