sre-postmortem-templates
Skill4 starsChanged 24 days agoMITView on GitHub
Use after an incident is resolved to draft or facilitate a blameless postmortem / incident review / RCA with a timeline from paging, monitoring and deploy history. Not for active incidents.
In this repository84 other files it gives its agents.
- AGENTS.mdAGENTS.md
- .github/copilot-instructions.mdCopilot instructions
- .cursor/rules/sre-safety-gates.mdcCursor rule
- sre-awsSkill
- sre-azureSkill
- sre-capacity-planningSkill
- sre-change-safetySkill
- sre-chaos-engineeringSkill
- sre-cloud-cost-reviewSkill
- sre-database-triageSkill
- sre-datadogSkill
- sre-dependency-mappingSkill
- sre-diagnosticsSkill
- sre-dr-backup-testSkill
- sre-dynatraceSkill
- sre-emudhra-ca-checkSkill
- sre-engineerSkill
- sre-expiry-auditSkill
- sre-gcpSkill
- sre-incident-commandSkill
- sre-incident-communicationSkill
- sre-k8s-safe-remediationSkill
- sre-k8s-triageSkill
- sre-k8s-upgrade-autopilotSkill
- sre-message-queue-triageSkill
- sre-migration-safetySkill
- sre-observability-cost-cutterSkill
- sre-observability-query-cookbookSkill
- sre-oncall-auditSkill
- sre-oncall-handoffSkill
- sre-pagerduty-opsgenieSkill
- sre-php2shell-checkSkill
- sre-production-readinessSkill
- sre-prometheus-grafanaSkill
- sre-remediation-actionsSkill
- sre-runbook-templatesSkill
- sre-safety-gatesSkill
- sre-security-incident-triageSkill
- sre-service-mesh-triageSkill
- sre-slo-alertingSkill
- sre-slo-burn-rate-rulesSkill
- sre-terraformSkill
- sre-toil-reviewSkill
- sre-awsSkill
- sre-azureSkill
- sre-capacity-planningSkill
- sre-change-safetySkill
- sre-chaos-engineeringSkill
- sre-cloud-cost-reviewSkill
- sre-database-triageSkill
- sre-datadogSkill
- sre-dependency-mappingSkill
- sre-diagnosticsSkill
- sre-dr-backup-testSkill
- sre-dynatraceSkill
- sre-emudhra-ca-checkSkill
- sre-engineerSkill
- sre-expiry-auditSkill
- sre-gcpSkill
- sre-incident-commandSkill
- The first 60 shown.
The file
We split this skill into the rules it gives an agent
7 rules. About 0 are common, 4 are distinctive, and 3 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈4
Distinctive · estimate
This author's own choices. Worth a look.
3
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 12
Wait for explicit human approval before any prod-changing command, MCP write tool, or external post.
- Distinctive: line 22
Collect from each source, in UTC: Keep each source's own timestamps and link every entry to its source.
- Very specific: line 24
Monitoring: when the SLI first left normal and when it recovered (error rate, latency, saturation), and the error budget spent …
- Very specific: line 26
Databases: if a database was involved, the evidence bundle captured during the incident by bash …
- Distinctive: line 87
Avoid 'why did you' and use 'how did the system'.
- Distinctive: line 89
Ensure the most senior person in the room is the one modeling blameless behavior.
- Very specific: line 94
Every action item has an owner, due date, and ticket before the postmortem is published.
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Postmortem Templates
Safety gates (REQUIRED)
- Read-only diagnosis first.
- Propose each change with risk (low/med/high) and reversibility.
- Wait for explicit human approval before any prod-changing command, MCP write tool, or external post.
- Act only as approved, verify, and stop and escalate if anything gets worse.
Full rules: sre-safety-gates. Read-only MCP setup: sre-safety-gates/references/mcp.md.
Blameless postmortem templates and facilitation guidance. Gather the evidence first, then fill the template.
Build the timeline (read-only)
- Confirm scope. Get the incident ID, the service, and the window: from first symptom (often before the first page) to resolution. Check that the incident is resolved; if it's still active, stop and use
sre-incident-command. - Collect from each source, in UTC. Keep each source's own timestamps and link every entry to its source.
- Alerts and pages: the incident log from
sre-pagerduty-opsgenie(trigger, acknowledge, escalate, resolve, notes):bash ../sre-pagerduty-opsgenie/scripts/pd_incidents.sh --since <ISO8601> --until <ISO8601> --status resolved(exit 2 means the data is unavailable, not that nothing happened), then that incident's log entries via api-reference-pagerduty.md. Also check alert history in the monitoring tool: when each alert first fired, including ones that fired before anyone was paged. - Monitoring: when the SLI first left normal and when it recovered (error rate, latency, saturation), and the error budget spent (
sre-slo-alerting). - Deploys and changes: CI/CD deploy history,
kubectl --context <ctx> -n <ns> rollout history deployment/<name>,helm --kube-context <ctx> -n <ns> history <release>,git log --since=<ISO8601> --until=<ISO8601> --oneline | head -n 50in the service and infrastructure repos, feature-flag and config change audit logs (sre-change-safety). - Databases: if a database was involved, the evidence bundle captured during the incident by
bash ../sre-database-triage/scripts/db-snapshot.sh(locks, connections, replication lag, slow queries at that moment), plus the managed service's event log and failover history. Use sre-database-triage read-only if the bundle is missing. - Chat and incident channel: declaration, decisions, who approved each action and when, status page and customer updates. Quote decisions, not people's mistakes.
- Alerts and pages: the incident log from
- Merge and check. Sort everything into one UTC timeline. Mark the key points: start of impact, detection, acknowledgement, mitigation, resolution. Compute TTD (time to detect: impact start to first alert), TTA (time to acknowledge: first alert to first human acknowledgement), TTM (time to mitigate: impact start to impact stopping) and total duration. Flag gaps and conflicting timestamps as open questions instead of guessing.
- Keep it a draft. Redact secrets and customer data. Publishing or sending the postmortem is an external post and needs approval (sre-safety-gates).
Postmortem principles
- Blameless: Focus on process and systems, not individuals.
- Evidence-based: Use data, logs, and timelines.
- Action-oriented: Every postmortem must result in meaningful follow-up items.
- Timely: Conduct the review within 48 hours of resolution.
Postmortem markdown template
# Postmortem: [Incident Title]
**Date:** [YYYY-MM-DD]
**Severity:** [e.g., SEV-1]
**Incident Commander (IC):** [Name]
## 1. Summary
Give a 2-3 sentence overview of what happened and the impact.
## 2. Impact
- **User Impact:** [e.g., 50% of users in US-East could not log in]
- **Duration:** [e.g., 45 minutes]
- **TTD (time to detect):** [impact start to first alert, e.g., 8 min]
- **TTA (time to acknowledge):** [first alert to first human ack, e.g., 5 min]
- **TTM (time to mitigate):** [impact start to impact stopping, e.g., 38 min]
- **SLO Impact:** [e.g., Consumed 15% of monthly error budget]
## 3. Timeline (UTC)
Sources: [PagerDuty/Opsgenie incident ID], [dashboards], [deploy history], [database evidence bundle], [incident channel]
- **09:52:** Impact starts: [SLI] leaves normal (from monitoring)
- **10:00:** Issue detected by [Alert Name]
- **10:05:** Incident bridge opened
- **10:15:** Root cause identified as [X]
- **10:30:** Mitigation [Y] applied
- **10:45:** Service restored
## 4. Root Cause Analysis (The Five Whys)
1. Why [Symptom]?
2. Why [X]?
...
## 5. What went well?
- [e.g., Monitoring detected the issue immediately]
- [e.g., Rollback was fast and successful]
## 6. Where did we get lucky?
- [e.g., Incident happened during business hours]
## 7. Action Items
| Action | Type | Owner | Due date | Ticket | Status |
|---|---|---|---|---|---|
| [Action] | [Mitigate / Prevent / Detect] | [Owner] | [YYYY-MM-DD] | [TICKET-123] | [Open] |
| [Preventative measure] | [Prevent] | [Owner] | [YYYY-MM-DD] | [TICKET-124] | [Open] |Facilitation tips
- Avoid 'why did you' and use 'how did the system'.
- Focus on the 'gap' between how the system was intended to work and how it actually worked.
- Ensure the most senior person in the room is the one modeling blameless behavior.
Output
- Blameless postmortem draft (unpublished until approved) with a UTC timeline that cites its sources, time to detect / acknowledge / mitigate, and open questions
- Every action item has an owner, due date, and ticket before the postmortem is published. In a draft, write
[TBD]for any that don't exist yet (tickets often don't), and list each[TBD]under open questions; publishing waits until none are left.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

