incident-postmortem-generator
Skill70 starsChanged 4 months agoMITView on GitHub
Generate a blameless incident postmortem from the current or recent SRE Agent investigation. Use after an incident is resolved, when asked for a postmortem, RCA report, incident review, or lessons learned document.
In this repository7 other files it gives its agents.
The file
We split this skill into the rules it gives an agent
11 rules. About 0 are common, 11 are distinctive, and 0 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈11
Distinctive · estimate
This author's own choices. Worth a look.
0
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 22
Evidence-based: Every claim backed by data (logs, metrics, timeline)
- Distinctive: line 23
Action-oriented: Every finding leads to a concrete action item
- Distinctive: line 39
When did the issue start? (first anomaly in metrics)
- Distinctive: line 40
When was it detected? (alert fired)
- Distinctive: line 41
When was it acknowledged? (engineer engaged)
- Distinctive: line 43
When was mitigation applied?
- Distinctive: line 44
When was full resolution confirmed?
- Distinctive: line 104
If the user provides a list of accepted exceptions, do not flag those items.
- Distinctive: line 114
When exceptions are provided:
- Distinctive: line 192
If Teams connector is available, offer to post to the incident channel
- Distinctive: line 193
If ServiceNow/PagerDuty is connected, offer to attach to the incident record
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Incident Postmortem Generator
Purpose
Automatically generate a structured, blameless postmortem document from the context of a resolved incident investigation. Leverages the timeline, root cause analysis, and mitigations already gathered by the SRE Agent.
When to use this skill
- User asks for a postmortem after an incident is resolved
- User asks for an RCA report, incident review, or lessons learned
- Automated trigger after incident resolution (via agent hook)
- User asks "write up what just happened"
Postmortem principles
- Blameless: Focus on systems and processes, never individuals
- Evidence-based: Every claim backed by data (logs, metrics, timeline)
- Action-oriented: Every finding leads to a concrete action item
- Learning-focused: What can we improve systemically?
Generation procedure
Step 1: Gather incident context
From the current conversation and agent memory, extract:
- Incident metadata:
- Incident ID (from PagerDuty/ServiceNow if connected)
- Severity level
- Duration (detection to resolution)
- Affected services and resources
- Impacted users/customers (if known)
- Timeline: Reconstruct from the investigation:
- When did the issue start? (first anomaly in metrics)
- When was it detected? (alert fired)
- When was it acknowledged? (engineer engaged)
- Key investigation milestones
- When was mitigation applied?
- When was full resolution confirmed?
- Root cause: From the agent's RCA:
- What failed?
- Why did it fail?
- Contributing factors
- Mitigation applied: What was done to resolve it?
Step 2: Assess detection and response
Analyze the incident response quality:
- Time to detect (TTD): From issue start to alert firing
- Was this acceptable? Could monitoring have caught it earlier?
- Time to acknowledge (TTA): From alert to human engagement
- Were on-call rotations effective?
- Time to mitigate (TTM): From engagement to mitigation
- Was the runbook adequate? Did the team have the right access?
- Time to resolve (TTR): Total duration
- Compare with SLA/SLO targets
Step 3: Five Whys analysis
Apply the 5 Whys technique to the root cause:
1. Why did the service return 500 errors?
→ The application pod was OOM-killed
2. Why was the pod OOM-killed?
→ Memory usage exceeded the 512Mi limit
3. Why did memory usage spike?
→ A new dependency introduced a memory leak in the connection pool
4. Why wasn't the leak caught before production?
→ Load testing doesn't cover the connection pool under sustained load
5. Why doesn't load testing cover this scenario?
→ Load test profiles haven't been updated since the architecture changeStep 4: Generate action items
For each finding, create a SMART action item:
| ID | Action | Owner | Priority | Due date | Status |
| AI-1 | Add memory-based autoscaling to payment-service | Platform team | High | +7 days | Open |
| AI-2 | Update load test to include sustained connection pool scenarios | QA team | High | +14 days | Open |
| AI-3 | Add memory utilization alert at 80% threshold | SRE team | Medium | +3 days | Open |
| AI-4 | Review connection pool settings across all services | Dev team | Medium | +14 days | Open |
Categories:
- Prevent recurrence: Fix the root cause
- Improve detection: Better monitoring/alerting
- Improve response: Better runbooks/automation
- Improve resilience: Architectural improvements
Step 5: Compile the postmortem document
Accepted exceptions (optional)
If the user provides a list of accepted exceptions, do not flag those items. Instead, note them in the report as Accepted Exception with the reason provided.
Example format the user may provide:
| Check | Reason |
| Time to detect | Alert delay was due to a known monitoring gap being addressed in AI-3 |
| Five Whys depth | Root cause was an external provider outage, further analysis not applicable |
| Missing runbook | Service is new, runbook creation is tracked in AI-5 |
When exceptions are provided:
- Skip the flagged checks in scoring
- List them in a separate "Accepted Exceptions" section at the end of the report
- Recalculate the overall score excluding excepted checks
Expected output
Report header (mandatory — use this exact format)
Incident Postmortem Report
| Field | Value |
| Subscription | (name + ID) |
| Incident Date | YYYY-MM-DD |
| Severity | Sev-X |
| Duration | Xh Ym |
| Status | Draft — pending team review |
Postmortem document structure
# Incident Postmortem: [Title]
## Executive summary
[2-3 sentences: what happened, impact, resolution]
## Impact
- **Services affected**: [list]
- **User impact**: [description and scope]
- **SLA/SLO impact**: [metrics affected]
- **Business impact**: [revenue, reputation, etc. if known]
## Timeline (all times UTC)
| Time | Event |
|------|-------|
| HH:MM | First anomaly detected in metrics |
| HH:MM | Alert fired: [alert name] |
| HH:MM | On-call engineer acknowledged |
| HH:MM | Investigation started |
| HH:MM | Root cause identified |
| HH:MM | Mitigation applied |
| HH:MM | Service fully recovered |
## Root cause
[Detailed technical explanation]
## Five Whys
[Analysis chain]
## Detection & response assessment
| Metric | Value | Target | Assessment |
|--------|-------|--------|------------|
| Time to detect | Xm | <5m | ✅/⚠️/❌ |
| Time to acknowledge | Xm | <15m | ✅/⚠️/❌ |
| Time to mitigate | Xm | <30m | ✅/⚠️/❌ |
| Time to resolve | Xm | <2h | ✅/⚠️/❌ |
## What went well
- [Positive aspects of the response]
## What could be improved
- [Areas for improvement]
## Action items
[Table from Step 4]
## Lessons learned
- [Key takeaways for the team]
---
*This postmortem was auto-generated by Azure SRE Agent and should be reviewed by the incident team before publishing.*Delivery
- Output the full document in the chat
- If Teams connector is available, offer to post to the incident channel
- If ServiceNow/PagerDuty is connected, offer to attach to the incident record
- Suggest scheduling a postmortem review meeting
References
- Postmortem Culture: https://learn.microsoft.com/en-us/azure/well-architected/operational-excellence/mitigation-strategy
- Azure Resource Health: https://learn.microsoft.com/en-us/azure/service-health/resource-health-overview
- Activity Log: https://learn.microsoft.com/en-us/azure/azure-monitor/essentials/activity-log
- Service Health: https://learn.microsoft.com/en-us/azure/service-health/overview
Sample output
The following is a redacted example of what the report looks like after an incident is resolved.
Incident Postmortem Report
| Field | Value |
| Subscription | contoso-prod-001 (a1b2c3d4-e5f6-7890-abcd-ef1234567890) |
| Incident Date | 2026-07-10 |
| Severity | Sev-2 |
| Duration | 1h 23m |
| Status | Draft — pending team review |
Executive summary
The payment-service API returned HTTP 500 errors for 1h 23m due to an OOM-killed pod in AKS cluster aks-app-prod. Root cause was a memory leak in the connection pool introduced in release v2.14.0. Impact: ~340 failed transactions affecting 120 customers.
Timeline (all times UTC)
| Time | Event |
| 14:12 | Memory usage spike detected on payment-service pod |
| 14:18 | Alert fired: "Pod OOM-killed in aks-app-prod" |
| 14:22 | On-call engineer acknowledged |
| 14:35 | Root cause identified: connection pool memory leak |
| 14:50 | Mitigation applied: rolled back to v2.13.2 |
| 15:35 | Service fully recovered, error rate at 0% |
Detection & response assessment
| Metric | Value | Target | Assessment |
| Time to detect | 6m | <5m | ⚠️ |
| Time to acknowledge | 4m | <15m | ✅ |
| Time to mitigate | 28m | <30m | ✅ |
| Time to resolve | 1h 23m | <2h | ✅ |
Action items (sample)
| ID | Action | Owner | Priority | Due |
| AI-1 | Add memory-based HPA to payment-service | Platform team | High | +7 days |
| AI-2 | Update load test for sustained connection pool scenarios | QA team | High | +14 days |
| AI-3 | Add memory utilization alert at 80% threshold | SRE team | Medium | +3 days |
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

