agentleFS
ricmmartinsricmmartins/azure-sre-agent-skills/skills/05-incident-postmortem/SKILL.md

incident-postmortem-generator

Skill70 starsChanged 4 months agoMITView on GitHub

Generate a blameless incident postmortem from the current or recent SRE Agent investigation. Use after an incident is resolved, when asked for a postmortem, RCA report, incident review, or lessons learned document.

The file

We split this skill into the rules it gives an agent

11 rules. About 0 are common, 11 are distinctive, and 0 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈11

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 0

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Evidence-based: Every claim backed by data (logs, metrics, timeline)

    line 22
  2. Distinctive:

    Action-oriented: Every finding leads to a concrete action item

    line 23
  3. Distinctive:

    When did the issue start? (first anomaly in metrics)

    line 39
  4. Distinctive:

    When was it detected? (alert fired)

    line 40
  5. Distinctive:

    When was it acknowledged? (engineer engaged)

    line 41
  6. Distinctive:

    When was mitigation applied?

    line 43
  7. Distinctive:

    When was full resolution confirmed?

    line 44
  8. Distinctive:

    If the user provides a list of accepted exceptions, do not flag those items.

    line 104
  9. Distinctive:

    When exceptions are provided:

    line 114
  10. Distinctive:

    If Teams connector is available, offer to post to the incident channel

    line 192
  11. Distinctive:

    If ServiceNow/PagerDuty is connected, offer to attach to the incident record

    line 193

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Incident Postmortem Generator

Purpose

Automatically generate a structured, blameless postmortem document from the context of a resolved incident investigation. Leverages the timeline, root cause analysis, and mitigations already gathered by the SRE Agent.

When to use this skill

  • User asks for a postmortem after an incident is resolved
  • User asks for an RCA report, incident review, or lessons learned
  • Automated trigger after incident resolution (via agent hook)
  • User asks "write up what just happened"

Postmortem principles

  • Blameless: Focus on systems and processes, never individuals
  • Evidence-based: Every claim backed by data (logs, metrics, timeline)
  • Action-oriented: Every finding leads to a concrete action item
  • Learning-focused: What can we improve systemically?

Generation procedure

Step 1: Gather incident context

From the current conversation and agent memory, extract:

  1. Incident metadata:
    • Incident ID (from PagerDuty/ServiceNow if connected)
    • Severity level
    • Duration (detection to resolution)
    • Affected services and resources
    • Impacted users/customers (if known)
  2. Timeline: Reconstruct from the investigation:
    • When did the issue start? (first anomaly in metrics)
    • When was it detected? (alert fired)
    • When was it acknowledged? (engineer engaged)
    • Key investigation milestones
    • When was mitigation applied?
    • When was full resolution confirmed?
  3. Root cause: From the agent's RCA:
    • What failed?
    • Why did it fail?
    • Contributing factors
  4. Mitigation applied: What was done to resolve it?

Step 2: Assess detection and response

Analyze the incident response quality:

  1. Time to detect (TTD): From issue start to alert firing
    • Was this acceptable? Could monitoring have caught it earlier?
  2. Time to acknowledge (TTA): From alert to human engagement
    • Were on-call rotations effective?
  3. Time to mitigate (TTM): From engagement to mitigation
    • Was the runbook adequate? Did the team have the right access?
  4. Time to resolve (TTR): Total duration
    • Compare with SLA/SLO targets

Step 3: Five Whys analysis

Apply the 5 Whys technique to the root cause:

1. Why did the service return 500 errors?
   → The application pod was OOM-killed
2. Why was the pod OOM-killed?
   → Memory usage exceeded the 512Mi limit
3. Why did memory usage spike?
   → A new dependency introduced a memory leak in the connection pool
4. Why wasn't the leak caught before production?
   → Load testing doesn't cover the connection pool under sustained load
5. Why doesn't load testing cover this scenario?
   → Load test profiles haven't been updated since the architecture change

Step 4: Generate action items

For each finding, create a SMART action item:

IDActionOwnerPriorityDue dateStatus
AI-1Add memory-based autoscaling to payment-servicePlatform teamHigh+7 daysOpen
AI-2Update load test to include sustained connection pool scenariosQA teamHigh+14 daysOpen
AI-3Add memory utilization alert at 80% thresholdSRE teamMedium+3 daysOpen
AI-4Review connection pool settings across all servicesDev teamMedium+14 daysOpen

Categories:

  • Prevent recurrence: Fix the root cause
  • Improve detection: Better monitoring/alerting
  • Improve response: Better runbooks/automation
  • Improve resilience: Architectural improvements

Step 5: Compile the postmortem document

Accepted exceptions (optional)

If the user provides a list of accepted exceptions, do not flag those items. Instead, note them in the report as Accepted Exception with the reason provided.

Example format the user may provide:

CheckReason
Time to detectAlert delay was due to a known monitoring gap being addressed in AI-3
Five Whys depthRoot cause was an external provider outage, further analysis not applicable
Missing runbookService is new, runbook creation is tracked in AI-5

When exceptions are provided:

  • Skip the flagged checks in scoring
  • List them in a separate "Accepted Exceptions" section at the end of the report
  • Recalculate the overall score excluding excepted checks

Expected output

Report header (mandatory — use this exact format)

Incident Postmortem Report

FieldValue
Subscription(name + ID)
Incident DateYYYY-MM-DD
SeveritySev-X
DurationXh Ym
StatusDraft — pending team review

Postmortem document structure

# Incident Postmortem: [Title]

## Executive summary
[2-3 sentences: what happened, impact, resolution]

## Impact
- **Services affected**: [list]
- **User impact**: [description and scope]
- **SLA/SLO impact**: [metrics affected]
- **Business impact**: [revenue, reputation, etc. if known]

## Timeline (all times UTC)

| Time | Event |
|------|-------|
| HH:MM | First anomaly detected in metrics |
| HH:MM | Alert fired: [alert name] |
| HH:MM | On-call engineer acknowledged |
| HH:MM | Investigation started |
| HH:MM | Root cause identified |
| HH:MM | Mitigation applied |
| HH:MM | Service fully recovered |

## Root cause
[Detailed technical explanation]

## Five Whys
[Analysis chain]

## Detection & response assessment

| Metric | Value | Target | Assessment |
|--------|-------|--------|------------|
| Time to detect | Xm | <5m | ✅/⚠️/❌ |
| Time to acknowledge | Xm | <15m | ✅/⚠️/❌ |
| Time to mitigate | Xm | <30m | ✅/⚠️/❌ |
| Time to resolve | Xm | <2h | ✅/⚠️/❌ |

## What went well
- [Positive aspects of the response]

## What could be improved
- [Areas for improvement]

## Action items
[Table from Step 4]

## Lessons learned
- [Key takeaways for the team]

---
*This postmortem was auto-generated by Azure SRE Agent and should be reviewed by the incident team before publishing.*

Delivery

  • Output the full document in the chat
  • If Teams connector is available, offer to post to the incident channel
  • If ServiceNow/PagerDuty is connected, offer to attach to the incident record
  • Suggest scheduling a postmortem review meeting

References

Sample output

The following is a redacted example of what the report looks like after an incident is resolved.

Incident Postmortem Report

FieldValue
Subscriptioncontoso-prod-001 (a1b2c3d4-e5f6-7890-abcd-ef1234567890)
Incident Date2026-07-10
SeveritySev-2
Duration1h 23m
StatusDraft — pending team review

Executive summary

The payment-service API returned HTTP 500 errors for 1h 23m due to an OOM-killed pod in AKS cluster aks-app-prod. Root cause was a memory leak in the connection pool introduced in release v2.14.0. Impact: ~340 failed transactions affecting 120 customers.

Timeline (all times UTC)

TimeEvent
14:12Memory usage spike detected on payment-service pod
14:18Alert fired: "Pod OOM-killed in aks-app-prod"
14:22On-call engineer acknowledged
14:35Root cause identified: connection pool memory leak
14:50Mitigation applied: rolled back to v2.13.2
15:35Service fully recovered, error rate at 0%

Detection & response assessment

MetricValueTargetAssessment
Time to detect6m<5m⚠️
Time to acknowledge4m<15m✅
Time to mitigate28m<30m✅
Time to resolve1h 23m<2h✅

Action items (sample)

IDActionOwnerPriorityDue
AI-1Add memory-based HPA to payment-servicePlatform teamHigh+7 days
AI-2Update load test for sustained connection pool scenariosQA teamHigh+14 days
AI-3Add memory utilization alert at 80% thresholdSRE teamMedium+3 days

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.