incident-runbook
Skill0 starsChanged 18 days agoMITView on GitHub
Guides a production incident from first alert to postmortem, covering severity, mitigation before root cause, status updates, a running timeline and a blameless write-up. Use when something is down or degraded in production, an alert fires, or the user asks for help writing a postmortem.
1 other project uses a version of this.
Adopted by
Most recently taken up by
- amanuelmamo7/Skillstheir own version, first seen Nov 7, 2026
Other versions1 project uses a version of this.
Word for word · 0
None found.
Edited · 0
None found.
Wrote their own version · 1
- amanuelmamo7/Skills★ 1same name
How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.
In this repository2 other files it gives its agents.
- load-test-planSkill
- migration-reviewSkill
The file
We split this skill into the rules it gives an agent
2 rules. About 0 are common, 2 are distinctive, and 0 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈2
Distinctive · estimate
This author's own choices. Worth a look.
0
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- could be automatic: a setting could enforce it
- Distinctive: line 15
Never run destructive commands (deleting data, dropping tables, force-pushing, clearing queues) during an incident without an explicit …
- Distinctive: line 51
Resolve: when metrics are back to baseline and have stayed there for a while (for example 30 minutes).
≈ “Common” is an estimate: at least 1 in 5 of the 350,495 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Incident runbook
During an incident the goal is to restore service first and understand it second. Keep every action reversible where possible, and write things down as they happen.
Ground rules for the agent
- Investigate read-only by default: logs, metrics, recent deploys, config diffs.
- Ask before any action that changes production: rollbacks, restarts, scaling, feature flags, data fixes. State the command, what it affects, and how to undo it.
- Never run destructive commands (deleting data, dropping tables, force-pushing, clearing queues) during an incident without an explicit go-ahead for that exact command.
- Log every action and finding in the timeline with a UTC timestamp.
Steps
Set severity so everyone knows how loud to be.
Sev Meaning Updates every SEV1 Core flow down for many users, or data loss/exposure 30 min SEV2 Degraded or partially down, workaround exists 60 min SEV3 Minor impact, can wait for working hours at resolution - First 15 minutes: what changed? Most incidents follow a change.
- Deploys and config changes in the last few hours (
git log, CI/CD history) - Feature flags flipped, migrations run, cron jobs added
- Traffic shape: spike, new client, bot, retry storm
- Dependencies: provider status pages, DNS, certificates expiring
- Deploys and config changes in the last few hours (
- Scope it. Which endpoints, regions, customers, and since when? Compare error rate, latency (p95/p99) and saturation (CPU, memory, DB connections, queue depth) against the same window yesterday.
- Mitigate before root cause. Pick the fastest reversible option:
- Roll back the last deploy, or turn the feature flag off
- Scale out, or shed load with rate limits
- Fail over to a healthy replica or region
- Restoring from backup is a last resort and needs explicit approval
Communicate on the cadence above:
[SEV2] Checkout errors: investigating Impact: ~8% of checkout requests fail with 502 since 14:05 UTC. Current action: rolling back release 2026.10.28-3 (deployed 13:58 UTC). Next update: 15:15 UTC or sooner if it changes.- Resolve when metrics are back to baseline and have stayed there for a while (for example 30 minutes). Announce it with start time, end time and impact.
- Postmortem within two working days, using the postmortem template. Fill the timeline from the log kept in step 1 onward.
Timeline format
14:05 Alert: checkout 5xx > 5% (pagerduty)
14:09 Scope: only /checkout, all regions; release 2026.10.28-3 went out 13:58
14:12 Decision: roll back (approved by @oncall)
14:19 Rollback complete; 5xx back under 0.5%
14:50 Resolved after 30 min stableDoes it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.