agentleFS
Sign inStart free
witi-projectswiti-projects/Skills/incident-response/SKILL.md

incident-response

Skill2 starsChanged 6 months agoMITView on GitHub

Handle production incidents — triage, mitigate, communicate, and write postmortems. Use when the user says "/incident-response", "incident response", or asks about incident response.

We split this skill into the rules it gives an agent

6 rules. 0 are common, 6 are distinctive, and 0 are very specific to this project.

  • 0

    Common

    Most of the 61 edited versions we read keep these.

  • 6

    Distinctive

    This author's own choices. Worth a look.

  • 0

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Check monitoring dashboards (error rates, latency, status page)

    line 26
  2. Distinctive:

    Check recent deployments: git log --oneline -10 — was anything deployed recently?

    line 27
  3. Distinctive:

    Scale up: Add more instances if it's a capacity issue

    line 36
  4. Distinctive:

    Verify the fix works in production (check metrics, not just absence of errors)

    line 55
  5. Distinctive:

    Write a blameless postmortem:

    line 60
  6. Distinctive:

    Don't assign blame in postmortems — focus on process improvements

    line 101

“Common” is measured: kept by at least 6 in 10 of the 61 edited versions of this file we could read. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Incident Response

Handle production incidents systematically.

Severity Levels

LevelDefinitionResponse TimeExamples
SEV1Service down, all users affectedImmediateDatabase crash, DNS failure, auth broken
SEV2Major feature broken, many users affected< 30 minPayments failing, search not working
SEV3Minor feature broken, workaround exists< 4 hoursExport button broken, slow dashboard
SEV4Cosmetic or low-impact issueNext business dayTypo in UI, minor styling bug

Incident Workflow

1. Detect & Triage (first 5 minutes)

  • Acknowledge the incident — "I'm looking into this"
  • Determine severity level
  • Check monitoring dashboards (error rates, latency, status page)
  • Check recent deployments: git log --oneline -10 — was anything deployed recently?

2. Mitigate (next 15-30 minutes)

The goal is to stop the bleeding, not find the root cause.

Quick mitigations:

  • Rollback: git revert <commit> && deploy — fastest option if a deploy caused it
  • Feature flag: Disable the broken feature
  • Scale up: Add more instances if it's a capacity issue
  • Failover: Switch to backup/secondary if primary is down
  • Block traffic: Rate-limit or block specific abusive traffic

3. Communicate

Internal:

  • Open an incident channel (#incident-2026-04-10)
  • Post status updates every 15-30 minutes
  • Assign roles: Incident Commander, Communicator, Engineers

External:

  • Update status page
  • Send email/notification to affected users if the outage is extended
  • Be honest: "We're experiencing issues with X. We've identified the cause and are working on a fix."

4. Resolve

  • Deploy the fix
  • Verify the fix works in production (check metrics, not just absence of errors)
  • Close the incident channel with a summary

5. Postmortem (within 48 hours)

Write a blameless postmortem:

# Incident: Payments failing for Stripe webhook
**Date:** 2026-04-10
**Duration:** 45 minutes (14:30 — 15:15 UTC)
**Severity:** SEV2
**Impact:** ~200 users unable to complete purchases

## Timeline
- 14:30 — Alert fires: payment success rate drops to 20%
- 14:35 — On-call engineer acknowledges, begins investigation
- 14:40 — Identified: Stripe webhook endpoint returning 500
- 14:45 — Root cause: migration added NOT NULL column without default
- 14:50 — Fix deployed: added default value to migration
- 15:00 — Payment success rate recovering
- 15:15 — Metrics back to normal, incident closed

## Root Cause
Database migration #47 added a `currency` column with NOT NULL 
but no DEFAULT value. Existing rows were fine (backfilled), but 
new webhook events failed because the insert didn't include `currency`.

## What Went Well
- Alert fired within 5 minutes of the issue starting
- Rollback was considered but the fix was faster

## What Went Wrong
- Migration wasn't tested with live webhook payloads
- No staging test for the webhook flow

## Action Items
- [ ] Add webhook integration test to CI (@alice, due 2026-04-17)
- [ ] Require DEFAULT for all new NOT NULL columns in migration review (@bob)
- [ ] Add runbook for payment failures (@charlie, due 2026-04-14)

Tips

  • Rollback first, investigate later — speed matters more than elegance
  • The most recent deploy is the most likely cause
  • Don't assign blame in postmortems — focus on process improvements
  • Maintain a runbook for common failure modes
  • Practice incident response with game days before real incidents happen

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

Reviews can't be read right now.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

Check your setup, free