Incident Response & Postmortem

Severity Classification

Incident Response Protocol

Phase 1: Assess (first 5 minutes)

1. What is broken? (symptom, not cause)
2. Who is affected? (all users, subset, internal only)
3. When did it start? (check monitoring, recent deployments)
4. What changed recently? (git log --oneline -10, recent deploys)

Phase 2: Decide — Fix Forward or Roll Back

Was there a recent deployment?
├── Yes → Does reverting the deploy fix the issue?
│   ├── Yes → ROLL BACK immediately
│   └── No → Fix forward (the deploy isn't the cause)
└── No → Fix forward (investigate root cause)

Rollback: revert the deploy, not just the newest commit. git revert HEAD undoes one commit and fails outright on a merge commit.

git checkout prod && git pull
git log --oneline --first-parent -5          # find the deploy's commit(s)
git revert -m 1 <merge-sha>                  # deploy landed as a merge commit
# or: git revert --no-edit <last-good-sha>..HEAD   # deploy was several plain commits
git push origin prod                         # confirm with the user first: this redeploys production

Phase 3: Mitigate

• Apply the smallest change that stops the bleeding
• This is NOT the final fix — it's a bandage
• Accept technical debt temporarily to restore service

Phase 4: Communicate

Status update template:

[INCIDENT] [SEV-X] [Service Name]

Status: Investigating | Identified | Monitoring | Resolved
Impact: [who is affected and how]
Start time: [when it began]
Current action: [what we're doing right now]
Next update: [when to expect the next update]

Phase 5: Resolve and Verify

• Confirm the fix resolves the root cause (not just the symptom)
• Monitor for recurrence (15 min, 1 hour, 24 hours)
• Mark incident as resolved only after monitoring period

Blameless Postmortem

After resolution, conduct a blameless postmortem within 48 hours. Use the postmortem template.

Rules

• Never blame individuals — focus on systems and processes
• Every postmortem produces at least 2 action items with GitHub Issues
• Action items must have owners and deadlines
• Share postmortems with the team — learning is the goal