Incident Response

An incident is a coordination problem before it is a debugging problem. Several people (and agents) act at once, facts arrive out of order, and users want to know what is happening. This skill gives the response a shape: one severity, named roles, one timeline, mitigation first, and a review that turns the outage into tracked work.

When to Use

• An alert, support report, or monitor shows users are failing, data is wrong, or a security boundary may be breached
• A deploy, migration, or config change is suspected of causing live harm
• More than one person or agent is already working the problem and nobody owns coordination
• After recovery, to run the post-incident review and record follow-up actions

When NOT to Use

Prerequisites

• A place everyone can read: an incident issue, a chat channel, or a shared document
• Access to the signals that show user impact (dashboards, logs, error tracker, status checks)
• Knowledge of the rollback or failover paths for the affected system

Workflow

1. Declare and Assign Roles

Declare the incident as soon as impact is likely; downgrading later is cheap, declaring late is not. Record a severity using the team's scale, or this default:

Name three roles, even if one person holds two:

• Incident lead — owns decisions and priorities; does not debug
• Operations lead — runs the technical investigation and changes
• Communications — writes status updates and keeps the timeline

Open the record with the GitHub CLI when the team tracks incidents as issues (the incident label must already exist — create it once with gh label create incident):

gh issue create --title "SEV2: checkout errors after 14:05 deploy" --label incident `
  --body "Lead: @a  Ops: @b  Comms: @c  Status: investigating"

2. Keep One Timeline

Append every observation and action with a timestamp in one place. Write facts, not conclusions: "14:12 error rate 8% on /checkout (dashboard X)" rather than "14:12 DB is overloaded". When an agent runs commands, record the command, the time, and the output that drove a decision. Redact credentials, personal data, and customer data; keep large raw output in an access-controlled place and link to it from the timeline.

3. Mitigate Before You Diagnose

Stop the user impact first, with the least risky reversible action:

1. Roll back the suspected change, or turn off its feature flag
2. Fail over, shed load, or scale the saturated component
3. Block the abusive or malformed traffic

Each mitigation is a change: state it in the timeline before running it, and say how you will know whether it worked. Root-cause work starts once impact is contained — hand it to systematic-debugging with the timeline attached.

4. Communicate on a Fixed Cadence

Post status at a fixed interval (for example every 30 minutes for SEV1, 60 for SEV2), even when nothing changed. Each update states: current impact, what is being done, and the time of the next update. Do not speculate about cause in external updates.

5. Resolve and Review

Close the incident only after the recovery checks below pass. Within a few days, run a blameless review:

• Timeline of detection, mitigation, and recovery, with times
• What made detection or mitigation slow
• Contributing causes (usually several), not a single culprit
• Actions, each with an owner and a due date, filed as issues

Common Rationalizations

Red Flags

• Nobody can say who the incident lead is
• Several people are making production changes without writing them down
• The status update promises a cause or an ETA the team does not have
• The review produces "be more careful" instead of owned, dated actions

Verification

• The user-facing signal that showed the impact is back to its baseline, not just a component health check
• The mitigation's side effects are checked (rolled-back data, disabled features, queued work)
• Any data written during the incident is confirmed correct, or a repair action is filed
• The timeline is complete enough for someone absent to follow the response
• Every review action has an owner, a due date, and an issue

If recovery checks fail after a mitigation, reopen the response at step 3 with the next mitigation rather than declaring partial success.

Tips

• Agents are useful as scribes and investigators: give one the timeline role and others bounded read-only investigations, and keep production changes with the operations lead
• Use the review actions as input to deployment-canary rollback triggers for the next release