Incident Response

Run a structured incident response: triage severity, coordinate communication, track resolution, and write a blameless postmortem.

Workflow

Phase 1: Triage

When an incident is reported, immediately assess severity.

Severity Matrix

Triage Questions

1. What is broken? (symptom, not guess at cause)
2. Who is affected? (all users, subset, internal only)
3. What is the business impact? (revenue, reputation, compliance)
4. When did it start? (check monitoring for the change point)
5. What changed recently? (deploys, config changes, traffic spikes)

Immediate Actions

• Assign an Incident Commander (IC) if SEV-1 or SEV-2.
• Open a dedicated incident channel or thread.
• Check if an obvious rollback would resolve the issue.
• Start a timeline document to capture events as they happen.

Phase 2: Coordinate

During the incident, maintain clear communication.

Status Update Template (post every 15-30 minutes for SEV-1/2):

INCIDENT UPDATE — <Service Name> — <SEV-X>
Time: YYYY-MM-DD HH:MM TZ
Status: Investigating | Identified | Monitoring | Resolved
Impact: <who is affected and how>
Current action: <what is being done right now>
Next update: <when>

Communication Channels

• Internal engineering: incident channel with real-time updates.
• Internal stakeholders (support, product): summary updates at key milestones.
• External (customers): status page updates for SEV-1 and SEV-2.

Coordination Rules

• The IC directs the response. Others execute and report back.
• One person communicates externally. Avoid conflicting messages.
• Document everything in the timeline. Memory is unreliable during incidents.
• Do not make unrelated changes during an active incident.

Phase 3: Resolution

When the immediate issue is resolved:

• Confirm the fix is stable (monitor for at least 15-30 minutes).
• Post a final status update marking the incident as resolved.
• Update the status page if one was created.
• Note the total duration and impact for the postmortem.
• Schedule the postmortem within 48 hours while memory is fresh.

Phase 4: Postmortem

Write a blameless postmortem focused on systemic improvements, not individual fault.

Postmortem Process

1. Gather the timeline from incident notes, logs, and chat history.
2. Identify contributing factors (not a single "root cause" — incidents are usually multi-causal).
3. Determine what went well and what could be improved.
4. Define concrete action items with owners and due dates.
5. Share with the broader team for review and learning.

Output Format

Triage Output

## Incident Triage

**Reported**: YYYY-MM-DD HH:MM TZ
**Severity**: SEV-X
**Service**: <affected service>
**Symptom**: <what is observed>
**Impact**: <who is affected and how>
**Suspected trigger**: <recent change, traffic spike, etc.>

### Immediate Actions
1. <first action to take>
2. <second action>
3. <third action>

### Rollback Assessment
- Can we rollback? Yes / No / Partial
- Last stable version: <tag or SHA>
- Rollback risk: <what could go wrong with rollback>

Postmortem Output

# Postmortem: <Incident Title>

**Date**: YYYY-MM-DD
**Duration**: X hours Y minutes
**Severity**: SEV-X
**Incident Commander**: <name>
**Authors**: <names>

## Summary

<2-3 sentence summary: what happened, impact, and resolution>

## Impact

- **Users affected**: <count or percentage>
- **Duration**: <start to resolution>
- **Revenue impact**: <if applicable>
- **Data impact**: <any data loss or corruption>
- **SLA impact**: <any SLA breaches>

## Timeline

All times in <timezone>.

| Time  | Event                                          |
|-------|------------------------------------------------|
| 14:00 | Deploy v2.3.1 rolled out to production         |
| 14:05 | Error rate alert fires                         |
| 14:08 | IC acknowledges, begins investigation           |
| 14:15 | Root cause identified: missing DB index         |
| 14:20 | Hotfix deployed                                |
| 14:25 | Error rate returns to baseline                  |
| 14:40 | Incident marked resolved after monitoring       |

## Contributing Factors

1. **<Factor>**: <explanation of how this contributed>
2. **<Factor>**: <explanation>

## What Went Well

- <positive observation about the response>
- <another positive>

## What Could Be Improved

- <gap in the response process>
- <missing detection or tooling>

## Action Items

| ID  | Action                                    | Owner  | Due Date   | Status  |
|-----|-------------------------------------------|--------|------------|---------|
| AI1 | Add DB index for users.lookup_by_email    | @eng   | 2024-02-01 | Done    |
| AI2 | Add load test for auth service             | @eng   | 2024-02-15 | Open    |
| AI3 | Improve deploy canary to catch latency spikes | @infra | 2024-02-15 | Open |

## Lessons Learned

<1-2 paragraphs on systemic takeaways that apply beyond this specific incident>

Examples

User: "Production is returning 500 errors on the checkout endpoint" Action: Triage the incident: assess severity (SEV-2 if checkout is broken for all users), suggest immediate actions (check recent deploys, look at error logs, assess rollback), and produce a triage document.

User: "Write a postmortem for last Tuesday's database outage" Action: Ask for the timeline, logs, and resolution details. Structure a blameless postmortem with contributing factors, what went well, improvements, and action items.

User: "Draft a status update for the ongoing API latency issue" Action: Produce a concise status update using the template, including current impact, what is being done, and when the next update will be.

Best Practices

• Blameless postmortems are non-negotiable. Focus on systems, processes, and tooling, not individuals. "Why did the system allow this?" instead of "Why did <person> do this?"
• Start the timeline immediately when an incident begins. Reconstructing it later is unreliable and time-consuming.
• Assign clear roles: Incident Commander, communicator, investigators. Uncoordinated response wastes time.
• Communicate early and often, even when you don't have answers. "We are investigating and will update in 15 minutes" is better than silence.
• Rollback first, investigate later. If a rollback can safely resolve the issue, do it. Root cause analysis can happen after users are unblocked.
• Action items must have owners and due dates. "We should improve monitoring" is not an action item. "Add p99 latency alert for auth service, owned by @team, due Feb 15" is.
• Review past postmortems before writing a new one. Repeated incidents with the same contributing factors indicate systemic issues.
• Keep postmortems concise. A 10-page document won't be read. A 2-page document with clear action items will drive change.