Incident Triage

Stabilise first. Understand second. The order is the whole skill.

Purpose

Use this when:

• A page fires, or someone reports production is broken.
• An outage or significant degradation is in progress.
• Multiple people are looking at a problem and nobody is coordinating.

Do not use this when:

• The system is stable and you are investigating a past event — use production-debugging or postmortem-analysis.
• The problem is contained in a non-production environment. Apply proportionate effort.

What this skill assumes: access to dashboards, deploy history, and the ability to roll back. If you lack any of these, saying so early is itself a useful triage output.

Operating Procedure

Mitigation comes before diagnosis. A user-visible outage is stopped first and understood afterwards. The instinct to find the root cause before acting is correct engineering and wrong incident response — it extends every outage by the length of the investigation.

The first 15 minutes

1. Declare. Say out loud, in the incident channel, that this is an incident and that you are coordinating. Ambiguity about who is driving is the most common cause of a slow response.
2. Establish impact (2 min). Who is affected, how many, doing what, since when. Impact drives severity; severity drives who gets woken up.
3. Check for a recent change (2 min). Deploys, config changes, feature flags, infrastructure changes, certificate expiries, scheduled jobs. Most incidents follow a change.
4. Mitigate (5 min). Roll back, fail over, disable the flag, shed load, scale up. Pick the action with the best ratio of expected relief to blast radius.
5. Verify the mitigation worked (2 min). Against the user-visible symptom, not against a dashboard you like.
6. Then investigate — hand off to production-debugging while someone keeps communicating.

Roles, when more than one person is involved

One person can hold several roles in a small incident. The commander should not be the person with their head in a terminal — that is how the coordination stops.

Everything you state must be labelled: Observed, Assumed, or Recommended. Under pressure, assumptions get repeated until they become facts, and the investigation follows them off a cliff.

Investigation

Step 1 - impact, in one line

Since 14:05 UTC, approximately 30% of checkout requests return 503.
Affects: all customers attempting checkout. Payments are unaffected.

Specifically: what is failing, how much, since when, for whom, and what is confirmed not affected. The last part is the one people skip and the one stakeholders ask about first.

If you cannot state impact, that is the first task — not a reason to skip to debugging.

Step 2 - severity

Err upward. Downgrading a Sev 1 that turned out to be a Sev 3 costs a little goodwill; discovering at hour three that a Sev 3 was always a Sev 1 costs considerably more.

Two escalation triggers that are not about severity: data loss or corruption is always Sev 1 regardless of how few users are affected, and any suspicion of a security breach goes to the security team immediately, before any remediation that might destroy evidence.

Step 3 - what changed

kubectl rollout history deployment/<name> -n <namespace>
kubectl get events -A --sort-by=.lastTimestamp | tail -30
git log --since='4 hours ago' --oneline --all

Check, in order of how often each is the cause:

1. Application deploys in the last few hours.
2. Config or feature-flag changes — these often bypass the deploy pipeline and the change log.
3. Infrastructure changes (Terraform applies, cluster upgrades, node pool changes).
4. Scheduled jobs — a batch job at the top of the hour is a classic.
5. Certificate or credential expiry — check if the start time is suspiciously round.
6. Upstream provider incidents — check their status page.
7. Traffic changes — a marketing campaign, a bot, a retry storm.

Nothing changed on our side is possible but is the least likely explanation, and treating it as the starting hypothesis costs a lot of time. Traffic, time-based expiry and dependency changes are changes too.

Step 4 - choose the mitigation

Rules for choosing:

• Prefer the reversible action. A rollback you can undo beats a clever fix you cannot.
• Change one thing at a time. Three simultaneous changes make the next 30 minutes unreadable — you will not know which helped, and one of them may be making it worse.
• Rolling back is not defeat. It is the fastest mitigation available for the most common cause.
• If data is being corrupted, stop the write path immediately. Availability can be restored; corrupted data often cannot. This is the one case where you accept downtime to mitigate.

Step 5 - communicate

14:12 UTC - Investigating elevated checkout errors. ~30% of checkout attempts failing.
            Payments and browsing unaffected. Next update 14:30.
14:26 UTC - Rolled back checkout-service to v2.4.1. Error rate falling. Monitoring.
            Next update 14:45.
14:41 UTC - Error rate normal since 14:31. Incident mitigated; root cause investigation ongoing.

Update on a stated cadence even when there is nothing new — "still investigating, next update in 15 minutes" prevents the stream of individual questions that otherwise consumes the responder. Always state the next update time, and always state what is not affected.

Decision Tree

Page fires / outage reported
├── Is it real? (check the SLI, not just the alert)
│   ├── No → monitoring problem → prometheus-troubleshooting; still write it up
│   └── Yes → continue
├── Data loss or corruption in progress?
│   └── YES → Sev 1. Stop the write path first. Availability is secondary.
├── Security incident suspected?
│   └── YES → escalate to security before remediating; preserve evidence
├── Did something change in the last few hours?
│   ├── Deploy      → roll back
│   ├── Config/flag → revert it
│   ├── Infra       → assess reverting; may not be reversible
│   └── Nothing obvious → check traffic, dependencies, expiry, scheduled jobs
├── Is it isolated?
│   ├── One zone/node → shift traffic away
│   ├── One tenant    → isolate or rate limit them
│   └── Everywhere    → shared dependency
└── Mitigated?
    ├── Yes → verify, communicate, then investigate → production-debugging
    └── No  → escalate; widen the group; consider a more aggressive mitigation

Common Causes

Safety

Read-only

Reading dashboards, logs, events, deploy history and status pages. Do all of it, freely, while deciding.

Note a real cost: during an incident, an expensive query can degrade the monitoring stack you are depending on. Keep incident queries bounded.

Potentially disruptive

These are mitigations. Each is a deliberate change with a blast radius, and every one must be announced in the channel before it is run so that others can interpret what happens next.

• Rollback — usually the safest and most effective. Confirm the previous version is actually good and that no irreversible migration shipped with the new one. A rollback across a schema migration can be worse than the outage.
• Restart / rollout restart — drops in-flight requests. Sometimes the right call for a leak or a stuck process; frequently a way to destroy evidence for no benefit.
• Scaling — safe upward, subject to quota and to whether the bottleneck is downstream. Scaling a service whose bottleneck is the database adds load to the database.
• Failover — verify the target is healthy first. A failover to a stale replica converts an outage into a data problem.
• Load shedding / rate limiting — deliberately rejects some users to save the rest. Legitimate and underused. Say explicitly who is being rejected.
• Disabling a feature flag — usually reversible and low risk; check for state written while the feature was on.

Destructive

During an incident, under time pressure, is the worst moment to take a destructive action. Each of these requires an explicit decision by the incident commander, stated in the channel, with the blast radius named:

• Failing over a database — may lose unreplicated writes and is frequently one-way. Required first: confirm replication lag, confirm the target's health, name who decided.
• Deleting a persistent volume, a namespace, or a stateful resource — destroys data. During an incident this is almost never the right action; the pressure to "just recreate it" is exactly the pressure to resist.
• Truncating a queue or dropping messages — discards work permanently. Sometimes correct for a poison-message storm. Required first: capture a sample of the messages for the investigation.
• terraform destroy or equivalent — see terraform-safe-changes. Not an incident action.
• Clearing data or caches whose loss shifts load elsewhere — a cold cache after a flush can take the database down as a second, worse incident.

If a destructive action seems necessary, pause and get a second person to agree. Losing thirty seconds to that conversation is cheap.

Verification

Verify the mitigation from the user's side.

Before: 30% of checkout requests 503; p99 12s; 14 customer reports
After:  0.1% error rate sustained 10 min; p99 240ms; no new reports

Check three things:

1. The user-visible symptom is gone, measured where the user is — a synthetic check or edge-level metrics, not an internal dashboard.
2. It stays gone for at least 10 minutes. Many mitigations produce a brief improvement as connections re-establish, then regress.
3. Nothing new broke. Scaling one service can starve another; a rollback can reintroduce a bug that was fixed in the version you just removed.

Only after all three, downgrade the severity and say so explicitly. "Mitigated" and "resolved" are different states — mitigated means the users are fine and the cause is still out there.

Anti-Patterns

Bad: Find the root cause before acting. Better: Mitigate first. Root cause analysis is not more important than the users who are currently unable to use the product, and mitigation almost never destroys the ability to investigate afterwards.

Bad: Everyone jumps into the same terminal and starts making changes. Better: One person makes changes and announces each one. Concurrent changes make the timeline unreadable and can interact badly.

Bad: Restart everything. Better: Sometimes a restart is the right mitigation. Doing it reflexively destroys evidence and often changes nothing — and on a crash-looping service it is what the system is already doing.

Bad: Wait until we know more before updating stakeholders. Better: Regular updates saying "no news yet, next update at X" cost nothing and stop a queue of people asking the responders for status individually.

Bad: Skip the incident record because it was short. Better: Short incidents are the cheapest possible way to learn. The ones that repeat are the ones nobody wrote down.

Bad: Keep debugging alone for 45 minutes because escalating feels like failure. Better: Set a time box at the start. Not escalating is the failure; the cost of one extra person looking is near zero compared with an extra half-hour of outage.

Composition

• production-debugging — after mitigation, to find the actual cause.
• observability-debugging — to scope impact using metrics, logs and traces.
• postmortem-analysis — afterwards, once the system is stable.
• kubernetes-troubleshooting, dns-troubleshooting, postgresql-troubleshooting and the rest — whichever layer the triage points at.

References

• Google SRE Book: Managing Incidents — the role structure used here.
• Google SRE Book: Emergency Response