Runbook

Structured incident investigation: symptom intake, triage, investigation, resolution, documentation. Every step collects evidence.

Core principle: Collect evidence before forming hypotheses. Do not fix what you have not diagnosed.

When to Use

• Production alert fires and needs investigation
• Service outage or degraded performance
• Error rate spike or new error pattern
• User-reported issue with unclear cause
• Post-deploy regression

Not for: planned maintenance, feature development, or issues with obvious root cause already known.

Process

1. Intake    — Capture symptoms, timeline, blast radius
2. Triage    — Severity assessment, who to notify, immediate mitigation
3. Investigate — Systematic evidence gathering, hypothesis testing
4. Resolve   — Apply fix, verify, monitor
5. Document  — Post-incident report with timeline and learnings

1. Intake

Capture before investigating:

• What — exact symptoms, error messages, affected endpoints
• When — first occurrence, pattern (constant, intermittent, escalating)
• Where — which services, environments, regions
• Who — affected users/systems, scope estimate
• What changed — recent deploys, config changes, infrastructure changes

# Quick checks to establish timeline
git log --oneline --since="2 hours ago"   # recent deploys
# Check monitoring dashboards, alert history
# Check infrastructure change logs

2. Triage

Immediate mitigation options (before root cause is known):

• Rollback last deploy
• Scale up / restart affected services
• Enable feature flag / circuit breaker
• Redirect traffic away from affected component

3. Investigate

Systematic, not ad hoc. Work from evidence, not hunches.

Evidence gathering order:

1. Logs around incident start time (correlate across services)
2. Metrics — error rates, latency, resource usage, queue depths
3. Recent changes — deploys, config, infrastructure
4. Dependencies — upstream/downstream service health
5. Reproduce — can you trigger the issue in a controlled way?

Hypothesis testing: Form hypothesis, predict what evidence would confirm or refute, check that evidence. Do not confirmation-bias toward the first theory.

Use arsyn:debugging for deep root cause analysis when the surface investigation identifies the failing component but not the underlying cause.

4. Resolve

• Apply targeted fix (not a workaround unless time-critical)
• Verify fix in production — monitor for 15-30 min
• Confirm affected metrics return to baseline
• Remove any temporary mitigations applied in triage

5. Document

Write post-incident report. See references/incident-template.md for format.

Key sections: timeline, root cause, impact, resolution, action items. The report serves future investigations — be specific about what worked and what did not during diagnosis.

Quick Reference

Integration

• arsyn:debugging — deep root cause tracing when surface investigation is not enough
• arsyn:verifying — verify fix meets requirements before closing
• arsyn:precision — careful execution when applying production fixes

Gotchas

• Fixing before diagnosing — Applying a "likely fix" without confirming root cause often masks the real issue or introduces new ones. Collect evidence first.
• Tunnel vision on first hypothesis — The most obvious explanation is not always correct. Actively seek disconfirming evidence.
• Skipping documentation — "We'll write it up later" means it never gets written. Capture timeline during investigation, not after.
• Incomplete blast radius assessment — Check downstream consumers, async jobs, and cache layers. Visible symptoms are often a subset of actual impact.
• Premature all-clear — Monitor for at least 15 minutes after fix. Some issues resurface under load or on the next cron cycle.
• Not checking what changed — Most production incidents correlate with a recent change. Always check deploy history and config changes early.