incident-investigation
Skill0 starsChanged 3 months agoMITView on GitHub
Use when investigating a production incident, anomaly, or error using observability tooling (Grafana, Sentry, Datadog, New Relic — via MCP or a plain API key) — from first symptom to a confirmed, evidence-backed root cause. Trigger on symptoms like an error-rate spike, latency increase, a failed alert, OOM/restart loops, or "something broke in production but I don't know where or when.
2 other projects use a version of this.
Adopted by
Most recently taken up by
- Rktmkalita/claude-skillstheir own version, first seen Nov 1, 2026
- SCStelz/security-investigatortheir own version, first seen Oct 8, 2026
Other versions2 projects use a version of this.
Word for word · 0
None found.
Edited · 0
None found.
Wrote their own version · 2
- SCStelz/security-investigator★ 252same name
- Rktmkalita/claude-skills★ 0same name
How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.
The file
We split this skill into the rules it gives an agent
3 rules. About 0 are common, 3 are distinctive, and 0 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈3
Distinctive · estimate
This author's own choices. Worth a look.
0
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 30
Before querying anything, get what the reporter already knows — don't start blind when a human already has half the evidence.
- Distinctive: line 92
If a deploy/annotation lines up with the incident start, that's a *lead*, not proof.
- Distinctive: line 120
If you catch yourself thinking:
≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Incident Investigation
Overview
This is a purely investigative skill: gather evidence from observability tooling, cross-reference it against the actual codebase, and arrive at a confirmed root cause backed by that evidence. It does not write tests, run code, implement a fix, or deploy anything — the deliverable is a root cause report with its evidence trail, handed off to whatever fixes code next.
Core principle: monitoring answers questions you already knew to ask (known-unknowns); observability lets you ask a question you didn't pre-build an alert for (unknown-unknowns). Most real incidents worth investigating are unknown-unknowns — reaching for the one dashboard you already have is monitoring behavior applied to a problem that needs observability.
The Iron Law
NO ROOT CAUSE CLAIM WITHOUT VERIFICATION AGAINST REAL EVIDENCEA hypothesis is not a conclusion. Before stating a root cause, cross-check it against the observability evidence and the actual code — both have to agree, not just sound plausible together.
Requires
Works with an MCP server for Grafana, Sentry, Datadog, or New Relic (all ship official servers as of 2026) — or with just an API key/token for any of these, no MCP needed. If you have neither an MCP connection nor an API key for the platform in question, say that explicitly rather than guessing what a dashboard "probably" shows.
Discover before using. Before calling any platform tool by name, check what's actually connected and available in this session — call the MCP server's own tool listing (list_tools or equivalent) rather than assuming. Tool names in this skill and in references/tool-reference.md are examples of the kind of capability to look for (query a time-series metric, search issues by error signature, correlate a deploy against an incident) — verified as of 2026, but these are actively evolving vendor products; the exact name may have changed since. Match by what the tool does, confirmed against what's actually available right now, not by pattern-matching the name written here.
Intake — Understand the Report First
Before querying anything, get what the reporter already knows — don't start blind when a human already has half the evidence. Ask (or extract from what was already said):
- What was observed, in their words — an error message, a screenshot, a customer complaint, an alert notification
- Since when — an exact time if they have one, otherwise "since roughly when" (this seeds the incident window for every path below)
- Scope — one user, one customer, one region, or everyone? A specific endpoint/service, or "the whole app"?
- What's already been tried — a restart, a rollback, a config change? If someone already acted, that action itself is now part of the timeline, not a clean baseline
- Any evidence already in hand — a log line, a trace ID, a request ID, a stack trace pasted into chat. Use it directly instead of re-deriving it from scratch.
A vague report ("it's broken") is still enough to start — go to Path D. Don't block on a perfect report; the paths below are exactly how you fill in what's missing. But don't skip asking, either — a report that already names the service and the approximate time turns a long investigation into a short one.
Triage by Primary Symptom
Pick the path that matches what was reported. Don't run every path — the symptom tells you where to start.
| Symptom | Start at |
| Error rate / exception spike, alert fired on errors | Path A |
| Slow endpoint, elevated latency, "it's slow" | Path B |
| OOM kill, restart loop, CPU/memory exhaustion | Path C |
| Vague report, no specific symptom yet | Path D |
Path A — Error Rate Spike
- Confirm the window and blast radius.
- Grafana:
query_prometheuswithrate(http_requests_total{status=~"5.."}[5m])— orcurl /api/ds/query(see tool-reference.md) - Sentry:
search_issuesfiltered to the suspected window — orcurl .../issues/?statsPeriod=24h - Datadog:
search_datadog_error_tracking_issues— orcurl /api/v2/logs/events/searchwithstatus:error - New Relic:
list_recent_issuesorexecute_nrql_querywithSELECT count(*) FROM TransactionError SINCE ...
- Grafana:
- Find the error pattern, not just the count. Grafana's Sift tool
find_error_pattern_logsgroups recurring error signatures automatically — built specifically for this (Grafana Cloud; not confirmed on self-hosted, see tool-reference.md). Sentry:get_event_stacktraceandget_issue_breadcrumbsfor the full picture, oranalyze_issue_with_seerto let Sentry's own AI triage it first. Datadog:analyze_datadog_logs. New Relic:analyze_entity_logsorlist_entity_error_groups. - Continue to Reconstruct the Timeline below once you have: the error signature, the exact time it started, and which service/endpoint it's on.
Path B — Latency / Slow Endpoint
- Confirm which service, not just "it's slow." RED per service (rate, errors, duration) narrows to the offending service before you look inside it.
- Find the slow spans. Grafana Sift:
find_slow_requests. Datadog:apm_latency_bottleneck_summaryorapm_query_trace. Sentry:get_trace_detailsandget_span_details. New Relic:analyze_transactionsoranalyze_golden_metrics. - Check the resource behind it (Path C, step 1) — a slow endpoint is often a starved resource wearing a latency costume.
- Continue to Reconstruct the Timeline once you have: the specific slow span (DB call, external API, lock wait) and when it started being slow, not just that it currently is.
Path C — Resource Exhaustion (OOM, Restarts, CPU)
- Pull the USE signals for the resource (utilization, saturation, errors) — CPU/memory over the suspected window. Grafana:
query_prometheusfor the container/pod metrics. Datadog:get_datadog_metric. New Relic:analyze_golden_metrics. - Correlate with restart/kill events, not just the metric shape —
get_annotations(Grafana),list_change_events(New Relic), or the platform's own event log often marks the OOM kill or restart directly. - Continue to Reconstruct the Timeline once you have: the resource that exhausted, the exact time, and whether it's a leak (climbs steadily) or a spike (sudden jump tied to one request/deploy).
Path D — No Specific Symptom Yet
- Sweep the Four Golden Signals (latency, traffic, errors, saturation) across the suspected time range first — this tells you which of Paths A/B/C actually applies before you commit to one.
- Once one signal stands out, jump to the matching path above.
When the Symptom and the Cause Are in Different Services
Any path above can dead-end at "this service is slow/erroring, but its own code looks fine" — that's the signal to check whether the real cause is downstream, in a service this one calls.
- Follow the trace ID across the service boundary, not just within one service. A distributed trace is a tree: a shared trace ID spans every service touched by one request, and each operation is a span with a parent-child link to the span that called it (Grafana Tempo, Sentry's
get_trace_details/get_span_details, Datadog'sapm_query_trace, and New Relic's distributed tracing all expose this). The parent-child chain across service boundaries is what tells you where in the call chain the slowness or error actually started, instead of just confirming it was visible where you first looked. - Use the platform's dependency-graph tool if one is connected, rather than manually chasing calls service by service — Datadog's
search_datadog_service_dependenciesand New Relic'slist_related_entitiesexist specifically for this. - Propagation only works if every hop actually forwards the trace context (commonly the
traceparentHTTP header, per the W3C Trace Context standard) — if a service in the chain doesn't propagate it, the trace breaks there, and that break point is itself useful evidence: it's either where instrumentation is missing, or a real place to keep looking manually. Seereferences/methodology.md#7.
Reconstruct the Timeline
Applies after any path above, before moving to root cause:
- Write it down as you go — what you queried, what it showed, when. Not from memory afterward.
- If a deploy/annotation lines up with the incident start, that's a lead, not proof. New Relic's
analyze_deployment_impactand Sentry'sfind_releasescorrelate this directly instead of you eyeballing two timestamps — use them before concluding a deploy caused it. - If timestamps from two different tools don't quite agree, that's expected, not necessarily a red flag by itself — independent systems don't share a clock, and logs/metrics/traces are typically ingested with different delay. Prefer a shared trace ID to line events up across tools when one is available, rather than trying to reconcile wall-clock timestamps down to the second. See
references/methodology.md#8. - Report what you actually observed, including your own missteps in the investigation so far — an edited timeline is a false one, and the next person investigating a similar incident reads this one first.
Root Cause: From Evidence to Code
The observability evidence above tells you that something broke and roughly where. This section finds why, by reading — not running — the code.
- Trace the evidence to an actual code path. The stack trace, span, or log line points at specific code — open it and read the whole relevant path, not just the line the error message names. A symptom two layers deep usually has its trigger further up the call chain, not at the line that finally threw.
- Check what changed in that code path, not just "was there a deploy" —
git log/git diffon the specific file(s) the evidence points to, cross-referenced against the timeline from the step above. - Form exactly one hypothesis, stated plainly: "I think
<X>is the root cause, because<evidence>shows<Y>." Not a list of three plausible guesses — one, falsifiable, tied to a specific piece of evidence gathered above.
Verify the Hypothesis Against the Evidence
The proof (or disproof) comes entirely from the data the investigation already pulled — the logs, traces, and metrics gathered in the paths above. Not a new test, not an experiment against local or production environments, not synthetic input: the same real data already sitting in front of you, read against the same real code.
- Walk the code path by reading it, using the exact conditions the evidence showed (the input, the record state, the timing) — does this logic, given those conditions, actually produce the observed symptom? If you have to imagine an input the evidence didn't show to make it fail, the hypothesis isn't confirmed yet.
- Check the timing lines up — the suspected code change (commit/deploy) happened before the incident window started, not after or so long before that something else is more likely.
- Check the error shape matches — does the exception type, the log message, the failed assertion in the code actually match what the observability evidence recorded? A hypothesis that requires explaining away a mismatch is wrong, not almost-right.
- If it doesn't hold up, go back to Root Cause step 1 with what this check revealed — don't force the hypothesis to fit by ignoring the part of the evidence that disagrees with it.
- State the confirmed root cause with its evidence trail — the specific log lines/trace IDs, the specific file and lines of code, the specific commit if one's implicated. This is the primary deliverable (optionally followed by Five Whys, below). Implementing and testing the fix is a separate step, outside this skill, using whatever process the codebase normally uses.
Push Past the Code Cause (Five Whys)
Once the root cause is confirmed, ask why one layer further: not "what line caused this" but "what let this reach production at all" — a missing test category, a review process gap, an alert that should have fired earlier and didn't. Toyota's Five Whys technique (Taiichi Ohno) treats this as the productive part of the exercise, not an afterthought — going to the gemba, the actual place the failure happened, rather than guessing from a meeting room. See references/methodology.md #5.
Red Flags — Stop and Gather Evidence
If you catch yourself thinking:
- "It's obviously the deploy, let's just say that's it" (without checking the diff or the error signature)
- "I already have a dashboard for this" (checking the one dashboard you have instead of asking the system a new question)
- "The error message says enough, no need for the trace/stack"
- "I'll write up the timeline after — I remember what happened"
- "The hypothesis sounds right, no need to check it against the actual code"
- "Let's bump the resource limit and see if it helps" (treating a symptom as if it were the cause)
- "The rollback fixed it, so that proves the deploy was the cause" (consistent with it, not proof of it)
- "Close enough" when the error shape or timing doesn't quite match the hypothesis
All of these mean: STOP. Go back to gathering evidence, or to verifying the hypothesis against it.
Common Rationalizations
| Excuse | Reality |
| "It's obviously the deploy" | Obvious ≠ verified — confirm the error signature actually matches the deploy's diff |
| "No time to write the timeline" | An untracked investigation gets redone from scratch by the next person who hits this same incident |
| "My dashboard shows nothing wrong" | One dashboard is monitoring, not observability — ask the system a new question instead of trusting the one you already built |
| "The hypothesis is plausible, that's enough" | Plausible ≠ verified — walk the actual code against the actual evidence before calling it root cause |
| "The rollback fixed it, that proves the cause" | Consistent with the deploy being the cause, not proof by itself — still confirm the error signature matches the diff |
| "The timing is close enough" | A few minutes off can mean it's the wrong commit — check it, don't eyeball it |
Quick Reference
| Phase | Key Activities | Success Criteria |
| Intake | Get what the reporter already knows: symptom, since when, scope, what's been tried | Have a starting time window and scope before querying anything |
| Triage | Match symptom to Path A/B/C/D | Correct path chosen without running all four |
| Evidence Gathering | Query metrics/logs/traces via MCP or curl | Exact error signature, service, and time window known |
| Cross-Service Check | Follow the trace ID / dependency graph if the service's own code looks clean | Confirmed the cause is in this service, or found which one it's actually in |
| Reconstruct Timeline | Log queries and findings as you go; check deploy correlation with the platform's own tool | Timeline written before the hypothesis is formed, not after |
| Root Cause | Trace evidence to code, check the diff, form exactly one hypothesis | Hypothesis stated with the specific evidence behind it |
| Verify Hypothesis | Cross-check the hypothesis against the code and the evidence, by reading — no execution | Error shape and timing both confirmed to match, or hypothesis revised |
| Five Whys | Push past the code cause to the process cause | Answer names a process gap, not just a line of code |
Common Causes & Solutions
| Symptom | Confirm with evidence | Don't jump straight to |
| Error spike right after a deploy | Error signature matches the deploy's diff; timing lines up | Assuming correlation = causation without checking either |
| Latency spike, DB-adjacent | Slow span points at a specific query/lock, not just "the DB" | Concluding "the DB is slow" without the specific query |
| OOM kill | Memory climbs steadily (leak) vs. jumps once (one bad request) — different root causes | Assuming it's a leak without checking the shape of the metric |
| Error rate normal, but users still report issues | Check a narrower segment (region, plan tier, feature flag) — aggregate signals can hide a localized problem | Concluding "nothing's wrong" from an aggregate dashboard |
Reference
- references/tool-reference.md — tool names seen per platform as of 2026 (confirm against
list_toolsbefore relying on them — see "Discover before using" above) +curlfallback for API-key-only access - references/methodology.md — sourcing for the underlying method: Google SRE Book, Charity Majors, PagerDuty, Toyota's Five Whys, John Allspaw/Etsy, Google's Dapper paper, W3C Trace Context, Leslie Lamport
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

