agentleFS
ChariereFiedlerChariereFiedler/ronce-racine/skills/production-incident-diagnostic/SKILL.md

production-incident-diagnostic

Skill9 starsChanged 25 days agoMITView on GitHub

Use when diagnosing a production incident - "prod is down", "la prod est down", an API responding 502/500/slow, abnormal behavior in prod, a user reporting something broken in production, alerting gone silent. Triggers on localizing the faulty layer of a live system.

The file

We split this skill into the rules it gives an agent

7 rules. About 0 are common, 2 are distinctive, and 5 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈2

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 5

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    This skill: when: a production incident - prod down, API misbehaving (502/503/500/slow), abnormal behavior in prod, user report, silent …

    line 16
  2. Very specific:

    ci-pipeline-orchestration: instead if: the CI pipeline is red but prod runs normally (build/release problem, not a live incident)

    line 17
  3. Very specific:

    recurring-bug-root-cause: instead if: a bug that recurs (2+ similar fixes) - class hunting, not a live incident

    line 18
  4. Very specific:

    detection-sweep: instead if: hunting static code problems across the whole project, outside an incident

    line 19
  5. Very specific:

    Look for: crash/panic/exception, OOM, "too many connections", exhausted descriptors, external dependency errors

    line 58
  6. Distinctive:

    Should an alert have fired?

    line 62
  7. Distinctive:

    If distributed tracing available: follow a faulty request end-to-end to locate the slow/failed hop

    line 65

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Production incident triage - from symptom to faulty layer

If the current repo has a specific prod-diagnostic skill, it wins - it knows the hosting, the observability tool, the endpoints and the project tracker.

This skill vs. others

  • This skill when: a production incident - prod down, API misbehaving (502/503/500/slow), abnormal behavior in prod, user report, silent alerting
  • ci-pipeline-orchestration instead if: the CI pipeline is red but prod runs normally (build/release problem, not a live incident)
  • recurring-bug-root-cause instead if: a bug that recurs (2+ similar fixes) - class hunting, not a live incident
  • detection-sweep instead if: hunting static code problems across the whole project, outside an incident

Quick rule: prod reachable but misbehaving → this skill; pipeline red, prod OK → ci-pipeline-orchestration.

Principle

Isolate the layer before fixing. An incident is traversed layer by layer - hosting/deployment → backend/API → alerting → frontend - until the faulty one is located. Evidence before conclusion: never "it's probably X" without a log, metric, trace or reproduction. The output = faulty layer + supported root cause + mitigation, not a blind restart.

Context to gather (before acting)

  • Scope: is everything affected, or one endpoint / one feature / a subset of users?
  • Since when: timestamp of the first symptom; does it correlate with a deployment, a migration, a traffic spike?
  • Suspected layer from the symptom (see triage table)
  • Access: health endpoint URLs, access to logs / metrics / traces (observability tool, dashboard), status of the last deployment, read-only DB access. If none of it is reachable (no dashboard, no shell, no endpoint), say so explicitly and run the diagnosis on the artefacts at hand - pasted logs, run history, the repository and its git history. A diagnosis without live access is a normal outcome, not a reason to stop; what it is not allowed to do is claim observations it could not make.
  • Project conventions: deploy/rollback commands, tracker for the incident ticket

Protocol

1. Quick triage - categorize the symptom

  • Reproduce the symptom (curl the endpoint, user journey) and note the exact code/behavior
  • Hit the health endpoint (liveness/readiness) if exposed
  • Without live access: reproduce locally if the failing thing can be run here; otherwise state that no reproduction was possible and delimit the scope from the artefacts (which runs failed, since when, what is still serving)
SymptomLikely layer
502/503Hosting / process down / reverse proxy
Health 200 but 500 on endpointsPartial backend (DB, external dependency)
Abnormal latency, timeoutsSaturation (DB pool, CPU, queue)
Blocked job/workerCrashed worker or deadlock
Silent alertingChannel config / rate limiting
Blank page, asset 404sCorrupted frontend build/deploy, cache/CDN
Errors specific to one userAuth / permissions / data

2. Check deployment & version

  • What version is actually running in prod? Compare to the expected commit (a "deploy success" serving the old image is a classic)
  • Last deployment / migration: timing vs incident start → rollback candidate

3. Logs

  • Filter the observability tool on the incident window and the suspected layer
  • Look for: crash/panic/exception, OOM, "too many connections", exhausted descriptors, external dependency errors

4. Metrics & alerts

  • Dashboard: CPU/memory, error rate, p95/p99 latency, connection pool saturation, queue depth
  • Should an alert have fired? If not, an observability gap to track too

5. Traces

  • If distributed tracing available: follow a faulty request end-to-end to locate the slow/failed hop

6. Layer bisection

  • Confirm/rule out each layer with evidence (health responds → process up; direct DB query responds → DB up). Descend to the layer that faults.

7. Root cause + mitigation

  • State the supported root cause (log/trace/repro). Decide mitigation: rollback (fast, reversible) vs forward-fix - when in doubt, ask the user
  • Any destructive action in prod (UPDATE/DELETE, restart, purge): explicit user confirmation, read-only access by default

Templates

  • templates/incident-report.md - incident report (timeline, layer, evidence, root cause, mitigation, follow-up)

Traps & rationalizations

ExcuseReality
"I'll restart, it'll come back"A restart without diagnosis erases the evidence and hides a cause that will return. Capture logs/metrics first.
"The symptom is gone, it's fixed"Fixing a symptom without a root cause = an incident that replays. No closure without a supported cause.
"It's probably the DB / the network"Unproven hypothesis. One piece of evidence per layer, otherwise you treat the wrong one.
"The deploy is green, prod is up to date"Verify the version actually served - a green deploy can serve the old image.
"I'll write up the incident later""Later" never comes. Report + ticket before closure.

Exit condition

  • Symptom reproduced and scope delimited (no live access → reproduced locally, or the absence of a reproduction stated and the scope established from the artefacts)
  • Faulty layer isolated with at least one piece of evidence (log, metric, trace or repro)
  • Root cause documented - never an unproven hypothesis
  • Mitigation applied or planned; destructive actions confirmed by the user
  • Prod re-checked: health endpoint OK and symptom gone (output pasted) - no live access → say so, and name the check whoever has access must run before the incident is closed
  • Incident report written + ticket created if prod was impacted

Tooling

  • templates/incident-report.md - report template to fill in
  • Generic tools: curl -v/-I on health endpoints, query the observability dashboard over the window, read-only DB query, trace following

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.