agentleFS
nfeuernfeuer/project-foundation-kit/.claude/skills/incident-capture/SKILL.md

incident-capture

Skill0 starsChanged 5 months agoMITView on GitHub

When a component fires fallback_activated (or crashes) repeatedly in a window, auto-open a pre-filled incident note so the 2am page starts with context instead of a blank page.

2 other projects use a version of this.

Adopted by

Most recently taken up by

The file

We split this skill into the rules it gives an agent

6 rules. About 0 are common, 5 are distinctive, and 1 is very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈5

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 1

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    If the count is below the threshold, exit 0 — nothing to open.

    line 47
  2. Distinctive:

    Before writing a new note, check whether an incident for this component is already active.

    line 50
  3. Distinctive:

    Keep this section factual — log facts and timeline, not speculative root-cause filler.

    line 65
  4. Distinctive:

    If the component last changed within the same git session, it is the primary suspect.

    line 92
  5. Very specific:

    Commit on main (ops notes) or a short-lived branch — whichever your project allows.

    line 115
  6. Distinctive:

    If the post fails, log the failure and continue — the note on disk is the source of truth.

    line 132

≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Incident Capture

Profile-driven. Alert destination is alerts.channel / alerts.target in .claude/kit.yaml. The SQLite fallback queries below use the table named by capabilities.llm.spend_table (default: invocation_log); substitute if your project differs.

A single fallback_activated event is expected; three in fifteen minutes is an incident. This skill detects the repetition pattern, gathers the correlated log context and the recent commits touching the component, writes an incident note from templates/incident_note.template.md, and posts a summary to chat. The person who picks up the page gets a filled-in timeline instead of a blank document. See the observability-check skill for the logging conventions this depends on, and templates/fallback_alert.py for the fallback_activated event contract and the FallbackNotifier dedup mechanism.

Workflow

1. Detect the trigger

Define "repeated" as N occurrences of the same event_type for the same component within a rolling window. Sensible defaults: N = 3, window = 15 m. Query Loki, the local structured log table, or the SQLite invocation log — whichever is available:

# Loki (LogQL) — adapt label selectors to your stack
logcli query \
  '{app="donna"} |= "fallback_activated" | json | component="<component>"' \
  --since=15m --output=jsonl \
  | jq -s 'length'

# SQLite fallback — table: capabilities.llm.spend_table (kit.yaml)
sqlite3 donna_tasks.db "
  SELECT COUNT(*) FROM invocation_log
  WHERE event_type = 'fallback_activated'
    AND component  = '<component>'
    AND created_at >= datetime('now', '-15 minutes');"

If the count is below the threshold, exit 0 — nothing to open.

2. Deduplicate — one open incident per component

Before writing a new note, check whether an incident for this component is already active. A sentinel file naming convention keeps the check simple:

mkdir -p docs/incidents/
ls docs/incidents/ 2>/dev/null | grep -q "^${component}-.*-open\.md$" && exit 0

This prevents a flapping component from generating a new note on every poll cycle. When the existing incident is resolved, rename the file (swap -open.md for -resolved.md) to re-arm the trigger.

3. Gather correlated log context

Pull the log lines for the window — include correlation_id, the error, and the fallback taken. These become the "Correlated logs" section of the note. Keep this section factual — log facts and timeline, not speculative root-cause filler. The commands below already cap the excerpt at 20 lines from the 15-minute window:

logcli query \
  '{app="donna"} |= "fallback_activated" | json | component="<component>"' \
  --since=15m --output=jsonl \
  | jq -r '"\(.ts)  [\(.correlation_id // "no-id")]  \(.error)  → fallback: \(.fallback)"' \
  | head -20

# SQLite fallback — table: capabilities.llm.spend_table (kit.yaml)
sqlite3 -csv donna_tasks.db "
  SELECT created_at, correlation_id, error, fallback
  FROM   invocation_log
  WHERE  event_type = 'fallback_activated'
    AND  component  = '<component>'
    AND  created_at >= datetime('now', '-15 minutes')
  ORDER  BY created_at;" | head -20

4. Find the last commits touching the component

# Replace the path with the actual module and config locations
git log --oneline -10 -- "src/<pkg>/<component>/" "config/<component>*.yaml"

If the component last changed within the same git session, it is the primary suspect. These hashes go into the "Related commits" section of the note. This skill documents the incident; it does not revert or patch the primary-suspect commit — that decision belongs to whoever responds.

5. Write the incident note from the template

component="<component>"
slug="${component}-$(date -u +%Y%m%d-%H%M%S)-open"
note="docs/incidents/${slug}.md"

# Populate with envsubst, sed, or a project helper; the template uses
# <PLACEHOLDER> markers — see templates/incident_note.template.md.
INCIDENT_COMPONENT="$component" \
INCIDENT_COUNT="$count"         \
INCIDENT_WINDOW="15m"           \
INCIDENT_TIMESTAMP="$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
  envsubst < templates/incident_note.template.md > "$note"

git add "$note"
git commit -m "incident: open ${component} $(date -u +%Y-%m-%d)"

Commit on main (ops notes) or a short-lived branch — whichever your project allows. Log a follow-up ID in docs/followups.md via the followup-tracking skill if the incident uncovers deferred work.

6. Post to chat

Send a brief alert to the debug/ops channel with a direct link to the note:

# kit.yaml → alerts.channel / alerts.target (destination; swap URL/payload for Slack)
curl -sX POST "$DISCORD_WEBHOOK_URL" \
  -H 'Content-Type: application/json' \
  -d "$(jq -n \
    --arg msg "**Incident opened** · \`${component}\` fired \`fallback_activated\` ${count}× in ${window}
→ \`${note}\`" \
    '{content: $msg}')"

If the post fails, log the failure and continue — the note on disk is the source of truth. Never let a broken notifier prevent the note from being written.

Output

## Incident Capture
- Component:    <name>
- Trigger:      <event_type> × <count> in <window>
- Note:         docs/incidents/<slug>-open.md
- Correlation IDs: <list>
- Related commits: <short hashes + subjects | none in window>
- Chat notified: yes | failed (<reason>)
- Deduped:      no — new note created | yes — existing open incident, skipped

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.