agentleFS
hiteshbandhuhiteshbandhu/skills-i-use/skills/chat-failure-audit/SKILL.md

chat-failure-audit

Skill1 starsChanged 5 months agoMITView on GitHub

Audit a real user transcript or agent session for failure modes — extract what went wrong, map each symptom to its root cause and the system layer that owns it, and rank by frequency × severity. Distinguishes symptom from cause (e.g. "it worked on the next turn" → root cause is activation not happening in-turn, not "the model is bad") and classifies each failure (capability gap, friction, state/lifetime bug, grounding/ confabulation, recovery gap, UX dead-end). Triggers on "analyse this chat", "what went wrong here", "audit this session", "why did the agent fail", "find the failure modes", "look at this transcript", "where did this break", "review this conversation". Use when you have a real session/log and want grounded failure analysis, not guesses.

The file

We split this skill into the rules it gives an agent

5 rules. About 0 are common, 4 are distinctive, and 1 is very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈4

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 1

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Do not analyse from a summary or from memory — read the actual turns.

    line 40
  2. Distinctive:

    Read the session turn by turn.

    line 49
  3. Distinctive:

    Read failure-classes.md. For each breakdown, push past the symptom:

    line 58
  4. Very specific:

    Write the report from audit-template.md to the output dir and update index.md.

    line 89
  5. Distinctive:

    Do not propose code fixes here — this skill diagnoses.

    line 97

≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Chat Failure Audit

Turn a real transcript into a ranked, root-caused failure list. The discipline: ground every finding in the actual session, and always push past the symptom to the cause and the layer that owns it.

Works with any agent. No vendor APIs. Reads the transcript/log you provide.

Supporting files (read when needed):

  • failure-classes.md — the six failure classes, symptom→cause prompts, severity rubric
  • audit-template.md — the failure-mode report format

Output: {SKILL_OUTPUT_DIR}/chat-failure-audit/ — see ../OUTPUT.md


Step 0 — Get the real session

Establish the source (ask only if missing):

  1. The transcript — a pasted log, a session file, an exported conversation, or a URL the agent can fetch. If it's behind auth and unfetchable, ask the user to paste it.
  2. What "success" was — what the user was trying to get done. Failures only mean something against an intended outcome.

Do not analyse from a summary or from memory — read the actual turns. If you can't see the real session, say so rather than inventing failures.

Resolve output dir per ../OUTPUT.md. Default ./skill-outputs/chat-failure-audit/.


Step 1 — Walk the turns, mark every breakdown

Read the session turn by turn. Mark each point where the outcome diverged from what the user wanted: a wrong answer, a stall, a re-ask, a tool that failed, a thing the user had to do twice, an abandonment. Capture the observable symptom verbatim — the actual message or behavior, not your paraphrase.


Step 2 — Symptom → root cause → layer

Read failure-classes.md. For each breakdown, push past the symptom:

Ask "why" until you hit a cause the system owns. "It worked on the next turn" is a symptom; the cause is "activation happens between turns, not within the turn." "The model gave a wrong number" may be a grounding failure (bad retrieval), not a model failure.

Record: symptom (observed) → root cause (the system behavior that produced it) → layer (which part owns the fix: prompt, retrieval, tool/transport, state/lifetime, UX, model choice).


Step 3 — Classify and rate

Tag each with one failure class:

  • Capability gap — the system genuinely can't do the thing yet.
  • Friction — it can, but made the user do avoidable work (→ hand to friction-audit).
  • State / lifetime bug — state was kept at the wrong scope or dropped across a boundary.
  • Grounding / confabulation — answered from nothing / stale / wrong source.
  • Recovery gap — a transient/recoverable error surfaced as a dead-end.
  • UX dead-end — failed or emptied without offering a next step.

Rate frequency (one-off / recurring / every-time) and severity (annoyance / blocks-task / wrong-and-trusted). Wrong-and-trusted outranks everything.


Step 4 — Output to user

Write the report from audit-template.md to the output dir and update index.md. Then in chat:

  1. The failure table: symptom → root cause → layer → class → freq × severity.
  2. The top failures to fix first, by frequency × severity (not by order of appearance).
  3. Explicit handoffs: friction → friction-audit; new/changed state → state-lifetime-decision; protocol/API mismatch → spec-grounded-design; big fork → architecture-review.
  4. Do not propose code fixes here — this skill diagnoses; design/implementation is a separate, confirmed step.

Edge cases

  • Only a summary, no raw turns — ask for the real transcript; flag that confidence is low without it.
  • One symptom, many causes — list each cause separately; they may need different fixes.
  • The "failure" is the user's intent being unclear — that's a real finding (a clarify gap), not a model failure; classify as UX dead-end or capability gap.
  • Everything blamed on "the model" — push harder; most "model failures" are grounding, state, or prompt failures the system owns and can fix.
  • Sensitive data in the transcript — never copy secrets/PII into the saved report; redact to the behavior.

Invocation examples

@chat-failure-audit analyse this chat: <url or pasted log>
what went wrong in this session — the user gave up at the end
find the failure modes in this agent run
why did this break? push past "the model was wrong"
audit this transcript and rank what to fix first

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.