chat-failure-audit
Skill1 starsChanged 5 months agoMITView on GitHub
Audit a real user transcript or agent session for failure modes — extract what went wrong, map each symptom to its root cause and the system layer that owns it, and rank by frequency × severity. Distinguishes symptom from cause (e.g. "it worked on the next turn" → root cause is activation not happening in-turn, not "the model is bad") and classifies each failure (capability gap, friction, state/lifetime bug, grounding/ confabulation, recovery gap, UX dead-end). Triggers on "analyse this chat", "what went wrong here", "audit this session", "why did the agent fail", "find the failure modes", "look at this transcript", "where did this break", "review this conversation". Use when you have a real session/log and want grounded failure analysis, not guesses.
In this repository51 other files it gives its agents.
- architect-enterprise-aiSkill
- author-agent-skillsSkill
- build-agent-harnessesSkill
- build-aiewf-2024-ragSkill
- build-generative-mediaSkill
- build-multimodal-productsSkill
- build-rag-search-stacksSkill
- build-robotics-aiSkill
- build-with-deepmindSkill
- build-with-microsoft-aiSkill
- build-with-openaiSkill
- deploy-public-sector-aiSkill
- design-agent-skillsSkill
- evaluate-with-braintrustSkill
- lead-ai-engineering-teamsSkill
- lead-ai-transformationSkill
- learn-summit-2023-remoteSkill
- learn-summit-2023-talksSkill
- navigate-aie-europe-2026Skill
- navigate-aiewf-2024-keynotesSkill
- observe-ai-productionSkill
- operate-agent-memorySkill
- operate-legal-ai-agentsSkill
- operate-openclaw-agentsSkill
- optimize-llm-inferenceSkill
- plan-sovereign-aiSkill
- price-ai-productsSkill
- run-aiewf-2024-evalsSkill
- run-llm-evalsSkill
- run-summit-2023-workshopsSkill
- secure-ai-agentsSkill
- ship-ai-as-pmSkill
- ship-aiewf-2024-agentsSkill
- train-llms-locallySkill
- architecture-reviewSkill
- create-a-skillSkill
- cross-model-reviewSkill
- cto-weekly-reviewSkill
- friction-auditSkill
- ig-carouselSkill
- pdf-reportSkill
- raise-prSkill
- reel-editSkill
- research-reportSkill
- ship-checkSkill
- spec-grounded-designSkill
- state-lifetime-decisionSkill
- ste100Skill
- scroll-engineeringSkill
- ui-uxSkill
- verify-like-a-userSkill
The file
We split this skill into the rules it gives an agent
5 rules. About 0 are common, 4 are distinctive, and 1 is very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈4
Distinctive · estimate
This author's own choices. Worth a look.
1
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 40
Do not analyse from a summary or from memory — read the actual turns.
- Distinctive: line 49
Read the session turn by turn.
- Distinctive: line 58
Read failure-classes.md. For each breakdown, push past the symptom:
- Very specific: line 89
Write the report from audit-template.md to the output dir and update index.md.
- Distinctive: line 97
Do not propose code fixes here — this skill diagnoses.
≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Chat Failure Audit
Turn a real transcript into a ranked, root-caused failure list. The discipline: ground every finding in the actual session, and always push past the symptom to the cause and the layer that owns it.
Works with any agent. No vendor APIs. Reads the transcript/log you provide.
Supporting files (read when needed):
- failure-classes.md — the six failure classes, symptom→cause prompts, severity rubric
- audit-template.md — the failure-mode report format
Output: {SKILL_OUTPUT_DIR}/chat-failure-audit/ — see ../OUTPUT.md
Step 0 — Get the real session
Establish the source (ask only if missing):
- The transcript — a pasted log, a session file, an exported conversation, or a URL the agent can fetch. If it's behind auth and unfetchable, ask the user to paste it.
- What "success" was — what the user was trying to get done. Failures only mean something against an intended outcome.
Do not analyse from a summary or from memory — read the actual turns. If you can't see the real session, say so rather than inventing failures.
Resolve output dir per ../OUTPUT.md. Default ./skill-outputs/chat-failure-audit/.
Step 1 — Walk the turns, mark every breakdown
Read the session turn by turn. Mark each point where the outcome diverged from what the user wanted: a wrong answer, a stall, a re-ask, a tool that failed, a thing the user had to do twice, an abandonment. Capture the observable symptom verbatim — the actual message or behavior, not your paraphrase.
Step 2 — Symptom → root cause → layer
Read failure-classes.md. For each breakdown, push past the symptom:
Ask "why" until you hit a cause the system owns. "It worked on the next turn" is a symptom; the cause is "activation happens between turns, not within the turn." "The model gave a wrong number" may be a grounding failure (bad retrieval), not a model failure.
Record: symptom (observed) → root cause (the system behavior that produced it) → layer (which part owns the fix: prompt, retrieval, tool/transport, state/lifetime, UX, model choice).
Step 3 — Classify and rate
Tag each with one failure class:
- Capability gap — the system genuinely can't do the thing yet.
- Friction — it can, but made the user do avoidable work (→ hand to
friction-audit). - State / lifetime bug — state was kept at the wrong scope or dropped across a boundary.
- Grounding / confabulation — answered from nothing / stale / wrong source.
- Recovery gap — a transient/recoverable error surfaced as a dead-end.
- UX dead-end — failed or emptied without offering a next step.
Rate frequency (one-off / recurring / every-time) and severity (annoyance / blocks-task / wrong-and-trusted). Wrong-and-trusted outranks everything.
Step 4 — Output to user
Write the report from audit-template.md to the output dir and update
index.md. Then in chat:
- The failure table: symptom → root cause → layer → class → freq × severity.
- The top failures to fix first, by frequency × severity (not by order of appearance).
- Explicit handoffs: friction →
friction-audit; new/changed state →state-lifetime-decision; protocol/API mismatch →spec-grounded-design; big fork →architecture-review. - Do not propose code fixes here — this skill diagnoses; design/implementation is a separate, confirmed step.
Edge cases
- Only a summary, no raw turns — ask for the real transcript; flag that confidence is low without it.
- One symptom, many causes — list each cause separately; they may need different fixes.
- The "failure" is the user's intent being unclear — that's a real finding (a clarify gap), not a model failure; classify as UX dead-end or capability gap.
- Everything blamed on "the model" — push harder; most "model failures" are grounding, state, or prompt failures the system owns and can fix.
- Sensitive data in the transcript — never copy secrets/PII into the saved report; redact to the behavior.
Invocation examples
@chat-failure-audit analyse this chat: <url or pasted log>
what went wrong in this session — the user gave up at the end
find the failure modes in this agent run
why did this break? push past "the model was wrong"
audit this transcript and rank what to fix firstDoes it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

