failure-analysis
Skill2 starsChanged 7 days agoApache-2.0View on GitHub
Triage a failing test — real product defect, or outdated test case after a deliberate product change? Classifies the cause from the JIRA change record and version window, proposes (never applies) test changes. Runs automatically on every FAIL before a bug is filed. Use for "/failure-analysis", "warum ist der Test rot", "is this test outdated". Not for filing the bug — /bug-report after classification.
1 other project uses a version of this.
Adopted by
Most recently taken up by
- feixiaoxu2022/agent-evaluation-skillstheir own version, first seen Jan 13, 2026
Other versions1 project uses a version of this.
Word for word · 0
None found.
Edited · 0
None found.
Wrote their own version · 1
- feixiaoxu2022/agent-evaluation-skills★ 4same name
How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.
In this repository34 other files it gives its agents.
- AGENTS.mdAGENTS.md
- CLAUDE.mdCLAUDE.md
- accessibility-testingSkill
- agentswitchSkill
- api-testingSkill
- bug-reportSkill
- bug-reproductionSkill
- demoSkill
- deploy-to-stageSkill
- exploratory-testingSkill
- model-setupSkill
- onboard-productSkill
- onboard-projectSkill
- perf-analysisSkill
- qa-helpSkill
- qa-metricsSkill
- qa-reportSkill
- re-boardSkill
- reconSkill
- release-readinessSkill
- report-to-jiraSkill
- retest-bugSkill
- retro-learnSkill
- risk-based-testingSkill
- scaffold-suiteSkill
- security-testingSkill
- skill-reviewSkill
- test-automationSkill
- test-case-managementSkill
- test-reliabilitySkill
- test-ticketSkill
- visual-testingSkill
- webdriverio-testingSkill
- whats-newSkill
The file
We split this skill into the rules it gives an agent
4 rules. About 0 are common, 2 are distinctive, and 2 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈2
Distinctive · estimate
This author's own choices. Worth a look.
2
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 16
Run it via subagent failure-analyst — the browser logs, JIRA issues and run history it reads have no business in the orchestrator's context.
- Very specific: line 30
Read run/AC/build/effective config from the capsule and re-derive none of them.
- Distinctive: line 31
Only a standalone invocation without a capsule uses the former explicit project/product paths.
- Distinctive: line 58
Compare the contracts: Put three things side by side: what the test asserts, what the run observed, and what the candidate ticket says …
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
/failure-analysis — why did this test fail?
Failure triage. Not test-gap analysis in the Teamscale sense (changed code without coverage) — the name collision is why this file says so twice.
Until now the FAIL path went straight from "a test failed" to "file a bug". Whether the product broke or the test aged out was decided by a human, every time, from scratch. This skill puts one evidenced step in between.
Run it via subagent failure-analyst — the browser logs, JIRA issues and run history
it reads have no business in the orchestrator's context. The caller supplies the
sealed run capsule path plus the failure packet. The analyst uses the capsule's exact
ACs, routing, build and effective config instead of PROJECT/PRODUCT full reads,
re-routing or another Learnings merge. Bounded log, JIRA, history, spec and targeted
new-symptom Learning reads remain part of triage. A stale capsule is returned as
STALE_CONTEXT; the analyst never regenerates it. Standalone use without a capsule
keeps the prior effective-project fallback.
Steps
- Intake from the handoff.
qa-manageror the direct calling orchestrator hands over the sealed capsule path, evidence dir and failure packet: every failing TC-ID with assertion and error text plusenvironment.version_or_build. Read run/AC/build/effective config from the capsule and re-derive none of them. Only a standalone invocation without a capsule uses the former explicit project/product paths. The caller is never an execution leaf. - Environment gate. Does the run even count? Compare the version against the
ticket's fix version and
deploy.to_version;greplogs/console.logandlogs/network.logfor 404s, 5xx,ECONNREFUSED, mixed bundle hashes;greprelevant capsule Learning excerpts for the signature; without a capsule, or for a genuinely new signature, grepmemory/learnings/env-gotchas.mdand the routed Learnings narrowly. Check whether any other case in the same run passed. A hit ends the analysis — reasoning about product vs. test on top of an invalid run is theory-building on sand. Classenvironment, verdict BLOCKED, no bug. - Determinism gate.
python3 .claude/skills/failure-analysis/tc-history.py --tc <ID> --evidence evidence/. Both outcomes at the same version, or a passing retry in this run → classflake, hand totest-reliability, stop. - Establish the change window. From the history:
last_pass_version→ the deployed version.never_passed: true→ a case that never passed cannot have become outdated; skip the "what changed" framing and compare against the ticket's own acceptance criteria. - Ask JIRA what changed in that window. One JQL —
project = <KEY> AND fixVersion in (<window>) AND statusCategory = Done, orresolutiondatebetween the last pass and now where the project does not use fix versions — plus the ticket under test and its links. On a project with JIRA access:browser, run the same JQL through the issue-search URL instead of the MCP tool —docs/rules/testing/jira-browser.md→ "Reading a ticket". Rank candidates, strongest signal first: the ticket itself → a linked issue → the linked MR's changed file paths (paths only, never diff content) → JIRA component → label → a feature-matrix row name → keyword overlap. Read at most 3 in full and say that the ranking is a heuristic, because it is. - Compare the contracts. Put three things side by side: what the test asserts,
what the run observed, and what the candidate ticket says should now happen —
quoted verbatim, never paraphrased. Then apply
references/classification.md. Paraphrase is where an agent smuggles in what it wants to be true. When a flow map exists (bridge file section## Flow map, orprojects/<KEY>/[products/<PRODUCT>/]flow-map.md): check the affected confirmed area's other flows for sibling cases asserting the same now-changed contract — a deliberate change rarely stales exactly one assertion. Name those siblings in the analysis; they are the human's ruling scope, not an automatic FAIL. - Return ≤ 60 lines. Class, confidence, basis, per-case mapping, and for
test-outdated/test-brittlethe proposal perreferences/proposal-template.md.qa-manageror the direct calling orchestrator writesfailure-analysis.mdinto the existing run directory and mergesfailure_analysisintoexecution_summary.json(.claude/rules/core/evidence-policy.md). The same merge declares the run'sskills_used— every skill the run actually used (failure-analysisincluded), never the ones merely planned.
What the class means downstream
| Class | Verdict | Action |
environment | BLOCKED | fix the environment and re-run; no bug, blocked_reason set |
flake | FAIL | no product bug — test-reliability for quarantine/root cause |
product-defect | FAIL | bug-reporter as before, analysis attached, FAIL transition on the original ticket per PROJECT.md |
test-outdated | NEEDS_REVIEW | proposal, no bug, no transition — a human rules |
test-brittle | NEEDS_REVIEW | proposal (locator/wait mechanics only) |
undetermined | FAIL | bug-reporter with the analysis attached; name the one artifact that would have decided it |
A NEEDS_REVIEW report always names the other branch: what follows if the human
rules product-defect instead (→ bug-reporter with this analysis attached) — the
person ruling should never have to ask "and if I disagree?".
NEEDS_REVIEW is interactive only. Headless — claude -p, CI — the same failure
stays FAIL with the analysis attached: a proposal with no human in the room is not a
proposal, and a pipeline that goes green on "the test was probably outdated" is worse
than one that goes red.
What this skill never does
Apply a proposed test change, or edit a spec — the subagent holds no Write or
Edit for exactly that reason. File a product bug for a failure it classified
environment, flake, test-outdated or test-brittle
(.claude/agents/bug-reporter.md rejects those). Conclude test-outdated from the
absence of a ticket, or from a paraphrased acceptance criterion. Name an introducing
commit without a declared app repo. Re-run the failing test itself.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

