agentleFS
Sn4kyGitSn4kyGit/qa-agent-blueprint/.claude/skills/failure-analysis/SKILL.md

failure-analysis

Skill2 starsChanged 7 days agoApache-2.0View on GitHub

Triage a failing test — real product defect, or outdated test case after a deliberate product change? Classifies the cause from the JIRA change record and version window, proposes (never applies) test changes. Runs automatically on every FAIL before a bug is filed. Use for "/failure-analysis", "warum ist der Test rot", "is this test outdated". Not for filing the bug — /bug-report after classification.

1 other project uses a version of this.

Adopted by

Most recently taken up by

The file

We split this skill into the rules it gives an agent

4 rules. About 0 are common, 2 are distinctive, and 2 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈2

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 2

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    Run it via subagent failure-analyst — the browser logs, JIRA issues and run history it reads have no business in the orchestrator's context.

    line 16
  2. Very specific:

    Read run/AC/build/effective config from the capsule and re-derive none of them.

    line 30
  3. Distinctive:

    Only a standalone invocation without a capsule uses the former explicit project/product paths.

    line 31
  4. Distinctive:

    Compare the contracts: Put three things side by side: what the test asserts, what the run observed, and what the candidate ticket says …

    line 58

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

/failure-analysis — why did this test fail?

Failure triage. Not test-gap analysis in the Teamscale sense (changed code without coverage) — the name collision is why this file says so twice.

Until now the FAIL path went straight from "a test failed" to "file a bug". Whether the product broke or the test aged out was decided by a human, every time, from scratch. This skill puts one evidenced step in between.

Run it via subagent failure-analyst — the browser logs, JIRA issues and run history it reads have no business in the orchestrator's context. The caller supplies the sealed run capsule path plus the failure packet. The analyst uses the capsule's exact ACs, routing, build and effective config instead of PROJECT/PRODUCT full reads, re-routing or another Learnings merge. Bounded log, JIRA, history, spec and targeted new-symptom Learning reads remain part of triage. A stale capsule is returned as STALE_CONTEXT; the analyst never regenerates it. Standalone use without a capsule keeps the prior effective-project fallback.

Steps

  1. Intake from the handoff. qa-manager or the direct calling orchestrator hands over the sealed capsule path, evidence dir and failure packet: every failing TC-ID with assertion and error text plus environment.version_or_build. Read run/AC/build/effective config from the capsule and re-derive none of them. Only a standalone invocation without a capsule uses the former explicit project/product paths. The caller is never an execution leaf.
  2. Environment gate. Does the run even count? Compare the version against the ticket's fix version and deploy.to_version; grep logs/console.log and logs/network.log for 404s, 5xx, ECONNREFUSED, mixed bundle hashes; grep relevant capsule Learning excerpts for the signature; without a capsule, or for a genuinely new signature, grep memory/learnings/env-gotchas.md and the routed Learnings narrowly. Check whether any other case in the same run passed. A hit ends the analysis — reasoning about product vs. test on top of an invalid run is theory-building on sand. Class environment, verdict BLOCKED, no bug.
  3. Determinism gate. python3 .claude/skills/failure-analysis/tc-history.py --tc <ID> --evidence evidence/. Both outcomes at the same version, or a passing retry in this run → class flake, hand to test-reliability, stop.
  4. Establish the change window. From the history: last_pass_version → the deployed version. never_passed: true → a case that never passed cannot have become outdated; skip the "what changed" framing and compare against the ticket's own acceptance criteria.
  5. Ask JIRA what changed in that window. One JQL — project = <KEY> AND fixVersion in (<window>) AND statusCategory = Done, or resolutiondate between the last pass and now where the project does not use fix versions — plus the ticket under test and its links. On a project with JIRA access: browser, run the same JQL through the issue-search URL instead of the MCP tool — docs/rules/testing/jira-browser.md → "Reading a ticket". Rank candidates, strongest signal first: the ticket itself → a linked issue → the linked MR's changed file paths (paths only, never diff content) → JIRA component → label → a feature-matrix row name → keyword overlap. Read at most 3 in full and say that the ranking is a heuristic, because it is.
  6. Compare the contracts. Put three things side by side: what the test asserts, what the run observed, and what the candidate ticket says should now happen — quoted verbatim, never paraphrased. Then apply references/classification.md. Paraphrase is where an agent smuggles in what it wants to be true. When a flow map exists (bridge file section ## Flow map, or projects/<KEY>/[products/<PRODUCT>/]flow-map.md): check the affected confirmed area's other flows for sibling cases asserting the same now-changed contract — a deliberate change rarely stales exactly one assertion. Name those siblings in the analysis; they are the human's ruling scope, not an automatic FAIL.
  7. Return ≤ 60 lines. Class, confidence, basis, per-case mapping, and for test-outdated/test-brittle the proposal per references/proposal-template.md. qa-manager or the direct calling orchestrator writes failure-analysis.md into the existing run directory and merges failure_analysis into execution_summary.json (.claude/rules/core/evidence-policy.md). The same merge declares the run's skills_used — every skill the run actually used (failure-analysis included), never the ones merely planned.

What the class means downstream

ClassVerdictAction
environmentBLOCKEDfix the environment and re-run; no bug, blocked_reason set
flakeFAILno product bug — test-reliability for quarantine/root cause
product-defectFAILbug-reporter as before, analysis attached, FAIL transition on the original ticket per PROJECT.md
test-outdatedNEEDS_REVIEWproposal, no bug, no transition — a human rules
test-brittleNEEDS_REVIEWproposal (locator/wait mechanics only)
undeterminedFAILbug-reporter with the analysis attached; name the one artifact that would have decided it

A NEEDS_REVIEW report always names the other branch: what follows if the human rules product-defect instead (→ bug-reporter with this analysis attached) — the person ruling should never have to ask "and if I disagree?".

NEEDS_REVIEW is interactive only. Headless — claude -p, CI — the same failure stays FAIL with the analysis attached: a proposal with no human in the room is not a proposal, and a pipeline that goes green on "the test was probably outdated" is worse than one that goes red.

What this skill never does

Apply a proposed test change, or edit a spec — the subagent holds no Write or Edit for exactly that reason. File a product bug for a failure it classified environment, flake, test-outdated or test-brittle (.claude/agents/bug-reporter.md rejects those). Conclude test-outdated from the absence of a ticket, or from a paraphrased acceptance criterion. Name an introducing commit without a declared app repo. Re-run the failing test itself.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.