agentleFS
Wolfe-JamWolfe-Jam/faf-skills/skills/wjttc-tester/SKILL.md

wjttc-tester

Skill4 starsChanged 4 days agoMITView on GitHub

First checks that a failing test still means a real failure, then runs the plan, reproduces and root-causes each failure, and writes a WJTTC report sorted by severity (Brake, Engine, Aero, Tyre, Pit) with a verdict. Use when someone wants tests run or a bug chased down: "run the tests and tell me what broke", "reproduce this bug", "can I trust our CI?". Runs and reports; wjttc-builder plans and writes the tests.

The file

We split this skill into the rules it gives an agent

2 rules. About 0 are common, 2 are distinctive, and 0 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈2

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 0

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Test Brake first. If the brakes don't work, nothing else matters.

    line 48
  2. Distinctive:

    Reproduce: every failure deterministically.

    line 83

≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md
<!-- faf:start -->
v: 1
needs:
  - "the project's test runner"
  - "faf-cli on Node 22+ (not installed: run `npx faf-cli <command>` in place of `faf <command>`)"
reads: [the test plan, the tests, CI results, the code under test]
writes: ["the WJTTC report in ./wjttc-reports/ (or a path the user gives)"]
network: []
publishes: false
done_when: "every test has run, each failure is reproduced and root-caused, and the WJTTC report is saved with its verdict"
project: project.faf
<!-- faf:end -->

WJTTC Championship Tester

"We break things so others never have to know they were broken."

Apply championship-grade standards to software testing. When brakes must work, so must the code. This skill executes test plans and files reports — it is the driver, not the engineer. To plan and generate the suite, use wjttc-builder.

When to use this skill

  • Running an existing or just-written test plan and reporting outcomes
  • Reproducing and root-causing a reported bug
  • Edge-case / error-handling / regression validation
  • Auditing whether the suite's CI signal can still be trusted
  • Producing a WJTTC report with a tier verdict

The WJTTC five tiers

Triage every test by blast radius. The first three set severity; Tyre and Pit cover durability and the release gate.

TierMeaningExamples
BrakeLife-critical — failure is catastrophicdata loss, auth bypass, payment errors, destructive ops without confirm
EnginePerformance-critical — wrong results / poor UXAPI accuracy, data transforms, calculations, format compliance, perf
AeroPolish & edge cases — minor inconvenienceUI quirks, rare message formatting, optional-feature edges, docs
TyreDurability under load — degradation over timestress/volume, concurrency, memory growth, large inputs
PitRelease gate — the stop that lets you gosmoke/regression suite, CI green, the WJTTC report filed

Test Brake first. If the brakes don't work, nothing else matters.

Step 0 — Signal Integrity pre-audit (run BEFORE adding/running anything new)

Red CI is a contract: it must always mean "stop, look, fix." A suite with high coverage but flaky reds is less trustworthy than a smaller suite with zero false alarms — because the team has stopped reading the reds. Fix the signal before you add more tests.

Method — classify the last 30 days of CI failures:

BucketDefinitionVerdict
Real bugRed mapped to a real defect; fixed by a code change✓ Signal worked
FlakeTiming/network/concurrency noise; passed on rerun, no code change✗ Test design defect
InfraMissing secret, runner image change, upstream dep — not the code✗ Workflow design defect

Signal Integrity Score: SI = Real bugs / (Real bugs + Flakes + Infra) × 100

SI %VerdictAction
100%✪Maintain — exemplary signal
95–99%★ ChampionshipAnnotate any flake immediately
85–94%◇ AcceptableSchedule the flake-class fix this sprint
70–84%● ErodingStop adding tests — fix flakes first
<70%○ Dead signalBlock merges until signal restored

Eliminate on sight: hard absolute-time perf asserts on shared runners (expect(t).toBeLessThan(30)) → move to a non-gating workflow; network calls in the main suite → mock at the boundary; concurrency tests without explicit ordering; secret-dependent steps that hard-fail when missing → grey-skip.

The inverse rule: green CI that passes while something is broken is equally a violation. If a real bug shipped despite green, write the regression test BEFORE the fix lands.

The conversation is the real gate. CI is supporting infrastructure for the human + AI audit; flaky CI wastes the audit's bandwidth. Signal Integrity keeps CI worthy of the conversation.

Execution loop

  1. Scope — what should it do? happy path, edges, failure modes, perf targets, tier of each.
  2. Audit signal (Step 0) before trusting or extending the suite.
  3. Run each test: set up, prepare data, execute, observe actual vs expected, record pass/fail/blocked, capture evidence on failure.
  4. Reproduce every failure deterministically; root-cause it; note the fix.
  5. Tier coverage check — confirm every test is tiered:
    faf wjttc --path tests          # audit tier coverage (vendor-neutral)
    faf wjttc --strict --json       # CI gate: non-zero if any test is untiered
  6. Report — file the WJTTC report (below), then surface the tier verdict.

WJTTC report format

Save reports to ./wjttc-reports/ in the project under test (or a path the user specifies). Never write to an absolute/personal path. Name files YYYY-MM-DD-{project}-{feature}-tests.yaml.

---
# WJTTC Test Report
project: "project-name"
feature: "feature-being-tested"
date: "2026-06-26"
tier: "Engine"            # Brake | Engine | Aero | Tyre | Pit
result: "PASS"            # PASS | FAIL | BLOCKED
environment: "OS, runtime version, key deps"
---

## Summary
objective: What was tested
totals: { total: 25, passed: 23, failed: 2, blocked: 0, pass_rate: "92%" }

## Failures
- name: "Long-string handling"
  tier: "Engine"
  status: "FAIL"
  steps: ["...", "..."]
  expected: "Handle gracefully"
  actual: "Crash"
  error: "RangeError: ..."
  root_cause: "Unbounded buffer"
  fix: "Cap input length / stream"

## Edge cases
- { case: "Empty string", input: "''", expected: "error", actual: "error", status: "PASS" }
- { case: "Unicode", input: "🏎️", expected: "stored", actual: "stored", status: "PASS" }

## Performance
- { op: "file read",  target: "<50ms", actual: "18ms", status: "PASS" }
- { op: "parse YAML", target: "<50ms", actual: "12ms", status: "PASS" }

## Bugs found
- id: 1
  title: "..."
  severity: "Brake"      # tier doubles as severity
  reproducibility: "Always"
  impact: "Who is affected, how serious"
  fix: "..."

## Coverage
tested:     ["happy path", "edges", "error handling", "perf"]
not_tested: ["concurrent access", "files >100MB"]

## Verdict
tier: "◆ Silver"          # from the tier table below
to_next: ["Fix 2 failing Engine tests", "Add Tyre concurrency tests"]

Tier verdict

Map the pass rate (or SI score) to the single canonical FAF tier ladder. No second ladder, no medals.

ScoreTierSymbol
100%Trophy✪
99%Gold★
95%Silver◆
85%Bronze◇
70%Green●
55%Yellow●
1%Red○
0%White♡

The FAF score is deterministic — same input, same score. A test report should be just as falsifiable: every verdict traces to a reproducible run. FAF doesn't lie.

WJTTC method notes

  • Test with real data, not just sanitized inputs — anonymized production data, messy inputs, production-like volume.
  • Document every failure so it can be reproduced: what failed, how to repro, why it matters, how to fix.
  • Tier before you test — severity is the tier, so triage first; faf wjttc enforces that nothing ships untiered.
  • Wire it into CI with TAF receipts so the report is part of the record, not a one-off:
    faf taf setup --write     # create .github/workflows/taf.yml (test receipts)
    faf score --json          # deterministic score snapshot for the receipt

Quick checklist (before release)

  • Signal Integrity audited (SI ≥ 85%)
  • Brake tests pass — zero tolerance
  • Edges + error handling tested
  • Tyre: behaves under load / concurrency
  • faf wjttc --strict green — every test tiered
  • Regression (Pit) suite passes
  • WJTTC report filed in ./wjttc-reports/
  • Pass rate ≥ 85% (◇ Bronze, production-ready)

Resources


Made by wolfejam.dev — "We break things so others never have to know they were broken."

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.