test-necessity
Skill0 starsChanged 18 days agoApache-2.0View on GitHub
Grade tests A–F by whether they can actually fail: for each test, break the code it claims to guard with a sabotage probe ({{cli}} probe) and record whether it goes red. A is kept, B/C are fixed up to A, D/F are deleted with evidence. Use to audit a suite, check new tests in a PR, or decide whether a specific test is still needed.
In this repository15 other files it gives its agents.
- ai-exchangeSkill
- cleanupSkill
- design-reviewSkill
- evidence-rulesSkill
- gradeSkill
- live-ai-checkSkill
- mine-skillsSkill
- open-prSkill
- parallel-workSkill
- plan-tasksSkill
- releaseSkill
- review-changeSkill
- ship-changeSkill
- skill-evalSkill
- write-specSkill
The file
We split this skill into the rules it gives an agent
5 rules. About 0 are common, 2 are distinctive, and 3 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈2
Distinctive · estimate
This author's own choices. Worth a look.
3
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 8
Ask one question of every test, and answer it with evidence:
- Distinctive: line 13
If the answer is "nothing", the test costs gate time, can fail for reasons unrelated to behaviour, and makes reviewers think something is …
- Very specific: line 31
Census: Run {{cli}} census. It reads every test case's syntax tree and flags cases with no assertion, assertions only under if/catch/an …
- Very specific: line 46
Run {{cli}} probe probes.json in your own worktree (it edits files and restores them.
- Very specific: line 58
Track: Keep a tracker file (for example docs/TEST-ASSESSMENT.md) with one row per file: grade, probe ids, action, commit.
≈ “Common” is an estimate: at least 1 in 5 of the 350,495 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Test necessity
Ask one question of every test, and answer it with evidence:
If this test were deleted today, what real defect could reach the main branch that cannot reach it now?
If the answer is "nothing", the test costs gate time, can fail for reasons unrelated to behaviour, and makes reviewers think something is covered when it isn't.
Grades
| Grade | Meaning | Outcome |
| A | A probe turns it red; it asserts behaviour; it is deterministic; its purpose is clear | Keep |
| B | Witnessed, but has weak assertions, shape-pinning, or redundant cases | Fix to A |
| C | Witnessed only partly (some claims stay green under their break) | Fix to A |
| D | Vacuous: no break of its subject turns it red | Delete, unless protected |
| F | Its subject is gone, it is skipped, or it only asserts mocks or status codes | Delete |
Protected (upgrade these; never delete them): security, authorization and tenancy tests; regression tests that cite an incident; contract and conformance tests.
Method
- Census. Run
{{cli}} census. It reads every test case's syntax tree and flags cases with no assertion, assertions only underif/catch/an unrun callback, status-only, mock-only, weak-only or snapshot-only checks, skips,.only, and duplicate bodies. It gives each file a ceiling, which caps the grade but never sets it. Probe the D/F-ceiling files first. Its ledger is keyed on blob sha, so later runs re-queue only changed files. State the denominator from its summary. - Probe. For each test, write one or more probes. Each probe makes a minimal,
realistic break in the code the test claims to guard. Typical breaks: flip a
comparison, drop a guard, return early, off-by-one, swallow an error.
With[{ "id": "auth-skip", "src": "src/auth.ts", "old": "if (!user) throw", "new": "if (false) throw", "test": "src/auth.test.ts", "expectRed": ["rejects anonymous"] }]probe.formatset (vitest/node),expectRedmust match a case that actually failed. A name merely printed in the output doesn't count. Run{{cli}} probe probes.jsonin your own worktree (it edits files and restores them; never run two at once in one worktree). Verdicts:witnessed: the test went red. The claim is backed.vacuous: the test stayed green. Before you blame the test, check the break was reachable (see{{skill:evidence-rules}}).other-red: something failed, but not the expected case. Investigate.error: red for the wrong reason (timeout, environment). This is not a witness.
- Grade and act. Fix B/C up to A: sharpen the assertions, cut redundant cases, use realistic fixtures, add a one-line header saying what the test guards and which probe witnessed it. Then probe again. Delete D/F in focused commits, with the probe result in each commit message, and remove any helpers or fixtures they leave orphaned.
- Track. Keep a tracker file (for example
docs/TEST-ASSESSMENT.md) with one row per file: grade, probe ids, action, commit. Later runs resume from it. State coverage as "N of M files graded".
In a PR
For each test the PR adds, write at least one probe and run it. A new test whose probe
is vacuous is a review BLOCKER.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.