Audit a test suite

A coverage number shows which lines ran, not which behaviour was checked. The most valuable finds are often tests that never run, tests that cannot fail, and risky behaviour nobody tested.

When to use

The task is about finding what to test, not writing one known test. Once you have the list, use write-tests for each item.

Steps

1. Run what exists. Follow orient-in-codebase, run the suite, and record how many tests ran and how many passed.
2. Check that every test runs. List all test files, then check that each one is reached by a CI job, directly or through a script it calls. Files that nothing runs are the cheapest gap to close. Wire each one in only if it passes.
3. Check what the coverage number means. Find which tests and source files are in scope, and what is excluded. Do not trust the percentage until you know that.

4. Look for tests that cannot fail. Sample tests in the riskiest areas:

   • Mocks of the code under test
   • Assertions that only check that something exists
   • Tests that read a copy of the code, or only match source text
   • Tests with no assertions, or skipped tests
   • Empty loops, such as a table that no longer has any rows

5. Map risk. List the areas by the damage a bug would cause:

   1. Money, credits, quota, entitlements
   2. Login, permissions, data belonging to other users
   3. Data loss and irreversible actions
   4. Secrets and personal data
   5. Schedules, retries, state changes
   6. Everything else
6. Compare risk with tests. For each of the top areas, use the list in step 2 of write-tests. Mark each item as tested, weakly tested or missing. Read the test before you mark it. A matching file name is not evidence.
7. Use coverage to find unexplored code. Look at uncovered branches in the top areas. Skip generated code and trivial getters.
8. Rank the gaps. Order by risk first, then by effort. Make each item a specific case, not an area.

Stop and ask when

• The suite cannot run locally.
• Fixing a gap needs a production change that you were not asked to make.

Common mistakes

Example

Weak finding: "Payment module has 62% coverage."

Strong finding:

1. High: a repeated refund webhook can credit twice.
   Evidence: `refund()` has no idempotency check; `refund.test` sends one event.
   Add: deliver the same event twice and concurrently, and assert one credit.

Report

Suite: <tests run, passed, failed, skipped>
Not running: <files no job reaches>
Cannot fail: <tests that check nothing, and why>
Coverage scope: <what the number includes and excludes>

Gaps, highest risk first:
1. <severity>: <defect that could ship>
   Evidence: <file and what you read>
   Add: <exact test case>