/test-health — Test Suite Health Audit

Audit the test suite for flaky tests, dead/trivial tests, coverage gaps on recent changes, missing assertions, and structural issues. Produce a health report with prioritized recommendations.

Arguments

• $ARGUMENTS may contain:
  • --flaky-runs N — number of times to run the suite for flaky detection (default: 5)
  • --coverage — only run the coverage gap analysis (skip flaky/dead detection)
  • --quick — skip flaky detection (most time-consuming), run everything else
  • No arguments — full audit

Phase 0 — Pre-flight

1. Confirm we're in the codegraph repo root
2. Verify vitest is available: npx vitest --version
3. Parse $ARGUMENTS:
   • FLAKY_RUNS=N from --flaky-runs N (default: 5)
   • COVERAGE_ONLY=true if --coverage
   • QUICK=true if --quick
4. Discover all test files:

   find tests/ \( -name '*.test.js' -o -name '*.test.ts' \) | sort

5. Count total test files and categorize by directory (integration, parsers, graph, search, unit)

Phase 1 — Flaky Test Detection

Skip if COVERAGE_ONLY or QUICK is set.

Run the full test suite FLAKY_RUNS times and track per-test pass/fail:

RUN_DIR=$(mktemp -d /tmp/test-health-XXXXXX)
for i in $(seq 1 $FLAKY_RUNS); do
  timeout 180 npx vitest run --reporter=json > "$RUN_DIR/run-$i.json" 2>"$RUN_DIR/run-$i.err"
  exit_code=$?
  if [ $exit_code -eq 124 ]; then
    echo '{"timeout":true}' > "$RUN_DIR/run-$i.json"
  elif [ $exit_code -ne 0 ] && [ $exit_code -ne 1 ]; then
    # Use jq to safely JSON-escape stderr content (may contain quotes, newlines, backslashes)
    stderr_content=$(cat "$RUN_DIR/run-$i.err")
    jq -n --argjson code "$exit_code" --arg stderr "$stderr_content" \
      '{"error":true,"exit_code":$code,"stderr":$stderr}' > "$RUN_DIR/run-$i.json"
  fi
done

For each run, parse the JSON reporter output from $RUN_DIR/run-$i.json to get per-test results.

After all runs are parsed and analysis is complete, clean up the temporary directory:

rm -rf "$RUN_DIR"

Analysis

Before analyzing, exclude invalid runs: skip any run file containing {"timeout":true} or {"error":true} — these runs produced no reliable per-test data and must not be counted as "all tests failed." Require a minimum of 2 valid runs (runs with parseable vitest JSON output) for flaky detection to be conclusive. If fewer than 2 valid runs remain, report that flaky detection was inconclusive due to too many errored/timed-out runs.

A test is flaky if it passes in some valid runs and fails in others.

For each flaky test found:

1. Record: test file, test name, pass count, fail count, failure messages
2. Categorize likely cause:
   • Timing-dependent: failure message mentions timeout, race condition, or test has setTimeout/sleep
   • Order-dependent: only fails when run with other tests (passes in isolation)
   • Resource-dependent: mentions file system, network, port, or temp directory
   • Non-deterministic: random/Date.now/Math.random in test or source

Timeout: Each full suite run gets 3 minutes (timeout 180). Exit code 124 indicates timeout — the run is recorded as {"timeout":true} and the loop continues.

Phase 2 — Dead & Trivial Test Detection

Skip if COVERAGE_ONLY is set.

Scan all test files for problematic patterns:

2a. Empty / no-assertion tests

Search for test bodies that:

• Have no expect(), assert(), toBe(), toEqual(), or similar assertion calls
• Only contain console.log or comments
• Are skipped: it.skip(, test.skip(, xit(, xtest(
• Are TODO: it.todo(, test.todo(

Pattern: test bodies with 0 assertions = dead tests

2b. Trivial / tautological tests

Detect tests that assert on constants or trivially true conditions:

• expect(true).toBe(true)
• expect(1).toBe(1)
• expect(result).toBeDefined() as the ONLY assertion (too weak)

2c. Commented-out tests

Search for commented-out test blocks:

• // it(, // test(, /* it(, /* test(
• Large commented blocks inside describe blocks

2d. Orphaned fixtures

Check if any files in tests/fixtures/ are not referenced by any test file.

2e. Duplicate test names

Search for duplicate test descriptions within the same describe block — these indicate copy-paste errors.

Phase 3 — Coverage Gap Analysis

Run vitest with coverage and analyze:

npx vitest run --coverage --coverage.reporter=json-summary 2>&1

3a. Overall coverage

Parse coverage/coverage-summary.json and extract:

• Line coverage %
• Branch coverage %
• Function coverage %
• Statement coverage %

3b. Uncovered files

Find source files in src/ with 0% coverage (no tests touch them at all).

3c. Low-coverage hotspots

Find files with < 50% line coverage. For each:

• List uncovered functions (from the detailed coverage data)
• Check if the file is in domain/ or features/ (core logic — coverage matters more)
• Check file's complexity with codegraph complexity <file> -T — high complexity + low coverage = high risk

3d. Recent changes without coverage

Compare against main branch to find recently changed files:

git diff --name-only origin/main...HEAD -- src/

For each changed source file, check if:

1. It has corresponding test changes
2. Its coverage increased, decreased, or stayed the same
3. New functions/exports were added without test coverage

Note: If the coverage tool is not configured or fails, skip this phase and note it in the report. Coverage is a vitest plugin — it may need @vitest/coverage-v8 installed.

Phase 4 — Test Structure Analysis

Skip if COVERAGE_ONLY is set.

Analyze the test suite's structural health:

4a. Test-to-source mapping

For each directory in src/:

• Count source files
• Count corresponding test files
• Calculate test coverage ratio (files with tests / total files)
• Flag directories with < 30% test file coverage

4b. Test file size distribution

• Find oversized test files (> 500 lines) — may need splitting
• Find tiny test files (< 10 lines) — may be stubs or dead

4c. Setup/teardown hygiene

Check for:

• Tests that create temp files/dirs but don't clean up (afterEach/afterAll missing)
• Tests that mutate global state without restoration
• Missing beforeEach resets in describe blocks that share state

4d. Timeout analysis

• Find tests with custom timeouts: { timeout: ... }
• Find tests that exceed the default 30s timeout in recent runs
• High timeouts often indicate tests that should be restructured or are testing too much

Phase 5 — Report

Write report to generated/test-health/TEST_HEALTH_<date>.md:

# Test Health Report — <date>

## Summary

| Metric | Value |
|--------|-------|
| Total test files | N |
| Total test cases | N |
| Flaky tests | N |
| Dead/trivial tests | N |
| Skipped tests | N |
| Coverage (lines) | X% |
| Coverage (branches) | X% |
| Uncovered source files | N |
| **Health score** | **X/100** |

## Health Score Calculation

- Start at 100
- -10 per flaky test
- -3 per dead/trivial test
- -2 per skipped test (without TODO explaining why)
- -1 per uncovered source file in `domain/` or `features/`
- -(100 - line_coverage) / 5 (coverage penalty)
- Floor at 0

## Flaky Tests
<!-- For each: file, name, pass/fail ratio, likely cause, suggested fix -->

## Dead & Trivial Tests
<!-- For each: file, line, issue, recommendation -->

## Coverage Gaps
<!-- Uncovered files, low-coverage hotspots with complexity -->

## Structural Issues
<!-- Oversized files, missing cleanup, timeout issues -->

## Recommended Actions

### Priority 1 — Fix flaky tests
<!-- List with specific suggestions -->

### Priority 2 — Remove or fix dead tests
<!-- List with specific suggestions -->

### Priority 3 — Add coverage for high-risk gaps
<!-- List uncovered functions in core modules, ordered by complexity -->

### Priority 4 — Structural improvements
<!-- Split large files, add cleanup, reduce timeouts -->

Phase 6 — Quick Wins

After writing the report, identify tests that can be fixed immediately (< 5 min each):

1. Remove .skip from tests that now pass (run them to check)
2. Add missing assertions to empty test bodies (if the intent is clear)
3. Delete commented-out test blocks older than 6 months (check git blame)

Do NOT auto-fix — list these as suggestions in the report. The user decides.

Rules

• Never delete or modify test files without explicit user approval — this is a read-only audit
• Flaky detection is slow — warn the user before running 5+ iterations
• Coverage requires @vitest/coverage-v8 — if missing, skip coverage and note it
• Order-dependent flakiness requires running tests both in suite and in isolation — only do this for tests that flaked in Phase 1
• Fixture files may be shared across tests — don't flag as orphaned if used indirectly
• Skipped tests aren't always bad — only flag if there's no TODO or comment explaining why
• Generated files go in generated/test-health/ — create the directory if needed
• This is a diagnostic tool — it reports problems, it doesn't fix them (unless the user opts in)