Mutation Testing

A green test suite tells you the tests passed. It does not tell you whether the tests would have caught a real defect. Mutation testing closes that gap: introduce a one-character defect in production code, run the suite, expect at least one test to fail. Survivors are coverage gaps.

Coverage measures whether the line was executed. Mutation measures whether the line was meaningfully asserted on. The two are not the same.

When to fire

• After authoring a new test file, before declaring it "covers" the module
• After adding tests to a previously-untested module
• Before publishing any release that newly depends on a module's correctness
• When a code review surfaces "this is tested, but I'm not sure the tests assert what we think they do"

Do NOT fire when:

• The code under test is throwaway (one-off scripts, scratch experiments)
• A formal mutation-testing tool (stryker, mutmut) is already in CI — that supersedes the manual protocol below
• The cost of running the suite once is high (long integration tests) — sample the high-risk mutations instead of running all of them

The protocol

For each load-bearing branch / guard / constant in the module:

1. Copy the project or a supported test fixture to a temporary location. The mutation target must be inside that copy, never the working tree.
2. Apply a one-character defect that flips the behavior (invert a comparison, change a numeric literal, replace a guard with if (false))
3. Run the suite against the copy — npm test or the project's canonical entry point, with any required root override targeting the copy.
4. Expect ≥ 1 failure. If the suite still passes, that branch has no real coverage.
5. Discard the copy before applying the next mutation.
6. Record the result: caught / survived / precondition-not-found, as one result line per mutation in a run-log file. The harness below already prints [Name] CAUGHT / [Name] *** SURVIVED *** / [Name] PRECONDITION-NOT-FOUND; capture those lines to the log so the tally (below) is a count, not a recollection.

The source files must remain unchanged. A mutation run that requires restoring a working-tree file is already too risky: an interrupted process, a failed write, or another tool reading the file can leave a real defect behind.

Isolated Mutation Harness

function Test-Mutation {
    param(
        [string]$ProjectRoot,
        [string]$RelativePath,
        [string]$Find,
        [string]$Replace,
        [string]$Name,
        [scriptblock]$RunTests
    )

    $sandbox = Join-Path ([System.IO.Path]::GetTempPath()) ("mutation-" + [guid]::NewGuid())
    try {
        New-Item -ItemType Directory -Force -Path $sandbox | Out-Null
        Copy-Item -LiteralPath $ProjectRoot -Destination (Join-Path $sandbox 'project') -Recurse
        $copyRoot = Join-Path $sandbox 'project'
        $target = Join-Path $copyRoot $RelativePath
        $original = Get-Content -LiteralPath $target -Raw
        if (-not $original.Contains($Find)) { "[$Name] PRECONDITION-NOT-FOUND"; return }

        Set-Content -LiteralPath $target -Value $original.Replace($Find, $Replace) -NoNewline
        $exitCode = & $RunTests $copyRoot
        if ($exitCode -ne 0) { "[$Name] CAUGHT" } else { "[$Name] *** SURVIVED ***" }
    } finally {
        Remove-Item -LiteralPath $sandbox -Recurse -Force -ErrorAction SilentlyContinue
    }
}

# The test command receives the disposable project root. Adapt its root override
# or fixture option to your project; do not silently fall back to the real root.
$runTests = {
    param($copyRoot)
    Push-Location $copyRoot
    try { npm test; return $LASTEXITCODE } finally { Pop-Location }
}

Test-Mutation $PWD 'lib/semver.js' 'if (pa.major !== pb.major)' 'if (pa.major === pb.major)' 'M2 major-equality inverted' $runTests

For a small validator, a purpose-built temporary fixture is often cheaper than copying the whole project. Steward's ONE-check suite uses that pattern: it generates disposable fixtures and temporary validator copies, then asserts the working validators' bytes remain unchanged.

Tallying the results

The mutations-caught ratio in the Quality bar comes from counting the run log's result lines mechanically, never from tallying outcomes as you go:

• PowerShell: $caught = (Select-String -Path $log -Pattern '\] CAUGHT$').Count, $survived = (Select-String -Path $log -Pattern 'SURVIVED').Count, $missing = (Select-String -Path $log -Pattern 'PRECONDITION-NOT-FOUND').Count
• bash: grep -c '\] CAUGHT$' run.log, grep -c 'SURVIVED' run.log, grep -c 'PRECONDITION-NOT-FOUND' run.log

Report caught / (caught + survived) from those counts, with survived and precondition-not-found listed separately.

Counter-Test Detection Rules

A detector needs two proofs. First, inject the live defect it exists to catch into an isolated active artifact and assert that it reports a failure. Then add the innocent case that previously caused a false positive and assert silence.

For example, a rule against a retired process must flag a new active route that revives the process, while allowing historical prose that says the process was retired. A counter-test for a detection rule keeps a correct historical record from looking like the defect itself.

High-value mutation patterns

Anti-patterns the protocol catches

1. Filter with no input data. A .filter(e => e.name !== 'local') test passes trivially when the test data contains zero local/ entries. Mutation: replace with e => true. If no test fails, the filter is unverified. Fix by seeding test data that would be filtered out and asserting it doesn't appear in the output.

2. Exports that production never calls. Easy to grep: Select-String -Path src/,extension.js -Pattern "exportedName" — if production callers are zero, the unit tests are testing dead code. Either wire the export into production or remove it.

3. Platform-skipped tests with no fallback. A symlink-cycle test that skips on Windows is no coverage on Windows. Look for t.skip(...) and ask: does the protected behavior have a deterministic, platform-independent test alongside it?

4. "Didn't throw" assertions. A test that builds a 60-level deep tree and asserts the call "didn't throw" is true even with the depth cap removed. Add a sharp assertion: with cap=50, the depth-60 leaf must be unreachable.

5. Frontmatter-only tests. A test that JSON.parse's a manifest and asserts keys.includes('spec_version') passes even when the field has the wrong value. Assert on the value, not just key presence.

What "good coverage" looks like

For each production module:

• Every load-bearing branch has a mutation that catches its removal. Not "every line is covered" — every branch where flipping the condition would ship a real defect.
• The mutation that catches it names the behavior (e.g., "stale-lock breaking disabled" → caught by stale lock (>10min mtime) broken atomically test).
• Active-filter tests where source data doesn't naturally exercise the filter. Seed the data; verify the filter fires.
• Reference-equality tests for null-result fast paths (e.g., a pure function that returns the input unchanged on null provenance — assert out === input).

Worked examples from the field

Quality bar

A mutation-testing pass is complete when:

• Every named branch in the production module has a mutation entry in your run log
• Every survivor was either closed by a new test or explicitly accepted with a reason (e.g., "behavior is internal-only, no public contract")
• The total mutations-caught ratio is recorded from the run-log counts (e.g., "15 of 16 mutations caught; M8 deferred"; see Tallying the results)
• The working tree's source files remain unchanged throughout the run, and the temporary copy is removed after each mutation

Anti-patterns

Falsifiability — would revise if

• Event-based: if a formal mutation-testing tool (Stryker, mutmut) is adopted in CI, this skill becomes documentation for the manual fallback only; trim the protocol section accordingly.

Origin

Codified 2026-05-31 after meta-testing a static-fetch test harness (commit 4163f67). The protocol caught two real coverage gaps (symlink-cycle protection and depth-cap enforcement in lib/fs-utils.js) that platform-skipped tests had been silently hiding. Applied a second time to another test harness, where it caught two more gaps (local/ filter on skills and instructions in build-edition-manifest.cjs). 12 of 12 caught after gap-closing — the protocol is the reason the final harnesses are trustworthy.

Related

• code-review/SKILL.md — pre-publish review; mutation testing is the final layer when code review surfaces test-quality concerns
• Release preflight discipline — releases that newly depend on a module's correctness should mutation-test it first (Alex ACT itself formalizes this in the Steward release ritual)