Code Testing Skill

Runs this repository's pytest suite through the justfile.

Prerequisite

/code-format and /code-check must be green first. If lint is dirty, go back to /code-check: an unused import or an undefined name is cheaper to surface there than through a test run, and a failing collection tells you less than a lint message about the same mistake.

Scope Selection

Pick the first row whose blast radius covers the change — the set of code paths a mistake here could plausibly break. Rows below it are escalations; take one only when a signal pushes the radius outward.

Signals that push the radius outward, from "pure logic" to wider:

• Changed a signature, an attribute, or the semantics of a type that other modules import — a finding, a check result, a parsed document.
• Changed a shared type that more than one module depends on, or anything one layer imports from another.
• Changed a registry, discovery of checks, or an __init__.py that re-exports.
• Changed a pyproject.toml — the root's dependency groups, pytest configuration or marker list, or a package's dependencies, console script, or how its version is derived. The last two reach tests/e2e/ and nothing below it.

If none fire, unit tests are enough.

Marker traps

Tests are selected by marker, and --strict-markers is on. Two consequences worth knowing before you filter:

• An undeclared marker is a collection error, not a skip. Adding @pytest.mark.slow without a matching entry in [tool.pytest.ini_options] markers fails the whole run at collection, and a command selecting -m slow fails before a single test executes. unit, it and e2e are the markers declared today; adding another means editing the root pyproject.toml in the same change.
• just test-unit filters on the marker, not on the directory. A test placed in a unit tests/ subpackage without @pytest.mark.unit is invisible to just test-unit and runs only under just test. If a new test never seems to execute, check its marker before you doubt the assertion.

Commands

Every recipe takes extra flags, passed through to pytest — each tier recipe is implemented as a dependency on test, so just test-unit runs uv run pytest -m unit with your flags appended. For a single file or a single test — which no recipe covers — call pytest directly:

uv run pytest src/lorecraft/<pkg>/tests/test_<module>.py -v
uv run pytest tests/it/test_<module>.py::Test<Subject>::test_<case> -v
uv run pytest tests/e2e/test_<module>.py -v

Snapshots

A snapshot test, through syrupy's snapshot fixture, compares output to a file checked in under __snapshots__/ beside its test module; tests/lib/snapshot.py stores each one as plain text. Command output is pinned this way, with whatever varies per build or machine, such as the version or the workspace root, swapped for a placeholder before it is compared. Every recipe above fails on a mismatch, and on a snapshot no test reads any more.

A failing snapshot is a finding, not a chore: when output changed on purpose, run just snapshot-update, read just snapshot-review, and commit the snapshot with the change. Never update to turn a run green without reading the diff. Updating one tier alone reads the other tiers' snapshots as unused, so the recipe never filters.

Coverage

Coverage says which lines and branches a tier runs — not whether a test checks what they do; that is mutation testing's question. It costs one ordinary run of the tier, so measure it when asked what the tests reach, or to find the code no test runs before writing tests for a module.

Each run overwrites .coverage. uv run coverage report reads it again, with the missing lines per file; uv run coverage html writes a browsable copy to htmlcov/. CI runs all four on every pull request and posts the report as one comment; it never fails the build.

Mutation testing

Expensive, and never the check after an edit. mutmut injects small faults into src/lorecraft/ — a flipped comparison, a changed constant, a dropped argument — and runs the tier once per fault. A fault the tests catch is killed; one they miss survives. The share killed is how effective the tests are, which line coverage cannot tell you: a line can run under a test that asserts nothing about it.

Run it when the user asks how strong the tests are, before hardening a module's tests, or to confirm new tests pin the behaviour they claim to. It costs minutes and every core.

Mutant names, as globs, narrow a run to one module: just test-mut 'lorecraft.vfs.disk.*'. Each run starts from an empty mutants/. The e2e tier has no recipe: it runs the console script in a subprocess, where mutmut cannot swap a fault in. CI runs just test-mut alone on every pull request and posts the report as one comment; it never fails the build.

Read references/mutation-testing.md before acting on a surviving mutant.

Notes

Unit tests sit beside the module they test, in a tests/ subpackage under src/: the tests for <pkg>/<module>.py are <pkg>/tests/test_<module>.py, and no wheel or sdist ships them. The it tier lives flat, in tests/it/. docs/code/tests-organization.md §2 owns the placement. The end-to-end tier lives in tests/e2e/ and covers the command line only: it drives the installed lorecraft console script. Its shared helpers live beside the suites in tests/lib/, imported as lib (from lib.cli import run_cli) through pytest's pythonpath = ["tests"]. A library layer has no e2e tier, because its end-to-end surface is its public API, which its it tier already covers. tests/e2e/ is the slowest by an order of magnitude and the only tier that spawns a process, which is the reason the default stays narrow.

pytest exits 5 when it collects nothing, and the test recipe preserves that failure. A tier with no collected tests therefore fails instead of reporting a misleading green run.

Collection is confined by testpaths to src/, tests/it/ and tests/e2e/. A unit tests/ subpackage has an __init__.py, so --import-mode=importlib imports it under its real dotted name and its relative imports resolve; the it and e2e directories carry none, and importlib mode keeps two test files of the same name apart. Document-shaped fixtures are checked in as real files, so a test reads the same thing an agent would; adding a fixture file changes no configuration, but the test that consumes it still needs its marker.

Anti-patterns

• Running tests before /code-check is green.
• Reaching for just test when the change is pure logic — use just test-unit.
• Putting a test that spawns a process, or one that needs the package installed, anywhere but tests/e2e/. docs/code/tests-organization.md owns the tier boundaries.
• Introducing a marker without declaring it in the root pyproject.toml.
• Calling pytest outside uv run — it is not installed in a system interpreter.
• Reading a zero-collected run as a passing run.
• Skipping tests on a behaviour change. Docs-only is the one acceptable skip, and it should be stated.

Debugging

Append pytest flags to either recipe: -x to stop at the first failure, -k <expr> to select by name, --log-cli-level=DEBUG to surface log output, -s to let prints through.

just test-unit -k <name-fragment> --log-cli-level=DEBUG -x

--log-cli-level works on the module-level loggers described in docs/code/logging.md; a module that logs through logging.getLogger(__name__) is reachable by name, and one that does not is not.

Next Steps

After tests pass: review the diff and commit, with lint and tests green.