test-suite-evaluator
Skill1 starsChanged yesterdayMITView on GitHub
Evaluate an automated test suite for effectiveness, trustworthiness, engineering quality, CI readiness, maintainability, coverage evidence, and operational health. Produces evidence-backed 1–10 scores, confidence ratings, critical caps, and prioritised remediation. Works with standalone test repositories and must not invent application-code coverage.
- Reads credentials
In this repository33 other files it gives its agents.
- CLAUDE.mdCLAUDE.md
- api-contract-testerSkill
- bug-reproducerSkill
- ci-failure-triagerSkill
- ci-suite-runnerSkill
- code-security-auditorSkill
- cross-browser-checkerSkill
- db-migration-reviewerSkill
- dead-test-finderSkill
- exploratory-charter-writerSkill
- flaky-test-hunterSkill
- hypercare-anomaly-watcherSkill
- i18n-checkerSkill
- journey-mapperSkill
- journey-to-testsSkill
- mobile-app-auditorSkill
- mutation-testerSkill
- performance-budget-auditorSkill
- post-deploy-validatorSkill
- privacy-flow-auditorSkill
- qa-orchestratorSkill
- release-readiness-gateSkill
- requirements-traceability-auditorSkill
- resilience-proberSkill
- risk-based-test-plannerSkill
- test-case-designerSkill
- test-data-auditorSkill
- test-impact-selectorSkill
- ticket-readiness-reviewerSkill
- ticket-to-tddSkill
- ui-crash-hunterSkill
- visual-regression-reviewerSkill
- wcag-auditorSkill
The file
Show its rules and text
We split this skill into the rules it gives an agent
29 rules. About 0 are common, 16 are distinctive, and 13 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈16
Distinctive · estimate
This author's own choices. Worth a look.
13
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 25
Never claim line, branch, condition, or function coverage without source-linked instrumentation or a credible coverage report.
- Distinctive: line 26
Never call test-derived behaviour inference “code coverage.” Use observed behavioural coverage, scenario representation, or test-intent …
- Distinctive: line 28
Every scored dimension must include evidence, confidence, and reasoning.
- Distinctive: line 30
Do not reward abstractions, retries, dashboards, test counts, or coverage percentages without assessing their actual value.
- Distinctive: line 69
Test files and framework configuration
- Distinctive: line 72
Test reports and historical artifacts
- Distinctive: line 86
Do not block the assessment merely because some evidence is absent.
- Distinctive: line 90
Prefer read-only or non-destructive commands.
- Very specific: line 115
Fixed ports: Before a run, look for hard-coded ports in configs and helpers (port: 4271, localhost:3000) and check they are free.
- Very specific: line 118
Coverage in a workspace: Add the coverage provider to the scratch copy only, at exactly the runner's version (@vitest/coverage-v8@<vitest …
- Very specific: line 122
A host OS that CI does not run: When CI runs only Linux but you are on Windows or macOS, tests that fail identically in every run on your …
- Distinctive: line 151
Before running anything, find out what the tests would touch:
- Very specific: line 153
Read the test setup files (vitest.setup.ts, jest.setup.js, conftest.py, globalSetup) and the test environment files (.env.test).
- Distinctive: line 162
If the tests need services you cannot provide safely (a real database, staging, credentials), do not run that layer.
- Very specific: line 246
Import mapping is test_derived.
- Very specific: line 252
Use the rubric in RUBRIC.md.
- Very specific: line 292
No suite: When discovery proved there are no tests (execution: no_tests), score the dimensions that describe the absence and leave the ones …
- Very specific: line 306
Default caps (with the rule id recorded in headline.applied_caps):
- Very specific: line 308
Test effectiveness below 4 → overall maximum 4 (effectiveness_below_4)
- Very specific: line 312
Test scope and coverage below 4 → overall maximum 4 (coverage_below_4): a suite that leaves major features or critical interfaces untested …
- Distinctive: line 340
Use one decimal place for weighted calculations, while warning that small decimal differences are not materially significant.
- Very specific: line 367
Test-count inflation (parametrised items counted as tests: report test functions as well as items.
- Distinctive: line 378
Test names implying behaviour not actually asserted
- Distinctive: line 414
Do not produce an unranked dump.
- Very specific: line 443
When terminal colour is supported, use ANSI colour through scripts/render_report.py:
- Distinctive: line 452
Never rely on colour alone.
- Very specific: line 475
1.1 → 1.2: add metadata.skill_version.
- Distinctive: line 479
Use direct language such as:
- Distinctive: line 486
Avoid vague praise. Do not describe a suite as robust, comprehensive, mature, or production-ready unless the evidence supports that …
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Test Suite Evaluator
Mission
Assess how much confidence an automated test suite deserves, not merely how polished its repository looks.
Produce a rigorous, readable assessment that distinguishes:
- Testing effectiveness — whether meaningful defects are likely to be detected.
- Suite trustworthiness — whether green and red results can be believed.
- Engineering quality — whether the suite is maintainable, reviewable, and appropriately designed.
- Operational health — whether CI history demonstrates reliability, speed, and sustained ownership.
- Evidence completeness — how much of the assessment is verified rather than inferred.
The evaluator may inspect files, run safe repository commands, execute tests, inspect reports, and analyse CI configuration when available. Never modify application or test code unless explicitly asked.
Non-negotiable rules
- Never claim line, branch, condition, or function coverage without source-linked instrumentation or a credible coverage report.
- Never call test-derived behaviour inference “code coverage.” Use observed behavioural coverage, scenario representation, or test-intent coverage.
- Missing evidence is
not_assessedorunknown, not automatically zero. - Every scored dimension must include evidence, confidence, and reasoning.
- Distinguish product failures, test failures, and environment failures.
- Do not reward abstractions, retries, dashboards, test counts, or coverage percentages without assessing their actual value.
- Penalise false-pass risk more heavily than cosmetic weaknesses.
- Static inspection alone cannot justify a 10/10 overall score.
- Do not reveal secrets found during inspection. Report only file location and secret type. A publicly documented development credential (a local storage emulator's well-known key, a framework's sample key) is not a secret leak, but treat it the same way: name its location and what it is, never its value.
Step 1: Establish assessment mode
Determine the strongest supported mode.
Black-box mode
Available evidence may include tests, fixtures, configuration, CI definitions, execution reports, screenshots, logs, and traces.
Can assess behavioural breadth, scenario depth, inputs, assertions, reliability indicators, maintainability, CI design, and diagnostics.
Cannot verify implementation branch coverage.
Grey-box mode
Adds one or more of: API specifications, route definitions, GraphQL schemas, protobufs, requirements, tickets, acceptance criteria, architecture documents, database schemas, or explicit risk models.
Can additionally assess specification coverage, interface coverage, requirement traceability, state coverage, role coverage, and integration-boundary coverage.
White-box mode
Adds application source and preferably runtime coverage or mutation reports.
Can assess implementation-linked coverage, untested modules, branch gaps, mutation effectiveness, and test-to-code alignment.
Record the mode and explain why it applies. The inventory must support it: white-box needs application source, grey-box needs requirements or specifications.
White-box with only source (no runtime coverage or mutation report) is weaker than white-box with instrumentation. Say which in metadata.notes; the validator warns when white-box rests on source alone. If the suite has a coverage provider configured (@vitest/coverage-v8, coverage.py, jest --coverage, JaCoCo; for Rust grcov, cargo-llvm-cov, or cargo-tarpaulin; for Go go test -cover, and the equivalent for other ecosystems), try a coverage run before settling for source only.
Coverage tools that install things. cargo llvm-cov needs the llvm-tools-preview rustup component; when it is missing, cargo-llvm-cov runs rustup component add llvm-tools-preview itself, without waiting for an answer, which downloads and installs into the user's shared toolchain. Check first with rustup component list --installed (look for llvm-tools). When it is not installed, ask the user before any coverage run; without their yes, record runtime_coverage as missing (not_assessed, never zero) or read CI's own coverage artefact (below). Never run a coverage tool outside the network guard: the refusing proxy stops such a download, and a refusal in the log is then the proof that it tried. The same holds for any tool that fetches on first use (cargo tarpaulin building its runtime, npx downloading a provider). A coverage report CI uploaded for the assessed commit (a coverage-*.json or lcov.info artefact, a Codecov upload) is runtime_coverage with origin ci and its as_of when you may read CI history; it is the way to see the CI's coverage gate when you cannot install the tool.
Step 2: Inventory available evidence
Inspect, where available:
- Test files and framework configuration
- Package manifests and lockfiles
- CI/CD workflows
- Test reports and historical artifacts
- Coverage and mutation reports
- Retry, quarantine, skip, and flaky-test data
- API schemas and requirement sources
- Application source
- Documentation and contribution guidance
- Secrets handling and environment configuration
Create an evidence inventory with status: available, partial, missing, or not_applicable. Give each entry a kind naming the RUBRIC evidence-completeness item it counts as (application_source, test_source, ci_configuration, recent_execution, historical_ci, runtime_coverage, mutation_report, requirements, defect_history, or other), and list all nine RUBRIC items, as missing when absent. Evidence completeness is computed from these entries. RUBRIC.md ("What counts as each evidence kind") says precisely what qualifies for each kind, including defect history and requirements; follow it rather than judging case by case.
application_source is available when all of the repository's own product code is readable, even if a library it imports lives in another repository you could not read (say so in details); it is partial when part of the system under test itself is unreadable (a service, a package, or a private submodule whose code the tests exercise).
Give an entry an origin (repository, ci, supplied, or assessor) and, for reports and logs, the date it describes (as_of). Your own runs during the assessment are recent_execution with origin assessor; they are never historical_ci, however many times you repeat them (the validator rejects that). Stale or hand-written artefacts committed to the repository (an old local run log, a hand-written per-route percentage summary) are other: mention them, but they earn no points and cannot support coverage claims.
Do not block the assessment merely because some evidence is absent. Continue with explicit limitations.
Step 3: Run safe checks when practical
Prefer read-only or non-destructive commands.
Examples:
- Dependency-aware lint or test commands already defined by the repository
- Test discovery/listing
- Dry runs
- Existing report generation
- Static searches for sleeps, retries, skips, weak assertions, hard-coded state, secrets, or unstable selectors
Do not access production systems, rotate secrets, delete data, or run destructive suites without explicit permission. Installing dependencies is fine when it goes into a scratch environment and nowhere else: a scratch virtual environment per Python component (python -m venv <tmp>/venv or uv venv, then the component's requirements, as CI does), or a scratch node_modules in the scratch copy. Never install into the checkout or a shared interpreter. Adding a measurement tool to that scratch environment only (pytest-cov or coverage.py, pytest-randomly) is also fine when the repository has none configured; say so in the run's notes.
Workspaces, several runners and dogfood suites
- One run per package. A pnpm, npm or yarn workspace often has a runner config per package (Vitest
projectssuch asunitandintegrationwith different worker counts) and more than one runner (Vitest in the packages,node --testfor scripts, the product's own runner for an end-to-end layer). Run each package's own test script from its directory (pnpm --filter <pkg> test, orvitest run --project unitinside it), keep a machine-readable report per package and runner, and add them up yourself. Count layers across packages: the unit projects of every package are one unit layer. Use--project unitfor the shuffled and coverage runs, which rarely need the slow layers. - Dogfood suites. A testing tool usually tests itself with itself: a suite run by the product's own CLI
(
node <pkg>/dist/cli/bin.js runin an app that depends on the workspace package). That suite is the repository's end-to-end layer: build the product first, run it as its package script does, and record it as its own layer with the product's runner as the tool. - Fixed ports. Before a run, look for hard-coded ports in configs and helpers (
port: 4271,localhost:3000) and check they are free; another process on the port givesEADDRINUSEor an "already running" failure that is yours, not the suite's. In a scratch copy, move the port and say so in the run's notes. - Coverage in a workspace. Add the coverage provider to the scratch copy only, at exactly the runner's version
(
@vitest/coverage-v8@<vitest version>; a mismatch refuses to start), with--filterfor the packages measured; it rewrites the scratch lockfile, never the checkout's. With projects, pass--coverage.include='src/**'so test helpers and configs are not counted. - A host OS that CI does not run. When CI runs only Linux but you are on Windows or macOS, tests that fail
identically in every run on your machine (paths,
os.devNull, shell quoting, file locking, line endings) areenvironmentfailures for the assessment, not reliability evidence. Run them again under the CI OS (WSL, a container) when you can, but only if that environment has the same isolation as your first run: a WSL distro with device tools (adb) on itsPATH, network access you did not guard, or the real home directory is not a safe place to run tests that control devices or write the profile. When you cannot, keep themenvironment, mark the layerpartial, and score what they show underportability(and the contributing guide's claims, if it documents that OS for development).
A scoped test command that compiles cleanly and exits zero while running zero tests is not a pass: check the reported test count, not just the exit code (for example cargo test -p <crate> when that crate's tests live elsewhere, or a path/pattern filter that matches nothing).
Rust (Cargo) workspaces
- Discover before building.
python scripts/cargo_project.py <repo> --out <scratch>/cargo.jsonreads the workspace without compiling: crates, targets (libs, binaries,tests/*.rs, benches, examples,required-features), helper binaries undertests/fixtures, features (default and optional), clap CLIs, and every test (unit tests in#[cfg(test)]modules, integration tests, doc tests) with itsignoredflag, itscfgexpressions and thefeaturesit needs. Those last two say which tests a plaincargo testwill not run. Compiling is the expensive step: build once into a scratchCARGO_TARGET_DIRoutside the checkout, with--locked(and--offlineonce the dependencies are in the cache), and reuse it for every later run. - Run what CI runs, per test.
cargo test --workspace --no-fail-fastwith the feature flags CI uses. Each test binary printsrunning N testsand atest result:line: add them up, check none is a scoped run of 0 tests (Step 8), and run doc tests too (cargo test --doc; nextest skips them). With cargo-nextest configured (.config/nextest.toml),cargo nextest run --workspace --no-fail-fast(--profile ciwhen CI uses it) writes JUnit under the target directory for the profile that setsjunit.path.cargo test -- --listlists the tests of each binary without running them. - Name the tests that did not run.
#[ignore]tests do not run unless asked (-- --ignored, only when you have checked what they touch); tests behind#[cfg(feature = "x")], binaries withrequired-features, and tests behind#[cfg(target_os = ...)]or#[cfg(unix)]compile out with the default features or on another platform. List each group fromcargo.jsonwith its count and record it as anot_run(orpartial) layer with the reason, or run it with--features x/--all-featureswhen the feature builds in your scratch environment (a GUI feature may need system libraries; say so when it does not). Never let them disappear into "all tests passed". - Guard the network for the test binaries. Neither in-process guard loads into a Rust test binary: wrap the command,
python scripts/net_guard_proxy.py run --log <scratch>/net_guard.log --home <scratch>/home -- cargo test --workspace --no-fail-fast --locked. It keepsCARGO_HOMEandRUSTUP_HOMEpointing at the real toolchain (so the empty home does not hide it) and setsCARGO_NET_OFFLINE=true. It guards only clients that honour the proxy variables (reqwest and ureq do by default);std::net::TcpStream::connect, a client built with.no_proxy(), raw UDP and DNS do not, so grep the tests and the code they call for those and record isolation aspartialevidence with that limit. - Random order on stable Rust. libtest's
--shuffleand--shuffle-seedare nightly-only (cargo +nightly test -- -Z unstable-options --shuffle-seed N), and cargo-nextest has no seeded shuffle. On a stable toolchain run the suite twice, and once more with-- --test-threads=1against the default parallel run: tests that pass alone and fail in parallel (or the reverse) share state, an isolation finding. Say that order-dependence was not tested by shuffling, rather than reporting "no order dependence". - Rust CLIs as interfaces.
scripts/list_interfaces.pylists clap sub-commands and options and the tests that run them (Step 4);required_featureson a binary says which builds have it.
Python components and unavailable dependencies
- One environment per component. A monorepo with several Python services usually has one requirements file and one virtual environment per service. Build each in the scratch area and run each service's tests with its own interpreter; record a layer per service if they differ.
- A private dependency you cannot fetch (a private git submodule, an internal package index): never fetch it with credentials you were not given. Tests that import it fail at collection, and pytest stops the whole session on a collection error unless you pass
--continue-on-collection-errors; pass it, so the rest of the suite still runs. Classify those failures asenvironment, and mark the layerpartialwith the reason (the tests that could not be collected did not run). You may stub the missing package to unlock the tests that only import it, but tests that read its data or behaviour still fail; say which results depend on the stub, keep those failuresenvironment, and keep the layerpartial. - Failures caused by your own limits (the unavailable dependency, a service you chose not to provide) say nothing about the suite's reliability; see the reliability anchors.
- Run the runner CI runs. When CI runs
python -m unittest discover(often ascoverage run -m unittest), that is the suite's runner, even if pytest would also collect the files. pytest collects any module-level function namedtest_*, including a helper (a@contextmanager def test_directory()), and reports it as a passing test (withPytestReturnNotNoneWarning), so its count can exceed what CI runs. Run the CI command for the recorded run; if you also run pytest (for JUnit or a shuffled run), compare the counts and name the extra items. The pytest plugin guard does not load under unittest: usescripts/py_guard.py(Step 3, item 4).
Check that running the tests is safe
Before running anything, find out what the tests would touch:
- Read the test setup files (
vitest.setup.ts,jest.setup.js,conftest.py,globalSetup) and the test environment files (.env.test). Look for defaults such asprocess.env.DATABASE_URL ??= ..., real API keys, and base URLs that point at shared or remote services. - Run in a scratch copy, not the checkout: export the tracked files (
git archive HEAD | tar -x -C <scratch>, or a plain copy without.git) and link the existing dependency directory into it rather than installing. Leave.gitout on purpose: tests of git-aware code call git on the enclosing checkout, and in a clone they find its index (one run left a stale.git/index.lockthat blocked every later git command there). If the tests need a repository, they create their own. Point every output at the scratch copy or a temporary directory (--outputFile,--coverage.reportsDirectory,--junitxml). Test runners write caches (node_modules/.vite,.pytest_cache), so the original tree must not be the working directory. - Supply fake configuration for anything the tests read: a loopback database URL, obviously fake keys (
sk_test_fake), andNODE_ENV=test. - Guard the network. Block every non-loopback connection for the run and count the attempts, with the guards this skill ships: for Node,
NODE_OPTIONS="--require <absolute path to this skill>/scripts/net_guard.cjs"(forward slashes on Windows; it wrapsnet.Socket.prototype.connectanddns.lookup, refuses anything other than127.0.0.1,::1orlocalhost, and also applies inside Vitest and Jest worker threads); for Python,scripts/net_guard.pyas a pytest plugin (-p net_guardwithPYTHONPATHreachingscripts/), which patchessocket.socket.connectandsocket.create_connection. The plugin guards the pytest process and its xdist workers only: a suite run withpython -m unittest(or any other runner), and the child Python processes tests start (subprocess.run([sys.executable, "scripts/x.py"])), are not guarded by it. For those, run the command underpython scripts/py_guard.py run --log <scratch>/net_guard.log -- python -m unittest discover -s tests: it puts a folder holding only asitecustomize.pyfirst onPYTHONPATH, so every Python process that inherits the variable loads the guard and writes its ownloadedline (count them against the processes you expect); an interpreter started with-I,-Eor-S, a child given an environment withoutPYTHONPATH, and non-Python children stay unguarded, so say so with the result. SetQA_NET_GUARD_LOG=<scratch>/net_guard.log; each guard writesnet_guard: loaded pid=<pid>for every process it loads in andnet_guard: refused <host>:<port>for each refusal (--self-teston either file checks it works here). Zero refused attempts is verified evidence for the isolation dimension; refused attempts are findings, with the hosts named. Check the log has aloadedline before claiming zero refusals: a guard that silently failed to load (a wrong path, aPYTHONPATHthe shell didn't convert) also refuses nothing, so with noloadedline the isolation evidence isnot_assessed. Neither guard loads into a Go, Rust or JVM test binary. For those, wrap the run withpython scripts/net_guard_proxy.py run --log <scratch>/net_guard.log --home <scratch>/home -- go test ./...: it starts a refusing loopback proxy, pointsHTTP_PROXY/HTTPS_PROXY/ALL_PROXYat it (NO_PROXYloopback), gives the command an empty home (HOME,USERPROFILE,XDG_*,APPDATA,LOCALAPPDATA) and lists what the tests wrote there; its log has the sameloadedandrefusedlines. It guards only clients that honour the proxy variables (Go's default transport does; a rawnet.Dial, a client with its own transport, DNS and UDP do not, and nothing is logged for them), so grep the tests for those and record network isolation aspartialevidence with that limit, never as verified. It keeps the toolchain homes where they were (GOCACHE/GOMODCACHE/GOPATH,CARGO_HOME/RUSTUP_HOME, package-manager caches) while it movesHOME, so the toolchain is still found; for Rust see "Rust (Cargo) workspaces" below. - Look for tests that control the host, before running them. Grep the tests and the helpers they call for commands run on the machine (
exec.Command,subprocess,child_process,Command::new) that start, kill or talk to devices and processes (adb,emulator,xcrun simctl,idevice*,pgrep/pkill/kill,taskkill,docker,systemctl), and package runners that download on first use (npx,pnpm dlx,pipx run,uvx,go run pkg@version), whose fetch happens in a child process the network guard does not see, and for writes under the user's home or profile (os.UserHomeDir,~,$HOME,Path.home(),os.homedir(), caches and downloads such as a browser fetched on first use). A test that kills "every matching process" or restarts a device server affects whatever else is running, including a real phone or emulator attached to the machine. Run with those tools offPATH(and their environment, such asANDROID_HOME, unset) unless the user agreed, and leave the tests that need them as anot_runlayer with the reason. - Run with a scratch home: set
HOMEandUSERPROFILE(Windows reads the profile fromUSERPROFILE, so a test that sets or reads onlyHOMEstill writes into the real profile), plusXDG_CONFIG_HOME,XDG_CACHE_HOMEandAPPDATA/LOCALAPPDATAwhere the code reads them, all pointing into the scratch area. Keep the toolchain homes where they are while the home moves (CARGO_HOME,RUSTUP_HOME,GOPATH/GOCACHE/GOMODCACHE, the npm, pip, Gradle and NuGet caches): with an emptyUSERPROFILE, rustup finds no default toolchain ("could not choose a version of cargo") and every test that runs cargo fails for the environment, not the code.python scripts/py_guard.py run --log <log> --home <scratch>/home -- <command>does both for a Python run (asnet_guard_proxy.py --homedoes for Go and Rust); a failure that names a missing toolchain is an environment failure, never a test result. Afterwards check the real home for files the run created; a write there is a finding (isolation), with the path.
Windows paths. From Git Bash (MSYS), a path argument is converted for Windows programs only when it is a whole argument: --cov-report=json:/c/Users/... (a path after a json: prefix) and a multi-entry PYTHONPATH=/c/a;/c/b are not converted, so Windows Python writes to C:\c\Users\... or silently ignores the entries (and with them the network guard). Pass Windows paths (C:/Users/...) to Windows programs, and check that outputs landed where you expected.
If the tests need services you cannot provide safely (a real database, staging, credentials), do not run that layer. Record it as not run with the reason.
Time budget
A full run of a large suite can take ten minutes or more, and a coverage run longer. Your own tool time limit is not evidence about the suite: if a command times out on your side, run it in the background and wait, or run part of it and record partial_run. Never score performance from your own timeout. Use a machine-readable reporter (vitest run --reporter=json --outputFile=<tmp>/run1.json, jest --json --outputFile, pytest --junitxml, Playwright --reporter=json): it gives per-file timings and makes comparing runs exact.
Run the suite twice when practical, and make one more run in random order (pytest -p randomly with pytest-randomly in the scratch environment, vitest run --sequence.shuffle, jest --randomize, go test -shuffle=on; Rust has a seeded shuffle only on nightly, see "Rust (Cargo) workspaces"), recording its seed. Tests that fail in one run and pass in the other are a verified flakiness observation for the reliability dimension; tests that fail only in the shuffled run depend on execution order, which is an isolation and reliability finding (name them, with the seed that reproduces it). Repeated runs during the assessment are still recent execution, not CI history.
A run that will not finish. Find the cause before waiting longer: pytest --durations=20 -p no:randomly on a bounded selection, or a sampling profiler (py-spy) on the busy worker, usually shows a function-scoped fixture redoing expensive work per case (re-reading a whole data directory for every parametrised item). For the assessment run, add a per-test timeout (pytest-timeout) and --junitxml, and with pytest-xdist --max-worker-restart 0. Killing an xdist run loses its JUnit file and its coverage data, so measure coverage on a bounded selection first, never on a run you may have to stop. A stopped run is still evidence: record it with stopped: true and not_reached (the collected count minus what it reported, from the progress output), and the layer it covered as partial unless a complete run covered it too; the validator enforces both.
Large data-validation suites. Parametrising over every data file (one test per JSON record, per content page) can turn a few hundred test functions into hundreds of thousands of items that test content, not code. Count test functions as well as collected items (pytest --collect-only -q, then group the ids by file::function) and report both; the item count alone inflates the suite (Step 8). Record the per-file data checks as their own layer (for example data_validation) beside the behaviour tests, so a runtime or a pass count dominated by them is not read as behaviour coverage. Run the behaviour layers in full; for the data layer and for the second and shuffled runs a fixed, recorded sample is acceptable (a seed and -k or a list of ids, the selection named in the run's command or notes), recorded as partial for that layer.
What CI runs versus what the runner config collects. Compare the commands in the CI workflows with what the runner's configuration collects (testpaths, --confcutdir, Vitest include, Jest roots). Tests CI runs outside the configured paths, and configured tests CI never runs, are separate layers; record each, with the command that runs it. CI history counts as evidence only for the layers those CI jobs ran.
CI gates. List the test and coverage gates CI enforces before judging them: python scripts/detect_conventions.py <repo> (the shared convention detector) writes ci_gates ({file, line, job, step, command, runner, fail_under} for cargo llvm-cov, cargo tarpaulin, cargo nextest, cargo test, coverage run -m unittest / -m pytest, coverage report --fail-under, go test) and, per Python area, python_runner (unittest or pytest, from the CI command). Read each gate's command, not just its threshold: a coverage gate measures only what that command builds and runs. cargo llvm-cov --workspace --fail-under-lines 90 with default features does not compile feature-gated code or its tests, and #[ignore] tests do not run under it, so the percentage says nothing about them; the same for a coverage run whose --source lists some modules. Report it as a finding (category other, dimensions ci and coverage) with the gate's file and line and the features or modules it cannot see, and recommend a second gate (--all-features, or --features <x> on the job that has the system libraries). A threshold is configuration, not a coverage claim.
Undeclared test dependencies. A public package the tests import but no manifest lists (an HTML parser, a browser driver) may be installed into the scratch environment, or into a --target directory put on PYTHONPATH for the run; record it in the run's notes. Its absence from the manifest is itself a portability finding.
When you measure coverage, tell the runner to write the report even when tests fail (Vitest --coverage.reportOnFailure=true; Jest writes it by default; pytest --cov writes it unless the session errors). Otherwise a single flaky failure leaves you with no coverage at all. Instrumentation slows the run and can add timeout failures; compare its failures with the uninstrumented run before blaming the product.
Record what ran and what it showed
Record commands attempted, exit status, duration, the environment that matters (runtime version, worker count, fake variables set, network guard on), and whether results are representative.
Record the strongest execution evidence as metadata.execution:
full_run: you ran every test layer the repository has during the assessmentpartial_run: you ran some layers or some tests, and at least one layer (for example an end-to-end layer that needs the app and a database) was not run or only partly runreports_only: you did not run it, but credible recent reports or CI results existstatic_only: nothing was run and no reports existnot_executable: discovery or execution was attempted and failed, and no credible reports existno_tests: discovery ran and found no tests at all (pytest --collect-onlycollected 0 items,vitest listlisted nothing, no test files exist). The absence is verified, so the evaluation is not provisional. Record the discovery run inmetadata.execution_result(an assessor run, every count zero, its layer named after the discovery, such aspytest discovery) and in the inventory asrecent_execution;test_sourceismissingwith details saying the absence was verified (the validator rejectsavailableorpartialbesideno_tests: test files that discovery did not collect mean the discovery failed, which isnot_executable). Score the dimensions as in "No suite" in Step 5A JavaScript or TypeScript repository with no test runner (no
testscript, no Vitest, Jest, Mocha or Playwright dependency or config) has no discovery command of its own. Search for test files first (*.test.*,*.spec.*,__tests__/,tests/,e2e/, outsidenode_modules), then run Node's built-in runner as the discovery, under the network guard:node --require <skill>/scripts/net_guard.cjs --test(Node 20 or later; by itself it finds*.test.*,*-test.*,*_test.*,test-*.*and files undertest/, TypeScript ones only where Node strips types: Node 22.6+ with--experimental-strip-types, by default from 23.6).node --testexits 0 when it finds no tests at all, so its exit status proves nothing: read the summary it prints (tests 0,suites 0;# tests 0with--test-reporter=tap) and record those counts as the discovery run, together with the file search. A file that fails to load is a failure in the summary, not zero tests. Only no test files found and a summary of zero tests together makeno_tests
The first three and no_tests need recent execution results in the inventory; static_only and not_executable must not claim them, and mark the evaluation provisional (metadata.provisional: true). provisional means exactly that: no execution evidence supports the scores. It is false for every other execution value.
For full_run and partial_run, record metadata.execution_result:
layers: every test layer the repository has (unit, component, integration, end-to-end...), eachrun,partial, ornot_run, with the reason when not fully run. A layer that exists and was not run makes the executionpartial_run, neverfull_run. A layer isrunonly when all its tests were collected and executed; tests that could not even be collected (a missing dependency, an import error) make itpartial.runs: each execution with itsorigin(assessor,ci_report,local_report), the layers it covered, and its passed, failed, skipped, and errored counts, duration, and whether it was instrumented for coverage. A run stopped before it finished hasstopped: trueandnot_reached; a layer only stopped runs covered ispartial.failures: every failing or erroring test, or a group of them with acount, classified asproduct(the system under test is wrong),test(the test is wrong, brittle, or timing-dependent),environment(the machine, a missing service, configuration), orunknown(not yet established; say what would settle it). Name the runs it failed in (runs_failed); a test that failed in some runs and not others is flaky.
A static review of a layer that did not run (for example Playwright specs read but not executed) may support design, assertion, maintainability and effectiveness judgements as test_derived evidence, but never execution, reliability or performance claims.
Step 4: Build a behaviour model
From each test or scenario, derive where possible:
- Feature or capability
- Starting state
- Actor or role
- Action
- Expected state transition
- Observable assertions
- Input classes
- Positive, negative, boundary, permission, concurrency, dependency-failure, and recovery dimensions
Aggregate this into behavioural coverage matrices (behavioural_coverage: happy path, negative, boundary, permissions, concurrency, dependency failure, recovery). State clearly that these are test-derived unless matched against specifications or source.
Interface coverage
When the application's interfaces can be listed, map each one to the tests that reach it and record the result in interface_coverage (total, tested, the untested ones, and how you matched them). This is cheap and often gives the strongest finding: an untested authentication, payment or data-deletion handler. It is interface coverage, not code coverage.
- Next.js App Router: list
app/**/route.ts(and.js) handlers; a handler is tested when a test file imports it (resolve@/and othertsconfig.jsonpathsaliases) or calls its URL. Count each exported method (GET,POST,PATCH,DELETE) when the tests show which method they exercise. Pages (page.tsx,.jsx,.js,.mdx) arekind: "pages", and metadata routes (robots,sitemap,manifest,favicon.ico,icon,apple-icon,opengraph-image,twitter-image, as static files or generators) are routes too; a content site with no API often has only these. Server actions ('use server',kind: "server_actions"), middleware, files inpublic/(a service worker is one) andnext.configrewrites and redirects are separate interfaces you map by hand. - FastAPI: count operations (method and path), not functions. List them from
app.openapi()["paths"]when the app imports in your scratch environment (note thatinclude_in_schema=Falseroutes are left out), or by walking each router's routes with itsAPIRouter(prefix=...)and theinclude_router(..., prefix=...)that mounts it. Do not walkapp.routesforAPIRouteobjects alone: newer FastAPI releases keep included routers as wrapper objects there, so the walk finds only the routes declared on the app itself (one of sixteen in one assessment). Match tests by method and path template (/items/{item_id}). - Express, Fastify, Flask, Django, Rails, Spring: the router or URL configuration lists the endpoints; match tests by URL and method.
- OpenAPI or GraphQL: operations from the specification (that makes it
specification_derived). - CLI and other multi-call binaries: when a tool dispatches to many sub-commands or utilities (a
git-style subcommand table, a coreutils-style multi-call binary, a plugin or handler registry), that dispatch table is the interface list. Map each sub-command or utility to the tests that invoke it (by name on the command line, or by calling its entry function), and record it withkind: "commands"(orkind: "cli"). An untested destructive sub-command (delete, overwrite, force) is often the strongest finding here. - Libraries, CLIs and tool servers: a package's public entry points (
package.jsonexportssubpaths, ormain) arekind: "package_exports"; itsbincommands and their sub-commands (commander, yargs, cac, argparse, click, typer;pyprojectscripts; Cargo binaries) arekind: "cli"; the tools an MCP or agent tool server registers arekind: "tools". For a CLI, SDK or MCP server these are the natural interfaces, not HTTP routes.
python scripts/list_interfaces.py <repo> --tests <test dir> lists FastAPI, Flask, Azure Functions, Next.js and Express routes (and a plain Node http.createServer as one entry), CLI commands, package exports and MCP tools from source (it never imports the app). Each entry carries its kind, the value for interface_coverage[].kind. In a Next.js project it also lists App Router pages (kind: "pages"; route groups dropped, [...x] and [[...x]] catch-alls, generate_static_params and dynamic_params recorded, a parallel-route slot page at its parent's URL with slot, an intercepting (.)x page at the URL it intercepts with intercepts), metadata routes (framework nextjs-metadata), pages/ router pages, and route handlers at the line of each exported method. What it found in a detected framework but does not read (Next.js server actions, middleware, public/ files, next.config rewrites and redirects) is listed in not_read and on stderr: the list is partial for those, so map them by hand. For a repository with no tests, pass --no-tests instead of --tests (every interface not reached, summary.tests: "none"); never point --tests at product code to get a run. Files under --tests that declare a listed interface are product code and are not read as tests, and when nothing under --tests looks like a test it warns on stderr and in tests_warnings. A route is reached when a test's URL literal matches it; for routers that prefer a rule without converters (Flask, Next.js, Go's ServeMux), a URL that a static route matches exactly (/robots.txt) reaches only that route, never a converter route it also fits (/<name>.txt). A Python module defining a bare handler(request, response) (a serverless function shape) is named in notes as not read. A command, tool or export when a test string names it or a test imports the export's source file. Naming a command is weaker than running it: read the hits before counting them. Computed names (tools registered in a loop over specs, a dynamic route prefix) come out as <dynamic> with a note; map those by hand. When it finds nothing it says so in notes ("no supported interface found"): that is not "no interfaces", so map them by hand or record interface coverage as not assessed.
It also lists what serves files and what guards requests: app.mount(path, StaticFiles(directory=...)) as a GET {path}/{path:path} route (framework starlette-static, with its directory) and Flask's static folder; app.add_middleware(...) and @app.middleware("http") under middleware (an authentication middleware is the "reached is not authorised" case below); and a route, router include or middleware that sits under an if (if args.uploads: app.include_router(...)) carries condition with the test, so a route that exists in one serving mode only says so. A flags-only argparse program is one cli entry with its options (a server's flags are the operator's entry point), and a hand-written command = sys.argv[1]; if command == "pack": dispatch is read as sub-commands. PHP is read by pattern only (preg_match and comparisons on REQUEST_URI in a front controller that .htaccess rewrites to) and named in notes, with the files it did not read. A dist/, build/ or out/ folder is skipped only when something in the repository produces it (a package.json build script, a bundler config, a .gitignore entry, generated-file headers): skipped folders are listed in skipped_dirs and on stderr, and --include DIR reads one anyway; a committed folder nothing produces is hand-written source and is read (source_build_dirs, with the reason in notes). --tests on a missing path exits 2; on a directory with no test files it warns, so "0 reached" is not read as a finding about tests.
It reads Go too: urfave/cli command tables (cli.App/cli.Command literals, Commands/Subcommands lists, commands held in variables or returned by functions), cobra (Use: and AddCommand), programs using the standard flag package (and flag.NewFlagSet sub-commands), net/http ServeMux routes (Go 1.22 "GET /path/{id}" patterns, gorilla .Methods), gin/echo/chi/fiber/httprouter routes with a literal path, listeners that register no route (go-listener: read them by hand, never counted as reached), and every importable package with an exported identifier (package_exports, its import path; internal/ and main are left out). Only _test.go files are read as Go tests: a command is reached when a test in its package names the variable that holds it, a route when a test requests its URL (httptest.NewRequest(http.MethodGet, "/x", ...)), a package by its own tests or an import of its path. Languages are counted by the shared lang_census.py: vendor/, third_party/, header trees, built bundles, submodules and embedded third-party packages (a directory with its own manifest and licence under drivers/, sdk/ and the like, or one whose manifest names another repository) are left out of the count and of the interfaces, and named in notes. When the dominant language is one it does not read, it says so loudly on stderr and in notes, records language_coverage (the census counts, status unsupported_dominant) and exits 3. Dominance is decided by lines of product code: test files and tooling folders (scripts/, tools/) are counted apart, so thirty short Python scripts and their tests do not outvote a Rust workspace.
It reads Rust in part, through the shared cargo_project.py (nothing is built): every Cargo binary (helper binaries under tests/fixtures are left out and named in notes) and, where clap parses its arguments (derive Parser/Subcommand/Args or the builder API), every sub-command and nested sub-command at its line, with options, option_lines, positionals, env and the binary's required_features; a binary with no clap parser is one cargo-bin entry to read by hand. A Rust command is reached only by a command line: a Cargo integration test that runs the binary (env!("CARGO_BIN_EXE_<name>"), assert_cmd's Command::cargo_bin("<name>"), std::process::Command on a path to the built binary, also through a helper module the test declares) with the sub-command's words among the strings it passes to .arg/.args or in a string array; a #[cfg(test)] module calling clap's parse_from/try_parse_from/get_matches_from with them; or a Python or JavaScript test whose process call has the binary in program position (the first element of the argv list, the value after --bin, a path under target/debug). Its name in a data string, a comment or an argument to another program never reaches it. Rust tests are read whenever --tests is given, wherever they live; options_reached lists the options those tests pass, and reached_by[].how says whether the test runs the binary or parses its arguments in process. Rust HTTP routes (axum, actix-web, rocket, warp), a library crate's public API and FFI exports are not read: a crate depending on one of those frameworks is listed in not_read. A repository whose product lines are mostly Rust is partial_dominant with a warning (exit 0), and one where Rust holds 30% or more of the product lines keeps partial but gets the warning too (partially_read says what was read): map what it does not read by hand, or record that part of interface coverage as not assessed.
Reached is not authorised. When fixtures replace the authentication or authorisation dependency (FastAPI app.dependency_overrides[get_current_user] = ... or a no-op permission check, a mocked auth() in Next.js, a test settings flag that disables auth), every route test runs without the real check, and "reached by a test" says nothing about access control. Record a second interface_coverage entry for the interfaces tested with the real dependency in place (for example "operations with a 401/403 test against the real permission check"); that is the security-relevant number, and an untested permission path is a finding.
List every untested interface in untested, or group them as {"group": "14 operations under /v1/items/{item_id}/...", "count": 14}; the entries must account for exactly total - tested (the validator checks).
Import mapping is test_derived; a coverage report listing each handler's executed lines makes it verified.
When you used list_interfaces.py, validate with --interfaces interfaces.json too: the validator then warns where interface_coverage departs from the script's list (a kind it lists that has no entry, a smaller total, a tested count other than its reached count). A hand mapping may rightly differ (you read the hits and dropped names that are not calls, or mapped tests the script cannot read); say how in the entry's method or notes.
Step 5: Score the dimensions
Use the rubric in RUBRIC.md. Score only supported dimensions.
Core dimensions and default weights:
| Dimension | id | Weight |
| Test effectiveness | effectiveness | 16 |
| Suite trustworthiness | trust | 14 |
| Reliability and determinism | reliability | 10 |
| Test design and level selection | design | 8 |
| Assertion and oracle quality | assertions | 8 |
| Test scope and coverage | coverage | 8 |
| Isolation and state management | isolation | 7 |
| Maintainability and appropriate reuse | maintainability | 7 |
| CI/CD integration | ci | 6 |
| Debuggability and reporting | debuggability | 4 |
| Performance and feedback speed | performance | 3 |
| Test data management | test_data | 3 |
| Environment portability and realism | portability | 2 |
| Security, privacy, accessibility, compatibility | security_a11y | 2 |
| Ownership and operational discipline | ownership | 2 |
Weights total 100. List all fifteen dimensions, each once. Reweight only when a dimension is genuinely not applicable: mark it applicable: false with a not_applicable_reason, give it weight 0 and score null, and rescale the other default weights proportionally so they total 100 again. Do not reweight merely because evidence is missing; keep the default weight, set the score to null, and lower evidence completeness and confidence instead.
For each dimension output:
score: integer 1–10, or null when not assessed (never 0). Use the anchors for the dimension inRUBRIC.md.confidence: high, medium, or lowevidence_level: verified, specification_derived, test_derived, mixed, or unknown- concise finding
- concrete supporting evidence with file paths or command results
- risks
- recommendations
A scored dimension must carry at least one piece of evidence and cannot have evidence level unknown; a dimension the evidence does not support is not assessed (score null, evidence level unknown), with the finding starting "Not assessed" and saying why. Placeholder findings are rejected: an empty or "n/a" finding, a marker such as TODO or TBD on its own or leading the text, or a short text holding one. A full sentence that uses such a word as the suite's own vocabulary ("the content rules check placeholder text") is not a placeholder.
Evidence levels belong to evidence, not to a whole dimension. The dimension's level must hold for every piece of its evidence. When a dimension rests on evidence of different levels (a coverage run and inferences read from tests), set evidence_level: mixed and write each evidence item as an object with its own level and an id: {"id": "e1", "level": "verified", "text": "vitest --coverage: lines 81.2%"}. A sentence of the finding, risks or recommendations that states a coverage figure must cite a verified item by its id ([e1]). Never pick verified for a dimension because some of its evidence is verified: that would license coverage wording for the inferred parts.
confidence: high needs at least two consistent evidence items and direct evidence (not test_derived alone).
No suite. When discovery proved there are no tests (execution: no_tests), score the dimensions that describe the absence and leave the ones that grade existing tests unassessed:
effectiveness,trust,assertionsandcoverageare 1 (absent), from the discovery run and the file search; their confidence may behigh, because the absence is directly verified (the validator checks they are 1).reliability,isolation,maintainability,debuggability,performance,test_dataandportabilityare not assessed (score null, "Not assessed: no tests exist"): there is nothing to grade, and unknown is not zero (the validator checks).design,ci,security_a11yandownershipare scored from their evidence: usually 1 (no test level, no pipeline running tests, no security check, no owner), higher only when something real exists (a CI pipeline that lints and builds, a CODEOWNERS entry for the futuretests/).
The overall is then 1.0 Dysfunctional with the effectiveness, trust, assertions and coverage caps listed (none binds). Write the first tests as the critical_risk recommendation.
Map file names and identifiers into evidence freely, including names that contain "coverage" (service-form-branch-coverage.test.tsx); write them in backticks when quoting. The coverage-language check ignores file names, identifiers and backticked text.
Step 6: Apply score caps and critical findings
Apply the lowest applicable cap after computing the weighted score.
Default caps (with the rule id recorded in headline.applied_caps):
- Test effectiveness below 4 → overall maximum 4 (
effectiveness_below_4) - Suite trustworthiness below 4 → overall maximum 4 (
trust_below_4) - Reliability below 4 → overall maximum 5 (
reliability_below_4) - Meaningful assertion/oracle quality below 3 → overall maximum 3 (
assertions_below_3) - Test scope and coverage below 4 → overall maximum 4 (
coverage_below_4): a suite that leaves major features or critical interfaces untested is Weak overall, however good its few tests are - Suite cannot be discovered or executed and no credible reports exist (
execution: not_executable) → mark overall as provisional; maximum 5 (not_executable) - Material secret or production-data exposure → critical finding (
category: secret_exposureorproduction_data_exposure, severitycritical); maximum 4 until remediated (secret_or_production_data) - Broad error swallowing, globally excessive retries, or systematic false-pass behaviour → critical finding with
category: systematic_false_pass; maximum 4 (systematic_false_pass) - Static-only assessment (
execution: static_onlyornot_executable) → maximum 8 (static_only)
List every cap whose condition holds, with its limit and an explanation, even when it does not bind because the score is already lower. The overall score is the raw weighted score lowered to the smallest limit. Explain every cap. Never silently modify the score. When no cap applies, say so in the report ("No caps applied").
The reliability anchors in RUBRIC.md decide whether the reliability cap applies: a suite that was red in every assessed run for reasons in the suite itself scores 3 or lower, not 4. Failures caused by the assessment's own limits (a private dependency you could not fetch, a service you chose not to provide) do not count towards "red in every run"; judge reliability on the tests that could run. A few identified order-dependent tests, found by a shuffled run, with the rest of the suite agreeing across runs, are a 4 (or higher when they are tracked and the rest is controlled), not a 3.
Thin suites. Reliability, assertions, isolation and performance grade the tests that exist. When the suite reaches little of the product (coverage below 4), those scores describe a small sample: their confidence is medium at most (the validator checks), their findings say how few tests they rest on, and the coverage_below_4 cap keeps the overall score from being carried by them. This does not apply to execution: no_tests: there is no sample, and a score of 1 for "no assertion exists" is directly verified.
critical_findings holds findings at critical, high, or medium severity; the report separates the findings of severity critical from the rest. Only the categories above cap the score. An untested security- or money-critical path (sign-in, two-factor, payments, data deletion, permissions) is at least a high finding with category: other, and lowers effectiveness and coverage through their anchors; it does not cap the score on its own unless the evidence shows systematic false passes. Give each finding the dimensions it affects.
Step 7: Calculate headline results
Report:
- Raw weighted score
- Capped overall score
- Testing effectiveness score
- Suite trustworthiness score
- Engineering quality score
- Operational health score, or
not_assessed - Evidence completeness, 0–100%
- Overall confidence
- Assessment mode
Use one decimal place for weighted calculations, while warning that small decimal differences are not materially significant.
How each is computed (see RUBRIC.md):
- Raw weighted score: the weighted mean of the scored dimensions. Unassessed dimensions are left out of the mean, never counted as zero.
- Testing effectiveness: the
effectivenessscore. Suite trustworthiness: thetrustscore. - Engineering quality: the weighted mean of
design,maintainability,debuggability,test_data, andportability. - Operational health: the weighted mean of
ci,performance, andownership, and only when historical CI outcomes are in the inventory; otherwise null (not_assessed). - A group score with no scored dimension is null.
- Evidence completeness: the RUBRIC points for the inventory, with partial evidence at half credit.
- Overall confidence cannot be
highwithout execution evidence.
Rating bands:
- 1.0–2.9: Dysfunctional
- 3.0–4.9: Weak
- 5.0–6.4: Serviceable
- 6.5–7.9: Strong
- 8.0–8.9: Excellent
- 9.0–10.0: Exceptional and evidence-backed
A static-only assessment cannot be labelled “Exceptional.”
Step 8: Detect gaming and misleading signals
Explicitly check for:
- Test-count inflation (parametrised items counted as tests: report test functions as well as items; helpers named
test_*that pytest collects in a suite CI runs with unittest) - Meaningless or tautological assertions
- Retries masking first-run failures
- Skips, quarantines, and expected failures without owner or expiry
- Skips conditioned on the product's own result (
if response.status_code != 200: pytest.skip(...), skip when a result is empty): a 404 or 500 from the code under test becomes a skip, not a failure - Coverage exclusions that hide difficult code
- CI files that do not run on meaningful triggers
- Page objects or helpers created only for appearance
- Generated expected results derived from the system under test
- Snapshots or golden files accepted without review discipline
- Reports that are attractive but diagnostically weak
- Test names implying behaviour not actually asserted
- Coverage-chasing tests: files named for coverage (
*-coverage.test.*), titles starting "covers …", comments citing source line numbers, tests that call code without asserting its outcome - Assertion-free tests (no
expect/assertin the body; theexpect-expectlint rule finds them) andtry { … } catch {}inside a test body - Configured thresholds that never run: coverage thresholds in
vitest.config.*orjest.config.*when CI runs tests without--coverage, or quality gates on a job that is not required
JavaScript and TypeScript stacks (Vitest, Jest, Next.js, Playwright) add signs that are easy to miss:
- A global
environment: 'jsdom'applied to API route or server tests, so server code runs with browser globals it never has in production - A global
vi.mock/jest.mockin a setup file that replaces a module (the database client, auth, a payment SDK) in every test, hiding integration behaviour everywhere - The database client (Prisma, Drizzle, TypeORM) mocked in every test while CI provisions a real database only for migrations, so no test exercises a real query
npm testscript environment variables (TZ=UTC,NODE_ENV) silently overridden in a setup file, or tests that depend onnpm_package_versionand other variables onlynpm runsets (they pass undernpm testand fail undernpx vitest)- An end-to-end layer (Playwright, Cypress) that CI never runs, or runs with retries that hide first-attempt failures
Python stacks (pytest, FastAPI, SQLAlchemy) have their own:
app.dependency_overrides[<auth dependency>] = <no-op>in a shared fixture or conftest, so every route test runs without authentication or permission checks (the Python twin of a globalvi.mockof auth); see "Reached is not authorised" in Step 4pytest.MonkeyPatch()created at module level in a conftest and never undone, oros.environ[...] = ...at import time in a conftest: the change leaks into every test in the sessionimportlib.reload(module)in a test: configuration the module read at import stays changed for later tests, and other tests silently depend on that order- Fixtures that drop and recreate a schema (
DROP SCHEMA public CASCADE,metadata.drop_all) on whatever database URL the environment names, with no check that it is a test database (name or host) - CI selecting tests by marker (
-m "not integration",-m integration) while some integration tests carry no marker, so they run in the wrong job or in none - The same test directory run by two targets (two Makefile targets or CI jobs that both include a shared tests folder), which double-counts its results
- Unit tests that read data files from a git submodule or another repository, so they fail wherever that is unavailable
Azure DevOps pipelines:
- No
pr:trigger in the YAML: pull-request validation then depends on branch policies that live in the project settings, not the repository. Recordci_configurationaspartialand the gating as unknown rather than assuming either way template:references to files in another repository or atemplates/folder you cannot read: the jobs they define are unknown, so say which stages you could not see
Other ecosystems have their own false-pass traps:
- Rust: a scoped command that compiles and reports success while running 0 tests (
cargo test -p <crate>when the integration tests live in a different crate, or a--test/name filter that matches nothing) is not a pass — read the "running N tests" line.#[ignore]on a test with no owner or expiry is a permanently silenced test, not a passing one.#[cfg(unix)],#[cfg(target_os = ...)], or feature-gated tests silently compile out on other platforms or without the feature, so CI on one target reports green while those tests never run anywhere; check which targets and feature sets CI actually exercises. A coverage gate (cargo llvm-cov --fail-under-lines) run with the default features inherits the same blind spot: the feature-gated code is not even in its denominator (see "CI gates" in Step 3). - Go:
go test -jsonis the machine-readable reporter (each package and test as events;gotestsum --junitfileturns it into JUnit),-count=1defeats the result cache (a cachedokran nothing now), and-shuffle=on(it prints the seed) is the random-order run. Tests gated byruntime.GOOSskips,//go:buildtags,testing.Short()or an environment variable compile out or skip on other targets, so a green Linux CI may never run the macOS or Windows paths; check whichGOOS, tags and-shortsettings CI uses. A test file that does not compile on oneGOOS(asyscallcall that exists only on Unix) makesgo test ./...report the whole package as[build failed]there: that package's tests did not run, which is a portability finding and anerroredrun layer, not N failures. A test whose body is onlyt.Skip(...)is permanently silenced.codecov.ymlignore:lists (andcoverage.ignorein other tools) can drop the weakest packages from the headline number: read them before quoting a coverage figure, and say which paths were excluded. A failingt.Runsubtest also fails its parent, so failure counts from per-test events count it twice. - Generally: tests gated behind a platform, OS, feature flag, environment variable, or build tag that the pipeline never sets are dead weight that still shows green; confirm the gated paths are run somewhere, and treat a run that executed no tests as
not_run, never a pass.
Step 9: Prioritise recommendations
Do not produce an unranked dump.
Return these groups, each recommendation carrying its group as category, and rank priority through the groups in this order:
- Critical confidence risks (
critical_risk): required when a finding of severitycritical, a finding in a capping category (secret_exposure,production_data_exposure,systematic_false_pass), or a cap other than the static-only caps applies. Ahighormediumfinding of categoryotherdoes not require one on its own. With no tests at all (no_tests) the score caps apply, so the firstcritical_riskis writing the first tests for the riskiest paths: the absence of tests is itself the critical confidence risk - Three highest-leverage improvements (
top_improvement): one to three - Quick wins (
quick_win): small effort only - Structural improvements (
structural) - Evidence needed for a stronger reassessment (
evidence_needed): required when any inventory item is missing or partial; name the inventory kinds it would supply inevidence_kinds
Each recommendation must include:
- impact: critical, high, medium, low
- effort: small, medium, large
- expected score dimensions affected (
dimensions_affected; optional for evidence needed) - concrete implementation suggestion
- acceptance criteria
Step 10: Produce beautiful output
Generate both:
evaluation.json, conforming toschemas/evaluation.schema.json- A concise human report in Markdown or terminal format
Run python scripts/validate_evaluation.py evaluation.json and fix every error before reporting. Besides the schema it recomputes the weights, the raw and group scores, the caps, the overall score and band, and evidence completeness, and rejects scores without evidence, a reliability score of 7 or more beside test-class failures the runs disagree on, high confidence in the quality of a thin suite's few tests, untested interface lists that do not account for every untested interface, evidence levels that do not hold for every item, high confidence without direct and multiple evidence, an execution value its layers and runs contradict, unclassified failures, assessor runs offered as CI history, recommendations outside the Step 9 groups, and any code-coverage claim made without a runtime coverage report or from evidence that is not verified. A stated coverage percentage is a claim whatever else its sentence says ("No tests for 2FA, so branch coverage is 83%" is a claim). With --interfaces interfaces.json (the list_interfaces.py output) it also warns where interface_coverage departs from that list (Step 4). Write and read the JSON as UTF-8.
Render the report with python scripts/render_report.py evaluation.json for the terminal, or --markdown report.md for a Markdown file. Both show every section below.
When terminal colour is supported, use ANSI colour through scripts/render_report.py:
- Green: 8–10
- Cyan: 6.5–7.9
- Yellow: 5–6.4
- Magenta: 3–4.9
- Red: 1–2.9 and critical findings
- Dim text: unknown or not assessed
Never rely on colour alone. Include text labels and symbols.
Recommended report order:
- Title and assessment mode
- Overall score card (raw and capped score, each cap or "No caps applied", the decimal-significance warning)
- Critical findings
- Dimension scorecard, with each dimension's evidence, risks and recommendations
- Behavioural coverage matrix and interface coverage
- Recommendations by group, with acceptance criteria
- Evidence limitations
- Appendix with execution results, commands and inspected artifacts
Migrating older evaluations
metadata.skill_version is required and names the contract an evaluation was written for. The validator warns when it differs from this skill's version. To bring an older evaluation forward:
- Before 1.1: rename dimension ids
trustworthiness→trust,oracles→assertions,scope→coverage,data→test_data,environment→portability,crosscutting→security_a11y; addmetadata.executionand akindfor every inventory entry. - 1.8 → 1.9: nothing in the JSON changes. The note answering the white-box warning must now say what reading source means for the claims ("implementation-linked claims rest on reading source, not instrumentation"); a note that only mentions reading source for another reason no longer silences it.
list_interfaces.pylists Rust clap sub-commands and options and no longer counts a Cargo binary as reached by its name in a test string: re-run it on a Rust repository, and pass its output to the validator with--interfaces. - 1.7 → 1.8: an evaluation with
execution: no_testsmust recordtest_sourceasmissing(one markedavailableorpartialis now rejected: if test files exist that discovery did not collect, the discovery failed and the execution isnot_executable). Nothing else changes;list_interfaces.pylists Next.js pages and metadata routes, so re-run it on a Next.js project for interface coverage. - 1.6 → 1.7: nothing to change.
metadata.executiongainedno_testsfor a repository whose discovery found no tests (re-record a verified empty discovery that was written asstatic_onlywithprovisional: trueasno_tests, not provisional). The white-box warning about reading source is answered by ametadata.notesentry saying implementation-linked claims rest on reading source, not instrumentation. - 1.3 → 1.4: nothing to change;
interface_coverage[].kindgainedpackage_exportsandtools(entries recorded asotherfor these may move to them). - 1.2 → 1.3: list the
coverage_below_4cap when coverage is below 4 (and recompute the overall score); lower tomediumthe confidence of reliability, assertions, isolation and performance when coverage is below 4; make eachinterface_coverageentry'suntestedaccount for exactlytotal - tested(names, or{"group", "count"}entries); reliability 7 or more cannot stand beside a test-class failure that some runs show and others do not. - 1.1 → 1.2: add
metadata.skill_version; addmetadata.execution_resultforfull_runandpartial_run; setprovisionaltrue only forstatic_onlyandnot_executable; give every recommendation acategory(andevidence_kindsfor evidence needed); set unassessed dimensions' evidence level tounknown; split mixed-evidence dimensions into per-item levels (mixed).
Required honesty language
Use direct language such as:
- “Instrumented code coverage was not measurable from the supplied evidence.”
- “This gap is inferred from test content, not verified against application source.”
- “Reliability confidence is low because historical CI outcomes were unavailable.”
- “The suite appears green partly because retries or exclusions suppress failures.”
Avoid vague praise. Do not describe a suite as robust, comprehensive, mature, or production-ready unless the evidence supports that statement.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

