sumo-qa-measuring-coverage
Skill7 starsChanged 40 days agoApache-2.0View on GitHub
Use after sumo-qa-deciding-approach routes here, when the user wants coverage or mutation numbers in the local QA report — "run coverage", "measure coverage", "run mutation testing", "record the survivors". Runs the repo's configured coverage/mutation tool, reads its output (any format), and persists a compact summary into the .sumo-qa artifacts the report loads.
- Installs packages
In this repository21 other files it gives its agents.
- AGENTS.mdAGENTS.md
- .github/copilot-instructions.mdCopilot instructions
- regen-eval-baselineSkill
- scaffold-sumo-qa-skillSkill
- sumo-qa-answering-testing-questionSkill
- sumo-qa-closing-qa-gapsSkill
- sumo-qa-creating-test-planSkill
- sumo-qa-deciding-approachSkill
- sumo-qa-executing-qa-rolloutSkill
- sumo-qa-finding-test-dataSkill
- sumo-qa-finishing-qa-workSkill
- sumo-qa-implementing-with-tddSkill
- sumo-qa-planning-qa-rolloutSkill
- sumo-qa-preparing-for-workSkill
- sumo-qa-reviewing-before-mergeSkill
- sumo-qa-security-testingSkill
- sumo-qa-strategisingSkill
- sumo-qa-strengthening-testsSkill
- sumo-qa-suggesting-external-skillSkill
- sumo-qa-triaging-test-failuresSkill
- using-sumo-qaSkill
The file
We split this skill into the rules it gives an agent
3 rules. About 0 are common, 1 is distinctive, and 2 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈1
Distinctive · estimate
This author's own choices. Worth a look.
2
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 15
NEVER fabricate a number. A coverage percentage or survivor count goes into an artifact ONLY when it came from a real run (or a report the …
- Very specific: line 28
NOT for: live PostToolUse mutation-survivor routing (that is #127's router — a different concern.
- Very specific: line 57
Run: pytest --cov=src/pkg --cov-report=json → coverage.json, 96.4% lines.
≈ “Common” is an estimate: at least 1 in 5 of the 350,495 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Measuring coverage & mutation
Turn a repo's coverage/mutation run into evidence the QA report can cite. The report reads .sumo-qa/coverage.json and .sumo-qa/mutation.json; this skill produces them. The host (you) runs the tool and reads its output; the MCP tools only validate + persist.
Output discipline (mandatory)
Inherits the global discipline from using-sumo-qa: never surface internal taxonomy labels; spend output on findings not framing; one question per turn; no preamble or closing pleasantries.
<HARD-GATE>
NEVER fabricate a number. A coverage percentage or survivor count goes into an artifact ONLY when it came from a real run (or a report the user pointed you at) in this session. No run / no report ⇒ record nothing, report "not available".
</HARD-GATE>The Iron Law
MEASURED, REPORTED, NEVER GATED. Coverage and mutation are evidence the report shows — they never become a pass/fail gate and never flip a readiness verdict. A 100% coverage signal does not make a risk "covered" or a blocked verdict "ready". If you catch yourself treating a percentage as a safety threshold, stop.
When to Use
- "run coverage and put it in the report", "measure coverage", "what's my coverage" (when the report should carry it)
- "run mutation testing", "record the survivors", "get mutation into the scorecard"
- As a follow-on to a QA rollout, before
sumo_qa_generate_qa_report, when you want the Coverage/Mutation rows populated instead of "not available".
NOT for: live PostToolUse mutation-survivor routing (that is #127's router — a different concern; see sumo-qa-strengthening-tests). Reading a saved report ≠ running the router.
Checklist
- Detect (configured tooling only). Inspect for tooling the project already pins — coverage:
pyproject [tool.coverage]/--covin pytest addopts /.coveragerc,package.jsonjest--coverage/nyc/c8,go test -cover, a committedcoverage.xml(Cobertura) /coverage.json(coverage.py) /lcov.info; mutation:[tool.mutmut]/mutmut, Stryker (stryker.conf.*,mutation.json), PIT (mutations.xml). Neverpip install/npm ia tool. Nothing configured → stop, report "not available" (step 5). Configured but unrun → offer a scoped run; if declined, fall back. - Run the detected command, scoped. Run the project's own command, preferring a machine-readable report and scoping to changed code when a diff is in play (e.g.
pytest --cov=<pkg> --cov-report=json;mutmut run+ its results summary). A run that errors (missing dep, misconfig) → report it plainly and fall back; never mask it. - Collect (LLM reads any format). Read the output — JSON, XML, lcov, or console text — and extract: coverage →
line_percent+ a compactdetailnaming uncovered/weak changed files/functions (tie gaps to.sumo-qa/diff-impact.jsonwhen present — changed code, not global vanity totals); mutation →survivors+killed+ adetailon where notable survivors live. Setfreshness="fresh"(you just ran it),source_tool= what you ran,generated_at= an ISO-8601 timestamp. - Persist. Coverage →
sumo_qa_record_coverage(root, coverage={...}); mutation →sumo_qa_record_mutation(root, mutation={...}). Each validates and writes the artifact, or returns an error envelope (fix the payload, don't hand-write the file). Then tell the user to (re)runsumo_qa_generate_qa_reportto see the Coverage/Mutation rows populate. - Fallback (no tool / no run). Report concisely: "Coverage/mutation not available — no coverage tool is configured in this repo." Record nothing. A first-class outcome, never a workflow failure, never a reason to invent a number.
Process Flow
See the Checklist above — that's the flow.
Red Flags — STOP
| Thought | Reality |
| "Coverage is 100%, so this risk is covered / it's safe to merge" | Iron Law. Coverage is reported, never a gate. It cannot discharge a risk or flip a verdict. |
"No coverage tool here, I'll just pip install one" | Never. Configured tooling only; absent tool ⇒ "not available". Adoption is a separate decision (sumo-qa-strategising), repo-pinned, never ad-hoc. |
| "I'll estimate the percentage from reading the code" | Fabrication. A number comes from a real run or a real report, or it doesn't get recorded. |
| "I'll re-run the suite to generate a mutation report for the live router" | Wrong skill. Reading a SAVED report here ≠ #127's PostToolUse router. Don't conflate them. |
"I'll hand-edit .sumo-qa/coverage.json" | Use sumo_qa_record_coverage — it validates the shape. A bad payload is an error to fix, not a file to forge. |
| "The run errored, I'll record the last known number" | Stale/fabricated. Report the failure, fall back to "not available". |
Example
Repo pins pytest --cov in addopts. User: "get coverage into the QA report."
- Detect:
[tool.pytest]addopts carry--cov=src/pkg. ✓ - Run:
pytest --cov=src/pkg --cov-report=json→coverage.json, 96.4% lines. - Collect:
line_percent=96.4,detail="changed-file gaps: pkg/api.py L40-44 (error branch uncovered)",freshness="fresh",source_tool="pytest-cov (coverage.py json)". - Persist:
sumo_qa_record_coverage(root=".", coverage={...})→.sumo-qa/coverage.json. Tell the user to regenerate the report.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.