corpus-qa-sweep: check every page, not one page

Why this skill exists

One artifact passing is not evidence that the corpus passes. A renderer change that looks perfect on the page you happened to open will meet content you did not imagine: a doc that is 90% tables, a heading that ends in the word you used as a marker, a version number in the middle of a sentence.

The second reason matters more, and is the part people skip. On a first sweep, most flags are defects in the harness, not the product. A sweep that reports findings without triaging them manufactures work and erodes trust in the next sweep. The triage step is not optional polish; it is the skill.

When to invoke

• A change to a renderer, template, generator, stylesheet, or shared component affects many outputs at once.
• Someone says "check all the pages", "use more test cases", "make sure there are no surprises", or asks for a pre-release regression pass.
• A single-sample verification passed and the evidence feels thin.
• A corpus has grown and nobody knows whether old items still hold.

Do not use this for a single artifact. One chart, one figure, one report goes to render-verify, which judges a rendered artifact against a failure catalog using human or model judgment. The two compose: sweep to find candidates, then judge one.

Step 1 — instrument the real output boundary

Capture what the system actually emits, at the last point before a human or a consumer sees it. Never re-run your own logic as the oracle; that only proves the logic agrees with itself.

Interception beats reimplementation because it survives refactors and cannot silently drift from the shipped path.

Step 2 — write invariants a machine can check

An invariant is a claim that must hold for every item. Prefer claims that fail loudly over claims that need a human to read them.

Count agreement is the strongest of these. "Twelve markers for twelve tables" is checkable; "the output looks reasonable" is not.

Step 3 — sweep the whole corpus

• Enumerate from the manifest, index, or router, never from a hand-written list. A hand-written list drifts and quietly stops covering new items.
• Batch the run to stay under tool and timeout limits.
• Bound every wait, and report whether the bound was hit. A truncated run looks exactly like a clean run with fewer results. Emit a guardHit flag and treat it as invalidating, not informational.

Step 4 — triage every flag before believing it

Treat a flag as a hypothesis about the product, and the harness as the first suspect. Look at the offending item before filing anything.

Only after the item is in front of you is the finding a defect. Report harness false positives explicitly. They are evidence about the sweep's own reliability and they stop the next person re-chasing them.

Step 5 — compare representation-insensitively

When the product deliberately rewrites content, a raw comparison reports loss that did not happen. Normalize both sides to the part that must survive.

Letters-only normalization is usually enough: strip everything except a-z, lowercase, then compare a distinctive slice. It tolerates rewrites, whitespace, and punctuation while still catching genuinely dropped content.

Step 6 — report clean count and violations separately

State how many items were swept, how many were clean, and what each violation was. Separate product defects from harness defects in the report. A sweep that says "7 violations" when 4 were its own bugs is worse than no sweep.

Worked example

The sweep that produced this skill, over a documentation shell's read-aloud feature:

Four of six first-pass findings were the harness. That ratio is the reason Step 4 exists.

Anti-patterns

Related skills

• render-verify — judge one rendered artifact against a failure catalog. Use it on the items this sweep flags.
• docs-shell — the shell whose manifest enumerates the corpus when sweeping a documentation site.
• figure-generator — contract tests pinning headline numbers, the same invariant idea applied to one generated figure.

Would Revise If

Revise this skill if any of the following fires:

• A sweep run by this method reports violations that are later found to be entirely harness defects, meaning Step 4 was performed but did not work.
• A corpus-wide sweep passes and a user still hits a defect of a class the invariants in Step 2 were meant to cover.
• The harness-first triage posture causes a real product defect to be dismissed as a false positive.
• Enumerating from a manifest proves impossible for a corpus this skill is invoked on, and Step 3 needs a discovery fallback.
• Zero sweeps are run by this method in the window, which would mean the skill is decorative for its intended audience.