qa-reviewer
Skill1 starsChanged 55 days agoMITView on GitHub
Review a phase-C section markdown for ONE chunk against the binding authoring rules from `config/templates/*.template.md` and the active `profile.yaml`. Runs deterministic mechanical checks (forbidden tokens, minimum sentence/word counts, citation footnote presence, identifier styling, no line-count prose, no `[FILE:Lx-Ly]` inline tags) and emits a `findings[]` JSON sidecar at `docs/_reviews/section-doc/<file>__<chunk_id>__<lang>.json`. USE FOR: review section doc, check section markdown, qa pass on phase-C output, validate citations, validate identifier styling, lint forbidden tokens in Summary / Purpose, audit forbidden openers, verify diagram presence, gate before phase-D rendering. DO NOT USE FOR: chunking (phase A — use cobol-chunking), facts extraction (phase B — use cobol-facts-extraction), per-chunk text/facts prep (use chunk-text-extraction / facts-slicing), authoring the section markdown itself (use section-doc-writer), file-level / diagram-renderer reviews (different scope in pipeline-dag.yaml phase C).
2 other projects use a version of this.
Adopted by
Most recently taken up by
- genfeedai/genfeed.aitheir own version, first seen Sep 12, 2026
- aico-platform/guardrails-demotheir own version, first seen Jul 4, 2026
Other versions2 projects use a version of this.
Word for word · 0
None found.
Edited · 0
None found.
Wrote their own version · 2
- genfeedai/genfeed.ai★ 4same name
- aico-platform/guardrails-demo★ 0same name
How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.
In this repository17 other files it gives its agents.
- chunk-text-extractionSkill
- cobol-chunkingSkill
- cobol-facts-extractionSkill
- diagram-rendererSkill
- facts-slicingSkill
- functional-analysis-finalizerSkill
- functional-analysis-group-finalizerSkill
- functional-analysis-group-writerSkill
- functional-analysis-writerSkill
- group-doc-finalizerSkill
- requirements-finalizerSkill
- requirements-group-finalizerSkill
- requirements-group-writerSkill
- requirements-writerSkill
- section-doc-finalizerSkill
- section-doc-writerSkill
- technical-analysis-writerSkill
The file
We split this skill into the rules it gives an agent
4 rules. About 0 are common, 1 is distinctive, and 3 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈1
Distinctive · estimate
This author's own choices. Worth a look.
3
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 18
Do not use this skill for:
- Very specific: line 106
All checks are pure regex / counting — no LLM call.
- Very specific: line 144
After section authoring, run scripts/review-section-set.py to validate the complete indexed section set:
- Very specific: line 156
Before invoking any checker, the sweep requires a current source manifest, a non-empty, unambiguous chunks index, every indexed markdown …
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
QA Reviewer (per-section, phase C)
Deterministic + LLM-assisted gate that fires immediately after section-doc-writer writes a per-section markdown. This skill owns the mechanical half — a Python checker that enforces the binding authoring rules from the templates and profile. The semantic prose review (does the Purpose paragraph actually describe the business intent? do the citations point at the right lines?) is the LLM qa-reviewer agent's job and consumes the same findings[] sidecar as its starting point.
When to Use
- Immediately after every
section-doc-writerper-section invocation (the pipeline-DAG order). - Re-validating an existing
docs/<lang>/<file>/sections/*.mdafter manual edits. - Pre-commit / CI gate before phase-D diagram rendering or phase-N portal build.
- The user asks to "review", "lint", "qa-check", "audit citations" on a section doc.
Do not use this skill for:
- chunking (
cobol-chunking), - facts extraction (
cobol-facts-extraction), - per-chunk text / facts prep (
chunk-text-extraction,facts-slicing), - writing the section markdown (
section-doc-writer), - per-file
index.md/complete.mdreview or diagram-renderer review — those are differentqa-reviewerscopes inpipeline-dag.yamlphase C.
Output Contract
For section markdown docs/<lang>/<basename-with-ext>/sections/<chunk_id_sanitized>.md reviewed at language <lang>:
docs/_reviews/section-doc/<basename-with-ext>__<chunk_id_sanitized>__<lang>.jsonShape:
{
"review_version": "1.0",
"section_doc": "docs/en/DEMO100.CBL/sections/…__B200-EX.md",
"section_doc_sha": "<64-hex>", // hash of reviewed bytes
"source_path": "repos/sample1/COBOL/DEMO100.CBL",
"chunk_id": "DEMO100/procedure-division/B200-CONTROLLO-PARAMETRO/B200-EX",
"language": "en",
"profile_path": "config/templates/section-doc.profile.yaml",
"thresholds": {
"min_total_words": 1500,
"min_purpose_sentences": 4,
"min_purpose_words": 80,
"max_tldr_words": 30,
"identifier_lock_regex": "[A-Z][A-Z0-9-]*(\\.[A-Z0-9-]+)?",
"require_citations": true
},
"ok": false,
"findings": [
{
"code": "TLDR_FORBIDDEN_TOKEN",
"severity": "error",
"section": "Summary",
"line": 12,
"message": "Summary contains forbidden COBOL jargon: 'PERFORM'."
}
/* zero or more; sorted (severity desc, line asc) */
],
"counters": {
"total_words": 842,
"tldr_words": 22,
"purpose_sentences": 5,
"purpose_words": 112,
"src_footnotes": 7,
"inline_file_tags": 0
}
}Exit codes mirror the verdict:
0—ok=true, zeroseverity=errorfindings.1— at least oneseverity=errorfinding (review file still written).4— required input missing (markdown not found or bundle unavailable).5— stale bundle (bundle.chunk_text_shamismatch vs. on-disk chunk text).
severity is one of error (blocks pipeline), warn (logged, non-blocking), info (advisory).
Mechanical Findings (enforced by the checker)
| Code | What triggers it | Severity |
TLDR_MISSING | No ## Summary block (or legacy ## TL;DR), or block has zero non-whitespace content. | error |
TLDR_TOO_LONG | Summary exceeds thresholds.max_tldr_words (default 30). | error |
TLDR_MULTI_SENTENCE | Summary contains more than one sentence (split on ., !, ?). | error |
TLDR_FORBIDDEN_TOKEN | Summary contains any of: PERFORM, EVALUATE, WORKING-STORAGE, GO TO, DECLARATIVES, or a line/verb count phrase. | error |
PURPOSE_MISSING | No ## Purpose block. | error |
PURPOSE_TOO_SHORT_SENTENCES | Purpose has fewer than min_purpose_sentences (default 4) sentences. | error |
PURPOSE_TOO_SHORT_WORDS | Purpose has fewer than min_purpose_words (default 80) words. | error |
PURPOSE_FORBIDDEN_OPENER | Purpose starts with one of: "This section contains the following…", "The section spans lines", "This PROCEDURE DIVISION section implements the following logic…", or equivalent transcript-style preface. | error |
PURPOSE_LINE_COUNT_PROSE | Purpose mentions a line count, line range, or verb count (PERFORM, CALL, GO TO counts). | error |
INLINE_FILE_TAG | Any occurrence of [FILE:Lx-Ly] style citation in body (must use [^src-N] footnotes). | error |
MISSING_SOURCES_BLOCK | require_citations=true and document has no ## Sources heading. | error |
MISSING_SRC_FOOTNOTES | require_citations=true and document has zero [^src-N] footnote references. | error |
IDENTIFIER_NOT_BACKTICKED | A token matching identifier_lock_regex appears in prose but is not wrapped in backticks. | warn |
BANNED_BANNER | The reviewed doc contains the forbidden "Self-contained document. Bundles… from facts.json…" banner. | error |
MIN_TOTAL_WORDS | Total word count below thresholds.min_total_words (default 1500). At per-section scope this is warn (the threshold is a per-document aggregate); at per-file scope it is error. | warn |
MISSING_DIAGRAM | Companion docs/_shared/<file>/diagrams/section-<chunk_id>.mmd is missing. | warn |
COPYBOOK_INCLUSION_SIDECAR_DRIFT | Copybook chunk: inline ```mermaid block under ## Inclusion sites → Inclusion graph is not byte-identical to bundle.output_paths.diagram. Sidecar must equal the inline source per copybook.template.md "DIAGRAM OWNERSHIP". | error |
COPYBOOK_INCLUSION_SIDECAR_UNEXPECTED | Copybook chunk: sidecar .mmd exists but the inclusion-graph gate is not met (< 2 includers in incoming_xref[via=copy] AND no REPLACING divergence). Sidecar must be pruned. | warn |
COPYBOOK_STRUCTURE_IN_SIDECAR | Copybook chunk: sidecar .mmd begins with classDiagram, erDiagram, or otherwise matches the structure-diagram contract instead of flowchart LR. The structure diagram is NEVER written to the sidecar (owned by diagram-renderer's er-fragment.mmd). | error |
All checks are pure regex / counting — no LLM call. The LLM qa-reviewer agent reads this JSON and produces additional severity entries with codes prefixed SEMANTIC_… (e.g. SEMANTIC_PURPOSE_OFF_TOPIC).
Procedure
Review exactly the requested bundle.languages at bundle.output_paths, passing
the selected language explicitly to the checker. Do not require an English
document when English was not requested. Cross-language structural parity applies
only when multiple languages are requested; shared neutral diagrams remain shared.
- Resolve inputs. Either pass the markdown path directly, or pass
--bundle <path>(preferred) and read the markdown path frombundle.output_paths. Two sibling artifacts:docs/_shared/<basename-with-ext>/_bundles/<chunk_id_sanitized>.bundle.json— authoritative; consumed by the mechanical checkercheck-section-doc.py(needs structured access tochunk_text_sha,facts_slice,output_paths,profile).docs/_shared/<basename-with-ext>/_bundles/<chunk_id_sanitized>.bundle.md— LLM input; deterministic Markdown projection used by the LLM semantic review pass for intent alignment. Never pass this to--bundle.
- Stale-input check. Compare
bundle.chunk_text_shawith a re-hash of the on-disk chunk text in the bundle. Mismatch → exit5, re-run the bundle prep step. - Run the checker. Invoke scripts/check-section-doc.py:
Flags (preferred → legacy):python .github/skills/qa-reviewer/scripts/check-section-doc.py ` --bundle docs/_shared/DEMO100.CBL/_bundles/DEMO100__procedure-division__B200-CONTROLLO-PARAMETRO__B200-EX.bundle.json ` --language en--bundle <path>— preferred. Readsource_path,chunk,chunk_text,chunk_text_sha,profile, andoutput_paths.<language>from the bundle. Markdown path =bundle.output_paths.<language>.--language <en|it|…>— language key intobundle.output_paths; defaulten.--section-doc <path>— override the bundle's resolved markdown path (rarely needed; skips the staleness check).--source <path> --chunk-id <id> --language <en|it|…>— legacy derivation path retained for ad-hoc runs without a bundle; does NOT perform thechunk_text_shastaleness check.
--profile <path>— override profile.yaml (defaultconfig/templates/section-doc.profile.yaml).--json-out <path>— override review sidecar path (defaultdocs/_reviews/section-doc/<file>__<chunk_id>__<lang>.json).--no-write— print review JSON to stdout, do not persist the sidecar.
- Report. Print one summary line:
OK …orFAIL: N error(s), M warn(s) → <sidecar path>. On failure, print one line pererrorfinding ([CODE] line=… <message>).
Determinism Rules
- Mechanical checks are 100% deterministic: byte-identical inputs → byte-identical sidecar.
- The sidecar JSON sorts
findings[]by(severity desc, line asc, code asc)and uses stable key order so re-runs are diff-clean. - The semantic LLM review run is separate from this skill and produces a sibling sidecar (or appends to the same JSON under a distinct
semantic_findings[]key). Do not mix the two in the same write — the LLM half is non-deterministic and should be re-runnable independently. - Profile thresholds are read once at the top of the run and copied verbatim into
thresholdsso the sidecar is self-describing without re-loading the profile.
Whole-source section sweep
After section authoring, run scripts/review-section-set.py to validate the complete indexed section set:
python .github/skills/qa-reviewer/scripts/review-section-set.py `
--source repos/sample/COBOL/ATCCDLGH.CBL- Languages come from
output.languagesin pipeline.yaml;--languages it,enoverrides them. English is the fallback only when the YAML language setting is omitted. Codes are normalized and deduplicated using the public runners' resolver; empty or malformed lists are rejected. - Before invoking any checker, the sweep requires a current source manifest, a
non-empty, unambiguous chunks index, every indexed markdown for every requested
language, and its authoritative
_bundles/<chunk-filename-stem>.bundle.json. Missing directories/documents/bundles and orphan markdown files block the sweep. Unrequested language trees are not checked or removed. - Bundle source, chunk ID, and language output paths must match the indexed
documents, and
facts_slicemust be present. Each checker receives--bundle, so incoming-reference rules and embedded-text hash checks are not bypassed. Re-stage bundles when adding languages or when inputs change. --source-rootand--profileremain available. Current child stdout/stderr and failure exit codes are printed. Existing sidecars are never read to infer the current run's findings; normal completed checks still write their sidecars.- Exit codes:
0complete success (warnings allowed);1content/checker failure;2invalid CLI/configuration;4missing or invalid prerequisites, incomplete sections, or orphan documents;5stale source/bundle. Mixed child failures prioritize4, then5, then1. Preflight failures stop before any reviews.
Resources
- scripts/check-section-doc.py — deterministic mechanical checker (entry point).
- scripts/review-section-set.py — complete indexed section-set gate.
Pipeline Wiring
config/pipeline-dag.yaml phase C lists qa-reviewer immediately after every per-section section-doc-writer step, with the per-chunk bundle JSON as the primary input alongside the section markdown and .mmd. This skill formalizes the deterministic mechanical-checker half of that step; the LLM semantic-review half is owned by .github/agents/section-reviewer.agent.md and is invoked by .github/agents/section-agent.agent.md after the mechanical checker passes.
Common Pitfalls
- Running QA before authoring. The checker fails with
4if the markdown does not exist — that is asection-doc-writerordering bug, not a reviewer bug. - Stale profile. Editing
profile.yamlchanges thresholds for every subsequent review. The sidecar copies the thresholds inline so old reviews keep their original verdict context, but new reviews will diverge — re-run the full phase-C QA after a profile bump. identifier_lock_regexfalse positives. The regex matches any[A-Z][A-Z0-9-]*token, including common English acronyms (SQL,CICS,DB2). The checker useswarnseverity (noterror) forIDENTIFIER_NOT_BACKTICKEDfor exactly this reason — escalate toerroronly at the per-file finalizer.- Missing diagram. A missing per-section
.mmdiswarnhere because the diagram renderer phase may not have run yet. The per-fileqa-reviewerfinalizer escalates it toerror.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

