Check Documentation Quality

Audit documentation freshness, completeness, and quality against the current codebase state. Read-only: never edit or write to any doc file.

Anti-hallucination detection

This skill exists to catch docs that lie. For every claim in a doc, verify it against the actual codebase rather than trusting the doc text:

1. Cross-reference claims: component/service names, described relationships
2. Verify counts: if a doc says "5 services", count the actual services
3. Check file references: confirm every referenced path exists
4. Validate diagrams: every Mermaid entity must exist in real code

Workflow

Phase 1: Scan docs/

1. Glob all documentation files (docs/, docs/adr/, docs/rfc/, top-level README.md, CONTRIBUTING.md).
2. Infer focus categories from the filenames actually present: don't assume a fixed set. A repo with docs/data-model.md gets a "data" category; one with docs/deployment.md and docs/docker-compose.yml docs gets "infrastructure"; group whatever's there under a name that matches its content. If the user names a focus that doesn't match anything found, say so and list the categories that do exist instead of silently no-op'ing.
3. Detect tech stack, database presence, deployment configs, and project type from the codebase (package files, Dockerfiles, etc.) to know what documentation should exist.

Phase 2: Parse arguments

Extract an optional focus keyword from the invocation and match it against the categories found in Phase 1. No argument means check everything found.

Phase 3: Verify claims against the codebase

For a small doc set (a handful of files, or a one-shot check like "does this file exist"), verify directly inline with Read/Grep/Glob/git: spawning a subagent for a single lookup adds latency for no benefit.

For a large multi-doc audit (a full docs/ tree, many ADRs, cross-referencing several files against the codebase), spawn one Explore subagent per document or logical section so each verifies its claims independently. Where no Task tool is available, fall back to processing each document sequentially inline instead: same verification steps, one document at a time, no parallelism. See references/verification-patterns.md for section-level verification patterns and bash commands.

Verification categories: component/service names, numeric counts, diagram entities, file/path references, technology claims (against package files), relationship claims.

Phase 4: Validation checks

Relevance: which docs are relevant given the detected stack; what's missing for detected technologies.

Freshness: compare each doc's last-modified date (git log) against related code changes. A doc untouched since before a significant change to the code it describes is stale.

Completeness: required sections present, no unreplaced {{PLACEHOLDER}} values, diagrams present where expected.

Quality: valid Mermaid syntax, no broken internal links, no empty sections.

Check Mermaid by rendering it. Needs mmdc: install with npm i -g @mermaid-js/mermaid-cli, or run it ad hoc by defining mmdc() { npx -y -p @mermaid-js/mermaid-cli mmdc "$@"; } first. If Chromium refuses to launch in a sandbox, add -p with a puppeteer config of {"args":["--no-sandbox"]}. Run this once over every doc found in Phase 1 (nothing it writes lands in the repo):

d=$(mktemp -d)
for md in $(find docs -name '*.md'); do
  awk -v out="$d/${md//\//__}" '/^```mermaid/{f=out "-" ++n ".mmd"; next} /^```/{f=""} f{print > f}' "$md"
done
for f in "$d"/*.mmd; do mmdc -q -i "$f" -o "${f%.mmd}.svg" || echo "FAIL $f"; done

Each FAIL <name>-N.mmd is an invalid diagram: report the doc and the Nth mermaid block as a finding with the parser error, and never edit the doc. Run it once, with no fix loop. If mmdc can't be installed, say so in the report and review the syntax by eye, marking those results unverified.

See references/verification-patterns.md for the exact commands.

Phase 5: Rate each document

Skip numeric scoring: a 0-100 breakdown per doc implies precision this check doesn't have. Give each document one coarse rating instead:

• Good: current, complete, no broken links or invalid diagrams
• Stale: accurate but outdated (freshness or completeness gaps, no false claims)
• Broken: contains hallucinations, broken links, or invalid Mermaid syntax
• Missing: expected given the detected stack but doesn't exist

See references/scoring-criteria.md for the full rubric per rating.

Phase 6: Generate report

• Status summary with counts per rating
• Documents grouped by rating (Good / Stale / Broken / Missing)
• Hallucination Report: claims that don't match reality, with evidence
• Quality issues with specific file:line locations
• Actionable recommendations naming a specific follow-up command (docs-update, docs-diagram) and why

Output Format

Status Summary

Documentation Health Report

Good (3 docs):
  docs/architecture.md
  docs/onboarding.md
  docs/adr/ (5 records)

Stale (2 docs):
  docs/data-model.md — last updated 2025-03-01, schema changed 2025-06-15
  docs/deployment.md — references removed staging environment

Broken (1 doc):
  docs/architecture.md — Hallucination: claims "6 microservices", 3 found

Missing (1 doc, expected for detected stack):
  docs/security.md — auth code present, no security doc

Hallucination Report

Hallucinations Detected:

docs/architecture.md:
  - Line 45: Claims "6 microservices" but only 3 found
    Verified with: find . -name "*service*" -type d | wc -l -> 3
  - Line 78: References "AuthService" which doesn't exist
    Verified with: grep -r "class AuthService" . -> no results

Recommendations Format

Priority Recommendations:

1. HIGH: docs/architecture.md claims 6 microservices, only 3 exist
   Command: docs-update architecture
   Reason: hallucinated count, misleads new engineers

2. MEDIUM: docs/data-model.md has 2 unreplaced {{PLACEHOLDER}} values
   Command: docs-update data-model
   Reason: incomplete since scaffolding

3. LOW: docs/security.md missing, auth module exists
   Command: docs-diagram security or docs-init security
   Reason: no security documentation for an app with auth

Usage Examples

docs-check                      # check everything found in docs/
docs-check core                 # focus on whatever category maps to "core" in this repo
docs-check focus on database documentation

Important Notes

• Non-destructive: only reads, never modifies documentation.
• Every number reported (counts, dates, "N days stale") must come from a command actually run in this session: never estimate or infer a count without running find/grep/git for it.
• Recommends docs-update or docs-diagram as the fix: this skill only reports.

When to Run

• Before onboarding new team members
• During documentation reviews
• After major refactoring
• As part of a pre-release checklist
• When documentation feels stale

Additional Resources

• references/verification-patterns.md: load when running Phase 3/4 verification, for exact bash/grep patterns and the subagent prompt templates
• references/scoring-criteria.md: load when assigning Phase 5 ratings, for the full Good/Stale/Broken/Missing rubric