agentleFS
Sign inStart free
2389-research2389-research/documentation-audit/skills/SKILL.md

documentation-audit

Skill1 starsChanged 5 months agoMITView on GitHub

This skill should be used when verifying documentation claims against codebase reality. Triggers on "audit docs", "verify documentation", "check docs", "docs accurate", "documentation drift", "before release", "after refactor", "docs don't match". Uses two-pass extraction with pattern expansion for comprehensive detection.

  • Reads credentials

We split this skill into the rules it gives an agent

3 rules. 0 are common, 3 are distinctive, and 0 are very specific to this project.

  • 0

    Common

    Most of the 4 edited versions we read keep these.

  • 3

    Distinctive

    This author's own choices. Worth a look.

  • 0

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Run Pass 1 extraction using parallel agents (one per doc)

    line 28
  2. Distinctive:

    Run Pass 2 expansion searches

    line 30
  3. Distinctive:

    After Pass 1, analyze false claims and search for similar patterns:

    line 50

“Common” is measured: kept by at least 6 in 10 of the 4 edited versions of this file we could read. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md
<!-- ABOUTME: Documentation audit skill for verifying claims against codebase --><!-- ABOUTME: Uses two-pass extraction with pattern expansion for comprehensive detection -->

Documentation Audit

Systematically verify claims in documentation against the actual codebase using a two-pass approach.

Overview

Core principle: Low recall is worse than false positives—missed claims stay invisible.

Two-pass process:

  1. Pass 1: Extract and verify claims directly from docs
  2. Pass 2A: Expand patterns from false claims to find similar issues
  3. Pass 2B: Compare codebase inventory vs documented items (gap detection)

Quick Start

  1. Identify target docs (user-facing only, skip plans/, audits/)
  2. Note current git commit for report header
  3. Run Pass 1 extraction using parallel agents (one per doc)
  4. Analyze false claims for patterns
  5. Run Pass 2 expansion searches
  6. Generate docs/audits/AUDIT_REPORT_YYYY-MM-DD.md

Claim Types

TypeExampleVerification
file_refscripts/foo.pyFile exists?
config_default"defaults to 'AI Radio'"Check schema/code
env_varSTATION_NAMEIn .env.example + code?
cli_command--normalize flagScript supports it?
behavior"runs every 2 minutes"Check timers/code

Verification confidence:

  • Tier 1 (auto): file_ref, config_default, env_var, cli_command
  • Tier 2 (semi-auto): symbol_ref, version_req
  • Tier 3 (human review): behavior, constraint

Pass 2 Pattern Expansion

After Pass 1, analyze false claims and search for similar patterns:

Dead script found: diagnose_track_selection.py
  → Search: all script references → Found 8 more dead scripts

Wrong interval: "every 10 seconds"
  → Search: "every \d+ (seconds?|minutes?)" → Found 3 more

Wrong service name: ai-radio-break-gen.service
  → Search: service/timer names → Found naming inconsistencies

Common patterns to always check:

  • Dead scripts: scripts/*.py references
  • Timer intervals: every \d+ (seconds?|minutes?)
  • Service names: ai-radio-*.service, *.timer
  • Config vars: RADIO_* environment variables
  • CLI flags: --flag patterns in bash blocks

Output Format

Generate docs/audits/AUDIT_REPORT_YYYY-MM-DD.md:

# Documentation Audit Report
Generated: YYYY-MM-DD | Commit: abc123

## Executive Summary
| Metric | Count |
|--------|-------|
| Documents scanned | 12 |
| Claims verified | ~180 |
| Verified TRUE | ~145 (81%) |
| **Verified FALSE** | **31 (17%)** |

## False Claims Requiring Fixes
### CONFIGURATION.md
| Line | Claim | Reality | Fix |
|------|-------|---------|-----|
| 135 | `claude-sonnet-4-5` | Actual: `claude-3-5-sonnet-latest` | Update |

## Pattern Summary
| Pattern | Count | Root Cause |
|---------|-------|------------|
| Dead scripts | 9 | Scripts deleted, docs not updated |

## Human Review Queue
- [ ] Line 436: behavior claim needs verification

Detailed References

For execution checklist and anti-patterns: checklist.md For claim extraction patterns: extraction-patterns.md

Download now (ZIP)

  • Other versions4 projects use a version of this.

    Word for word · 0

    None found.

    Edited · 1

    1. michellepellon/superpowers★ 146% same

    Wrote their own version · 3

    1. int2t05/engineering-skills★ 3same name
    2. kontourai/station★ 3same name
    3. DojoGenesis/openclaw-plugin★ 0same name

    How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.

  • In this repositoryOne other file it gives its agents.

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

Reviews can't be read right now.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

Check your setup, free