skill-deep-audit — Generic Skill Auditor

A read-only, multi-dimensional quality auditor for agent skills. Runs static analysis + optional dryRun reachability checks and produces a scorecard.

• Version: 1.1.0
• License: MIT
• Author: Evan Song · github.com/Songhonglei
• Repository: https://github.com/Songhonglei/build-better-skills
• Part of: build-better-skills suite — see Stages for the lifecycle map.

Design principles

Can it run?            → D3 Portability + D4 Usability conventions
Does it run correctly? → D1 Process closure + D6 Code & doc quality
Is it safe to run?     → D5 Security & op risk
Is it well-conformed?  → D2 Tool & command conventions
Is the whole healthy?  → D7 Dependency & footprint

NEVER DO

• Do not execute any write operation of the audited skill (read-only, reachability checks, and static analysis only).
• Do not modify the audited skill's files (read-only and report-only; --fix is the one exception and requires explicit user authorization).
• Do not fabricate check results — when undecidable, mark "cannot confirm, manual verification needed".

MUST DO

• At the start of every audit, ask the user to choose the check depth (L1 / L2 dryRun) and wait for the choice.
• Any ERR → final result is FAIL, regardless of total score.
• After the audit, write the scorecard MD file.
• If an L2 dryRun check fails because an external dependency is unavailable, downgrade to WARN with the reason — do not abort.

Dependencies of this skill (all soft — degrade gracefully if missing)

The core of this skill is pure static / read-only analysis. There are no hard external dependencies — L1 static audit works even if no tooling is installed.

Bundled checker: scripts/check_path_boundary.py (the D3-E5 (1) judge) needs python3 ≥ 3.8 and uses the standard library only (re / json / sys / pathlib) — no pip install, runs offline, strictly read-only. If python3 is unavailable, fall back to the greps in D3-E5 and verify endpoints by hand; the rest of the audit is unaffected.

Step 0: Ask for check depth

At the start of the audit, present the following options and wait for the user to choose explicitly:

Please choose check depth:

L1 Static analysis (~2 min)
   File read, structural check, keyword scan, syntax check.
   Max 117 (skips items that need to touch external systems). Pass line ≥ 90.
   Good for: quick first-draft check.

L2 dryRun (~5 min, recommended) ⭐
   L1 + Hub existence check + dependency existence check + branch reachability
   simulation (file existence / env config / read-only verification of
   unhit branches).
   Max 120. Pass line ≥ 90.
   Good for: pre-release / pre-ship full acceptance.

Default recommendation: L2 dryRun. Reply 1 / L1 for static, 2 / L2 for dryRun
(empty Enter = L2).

⚠️ Note: L2 dryRun does ONLY read-only queries and reachability checks. It
   performs no writes / updates, and it does NOT actually run the audited
   skill's business workflow.

Step 1: Locate the skill directory

The user may provide:

• A skill name (e.g. my-skill) → look for a same-named folder under <skills-dir>/. <skills-dir> is the agent's skills directory and may be e.g. ~/.claude/skills/, ~/.openclaw/workspace/skills/, or any path the user specifies. Some agents use a different layout — adjust to what's actually on disk.
• A relative or absolute path (e.g. skills/my-skill/) → use directly.

ls {skill-path}/
cat {skill-path}/SKILL.md | head -20

If the directory cannot be found → tell the user and stop. Do not guess.

Step 2: Static analysis (runs at every depth)

Execute each check defined in references/check-rules.md in order.

⚖️ Determinism guarantee: each rule's hit/miss decision uses the grep pattern, keyword list, and numeric thresholds defined for it in check-rules.md — not the agent's subjective judgement. Edge cases not explicitly covered by a rule are handled by the "False-Positive General Rules" section (marked "manual verification needed", not hard-judged). This guarantees stable, repeatable results across different agents / re-runs.

2.1 Collect file list

# Script extensions must cover mixed-language skills — .js/.cjs/.mjs/.ts cannot be missed
find {skill-path} -type f \( \
  -name "*.py" -o -name "*.sh" -o -name "*.md" -o -name "*.json" -o -name "*.yaml" \
  -o -name "*.js" -o -name "*.cjs" -o -name "*.mjs" -o -name "*.ts" \) \
  | grep -v __pycache__ | grep -v node_modules | grep -v .git

⚠️ Extension coverage blind-spot: find -name "*.js" does not match .cjs / .mjs / .ts. Python scripts often subprocess-call a sibling xxx.cjs — if the file list misses .cjs, the auditor will wrongly report "called script does not exist" (false positive on D6-E4 / D6-E6). All later extension-scoped scans must include the full set.

2.2 Per-dimension static scan

Execute in order: D1 → D2 → D3 → D4 → D5 → D6.

ℹ️ D7 is not in this step: D7 (dependencies & footprint) needs the code stats (see 2.4) plus Hub / existence checks, so it is consolidated into Step 4. Step 2 only scans D1–D6.

Execution-level convention:

• Rule title contains L1 → runs at all depths.
• Rule title contains L2 dryRun → runs only at L2 dryRun; for L1, mark as ➖ skipped (L2 dryRun item).
• D4-E5 scan must exclude {skill-path}/AUDIT-*.md — audit reports are produced by this tool itself and are not part of the audited skill's package.

For each rule:

1. Check whether its execution level matches the current depth; if not, record ➖ skipped.
2. Run the corresponding scan command (grep / regex / file parse / agent-read judgement).
3. Record: pass ✅ / fail ❌ / skipped ➖.
4. Accumulate deductions.

The four 0-point ERRs (D3-E4 / D3-E5 / D3-E6 / D4-E6) deduct nothing, so a hit is invisible in the score and must be surfaced in the Step 7 summary. D3-E5 runs via the bundled checker:

python3 <this-skill>/scripts/check_path_boundary.py {skill-path}   # exit 1 = ERR

Record each as pass ✅ / ERR ❌ (0 pts). Full run instructions and the per-function / per-Step-block judgement caveats are in check-rules.md.

2.3 D6-E1 script syntax check

# Python
for f in $(find {skill-path}/scripts -name "*.py" 2>/dev/null); do
  python3 -m py_compile "$f" 2>&1 && echo "OK: $f" || echo "SYNTAX ERR: $f"
done

# Shell
for f in $(find {skill-path}/scripts -name "*.sh" 2>/dev/null); do
  bash -n "$f" 2>&1 && echo "OK: $f" || echo "SYNTAX ERR: $f"
done

2.4 Code-size stats (prerequisite for D7)

Script-count, total-line, and skill-on-skill dependency extraction commands → read references/scan-commands.md. Feed the results into Step 4 (D7) and report section "VI. Skill Dependencies".

Step 3: Hub existence check (runs at L2 dryRun)

Pre-check: this step requires a skill-hub query tool (e.g. clawhub). If unavailable → skip Hub checks; mark D7-W1 as "cannot verify (no hub tooling)", downgrade to WARN, do not abort.

1. Extract the name field from frontmatter.
2. Use the available skill-hub query tool to check whether the skill is already published.
3. Record the result against D7-W1 (not published → WARN, not ERR).

Step 4: Dependency & footprint analysis (D7)

• D7-W1 Hub publish status (consolidated from Step 3).
• D7-W2 Precise dependency-skill list + purpose annotation + existence check (local ✅ / hub-has-not-installed ⚠️ / not-found ❌). Output a full list in report section "VI. Skill Dependencies" regardless of count; ≥ 5 deps → WARN; depending on a "not found ❌" skill → ERR; depending on a "hub-has-not-installed ⚠️" skill → WARN.
• D7-W3 Code ≥ 5000 lines or scripts ≥ 10 → identify high-cohesion modules and suggest a split direction.

Step 5: Aggregate scoring

Total 120 points

📊 Scoring convention: ERR is uniformly 3 points (a hit means FAIL; the point value carries no real meaning). WARN uses three priority tiers (high 3 / mid 2 / low 1) — the difference is meant to guide fix order.

Exception — four 0-point ERRs: D3-E4 (cross-skill reference base), D3-E5 (relocation-safe path derivation), D3-E6 (file I/O failure handling), D4-E6 (SKILL.md length control) consume no budget and deduct nothing — they rely only on the "zero ERR" half of the test below. All four detect silent failure modes, where blocking is the right instrument and a numeric penalty is not.

Dual-judgement (both conditions must hold for PASS):

Pass line is uniformly 90 at both depths (skipped items don't count toward the actual max but don't change the pass line):

Step 6: Produce the scorecard MD file

Generate the full report using references/output-template.md.

Write path: {skill-path}/AUDIT-{YYYY-MM-DD}.md

AUDIT-*.md should not be packaged with the skill (D4-E5 will detect this).

Step 7: Output summary

📋 Audit complete: {skill-name}
─────────────────────────────────────
Total score: {score}/{max}   {PASS ✅ / FAIL ❌}  (L1 max 117 / L2 dryRun max 120)
Pass line:   ≥ 90 (uniform across L1 / L2 dryRun)  AND  zero ERR (dual-judgement)
Depth:       {L1 static / L2 dryRun}

🔴 ERR: {n} (incl. {z} zero-point)   |   🟡 WARN: {n}
Path & failure handling: D3-E4 {✅/❌} reference base | D3-E5 {✅/❌} path derivation
                         D3-E6 {✅/❌} I/O failure   | D4-E6 {✅/❌} SKILL.md length
                         (all four are 0-point ERRs — a hit forces FAIL)
Highest-priority fix: {ID and name of the highest-deduction ERR}

Estimated score after fixing all ERR: {estimated}/{max}

📁 Scorecard: {skill-path}/AUDIT-{date}.md

🔧 Fix: {N} items auto-fixable / {M} items need human confirmation
   Reply "fix" to start auto-fix (the skill folder is backed up first).

Step 8: Auto-fix (--fix) behavior spec

⚠️ This is the only step in skill-deep-audit that is allowed to modify the audited skill's files, and only after explicit user authorization. The day-to-day audit (Step 0–7) strictly observes the "audit-only, never fix" red line.

Trigger conditions

• The user explicitly replies "fix", "apply fix", "--fix", "fix 5.1", etc. after the report is delivered.
• Without explicit user authorization, never auto-fix. The report only recommends; it does not execute.

Fix scope tiers (corresponds to report section "V. Fix Recommendations")

Execution flow (strict order)

1. Mandatory pre-fix backup:
   • Copy the entire audited skill directory to a backup path: {skill-path}.bak-{YYYYMMDD-HHMMSS}
   • Immediately tell the user the full backup path.
   • If backup fails → abort the fix, don't touch anything.

   BACKUP="{skill-path}.bak-$(date +%Y%m%d-%H%M%S)"
   cp -r "{skill-path}" "$BACKUP" && echo "✅ Backed up to $BACKUP"

2. Apply fixes item by item:
   • 5.1 items: edit directly per the report's "③ Fix" section. After each change, briefly report ✅ Fixed [ID].
   • 5.2 items: only change after the user explicitly confirms that item; items the user hasn't approved are not touched.
3. Do not auto-re-audit:
   • After fixes, prompt the user: 🔧 Fixed {n} items. Re-run the audit now to verify? (reply "re-audit" to start)
   • Wait for the user to confirm "re-audit" before re-running Steps 0–7.
4. Fix record: in the report or reply, list "which files / which items were changed + backup path" so the user can roll back.

Red lines (also apply during fix)

• Do not execute any write operation.
• Do not delete any file (even if it looks redundant); if deletion is needed ask the user separately.
• 5.2 business-logic items are never changed unilaterally, even if they "look safe".
• If the user wants to roll back after fix: instruct the user to restore by copying the backup directory over.

Part of build-better-skills

This skill belongs to the build-better-skills suite. For the full lifecycle map (Install → Audit → Release → Testing → Sediment), all sibling skills, and their current status, see the Stages table on the suite repo home — kept as the single source of truth (this file does not duplicate it).

Rule references

• Full rule decision logic → references/check-rules.md
• False-positive / boundary general rules → references/check-rules.md "False-Positive General Rules" section
• Controlled-domain config (D2-E1, default empty) → references/controlled-domains.md
• Report MD template → references/output-template.md