Auditing AI-built code

Core principle

A finding is not a finding until you tried to disprove it.

Fan-out audits are cheap to run and produce confident, well-formatted, file:line-cited claims. A meaningful share of them are wrong. At the scale where you need an audit, you cannot personally re-derive every claim - which is exactly when a plausible-but-false finding gets acted on. The value of this skill is not the hunting. It is the disproving.

When to use

• A repo grew faster than anyone's mental model and you need a map plus a cleanup plan
• Pre-launch sweep of a codebase largely written with AI assistance
• Onboarding to an unfamiliar large codebase
• You are about to dispatch parallel finder agents at a repo
• Someone handed you a findings list and the next step is acting on it

Not this skill:

• cil:untangle - structural over-engineering (parallel state, callback chains, god files). Complementary; run it for the "too much code" question. This skill asks "what is wrong or fake, and is the finding real?"
• cil:audit - the full sweep that runs this lens plus untangle and product-coherence in one pass. Reach for it when you want all three at once, not just code.
• /code-review, /simplify - line-by-line correctness on a diff you already have

Step 1: Orientation before hunting

Never start by reading files. Establish scale, churn, and the spine first - it tells you which findings matter. Run these and keep the output:

# Scale per area
for d in <dirs>; do echo "$d: $(find $d -name '*.ts' -o -name '*.tsx' | wc -l) files"; done

# Churn: where the pain actually is (most-changed files)
git log --since="6 months ago" --name-only --pretty=format: -- '<globs>' \
  | grep -v '^$' | sort | uniq -c | sort -rn | head -30

# The spine: most-imported modules. This is the reading list for a newcomer.
grep -rhoE "from ['\"](@alias)/[^'\"]+" <dirs> | sed "s/from ['\"]//" | sort | uniq -c | sort -rn | head -40

# Growth rate - calibrates how much drift to expect
git log --since="12 months ago" --pretty=format:'%ad' --date=format:'%Y-%m' | sort | uniq -c

# Biggest files, dead code, type baseline
find <dirs> -name '*.ts*' -exec wc -l {} + | sort -rn | head -30
<dead-code-scanner>   # knip / ts-prune / etc
<typecheck>           # record the CURRENT error count as a baseline, per config

Record the typecheck baseline before changing anything. Without it you cannot tell new breakage from pre-existing.

Also check what is not in the code: stale worktrees, branch divergence from the deploy branch, and unmerged branches. Work stranded outside the main line is invisible to every file-based survey. A tested fix sitting in a forgotten worktree is a real finding.

Step 2: Fan out finders

Dispatch parallel read-only agents on distinct angles (see failure-modes.md for the catalogue). Give each one: the repo conventions file, one angle, and the finding contract below. Tell them explicitly to skip essential complexity and say why they skipped it.

Step 3: The finding contract

This is the load-bearing part of the skill. Every finding - yours or a subagent's - is reported in this shape. The evidence line is required; a finding without one is not reportable.

FINDING: <one sentence, what is wrong>
LOCATION: path/to/file:line  (+ every other copy, if duplication)
CLAIM TYPE: bug | dead | duplicate | perf | drift | gap
DISPROOF ATTEMPTED: <the specific check you ran to try to kill this finding, and its result>
CONFIDENCE: verified | inferred
FIX: <the concrete simpler/correct form>

CONFIDENCE: verified means you ran the disproof and it failed to kill the finding. inferred means you read the code and reasoned. Both are allowed to be reported. Silently mixing them is not. The reader must be able to tell them apart at a glance.

What counts as a disproof attempt

Matched to claim type. These are derived from real false findings:

Verify the high-severity claims yourself

Do not batch-trust. Before reporting to a human, personally re-derive every finding marked HIGH or "bug". That is a small number of checks and it is where the credibility of the whole audit lives.

Step 4: Triage, then tier

Sort every surviving finding:

• Accidental - duplicates something that exists, supports a non-load-bearing requirement, or is plumbing composition would remove. Flag it.
• Essential - real concurrency/locking, non-obvious SQL, list batching, intentional duplication with a documented sync marker, long fixtures. Skip it and say why, so the next audit does not re-flag it.

Then tier by risk, not by line count:

1. Tier 0 - deletions and dead weight. Zero risk, immediate relief.
2. Tier 1 - correctness. Things that would embarrass you in front of a user.
3. Tier 2 - dedup. One commit per duplication group.
4. Tier 3 - structural splits. One at a time, each with its own verification run.
5. Deferred - anything that changes a contract a shipped client depends on. That is an API migration with a deprecation window, not cleanup. Never bundle it with the safe tiers.

Keep a "disproven - do not re-file" section in the output. Findings that died during verification will be rediscovered by the next audit unless you write down why they are wrong.

Calibration

A disciplined codebase yields few real findings. Most suspicious-looking code turns out essential. If every file looks like a problem, you are pattern-matching on surface shape rather than tracing behavior - recalibrate before reporting.

Expect the audit to disconfirm the "it's all AI slop" hypothesis in the areas that are expensive to fix later (types, migrations, deprecation discipline, test placement) and confirm it in the one that is cheap but spreads: duplication. Report both halves honestly.

Common mistakes