Repo Pincer

A system's real behavior is not what it claims about itself — it's the intersection of what it claims and what it does. Most code review reads one direction (docs, or code) and stops. This methodology reads both independently and closes them like a pincer: the discrepancies between the two passes are the actual finding.

The methodology is domain-general; the one mechanical accelerant is not — confirmed against a real, external target (a NixOS dotfiles flake repo, no relationship to this repo's own conventions). The two-pass discipline, the explicit mandate to hunt Silent findings as their own direction rather than only verifying existing claims, and the outcome-anchored severity rubric below all transferred cleanly and produced real, fixed findings an unstructured "read the docs and check what looks off" pass had already missed. check_structural_claims.py (Pass 2 step 0) did not transfer the same way: it's tuned to this repo's own test-count-claim tree-block convention, and reporting "0 claims found" on an unrelated target is a real, expected result — not a failed check, and not evidence the target's docs are clean (see that step's own trust-boundary note and SKILL.md's "What's NOT built here"). If you find yourself wanting this pattern — a narrow, cheap, mechanical check run before any conversational judgment — for a different domain (a different kind of drift entirely, not doc-vs-code), that instinct is usually right about the shape being worth reusing, but wrong to reach for this script: build a new, narrowly-scoped checker for that domain instead of trying to stretch this one, and weigh its ongoing maintenance cost against how often the problem it'd catch actually recurs — a same-day one-line fix recurring rarely may not justify new permanent tooling with its own blind spots and staleness risk, even when the "cheap mechanical check first" shape is exactly right.

How this works

Two independent passes that must reconcile. Do not let the second pass be informed by the first — bottom-up findings should stand on their own before being checked against the claims ledger.

Pass 1 — Top-down (claims)

0. If a findings ledger exists from a prior run against this target, load it first:

   python ${CLAUDE_PLUGIN_ROOT}/scripts/track_findings.py --ledger-file findings-ledger.json --repo-root . check

   Every unchanged record's conclusion still holds — its cited source hasn't moved since it was last verified, so don't re-derive it from scratch; carry it straight into this run's report. Only changed and missing_source records need fresh Pass 2/3 judgment, same as a brand-new claim. This is the expensive-judgment analog of Pass 2 step 0's mechanical check: that script skips re-deriving a cheap fact when nothing's changed, this one skips re-deriving an expensive one. No prior ledger is the common case for a first run — say so and proceed normally.
1. Read outward-facing claims in order of authority: root README → architecture/design docs → package manifests (package.json, pyproject.toml, Cargo.toml, etc.) → per-module docs → public API/CLI surface (exported symbols, defined commands).
2. Build a claims ledger: for each claim, record what it asserts, exactly where (file + line or section), and how falsifiable it is. A specific claim ("retries 3× with exponential backoff") is worth more than a vague one ("handles errors gracefully") — note the difference, don't just list claims flatly.
3. Do not verify anything yet. This pass only records what the system says about itself.

Pass 2 — Bottom-up (reality)

⚠ Trust boundary — read before running the mechanical check on an unvetted target. check_structural_claims.py executes every test_*.py file it finds under the target's skill directories via subprocess.run(), with no sandboxing — no network isolation, no resource limits, no restricted filesystem access. That's safe against a target you already trust (this repo's own skills, a codebase you maintain), because you're just running code you already trust in a different way. It is not safe against "evaluating a dependency or vendor repo before committing to it" — one of this skill's own documented use cases — because that target's test files are exactly the untrusted code the audit exists to evaluate, and running them unsandboxed hands them this session's full privileges. Only run the mechanical check against a target you'd already be willing to run arbitrary code from; for a genuinely unvetted target, read Pass 2's entry points and tests statically instead of executing them, at least until they've been read.

0. Run the mechanical check first, before any reading. scripts/check_structural_claims.py verifies one narrow, high-value claim type — "N-test suite" / "N-test regression suite" claims — by actually running the referenced tests and comparing counts, not estimating:

   python ${CLAUDE_PLUGIN_ROOT}/scripts/check_structural_claims.py --claims-file README.md --skills-dir skills

   This exists because exact-number claims are simultaneously the cheapest to verify (one command, no judgment) and the most likely to drift silently — nobody re-counts "26 tests" by hand every time a test gets added. Confirmed empirically: a repo-wide pass against this very repo found three real Drift findings this way in seconds, before any conversational tracing started. A fourth output bucket, errored, is not Drift — it means a test file crashed or timed out, so the actual count isn't trustworthy enough to call a numeric mismatch; that's a different, usually more urgent finding worth its own line in the report, not one folded silently into a Drift number. Narrow on purpose — see "What's NOT built here." Add --fail-on-drift to use this as a direct CI gate — it exits 1 if any claim is Drift or Errored, rather than only ever reporting and exiting 0. Missing-skill and no-tests-found findings don't gate: those more often mean the line's skill-directory association is ambiguous than that something regressed, so they stay report-only either way.
1. Identify entry points first: exported/public functions, CLI commands, API routes, error-handling paths. Start here, not with every private helper — this is where claims are made and where drift matters most.
2. For each entry point, read the actual implementation. Record real behavior — inputs, outputs, side effects, error handling — from the code itself, not from comments or docstrings (those are claims, and belong in Pass 1 if load-bearing).
3. Trace the call graph outward from each entry point only as far as needed to confirm or refute a specific claim — not exhaustively. Depth follows the claim being checked, not a fixed crawl.
4. Build the as-built model: what the system actually does and how the pieces actually connect, independent of what Pass 1 said.
5. Silent findings are the most expensive direction to check, and the first thing to get skipped under time pressure. Finding an undocumented capability means reading code looking for behavior nobody claimed exists — there's no claim to start from, unlike Confirmed/Drift/Aspirational, which all start from something already written down. At repo scale (many targets, limited time), this is exactly the direction that silently gets shortchanged first. If time pressure forces a choice, say so in the report's "Open questions" section rather than letting thin Silent-coverage look the same as thorough coverage.

Pass 3 — Reconciliation

1. Two directions, not one — a Silent finding has no claim to walk from, so it can't surface from the claims ledger alone:
   • Walk the claims ledger against the as-built model. Classify each claim as Confirmed (matches reality), Drift (was true once; reality has since moved — version skew, partial refactor), or Aspirational (describes something not yet implemented — a TODO in disguise).
   • Walk the as-built model for anything with no corresponding claim at all. That's Silent — reality does something the docs never mention.

2. Rank each Drift/Aspirational/Silent finding High/Medium/Low against concrete consequences, not a vague sense of how bad it feels — borrowed from cloudflare/security-audit-skill's severity-anchor pattern (outcome anchors, not adjectives) and re-grounded in what a documentation/reality mismatch actually costs:

   For Drift and Aspirational (there's a specific false claim — ask what a reader loses by trusting it):

   • High — trusting the claim leads to a wrong safety- or security-relevant assumption, or a wrong assumption about a write/destructive path: a doc says input is sanitized here and it isn't, says something is encrypted at rest and it isn't built, says a script only reads and it actually writes or deletes.
   • Medium — trusting the claim costs real time or produces a real but recoverable mistake, with nothing safety/security-relevant at stake: a documented capability doesn't exist (someone builds around a phantom feature, or files a duplicate issue for something already claimed-but-missing), a documented behavior differs enough to break an integration built against the doc.
   • Low — cosmetic: stale prose, a renamed variable, an outdated example whose surface syntax drifted but whose actual usage is still correct. No reader reaches a materially wrong outcome from trusting it.

   For Silent (there's no claim to trust — ask what not knowing costs instead):

   • High — the undocumented behavior sits on a write path or touches security/safety-relevant behavior: silently sends data externally, deletes or mutates something, bypasses a documented safeguard — and nobody would know without reading the code.
   • Medium — a real, useful capability that's genuinely undocumented but inert or non-destructive: an extra CLI flag nobody knows exists.
   • Low — a trivial internal implementation detail with no user-facing consequence either way, documented or not.

   The discriminator, same shape as Cloudflare's "if you can't state the concrete damage, the severity is lower than it feels": if you can't name what a reader would get wrong, lose, or break by trusting a Drift/Aspirational claim — or what goes unnoticed because a Silent behavior stayed undocumented — the severity is lower than it feels, not higher.

3. The ranked discrepancy list is the primary deliverable — report it before either summary. Confirmed claims aren't listed individually; note them only in aggregate ("14 of 18 claims confirmed").
4. Record each new or changed Drift/Aspirational/Silent/Confirmed finding in the findings ledger, so a later run against this same target doesn't re-derive it from scratch:

   python ${CLAUDE_PLUGIN_ROOT}/scripts/track_findings.py --ledger-file findings-ledger.json --repo-root . upsert \
     --fingerprint <skill-or-area>:<verdict>:<short-slug> --verdict drift \
     --title "<one-line summary>" --source-ref <file that this conclusion depends on> \
     [--source-ref <another file>] --notes "<why, briefly>"

   Pick the fingerprint once and keep it stable across runs — it's how the next run recognizes "this is the same finding," not a hash it computes for you. --source-ref is every file whose current content this specific conclusion depends on; if any of them changes before the next run, check (Pass 1 step 0) will flag it for revalidation instead of silently carrying forward a conclusion the changed source might have invalidated. A Confirmed claim worth recording is one specific enough that a later change to its source would be worth knowing about — most Confirmed claims aren't, and don't need an entry. changed/ missing_source records from Pass 1 step 0 that were revalidated this run get re-upserted here like any other finding, refreshing their evidence baseline.

/pincer [path or repo] [--depth quick|standard|thorough]

Default scope is a single skill/module/directory, not the whole repo — widen only if asked, since a full pass on a large repo in one shot is rarely what's useful. This default applies when a specific path/skill/module is named or otherwise obvious from context. A bare, unqualified target with no path named at all — "run repo-pincer on this repo," "audit this repo," "reverse engineer this repo" — has no single module to default to, and is a real, common way people ask for this without meaning to override anything. Treat that phrasing as an explicit repo-wide request (confirmed in practice: this is exactly how one real invocation was phrased, and it resolved to scanning every top-level doc, not one module), using the breadth strategy below — not as license to silently pick one module and scope down without saying so.

Depth controls how far Pass 2 traces within one target:

• quick — top-level README + entry points only, no call-graph tracing
• standard (default) — trace only as far as needed to confirm or refute each ledger claim
• thorough — trace the full call graph from every entry point

None of these address breadth — how to allocate effort across many targets when a repo-wide pass genuinely is what's asked for, despite the default guidance above. Depth is about one target; a multi-target pass needs a different strategy, found only by actually running this against a real large repo, not by reading the methodology: run the mechanical check (Pass 2 step 0) across every target first, then spend conversational tracing effort on the highest-authority sources (root README, top-level manifests) before per-module depth, rather than attempting uniform standard-depth tracing across everything at once — that doesn't scale and produces thin, uneven coverage without ever admitting it's thin.

1. Run Pass 1, then Pass 2, then Pass 3, in that order.
2. Compose the output (see schema below).
3. If an obsidian-vault MCP server is connected, offer to write the output to Sources/raw/<repo>-pincer-<date>.md — the existing raw-capture convention wiki-synthesizer/wiki-operator /source already compiles. This is optional; the skill runs standalone with no MCP dependency.

Output schema

If writing into a wiki vault, use this frontmatter — one new field (subtype) on the existing source note type, no other schema changes:

type: source
subtype: codebase
status: draft
confidence: medium   # findings are provisional until cross-checked with a maintainer or the test suite
updated: YYYY-MM-DD
repo: owner/name
commit: <sha analyzed>

Body sections:

• ## Top-down summary — the condensed claims ledger
• ## Bottom-up summary — the condensed as-built model: entry points, core abstractions, key call-graph findings
• ## Discrepancies — the ranked Drift/Aspirational/Silent list, each with a file:line pointer; Confirmed claims noted only in aggregate
• ## Open questions — anything genuinely ambiguous even after both passes

If no vault is connected, present the same structure directly in the conversation rather than writing a file.

What's NOT built here

• check_structural_claims.py only checks test-count claims. Not path-existence claims, not any other kind of specific number (tool counts, rule counts) — those were checked by hand in the pass that motivated this script and found no drift, but auto-extracting "things that look like a path" from arbitrary backtick spans in prose is meaningfully noisier (backticks cover code snippets and command names too, not just paths) and risks false positives if generalized carelessly. A real, scoped follow-up, not attempted here.
• Tuned to this repo's own documentation conventions, not portable to an arbitrary target repo's arbitrary claim phrasing. It assumes a claim and its skill-directory name share one line, tree-block style (├── name/ # ... N-test suite). An honest limit, not really a meaningful one in practice — a different repo phrases these claims differently regardless, so a checker built for one repo's convention was never going to be zero-effort to point at another one. On a target that doesn't use this convention at all, the script reports "0 test-count claims found" with an explicit note that this means the check doesn't apply here — not "0 checked, no discrepancies," which would read identically to a real clean pass. Confirmed as a real, live ambiguity on a first run against an external target (a NixOS dotfiles repo) before this distinction existed; both the text output and the JSON output's zero_claims_found/claims_checked fields make the two cases distinguishable now. --fail-on-drift does not gate on this case either way.
• No semantic or behavioral checking whatsoever. Never touches Silent findings or whether code still does what it claims to do — that stays Pass 2's conversational job, unchanged. This script only accelerates the narrow, exact-number slice of it.
• Can't verify a test-count claim about repo-pincer itself. Its own test file includes a live check that re-invokes this script against the real README — counting it via subprocess would recurse (verify the claim → run the test file → which runs this script again → which tries to verify the same claim → ...). Caught as an actual runaway subprocess tree while first verifying this script, not a hypothetical risk. count_actual_tests() excludes test_check_structural_claims.py by name unconditionally, so this fails safe (reports "no test files found") rather than recursing even if a numbered claim is ever written for repo-pincer again — which is exactly why the README describes this skill's own suite generically ("test suite"), not with a number this tool would just report as unverifiable.
• A per-test-file timeout (TEST_FILE_TIMEOUT_SECONDS, 30s) is the general backstop for the class of bug the recursion above turned out to be one instance of. The filename exclusion defuses that one specific known trigger; the timeout defends against any other future test file that blocks — a stray network call, a leftover input(), an unrelated infinite loop — turning a hang into a reported errored finding instead of a wedged process. Confirmed with a deliberately-hanging fixture test file in test_check_structural_claims.py.
• Only recognizes stdlib unittest's own summary line ("Ran N tests") as evidence a test file ran. Every skill in this repo uses unittest today, so this is untested against any other runner's output format (e.g. pytest's "N passed") — a test file using a different runner would currently show up as errored (no recognized "ran" line), not silently miscounted, but it's still a real scope boundary, not a hypothetical one, if a future skill's tests are ever written differently.
• No sandboxing around the code it executes. count_actual_tests()'s subprocess.run() has a timeout but no network isolation, no resource limits, and no filesystem restriction — see the trust-boundary warning at the top of Pass 2 for what this means in practice: safe against a target you already trust, not safe as a way to evaluate an unvetted third-party repo's code by running it.
• track_findings.py hashes whole files, not line ranges. An unrelated edit anywhere in a cited file reports changed even if the specific fact the finding depends on didn't move — a false positive, but the safe direction to be wrong in: it costs an unnecessary revalidation, never a silently stale carry-forward. Line-range hashing would be more precise and is a real, scoped follow-up, not attempted here — same relationship as check_structural_claims.py's own path-existence-claims gap.
• Nothing extracts findings into the ledger automatically. Recording a finding is an explicit upsert call Pass 3 makes for each one — there's no step that reads a written report and back-fills the ledger from it. Skipping the upsert step doesn't lose the finding from that run's report, only from the next run's cross-run memory of it.
• The ledger has no notion of "this finding was fixed." A changed or missing_source record just means its evidence moved, not which direction — Pass 3 still has to look and decide whether the finding still holds, was fixed, or needs a new fingerprint entirely (e.g. a Drift finding that got fixed and is now Confirmed should be re-upserted with the same fingerprint but a new verdict, not treated as resolved by omission).
• The ledger's cross-run mechanics are untested in the wild. The first real external-target run (see the domain-general note above) declined to use it ("skip the ledger, the PR is enough") — a reasonable call for a one-off audit, but it means check's unchanged/ changed/missing_source classification has real unit-test coverage only, not yet a real second-run confirmation against an actual evolving target. Untested, not known-broken.

Output discipline

• Never present a Pass 1 claim as verified until Pass 2 has actually checked it. A claims ledger is an input, not a finding.
• Never trace exhaustively when a targeted check would confirm or refute a claim faster.
• Lead with High-severity discrepancies. A report with one High finding buried at the bottom is worse than a short one that leads with it.
• If nothing to reconcile is found — the docs are accurate and complete for the scope checked — say so plainly. Don't manufacture findings to seem thorough.

Files