agentleFS
shawnclyborshawnclybor/clybor-claude-tooling/.claude/skills/build-probe/SKILL.md

build-probe

Skill6 starsChanged 29 days agoMITView on GitHub

Turn a PRD's asserted facts into EMITTED facts. Writes one executable probe per measured claim in the PRD — each prints a single value — plus a lockfile pinning every source the PRD cites. PASS = every probe runs and prints exactly what the PRD says, and no cited source has moved. Runs after build-prd and before build-validate, and re-runs in seconds at every later stage as the drift detector. Use when a PRD asserts a number, a count, a filename, a line reference, or an "every X / none of X" claim. Triggers — "probe the PRD", "prove the anchors", "are the numbers still true", "recheck the sources", "/build probe".

The file

We split this skill into the rules it gives an agent

11 rules. About 0 are common, 8 are distinctive, and 3 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈8

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 3

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    After build-prd, before build-validate.

    line 38
  2. Very specific:

    Before build-plan, build-execute and build-evaluate — re-run, seconds, as the drift gate.

    line 39
  3. Distinctive:

    Read the PRD and list every claim that a script could settle.

    line 65
  4. Distinctive:

    every number, sum, count, total, delta

    line 67
  5. Distinctive:

    every filename, path, and line reference (line numbers drift — a probe re-checks them for free)

    line 68
  6. Distinctive:

    every quoted string or label attributed to a document

    line 69
  7. Distinctive:

    every suite or baseline count — probe it by RUNNING the suite, never by reading a prior report

    line 72
  8. Very specific:

    Every subsequent stage begins with bash probes/run.sh.

    line 159
  9. Distinctive:

    Do NOT re-harvest claims at every stage.

    line 162
  10. Distinctive:

    Do not probe what the build will produce.

    line 183
  11. Distinctive:

    If harvesting turns up more than ~20 checkable claims, the PRD is probably asserting things it does not need.

    line 184

≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Build Probe

A number in a PRD is a rumour until a script prints it.

This stage exists because of a measured fact about the previous pipeline: across 21 recorded review rounds, a freshly-written plan carried a mean of 11.9 must-fix, and roughly half of every finding was a checkable claim about the world that nobody had checked. A five-agent panel found those at a cost of ~45 minutes. A 20-line script finds them in seconds, finds them the same way every time, and — unlike a panel — can be re-run tomorrow.

What this replaces

The old Stage 3 ran one detector (an LLM reviewer panel) against three different failure classes. They have wildly different cheapest detectors:

ClassCheapest detectorStage
Facts — numbers, counts, filenames, line refs, "every/none" claimsa scriptbuild-probe (this)
Plan incoherence — unwritten fields, dependency inversions, unfalsifiable checkscheck-plan-soundness.pymechanical gate
Premises — is the frame right, is this the correct oraclea human, or one scoped reviewbuild-validate

Probe first. What survives is judgment, and judgment is all build-validate should ever see.

When to use

  • After build-prd, before build-validate. Every measured claim in the PRD gets a probe.
  • Before build-plan, build-execute and build-evaluate — re-run, seconds, as the drift gate.
  • Any time someone asks "is that still true".

When NOT to use

  • A claim that is a judgment, not a measurement ("this is the right oracle", "the frame is audit not generate"). Probes cannot settle those. That is build-validate's one job.
  • A claim about something not yet built. A probe measures the world as it is; if the PRD asserts what the build will produce, that is the acceptance driver's job, not a probe's.

The layout

.claude/PRPs/{slug}/probes/
  run.sh            runs every probe, diffs emitted vs expected, verifies the lock, exits non-zero on any mismatch
  lock.sh           adds/refreshes sources.lock entries WITHOUT dropping existing ones
  expected.tsv      probe_id <TAB> expected_value <TAB> what it proves     (one line per claim)
  p-<id>.sh         one probe per claim. Prints ONE value on stdout and nothing else.
  sources.lock      path <TAB> sha256 <TAB> size    for every file the PRD cites

Copy templates/run.sh and templates/probe.sh from this skill's directory. Do not invent a different shape — the point is that any reader can run bash probes/run.sh without instructions.

Phase 1: HARVEST every checkable claim

Read the PRD and list every claim that a script could settle. Be exhaustive; this is the cheap part.

  • every number, sum, count, total, delta
  • every filename, path, and line reference (line numbers drift — a probe re-checks them for free)
  • every quoted string or label attributed to a document
  • every "every X" / "none of X" claim — these break most often, because a subset was checked and generalised
  • every suite or baseline count — probe it by RUNNING the suite, never by reading a prior report

A claim you cannot write a probe for is either a judgment (hand it to build-validate) or unfalsifiable (which is itself a finding — say so, do not quietly drop it).

Phase 2: WRITE one probe per claim

Each p-<id>.sh prints exactly one value on stdout. No commentary, no formatting, no exit-code signalling — run.sh owns comparison and exit codes.

#!/usr/bin/env bash
# p-anchor1-rows.sh — PRD Anchor 1 item 1: the roster has 7 provider rows
set -euo pipefail
python3 - <<'PY'
import re, pdfplumber
with pdfplumber.open("...") as pdf:
    t = "\n".join((p.extract_text() or "") for p in pdf.pages)
print(len([l for l in t.splitlines() if ROW.match(l)]))
PY

Rules that keep probes honest:

  1. A probe reads the SOURCE, never the record. Probing a value out of a file the build wrote proves the build agrees with itself. Read the PDF, the corpus, the code.
  2. One value, one probe. A probe printing three numbers cannot say which one moved.
  3. Never write. Probes are read-only. A probe that mutates a case tree has corrupted its own next run.
  4. Prints, does not judge. expected.tsv holds the expectation, so the same probe re-used under a different expectation stays honest.
  5. A probe that cannot be written is a finding. Record it in expected.tsv with expected value UNPROVABLE and a reason. run.sh counts these and prints them; it does not fail on them, and it never hides them.

Phase 3: LOCK the sources

For every file the PRD or plan cites — including other slugs' plan.md and prd.md — record path, sha256, and size in sources.lock.

This catches the class no reviewer can: a cited document being rewritten underneath you. It has happened; a plan was folded from a sibling slug's task bodies while a concurrent session cut that slug from 14 tasks to 10, and every citation silently pointed at different work.

bash probes/lock.sh <path> [<path> ...]

⚠ Never build the lock by truncating it. : > sources.lock in a loop is the obvious way and it silently destroys pins someone else added — measured 2026-09-02, a concurrent session had pinned three cited sibling plans and a probe repoint took those pins with it. Nothing detected it, because a lock with fewer entries still verifies clean. lock.sh merges, and refuses to re-pin a source whose hash has changed: re-pinning a moved file is a decision, never a side effect of adding one.

Phase 4: RUN and report

bash .claude/PRPs/{slug}/probes/run.sh

Output is one line per probe — OK, MISMATCH expected/got, or UNPROVABLE reason — then the lock verification, then a summary. Exit 0 only when every probe matches and the lock verifies.

PASS = every probe OK, lock verified, zero MISMATCH. UNPROVABLE entries do not block, but they are carried into build-validate as premises, because that is what they are.

On MISMATCH the PRD is wrong, not the probe — fix the PRD and re-run. Never adjust expected.tsv to match a probe's output; that converts the gate into a mirror. If a probe itself is wrong, fix the probe and say so in the commit.

Phase 5: REPORT

Write .claude/PRPs/{slug}/probe.md:

## Probe: {slug}  —  {ISO date}

| probe | claim | expected | emitted | verdict |
|---|---|---|---|---|

**Sources locked:** N files, sha256 recorded.
**Verdict:** PASS / FAIL ({N} mismatches)
**Carried to build-validate as premises:** {list of UNPROVABLE entries — the judgment calls}

Re-running later

Every subsequent stage begins with bash probes/run.sh. It costs seconds. A red probe at plan time means a number moved or a source was rewritten, and that is worth more than any amount of re-reading.

Do NOT re-harvest claims at every stage. Harvest once, at PRD time. New claims arrive only when the PRD changes, and a PRD change is exactly when a new probe should be written.

Output

## Probes: {slug}

**Written:** N probes, M sources locked
**Result:** {N} OK · {N} MISMATCH · {N} UNPROVABLE
**Verdict:** PASS / FAIL

### Next step
PASS → `build-validate {slug}` — premises only, and the UNPROVABLE list above is its agenda.
FAIL → fix the PRD (not the expectations) and re-run.

Guidelines

  • Probes are committed. They are the build's memory of why anyone believed a number.
  • A probe is cheap to write and free to re-run. When in doubt, write it.
  • Do not probe what the build will produce. Probe what the world already contains.
  • If harvesting turns up more than ~20 checkable claims, the PRD is probably asserting things it does not need. That is a signal about the PRD, not a reason for 40 probes.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.