build-probe
Skill6 starsChanged 29 days agoMITView on GitHub
Turn a PRD's asserted facts into EMITTED facts. Writes one executable probe per measured claim in the PRD — each prints a single value — plus a lockfile pinning every source the PRD cites. PASS = every probe runs and prints exactly what the PRD says, and no cited source has moved. Runs after build-prd and before build-validate, and re-runs in seconds at every later stage as the drift detector. Use when a PRD asserts a number, a count, a filename, a line reference, or an "every X / none of X" claim. Triggers — "probe the PRD", "prove the anchors", "are the numbers still true", "recheck the sources", "/build probe".
In this repository30 other files it gives its agents.
- adversarial-re-readSkill
- adversarial-reviewSkill
- build-estimateSkill
- build-evaluateSkill
- build-executeSkill
- build-planSkill
- build-polishSkill
- build-prdSkill
- build-validateSkill
- excel-connector-governanceSkill
- five-whysSkill
- insight-builderSkill
- insight-crystallizerSkill
- insight-promotionSkill
- multi-sessionSkill
- notion-governanceSkill
- oodaSkill
- prd-writerSkill
- promote-to-toolingSkill
- quality-reviewSkill
- ralph-implementSkill
- ralph-loopSkill
- search-governanceSkill
- session-outputSkill
- skill-validatorSkill
- task-planSkill
- verifySkill
- voice-profile-testSkill
- why-diagnosticSkill
- writing-qualitySkill
The file
We split this skill into the rules it gives an agent
11 rules. About 0 are common, 8 are distinctive, and 3 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈8
Distinctive · estimate
This author's own choices. Worth a look.
3
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 38
After build-prd, before build-validate.
- Very specific: line 39
Before build-plan, build-execute and build-evaluate — re-run, seconds, as the drift gate.
- Distinctive: line 65
Read the PRD and list every claim that a script could settle.
- Distinctive: line 67
every number, sum, count, total, delta
- Distinctive: line 68
every filename, path, and line reference (line numbers drift — a probe re-checks them for free)
- Distinctive: line 69
every quoted string or label attributed to a document
- Distinctive: line 72
every suite or baseline count — probe it by RUNNING the suite, never by reading a prior report
- Very specific: line 159
Every subsequent stage begins with bash probes/run.sh.
- Distinctive: line 162
Do NOT re-harvest claims at every stage.
- Distinctive: line 183
Do not probe what the build will produce.
- Distinctive: line 184
If harvesting turns up more than ~20 checkable claims, the PRD is probably asserting things it does not need.
≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Build Probe
A number in a PRD is a rumour until a script prints it.
This stage exists because of a measured fact about the previous pipeline: across 21 recorded review rounds, a freshly-written plan carried a mean of 11.9 must-fix, and roughly half of every finding was a checkable claim about the world that nobody had checked. A five-agent panel found those at a cost of ~45 minutes. A 20-line script finds them in seconds, finds them the same way every time, and — unlike a panel — can be re-run tomorrow.
What this replaces
The old Stage 3 ran one detector (an LLM reviewer panel) against three different failure classes. They have wildly different cheapest detectors:
| Class | Cheapest detector | Stage |
| Facts — numbers, counts, filenames, line refs, "every/none" claims | a script | build-probe (this) |
| Plan incoherence — unwritten fields, dependency inversions, unfalsifiable checks | check-plan-soundness.py | mechanical gate |
| Premises — is the frame right, is this the correct oracle | a human, or one scoped review | build-validate |
Probe first. What survives is judgment, and judgment is all build-validate should ever see.
When to use
- After
build-prd, beforebuild-validate. Every measured claim in the PRD gets a probe. - Before
build-plan,build-executeandbuild-evaluate— re-run, seconds, as the drift gate. - Any time someone asks "is that still true".
When NOT to use
- A claim that is a judgment, not a measurement ("this is the right oracle", "the frame is
audit not generate"). Probes cannot settle those. That is
build-validate's one job. - A claim about something not yet built. A probe measures the world as it is; if the PRD asserts what the build will produce, that is the acceptance driver's job, not a probe's.
The layout
.claude/PRPs/{slug}/probes/
run.sh runs every probe, diffs emitted vs expected, verifies the lock, exits non-zero on any mismatch
lock.sh adds/refreshes sources.lock entries WITHOUT dropping existing ones
expected.tsv probe_id <TAB> expected_value <TAB> what it proves (one line per claim)
p-<id>.sh one probe per claim. Prints ONE value on stdout and nothing else.
sources.lock path <TAB> sha256 <TAB> size for every file the PRD citesCopy templates/run.sh and templates/probe.sh from this skill's directory. Do not invent a
different shape — the point is that any reader can run bash probes/run.sh without instructions.
Phase 1: HARVEST every checkable claim
Read the PRD and list every claim that a script could settle. Be exhaustive; this is the cheap part.
- every number, sum, count, total, delta
- every filename, path, and line reference (line numbers drift — a probe re-checks them for free)
- every quoted string or label attributed to a document
- every "every X" / "none of X" claim — these break most often, because a subset was checked and generalised
- every suite or baseline count — probe it by RUNNING the suite, never by reading a prior report
A claim you cannot write a probe for is either a judgment (hand it to build-validate) or
unfalsifiable (which is itself a finding — say so, do not quietly drop it).
Phase 2: WRITE one probe per claim
Each p-<id>.sh prints exactly one value on stdout. No commentary, no formatting, no exit-code
signalling — run.sh owns comparison and exit codes.
#!/usr/bin/env bash
# p-anchor1-rows.sh — PRD Anchor 1 item 1: the roster has 7 provider rows
set -euo pipefail
python3 - <<'PY'
import re, pdfplumber
with pdfplumber.open("...") as pdf:
t = "\n".join((p.extract_text() or "") for p in pdf.pages)
print(len([l for l in t.splitlines() if ROW.match(l)]))
PYRules that keep probes honest:
- A probe reads the SOURCE, never the record. Probing a value out of a file the build wrote proves the build agrees with itself. Read the PDF, the corpus, the code.
- One value, one probe. A probe printing three numbers cannot say which one moved.
- Never write. Probes are read-only. A probe that mutates a case tree has corrupted its own next run.
- Prints, does not judge.
expected.tsvholds the expectation, so the same probe re-used under a different expectation stays honest. - A probe that cannot be written is a finding. Record it in
expected.tsvwith expected valueUNPROVABLEand a reason.run.shcounts these and prints them; it does not fail on them, and it never hides them.
Phase 3: LOCK the sources
For every file the PRD or plan cites — including other slugs' plan.md and prd.md — record path,
sha256, and size in sources.lock.
This catches the class no reviewer can: a cited document being rewritten underneath you. It has happened; a plan was folded from a sibling slug's task bodies while a concurrent session cut that slug from 14 tasks to 10, and every citation silently pointed at different work.
bash probes/lock.sh <path> [<path> ...]⚠ Never build the lock by truncating it. : > sources.lock in a loop is the obvious way and it
silently destroys pins someone else added — measured 2026-09-02, a concurrent session had pinned
three cited sibling plans and a probe repoint took those pins with it. Nothing detected it, because
a lock with fewer entries still verifies clean. lock.sh merges, and refuses to re-pin a source
whose hash has changed: re-pinning a moved file is a decision, never a side effect of adding one.
Phase 4: RUN and report
bash .claude/PRPs/{slug}/probes/run.shOutput is one line per probe — OK, MISMATCH expected/got, or UNPROVABLE reason — then the lock
verification, then a summary. Exit 0 only when every probe matches and the lock verifies.
PASS = every probe OK, lock verified, zero MISMATCH. UNPROVABLE entries do not block, but they
are carried into build-validate as premises, because that is what they are.
On MISMATCH the PRD is wrong, not the probe — fix the PRD and re-run. Never adjust expected.tsv to
match a probe's output; that converts the gate into a mirror. If a probe itself is wrong, fix the
probe and say so in the commit.
Phase 5: REPORT
Write .claude/PRPs/{slug}/probe.md:
## Probe: {slug} — {ISO date}
| probe | claim | expected | emitted | verdict |
|---|---|---|---|---|
**Sources locked:** N files, sha256 recorded.
**Verdict:** PASS / FAIL ({N} mismatches)
**Carried to build-validate as premises:** {list of UNPROVABLE entries — the judgment calls}Re-running later
Every subsequent stage begins with bash probes/run.sh. It costs seconds. A red probe at plan time
means a number moved or a source was rewritten, and that is worth more than any amount of re-reading.
Do NOT re-harvest claims at every stage. Harvest once, at PRD time. New claims arrive only when the PRD changes, and a PRD change is exactly when a new probe should be written.
Output
## Probes: {slug}
**Written:** N probes, M sources locked
**Result:** {N} OK · {N} MISMATCH · {N} UNPROVABLE
**Verdict:** PASS / FAIL
### Next step
PASS → `build-validate {slug}` — premises only, and the UNPROVABLE list above is its agenda.
FAIL → fix the PRD (not the expectations) and re-run.Guidelines
- Probes are committed. They are the build's memory of why anyone believed a number.
- A probe is cheap to write and free to re-run. When in doubt, write it.
- Do not probe what the build will produce. Probe what the world already contains.
- If harvesting turns up more than ~20 checkable claims, the PRD is probably asserting things it does not need. That is a signal about the PRD, not a reason for 40 probes.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

