agentleFS
jamie8johnsonjamie8johnson/cqs/.claude/skills/recall-gate/SKILL.md

recall-gate

Skill18 starsChanged 5 months agoMITView on GitHub

Run the retrieval recall gate with dead-gold triage and the binary A/B regression test. Required before release tags; use after any retrieval-adjacent merge.

The file

Files
SKILL.md

Recall Gate

Answers one question with evidence: did retrieval change? Numbers moving is NOT the answer — fixture drift and corpus drift both move numbers without any retrieval change. Classify before believing.

1. Run both splits

CQS_NO_DAEMON=1 cqs eval evals/queries/v3_test.v2.json 2>/dev/null | grep OVERALL
CQS_NO_DAEMON=1 cqs eval evals/queries/v3_dev.v2.json  2>/dev/null | grep OVERALL

Aggregate = mean of the two (equal n=109). Compare against the latest recorded gate in ~/training-data/research/eval.md — NEVER against numbers from memory. Run-to-run noise is ±1pp with an active daemon.

Pin the substrate. Eval numbers depend on WHICH index computed them — a bare cqs eval relays to the daemon's RESIDENT index (index_substrate: "resident" in --json), while CQS_NO_DAEMON=1 forces a fresh CLI-side build ("fresh-built"). The two differ within the ~1pp HNSW-reconstruction noise, so comparing a resident run against a fresh-built baseline manufactures a phantom delta. Either pin CQS_NO_DAEMON=1 on every run (as above — fresh-built always) OR assert the --json index_substrate marker is uniform across every run you compare (including the recorded baseline). The recorded gate must note which substrate it was taken on.

2. If any metric dropped: check dead golds FIRST

The matcher is (origin, name) — file moves, splits, renames, and deletions kill golds and cap R@K, looking exactly like a recall regression:

# per split: for each q['gold_chunk'], does the index hold a chunk with that (origin, name)?
sqlite3 .cqs/slots/gemma/index.db "SELECT 1 FROM chunks WHERE origin=? AND name=?"

Re-pin dead golds: moved function → same name, new origin. Deleted function → re-point to the nearest current equivalent that genuinely answers the query text (read the query!), and record a repinned_<date> note in the query's metadata. Re-run step 1.

3. If a delta survives re-pinning: same-corpus binary A/B (definitive)

git worktree add /tmp/cqs-ab <baseline-tag>
cd /tmp/cqs-ab && CARGO_TARGET_DIR=$HOME/.cargo-target/cqs-ab cargo build --release --features cuda-index
CQS_NO_DAEMON=1 $HOME/.cargo-target/cqs-ab/release/cqs eval <fixture>   # both splits
CQS_NO_DAEMON=1 cqs eval <fixture>                                       # both splits, new binary

Pin BOTH sides to CQS_NO_DAEMON=1 (daemon vs CLI path differences pollute the comparison). Bit-identical output = zero retrieval change; the inter-capture delta is corpus drift — say so and pass the gate. A real difference = regression; bisect the suspect PRs ("no ranking change" claims with pins are where to start). Clean up the worktree + target dir after.

4. Record and sync

  • Append the gate block to ~/training-data/research/eval.md (table, verdict, re-pins).
  • On release gates, sync the three public surfaces to the new snapshot: README TL;DR line, Cargo.toml description, gh repo edit --description.
  • Fixture re-pins commit with the gate (release branch or a docs PR).

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.