rag-pipeline-v1
CLAUDE.md0 starsChanged 35 days agoMITView on GitHub
Working spec lives in IMPLEMENTATION_PLAN.md. Design rationale lives in docs/decisions.md. Current phase is stated at the top of decisions.md.
- Reads credentials
In this repository4 other files it gives its agents.
- AGENTS.mdAGENTS.md
- bake-offSkill
- phase-gateSkill
- plan-gateSkill
The file
We split this CLAUDE.md into the rules it gives an agent
7 rules. About 0 are common, 3 are distinctive, and 4 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈3
Distinctive · estimate
This author's own choices. Worth a look.
4
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- could be automatic: a setting could enforce it
- Very specific: line 26
Run tests: python -m pytest tests/ -q (must pass before any commit)
- Very specific: line 62
NEVER commit anything in data/, any *.pdf, .env, chroma_db/, or eval/private/ (D65).
- Very specific: line 89
Every design choice (chunk size, pattern, library, threshold) gets a short entry appended to docs/decisions.md: what / why / what was …
- Distinctive: line 91
If implementation diverges from the approved plan, STOP and return to plan mode — do not improvise silently.
- Distinctive: line 96
No new dependencies without stating why in the plan.
- Distinctive: line 115
Treat Codex findings like pressure-tester findings: verify each against the code before acting.
- Very specific: line 119
NEVER paste corpus text (handbook extracts, chunk contents, held-out eval questions) into a Codex prompt, and always include the …
≈ “Common” is an estimate: at least 1 in 5 of the 27,641 CLAUDE.md files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
CLAUDE.md — legal RAG pipeline
What this project is
RAG pipeline over an OCR-scanned Irish conveyancing handbook (~800 pages, decimal
paragraph numbering: Chapter 3 → 3.2 → 3.2.1). The product's entire point is
grounded answers with chapter/paragraph/page citations and explicit refusal
when the answer is not in the corpus. Portfolio piece for the Claude Corps
Fellowship — submitted 17 Jul 2026 at v2.1.1. Now on the post-submission
production-hardening track; the current phase is stated at the top of
docs/decisions.md.
Working spec lives in IMPLEMENTATION_PLAN.md. Design rationale lives in docs/decisions.md. Current phase is stated at the top of decisions.md.
Harness
The workflow itself (gates, critics, bake-offs) is documented in docs/harness.md — read it before changing how work gets done here. Fresh-context critics live in .claude/agents/ (plan-auditor, pressure-tester); gates and the design tournament in .claude/skills/ (plan-gate, phase-gate, bake-off). Designs too big for a decisions.md entry, and all bake-off briefs, are artifacts in docs/designs/ (see its README). Bake-offs are judged by eval/golden_set.jsonl — never by a model's opinion, never on the held-out set.
Commands
- Run tests:
python -m pytest tests/ -q(must pass before any commit) - Index corpus:
python -m src.pipeline index ./data/Conveyancing_Handbook.pdf --type handbook - Query:
python -m src.pipeline query "..." --top-k 6 - Extraction QA:
python scripts/extraction_qa.py ./data/Conveyancing_Handbook.pdf - Eval (canonical v5, now historical: heldout-v1 retired to regression in
16A-2a, D73; the headline becomes canonical v6
eval/results_v6.md, written only by a slot-0/0b sealed run,pipeline eval --sealed-slot; makes live API calls incl. Haiku query expansion):python -m src.pipeline eval --heldout eval/heldout_set.jsonl --realistic eval/realistic_set.jsonl --judge— writes the committedeval/results.mdonly on a canonical run, i.e. ALL of: a held-out set AND a realistic set, each with answerable hybrid results, at distinct file identities; all 4 modes incl. hybrid+rewrite; refusals AND completeness scored; top_k=6; zero generation errors; zerogeneration_incompleterows (truncated/declined/incomplete stop reasons) and zerounknown-status rows (v5, H1c); query expansion attempted (>=1) with zero fallbacks; the BM25 sidecar actually loaded whenever a default retrieval/generation path ran (D51); and the judge ran on every set, judged=1 item and had zero judge API/parse errors (D51). Else the gitignored
eval/results_partial.md. - Validate an eval set (schema v1/v2, errors by line and field, never text):
python scripts/validate_eval_set.py <path> - Leak scan (D65):
python scripts/scan_leaks.py --output <files> --run <id>before releasing a private run's output;--merge-gate --base mainbefore a push when private sets exist (add--pr <n>once the branch has a PR). - Eval offline / CI (no API key, retrieval ablation only):
python -m src.pipeline eval --skip-refusals --skip-completeness— both skips are required to make ZERO API calls: they suppress generation AND disable Haiku query expansion (the hybrid+rewrite row then renders as the disabled fallback, equal to raw hybrid). Exception (16A-2a, D73): an explicit--expansion build:<path>under both skips makes metered Haiku rewrite calls only, and a replayed artifact under both skips runs retrieval only with zero calls. - Query without expansion (debug/offline): add
--no-rewriteto the query command.
Hard rules
- NEVER commit anything in
data/, any*.pdf,.env,chroma_db/, oreval/private/(D65). The corpus is copyrighted; the repo is public. - Private eval data (D65): eval inputs are
public/private/sealed(src.eval_sets.classify). Private material lives only undereval/private/; a private run prints aggregates and opaque ids only and writes only undereval/private/runs/<run id>/. Never pass a weakerprivacythan the floor, never add a registry/classifier parameter tosrc/orscripts/, and with private sets present runpython scripts/scan_leaks.py --merge-gate --base mainbefore every push, with--pr <n>once a PR exists (a hit in pushed history is a D64 owner stop). - Sealed eval data (D72, 16A-2a): sealed input runs only inside a claimed
slot (
pipeline eval --sealed-slot <n>) and is read only throughsrc/eval_sealed_read.py; every other run, retrieval, generation or replay entry refuses it (exit 4). Onlyscripts/sealed_tool.pyremoves a sealed marker (release). The orchestrator never openseval/private/sealed/, the sealed processor's staging root (~/.local/state/claudecode/sealed_sp/) orTutorial_Docs_for_review/(beyond the 12 Oct table-of-contents read); sealed rows reach it only assealed_toolcounts, digests and opaque ids. Runpython scripts/sp_launch.py --sweepbefore any Codex call. - Eval spend (D70): every live eval call goes through the spend meter
(
src/spend.py, ledger~/.local/state/claudecode/anthropic_spend.jsonl); the €40/UTC-week cap is hard — exit 3 is a D64 owner stop, never lift it without--owner-approved-eur X --approval-ref <ref>after the owner approves X. - Evidence before claims: run the code/tests and show the output before saying something works. "Should work" is not done.
- Every design choice (chunk size, pattern, library, threshold) gets a short entry appended to docs/decisions.md: what / why / what was rejected.
- If implementation diverges from the approved plan, STOP and return to plan mode — do not improvise silently.
- Scope: one phase of IMPLEMENTATION_PLAN.md per branch and PR. Under the standing go (D64) a session may chain phases in roadmap order (design 003 §9); without it, do not start the next phase without being asked.
- No new dependencies without stating why in the plan; pin exact versions in requirements.txt.
Adversarial review (Codex)
A second-vendor reviewer (OpenAI Codex CLI, model configured by the user) is
installed: use the codex MCP tool when present, otherwise
codex exec --sandbox read-only "..." via Bash. ALWAYS pass
--sandbox read-only — the default sandbox is workspace-write, i.e. Codex
could otherwise edit this repo; a reviewer must not touch the working tree.
It can read files and run read-only commands (e.g. git diff) itself — pass
it pointers (file paths, branch names), not pasted content.
- Plan gate: after plan-auditor passes a phase plan, get a Codex critique of the plan file; reconcile both sets of findings before implementing.
- Merge gate: before requesting merge of a phase branch, get a Codex review of the diff vs main; fix findings forward or rebut them explicitly in the PR description.
- Canonical call:
codex exec --sandbox read-only "Adversarially review <plan file | the diff vs main> for phase N of IMPLEMENTATION_PLAN.md: real bugs, missing steps, spec divergence, weak tests. Cite file:line. Where a fix is small and mechanical, include a proposed unified diff in the finding (text only — you cannot apply it). Do NOT read data/, chroma_db/, chroma_db_arm_*/, eval/bakeoff/, eval/private/, Tutorial_Docs_for_review/, ~/.local/state/claudecode/sealed_sp/, or held-out eval files." - Treat Codex findings like pressure-tester findings: verify each against the code before acting; it can be wrong or out of scope. Proposed diffs are suggestions, not patches: verify and apply them yourself — Codex never writes to the tree.
- NEVER paste corpus text (handbook extracts, chunk contents, held-out eval
questions) into a Codex prompt, and always include the do-not-read clause
above — the corpus is copyrighted and must not be shipped to a third-party
model (same reason as the
data/commit ban; notechroma_db/and the Phase 15 bake-off arm indexeschroma_db_arm_*/contain the full corpus text too;eval/private/holds private and sealed eval material, D65).
Conventions
- Python 3.11+, type hints and docstrings on all public functions.
- Structural regexes anchor to line starts; no IGNORECASE on structural markers (learned from the PART false-positive).
- Chunk metadata keys: chapter_number, chapter_title, section_number, heading, page_start, page_end, source, title, document_type.
- Tests mock all IO and models (see the FakeEmbeddings pattern in tests/test_embedder.py, imported by the other test files that need it); no network or API calls in the test suite.
- Explain non-obvious diffs at the level of someone re-learning Python.
Context that saves you time
- Corpus PDF has an OCR text layer: expect header/footer pollution, hyphenation breaks, occasional character errors. Cleaning happens in ingest, driven by scripts/extraction_qa.py findings.
- ingest returns (clean_text, page_map) — page_map carries per-page char offsets; the chunker uses it to assign page_start/page_end.
- Embedding model: sentence-transformers/all-MiniLM-L6-v2, recorded in the Chroma collection metadata; corpus and queries must use the same model.
- Retrieval is hybrid: BM25 + vector, fused with reciprocal rank fusion.
- The legislation chunker (PART/Section patterns) is retained and routed by document_type; the handbook strategy is the default for this corpus.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

