sandbox-isolation
Skill1 starsChanged 16 days agoMITView on GitHub
Stage an isolated, answer-free sandbox for a blind implementation and keep the run decontaminated. Use when setting up a blind eval unit, preventing answer leakage into a "blind" run, or decontaminating skills before a first run or a retest. Builds a history-free sandbox at the fork-point (no .git, no human ref, opaque slug) holding only the skills + SPEC + BUSINESS_REQUIREMENT, launches a FRESH general-purpose implementer (never a fork), and runs the leak-gate that aborts staging if any answer token survives in the sandbox skills. Triggers "stage the eval sandbox", "prevent answer leakage", "decontaminate the skills", "set up a blind run", "why is my eval contaminated", "prepare a retest sandbox". Do NOT use to score the result (that is adversarial-review-loop) or to drive the whole eval (that is blind-eval).
In this repository15 other files it gives its agents.
- codebase-mapping-forgeSkill
- conformance-lintSkill
- skill-editorSkill
- spec-style-forgeSkill
- honesty-vocabularySkill
- karpathy-guidelinesSkill
- reuse-before-createSkill
- simplicity-firstSkill
- small-diff-surgical-changeSkill
- surface-assumptionsSkill
- trace-to-consumerSkill
- verification-before-advanceSkill
- adversarial-review-loopSkill
- blind-evalSkill
- skill-improvement-loopSkill
The file
We split this skill into the rules it gives an agent
2 rules. About 0 are common, 0 are distinctive, and 2 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈0
Distinctive · estimate
This author's own choices. Worth a look.
2
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 47
${CLAUDE_PLUGIN_ROOT}/core/schema/STRIP_SET.manifest: after copying skills.source_dir — a manifest, not a hard-coded list, so a renamed … - Very specific: line 117
Run ${CLAUDE_PLUGIN_ROOT}/core/lib/extract_ai_diff.sh to diff the sandbox tree against its fork-point baseline: that is precisely what the …
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Sandbox isolation — keep the blind run blind
Contamination is the single most-repeated failure class in this kind of eval: the "blind" run quietly sees the answer, and the score becomes circular. The answer leaks through three doors — the orchestrator's own context (which holds the human diffs), the git history/ref left in the sandbox, and skills that were tuned from earlier eval units and now literally name those units' answers. This skill shuts all three: a history-free fork-point sandbox, a fresh implementer that never inherits your context, and a leak-gate that greps the sandbox skills for the unit's answer tokens and aborts staging unless the count is 0.
Everything project-specific comes from PROJECT.yaml (${CLAUDE_PLUGIN_ROOT}/core/schema/PROJECT.schema.yaml); the
scripts are ${CLAUDE_PLUGIN_ROOT}/core/lib/* and are project-agnostic.
When to use
- Setting up any unit of work for a blind implementation in an eval.
- Before any retest of a previously-failed unit — the improved skills are the most likely to contain that unit's answer.
- Whenever you suspect a "blind" run wasn't blind, or a score looks suspiciously good.
What goes in the sandbox — and what must not
| Include | Exclude (leaks the answer) |
The fork-point tree of each touched repo (repos[].name from PROJECT.yaml) | .git / any history — strip it entirely |
The project's skills (skills.source_dir), sanitized (below) | The human ref / diff — the reference lives with the scorer, never here |
SPEC.md (the design doc, from the spec adapter) | Any eval-run / prior-run data, notes, or score.json |
BUSINESS_REQUIREMENT.md (the real ask, from the tracker unit) | The unit id anywhere in the path — use an opaque slug |
The path must not reveal the unit. Stage under output.sandbox_root with a random/opaque slug so the
implementer cannot infer the unit (and cannot map the path back to a known answer).
The engine skills never enter the sandbox. This skill and its siblings (blind-eval,
adversarial-review-loop, sandbox-isolation, skill-improvement-loop, and the adapt/* forge
skills) describe the eval itself. If they were staged, the "blind" implementer would know it is being
tested and how. ${CLAUDE_PLUGIN_ROOT}/core/lib/stage_sandbox.sh strips any skill whose directory basename appears in
${CLAUDE_PLUGIN_ROOT}/core/schema/STRIP_SET.manifest after copying skills.source_dir — a manifest, not a
hard-coded list, so a renamed or newly-added engine skill is not silently left behind (CI's
zero-residue sandbox test asserts none of them survive staging). This holds even if an operator
installed the engine skills alongside the project's skills under .claude/skills/.
Stage it — ${CLAUDE_PLUGIN_ROOT}/core/lib/stage_sandbox.sh
Reads the repos from config, checks out each touched repo at the unit's fork-point (via the vcs
adapter's archive), removes .git, copies the skills into the sandbox's .claude/skills/ minus the
STRIP_SET entries and sanitizes them, copies skills.index_file (if set; it lives inside
skills.source_dir) to the sandbox root as well, and writes SPEC.md + BUSINESS_REQUIREMENT.md.
It runs the leak-gate as its last step and aborts on a hit — staging is not "done" until the gate
is green. The gate scans .claude/skills, so it checks the index file through its identical copy
there — unless the index sits inside a stripped or derived (sanitized) skill dir, whose copy is
removed; keep it at the top of skills.source_dir. A gate with nothing to check prints
NOTHING CHECKED and still exits 0, so staging goes ahead — read the gate's output, not the exit code.
The leak-gate MUST pass — ${CLAUDE_PLUGIN_ROOT}/core/lib/sanitize_skills.sh + ${CLAUDE_PLUGIN_ROOT}/core/lib/leak_gate.sh
Skills derived from earlier eval units encode those units' answers: a "trace to the consumer" skill built from a unit will name that unit's consumer field; a worked example will name the exact symbol. Handing that to the "blind" implementer is handing it the answer.
- Sanitize the derived skills — for each
skills.derived_skillsentry,${CLAUDE_PLUGIN_ROOT}/core/lib/sanitize_skills.shswaps the sandbox copy for a curated method-only<name>/SKILL.sanitized.mdthat you author once per derived skill (a version with the consumer maps, worked examples, and named symbols removed but the technique kept). It does not auto-strip prose — so if a derived skill has noSKILL.sanitized.md, the script fails closed (non-zero exit) and staging aborts rather than shipping the answer-bearing skill. (Reviewers may keep the full skills; the sandbox copy must be the sanitized one.) Authoring oneSKILL.sanitized.mdper derived skill is a required onboarding step. - Gate it —
${CLAUDE_PLUGIN_ROOT}/core/lib/leak_gate.shgreps the sandbox skills for the unit's answer tokens — both the curateddecontamination.answer_tokens[<unit_id>](fields/symbols/consumers the skill must not hand over) and distinctive identifiers auto-derived from the unit's human diff using the repo's language-aware heuristic (repos[].language), so a skill that leaks the answer in prose using a symbol nobody tokenised is still caught. The count MUST be 0 or staging aborts. Quote yourgrep --includeglobs (--include="*.md"), and read the real count, not a wrapper exit code.
This runs before every blind run and ALWAYS before a retest. A retest re-runs failed units with
the improved skills, which were written from the findings and therefore carry the answers — the gate
is what makes a retest a fair test rather than an open-book one. (Retest flow:
skill-improvement-loop.)
The implementer — FRESH, never a fork
- Launch a FRESH
general-purposesubagent. NEVER afork. A fork inherits the orchestrator's context, which contains the human diffs and the answers — an instant, invisible leak. (general-purpose, not a fork, is the whole point.) - The prompt is neutral and bounded. It must:
- Implement the unit from
SPEC.md+BUSINESS_REQUIREMENT.md+ the sandbox skills only. - Read and follow any skill whose
descriptionmatches the unit — the skills are under eval; the implementer must actually use them. - No git, no network, no external/MCP tools; stay inside the sandbox.
- No
Skilltool calls. The sandbox's.claude/skillsfiles are the only skills the arm has — read them as files. Session-loaded skills (themahoraga:*plugin pack, the live repo's.claude/skills) sit outside the arm; invoking one puts GUARD into the baseline arm and voids the ablation. (This rule is for subagent implementers inside a session. Undermahoraga loopeach arm is its own container with a fresh config, its staged content is delivered always-on, and theSkilltool can reach nothing but what was staged.) - Carry no leading question that hands over the answer — judge the agent by its code, not by a hint. If a prompt leaks the answer, the run is void: re-stage on a fresh slug and relaunch.
- Implement the unit from
Reviewers may see the human ref; implementers may not
The asymmetry is deliberate. The scorer needs the human diff as its comparison baseline, so keep it available to the review step — but never inside the sandbox, where the implementer would see it. Blind for the implementer; sighted for the reviewer.
After implementing — extract exactly what changed
Run ${CLAUDE_PLUGIN_ROOT}/core/lib/extract_ai_diff.sh to diff the sandbox tree against its fork-point baseline: that is
precisely what the agent changed, and it is what the verify step and
adversarial-review-loop consume. Record the slug and phase in
the run state file so the unit resumes cleanly.
Related skills
blind-eval— the orchestrator that calls this per unit and runs implementers in low-concurrency waves.adversarial-review-loop— scores the extracted AI diff against the human reference; re-stage here if it finds a leaked prompt.skill-improvement-loop— its validation retest depends on this skill's sanitize + leak-gate to stay honest.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

