bench

Arguments: $ARGUMENTS

Bench runs as node "${CLAUDE_PLUGIN_ROOT}/engine/ep.mjs" bench <subcommand> --json; bench help prints every flag and the arms.json format. Its data lives in the state folder that data.benchDir names, never in the repository.

Bench is not a sandbox. validate runs the repository's own test and prepare commands, and run starts agent sessions that load its hooks, MCP servers, commands and skills. All of it runs with the user's full rights, in scratch clones. For a repository the user doesn't trust, suggest WSL or a dev container instead.

validate and run can't be started from this session. Each refuses inside an agent session (runs started here would inherit this session's variables), lists in full what it will execute, prints a fresh random code and goes on only when a person types that code back at a terminal; without one it stops with exit 3. Nothing is remembered, so each asks again. Never unset those variables or work around a refusal, never run claude from here (not even --version), and never hand an agent a terminal to drive: an agent on a pseudo-terminal could type the code itself, which is the one thing the prompt can't stop. This skill prepares and reports; the user starts validate and run.

1. Preconditions: this is a git repository, and claude resolves on PATH (check with Get-Command claude or command -v claude; don't run it). If it's missing, give the user-level install command and stop.
2. Safety: say the not-a-sandbox paragraph in two sentences, then wait for an explicit yes before tasks or validate.
3. Tasks: ask for the full acceptance command (for example npm test) and, if the tests need it, a prepare command (for example npm ci). Run bench tasks --accept "<cmd>" [--prepare "<cmd>"] --limit 20 --tuning 2. Show the drafted prompts (tasks.jsonl in the bench folder) and let the user correct them. A prompt states the change the way the user would ask for it, never the commit's diff or tests. A task marked "promptNeedsReview": true keeps plan from working until the user has rewritten the prompt and set that to false.
4. Validate: give one line to paste into a terminal the user opens themselves, not one started from this session: node "<full path to ep.mjs>" bench validate --repo "<repository path>". It can take many minutes. The user reads the commands it prints and types the code to go on. Afterwards read the gate results from tasks.jsonl and report each dropped task with its gate: G1 history-free clone, G2 fails at base, G3 passes at reference, G4 stable verdicts.
5. Arms: help the user write arms.json in the bench folder. Use a baseline arm plus one arm with the change, with model, effort, permission mode and tools the same in both. Bench compares configurations only, and it adds a lower-effort reference arm.
6. Run bench plan --k 2 --cap-usd <per-run cap>. Show the maximum spend, the minimum detectable effect and the plan hash. Say the runs draw on the user's plan like any session. With an API key or a custom endpoint set, in the environment, the settings or an arm, bench refuses unless the user adds --allow-api-billing themselves. Wait for a yes on the spend.
7. Give one line to paste into a terminal the user opens themselves: node "<full path to ep.mjs>" bench run --plan <hash> --repo "<repository path>". It lists the repository's settings, hooks, MCP servers, commands, skills and agents, each arm's settings, plugin files and appended prompt, and the tools it pre-approves, then prints a code. Tell the user to read the list and type the code only if they accept all of it: a part marked NOT SHOWN or UNCHECKED was not read for them, so they open that file themselves first. Ctrl-C stops the agent and everything it started, and the same line resumes the plan; a run cut short counts at its per-run cap. Then stop and wait.
8. When the user is back, run bench report and show the per-task cost ratios, the geometric mean with its interval, the sign count, pass^k and the validity threats. Any drop in pass rate is a reason not to keep the change. Cost is an estimate at API prices. Report only what the plan's decision rule says; never call a difference significant on your own.