measure

Two claude -p runs on the same task: without (this plugin switched off, everything else as usual) and with (--plugin-dir plus a pre-written local config). Each arm gets a fresh clone and the same plain prompt, runs once, and is capped by tokens, by dollars, or by both (below). The harness reads tokens, cost and time from the stream, runs the gate in each clone and writes the report. Every number you quote later comes from that report.

"Stop" and "ask" mean: end your turn with the question and wait.

Before anything runs

Ask the owner to confirm each of these in one message. Never start on a guess:

1. Task. The repo (git URL or path), the exact commit <sha> both arms start from, the issue number N and the branch name B. It should be small enough to finish in about 15 minutes.
2. Prompt. Default, filled in and shown back verbatim: Work on issue #N on branch B. Commit locally, do not push or open a PR. Report what you built, the gate results, decisions to confirm, risks and anything unverified. Both arms get exactly this text. The with arm therefore runs the whole start-task loop; headless, its questions come out as written decisions. The report's questions row counts AskUserQuestion calls only, so those written decisions are not in it.
3. Model. sonnet unless the owner names another.
4. Cap per arm. At least one is required; the harness refuses to run without one.
   • On a subscription plan (limits are in tokens, not dollars) use --max-tokens: default suggestion 1,500,000 weighted tokens per arm on sonnet. Weighted means input + output + cache creation + 0.1 x cache read (cache reads count 0.1, as plan and API limits weight them); raw cache reads are about 98% of the stream and would stop a run far too early. The harness counts this from the stream and kills the arm when the count reaches the cap; the arm keeps its partial stream, still gets the gate, and the report row stopped at token cap says yes. The count is checked per assistant message, so an arm can overshoot a little. A killed arm has no result line, so the report estimates its output tokens and takes its time from the stream timestamps, and says so.
   • On API billing use --budget (--max-budget-usd): default 3 USD.
   • Both can be given; both then apply. Say the approximate total cap for each one that applies: 2 x budget in USD, 2 x --max-tokens in weighted tokens. Above 20 USD or 20,000,000 tokens per arm needs --allow-high-budget, and only on the owner's explicit say-so.
5. Gate command to run in each clone afterwards (the repo's own test command), an optional --setup "<cmd>" run once in each clone before claude starts (for example npm ci; its time and output are not part of the arm's numbers, and a failing setup stops the run before anything is spent). The setup runs with the full environment, outside the allowed-tools list (package lifecycle scripts run), and is killed after --timeout-min, and an optional --local-config <json file> for the with clone. Without it the with clone gets { "gate": { "gate": "<cmd>" } } from --gate ({} with no gate): guided local mode.
6. Report path. Public repo: docs/measure/YYYY-MM-DD-<slug>.md. A repo that is not public: that project's git-ignored work directory.

Do not run the harness without an explicit yes to the model, whichever caps apply (tokens, budget or both) and their approximate totals.

Both arms run headless with --permission-mode acceptEdits, a fixed allowed-tools list (override with --allowed-tools "<space-separated>"; read-only gh, no push) and a clone whose push URL is disabled. Only the disabled push URL is enforced; every other "no writes" rule is the allowed-tools list by convention (node and npm can run anything). Headless, a PreToolUse ask becomes a deny: the plugin's scope and approval asks refuse the edit instead of asking (probed on Claude Code 2.1.314). An arm that runs longer than --timeout-min (default 30) is killed and marked.

Run

1. Write the prompt to a file with Write (a temp file, not in a clone).
2. Run, in one call, with a long Bash timeout (each arm can take 15 minutes): node "${CLAUDE_PLUGIN_ROOT}/scripts/measure/run.mjs" --repo <repo> --ref <sha> --prompt-file <file> --model <model> [--max-tokens <n>] [--budget <usd>] --out <report.md> --gate "<cmd>" [--setup "<cmd>"] [--local-config <file>] (at least one of --max-tokens and --budget)
3. Exit 1 with a FAILED report means claude could not be started, or an arm's claude exited non-zero with no result line, or --setup failed (the reason is on top of the report); exit 2 means a missing flag or cap: fix the arguments and ask before running again. Any other failure: report it and stop. Never re-run on your own; a re-run costs the same money again.

After

The run prints WARNING: lines when the report or a stream contains the home directory, the temp directory, the plugin's real path or a forbidden term. Show every such line to the owner before anything is committed or published; the raw .jsonl streams are never committed (docs/measure/*.jsonl is git-ignored).

Read the report and show the owner the table. An arm marked stopped at token cap is partial: its numbers are not comparable with a finished arm, so say so. State only what it says, including n=1 per arm; not a benchmark. The raw streams sit next to it as <report>.with.jsonl and <report>.without.jsonl. Do not commit or publish anything unless asked.