run-evals
Skill4 starsChanged 35 days agoMITView on GitHub
Run, re-run, or extend the SWE-bench Verified evaluations of reasoning-router (classifier-routed effort versus fixed effort) with the harness in eval/, and publish metadata-only reports to eval/results/. Use whenever the task is to reproduce an eval report, compare a classifier (Jev, Clef, Laya, Kev, OpenAI Decisions) against fixed medium/high/xhigh effort, measure solve rate or token use on coding tasks, add a task set, or summarize eval runs, even if the user doesn't name this skill. Not for unit-testing the router or the plugin smoke tests.
- Reads credentials
9 other projects use a version of this.
Adopted by
Most recently taken up by
- stellar-experimental/stellar-raventheir own version, first seen Nov 15, 2026
- danielvm-git/bigpowerstheir own version, first seen Nov 6, 2026
- PyLops/pylops-agent-skillstheir own version, first seen Nov 1, 2026
- google/skills_lint.darttheir own version, first seen Oct 30, 2026
- lycorp-jp/sim-usetheir own version, first seen Oct 21, 2026
- on-par/software-factorytheir own version, first seen Oct 20, 2026
Other versions9 projects use a version of this.
Word for word · 0
None found.
Edited · 0
None found.
Wrote their own version · 9
- lycorp-jp/sim-use★ 1.5ksame name
- danielvm-git/bigpowers★ 292same name
- Azure/azure-sdk-tools★ 134same name
- BariBariGood/manzanas★ 22same name
- stellar-experimental/stellar-raven★ 17same name
- google/skills_lint.dart★ 8same name
- on-par/software-factory★ 6same name
- PyLops/pylops-agent-skills★ 1same name
- rmatx/multi-agent-cs-crew★ 0same name
How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.
In this repository2 other files it gives its agents.
- AGENTS.mdAGENTS.md
- add-classifierSkill
The file
We split this skill into the rules it gives an agent
8 rules. 0 are common, 4 are distinctive, and 4 are very specific to this project.
0
Common
Most of the 8 edited versions we read keep these.
4
Distinctive
This author's own choices. Worth a look.
4
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 16
Live runs cost money: Every agent attempt calls the upstream model.
- Very specific: line 30
Check these before spending anything (details in eval/README.md):
- Distinctive: line 45
If a credential is missing, stop and ask.
- Very specific: line 80
Commit first: schedule.json records the commit and flags a dirty tree, and the driver builds the packages before the first attempt.
- Distinctive: line 82
Start small (one or two rounds) to screen tasks, then decide which tasks merit a full set.
- Distinctive: line 100
Add interpretation and scope limits by hand (selection bias, small samples, concurrency effects on latency).
- Very specific: line 103
Add a row to the Reports table in eval/README.md.
- Distinctive: line 104
Check the report contains no prompt, patch, or credential.
“Common” is measured: kept by at least 6 in 10 of the 8 edited versions of this file we could read. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Run evals
The harness in eval/ runs OpenCode with this repo's built
@reasoning-router/opencode plugin on pinned SWE-bench Verified tasks,
grades each patch with the official Docker harness, and checks that the
router's decision log accounts for every model call. eval/README.md is the
reference; if it and this skill disagree, the README and the scripts win.
Ground rules
- Live runs cost money. Every agent attempt calls the upstream model;
routed arms also call the classifier. Preparing, preflighting,
--prepare-only, summarizing, andnpm testdo not. Confirm the run set's size (tasks × arms × rounds) and the user's quota before starting one. - Never replace an attempt. A failed, timed-out, or ungraded attempt is
data. Don't delete its directory or rerun it under the same run-set name;
run-set.mjsrefuses to reuse a name for this reason. - Publish metadata only.
eval/runs/holds prompts, patches, sessions, and grader logs and is ignored. Commit onlyeval/results/*.md. - Keep the arms matched. Same model, task commit, prompt, timeout, grader, and router commit across arms. Don't change any of them mid-set.
Prerequisites
Check these before spending anything (details in eval/README.md):
- Node 24 and
npm ci; OpenCode V2 onPATH(opencode --version) orEVAL_OPENCODE_BIN. - A Python venv with
datasetsandswebench(for exampleeval/runs/swebench-venv), and a working Docker daemon. REASONING_ROUTER_UPSTREAM_BASE_URLandREASONING_ROUTER_UPSTREAM_API_KEYfor a Responses endpoint serving the model.- For a routed arm, the classifier settings from
docs/environment.md.jevneedsREASONING_ROUTER_CLASSIFIER_API_KEY(plusREASONING_ROUTER_CLASSIFIER_BASE_URL=https://ai-gateway.vercel.sh/typesafefor a Gateway key);layaandkevneed a server the user runs. UnsetREASONING_ROUTER_CLASSIFIER_MODELleft over from another provider.
If a credential is missing, stop and ask; never print or log one.
1. Pick or add a task set
Task sets are eval/tasks/<set>.json (id, repo, base commit, "grade":
"swebench", optional agentTimeoutMinutes) with a matching
<set>.sha256 over the selected dataset rows in file order. To add one,
write the selection, compute the digest once from the pinned revision, and
commit both. Choose tasks before seeing results; if you select a task
because of an observed gap, say so in the report.
2. Prepare and preflight
PY=$PWD/eval/runs/swebench-venv/bin/python
$PY eval/prepare.py <set> # writes eval/runs/swebench-<set>.json and tasks-<set>.json
$PY eval/preflight.py <set> # base must fail, reference patch must passDrop any task that fails preflight; don't run agents against it. A quick
node eval/run.mjs --manifest eval/runs/tasks-<set>.json --task ID --model
gpt-6-astra --arm jev --prepare-only confirms the generated config without
calling anything.
3. Run a matched run set
SWE_BENCH_DATASET_PATH=$PWD/eval/runs/swebench-<set>.json SWE_BENCH_PYTHON=$PY \
node eval/run-set.mjs --name <set>-<date> --manifest eval/runs/tasks-<set>.json \
--model gpt-6-astra --arms medium,high,xhigh,jev --rounds 5 --concurrency 2- An arm is a fixed effort or a classifier name;
jevis the default classifier. Routed arms run with fallback disabled, so every decision is the classifier's. - Commit first:
schedule.jsonrecords the commit and flags a dirty tree, and the driver builds the packages before the first attempt. - Start small (one or two rounds) to screen tasks, then decide which tasks merit a full set. Prefer tasks where fixed arms disagree; an all-pass or all-fail task can't show a difference.
- Long sets: run in the background and check the per-attempt logs in
eval/runs/<name>/. If the driver dies, don't restart it under the same name; summarize what finished or start a new, disclosed run set.
4. Summarize into eval/results/
node eval/summarize.mjs --run-set <set>-<date>It fails on a missing or duplicate attempt. Then:
- Review
eval/results/<name>.md. Report solved counts with denominators, evidence-valid counts, and the routed arms' effort distribution and fallbacks. Token means use evidence-valid attempts only. - Add interpretation and scope limits by hand (selection bias, small samples, concurrency effects on latency). Don't claim savings or equal solve rates the numbers don't support.
- Add a row to the Reports table in
eval/README.md. - Check the report contains no prompt, patch, or credential.
Cost and quota cautions
- Size before running: tasks × arms × rounds attempts, each up to the task's agent timeout (15 minutes by default) plus up to 30 minutes of grading.
- Subscription upstreams (such as a CLIProxyAPI account) have rolling quota windows. Check remaining quota first and leave headroom for concurrent attempts; an attempt cut off by quota still counts as initiated.
- Concurrency above two shares upstream rate limits and skews latency.
- Classifier calls are billed per request on Jev, Clef, and OpenAI Decisions; local Laya or Kev servers are not.
Verify harness changes
Changes to eval/*.mjs must keep npm run check passing; it runs
eval/test/ with fake opencode and grader binaries. Those tests prove the
harness logic, not that a live model, classifier, or grader accepts it. Say
which parts you verified live.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

