run-evals
Skill6 starsChanged 2 days agoMITView on GitHub
Run the software-factory eval suite from the repo root — deterministic stub mode (the CI PR gate), single-case filtering, full LLM-judge mode, baseline regression comparison, and trend-history appends. Use when asked to "run the evals", "check the eval baseline", before committing prompt/constitution/golden-case changes, or when investigating an eval regression.
9 other projects use a version of this.
Adopted by
Most recently taken up by
- robertn702/reasoning-routertheir own version, first seen Nov 16, 2026
- stellar-experimental/stellar-raventheir own version, first seen Nov 15, 2026
- danielvm-git/bigpowerstheir own version, first seen Nov 6, 2026
- PyLops/pylops-agent-skillstheir own version, first seen Nov 1, 2026
- google/skills_lint.darttheir own version, first seen Oct 30, 2026
- lycorp-jp/sim-usetheir own version, first seen Oct 21, 2026
Other versions9 projects use a version of this.
Word for word · 0
None found.
Edited · 0
None found.
Wrote their own version · 9
- lycorp-jp/sim-use★ 1.5ksame name
- danielvm-git/bigpowers★ 292same name
- Azure/azure-sdk-tools★ 134same name
- BariBariGood/manzanas★ 22same name
- stellar-experimental/stellar-raven★ 17same name
- google/skills_lint.dart★ 8same name
- robertn702/reasoning-router★ 4same name
- PyLops/pylops-agent-skills★ 1same name
- rmatx/multi-agent-cs-crew★ 0same name
How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.
In this repository5 other files it gives its agents.
- AGENTS.mdAGENTS.md
- docs/agents.mdAGENTS.md
- CLAUDE.mdCLAUDE.md
- merge-policySkill
- verifySkill
The file
We split this skill into the rules it gives an agent
1 rule. 0 are common, 0 are distinctive, and 1 is very specific to this project.
0
Common
Most of the 8 edited versions we read keep these.
0
Distinctive
This author's own choices. Worth a look.
1
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 8
Run every command from the repo root.
“Common” is measured: kept by at least 6 in 10 of the 8 edited versions of this file we could read. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Run the evals
Run every command from the repo root. Golden cases live in evals/golden/*.md; the committed baseline is evals/baseline.json. See evals/README.md for the golden-case file format.
Quick check — deterministic stub mode (start here)
npm run eval -- --stubRuns the deterministic stub subset at zero model cost — the exact check CI runs on every PR (and the last step of bash scripts/verify.sh). Run it before committing any prompt, skill, constitution, or golden-set change. Stub mode never invokes the LLM judge. Exit code is non-zero if any case fails.
Run a single case
npm run eval -- --stub --filter <id-substring> --no-judge--filter keeps cases whose id contains the substring. The canonical smoke case is:
npm run eval -- --stub --filter local-small-first-green --no-judgeFull run — real harness + LLM judge (local/nightly, costs money)
npm run evalRuns all golden cases through the real model router with the LLM judge. Requires the provider CLIs/keys that packages/config routes to. Useful flags: --no-judge (skip the judge), --judge-k <n> (judge each case n times), --plan-model <id> / --judge-model <id> (pin models), --report eval-report.json (write the JSON summary), --dir <path> (alternate case directory). Flags not listed here are rejected.
Baseline comparison (regression gate)
npm run eval -- --baseline evals/baseline.jsonPrints NOTE: and REGRESSION: lines and exits 1 if the run regresses against the committed baseline. To intentionally refresh the baseline after an accepted change:
npm run eval -- --write-baseline evals/baseline.jsonThis preserves the existing tolerance and budgets fields. Commit the updated evals/baseline.json in the same PR and say why it moved.
Trend history
npm run eval -- --report eval-report.json
npm run eval-history -- --report eval-report.json --history history.jsonleval-history appends a dated record to the JSONL history (flags: --run-url, --date YYYY-MM-DD). If the report file is missing it prints a skip message and exits 0.
Reading the output
Each run prints a per-case table, then one summary line: pass-rate P/T (…%) · route-accuracy …% (C/A) · est. cost $… · total latency …s. Non-zero exit means a failed case or a baseline regression — fix or explain before committing.
Readiness suite
npm run eval -- --stub --suite readiness
npm run eval -- --stub --suite readiness --baseline evals/readiness-baseline.json
npm run eval -- --suite readiness --readiness-model <id> --judge-model <id> --baseline evals/readiness-baseline.json--suite plan|readiness selects the suite (default plan; any other value is rejected). The readiness suite runs evals/readiness/*.md through the readiness_judge route, and --readiness-model <id> pins the model for it. The stub run is also in bash scripts/verify.sh and CI. It prints accuracy, false-ready, false-not-ready, must-ask and judge-errors. Any rise in false-ready fails the baseline comparison; false-not-ready tolerates 0.10. Refresh with --write-baseline evals/readiness-baseline.json. Case format: evals/README.md.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

