agentleFS
on-paron-par/software-factory/.claude/skills/run-evals/SKILL.md

run-evals

Skill6 starsChanged 2 days agoMITView on GitHub

Run the software-factory eval suite from the repo root — deterministic stub mode (the CI PR gate), single-case filtering, full LLM-judge mode, baseline regression comparison, and trend-history appends. Use when asked to "run the evals", "check the eval baseline", before committing prompt/constitution/golden-case changes, or when investigating an eval regression.

9 other projects use a version of this.

Adopted by

Most recently taken up by

The file

We split this skill into the rules it gives an agent

1 rule. 0 are common, 0 are distinctive, and 1 is very specific to this project.

  • 0

    Common

    Most of the 8 edited versions we read keep these.

  • 0

    Distinctive

    This author's own choices. Worth a look.

  • 1

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    Run every command from the repo root.

    line 8

“Common” is measured: kept by at least 6 in 10 of the 8 edited versions of this file we could read. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Run the evals

Run every command from the repo root. Golden cases live in evals/golden/*.md; the committed baseline is evals/baseline.json. See evals/README.md for the golden-case file format.

Quick check — deterministic stub mode (start here)

npm run eval -- --stub

Runs the deterministic stub subset at zero model cost — the exact check CI runs on every PR (and the last step of bash scripts/verify.sh). Run it before committing any prompt, skill, constitution, or golden-set change. Stub mode never invokes the LLM judge. Exit code is non-zero if any case fails.

Run a single case

npm run eval -- --stub --filter <id-substring> --no-judge

--filter keeps cases whose id contains the substring. The canonical smoke case is:

npm run eval -- --stub --filter local-small-first-green --no-judge

Full run — real harness + LLM judge (local/nightly, costs money)

npm run eval

Runs all golden cases through the real model router with the LLM judge. Requires the provider CLIs/keys that packages/config routes to. Useful flags: --no-judge (skip the judge), --judge-k <n> (judge each case n times), --plan-model <id> / --judge-model <id> (pin models), --report eval-report.json (write the JSON summary), --dir <path> (alternate case directory). Flags not listed here are rejected.

Baseline comparison (regression gate)

npm run eval -- --baseline evals/baseline.json

Prints NOTE: and REGRESSION: lines and exits 1 if the run regresses against the committed baseline. To intentionally refresh the baseline after an accepted change:

npm run eval -- --write-baseline evals/baseline.json

This preserves the existing tolerance and budgets fields. Commit the updated evals/baseline.json in the same PR and say why it moved.

Trend history

npm run eval -- --report eval-report.json
npm run eval-history -- --report eval-report.json --history history.jsonl

eval-history appends a dated record to the JSONL history (flags: --run-url, --date YYYY-MM-DD). If the report file is missing it prints a skip message and exits 0.

Reading the output

Each run prints a per-case table, then one summary line: pass-rate P/T (…%) · route-accuracy …% (C/A) · est. cost $… · total latency …s. Non-zero exit means a failed case or a baseline regression — fix or explain before committing.

Readiness suite

npm run eval -- --stub --suite readiness
npm run eval -- --stub --suite readiness --baseline evals/readiness-baseline.json
npm run eval -- --suite readiness --readiness-model <id> --judge-model <id> --baseline evals/readiness-baseline.json

--suite plan|readiness selects the suite (default plan; any other value is rejected). The readiness suite runs evals/readiness/*.md through the readiness_judge route, and --readiness-model <id> pins the model for it. The stub run is also in bash scripts/verify.sh and CI. It prints accuracy, false-ready, false-not-ready, must-ask and judge-errors. Any rise in false-ready fails the baseline comparison; false-not-ready tolerates 0.10. Refresh with --write-baseline evals/readiness-baseline.json. Case format: evals/README.md.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.