run-evals
Skill17 starsChanged 2 days agoApache-2.0View on GitHub
Plan and run scoped evaluations, check routing gates, compare answer quality or client models, review stored verdicts, or close an eval round. Name the decision first, run the smallest instrument that can change it, call the result early, and route evidence-backed findings.
9 other projects use a version of this.
Adopted by
Most recently taken up by
- robertn702/reasoning-routertheir own version, first seen Nov 16, 2026
- danielvm-git/bigpowerstheir own version, first seen Nov 6, 2026
- PyLops/pylops-agent-skillstheir own version, first seen Nov 1, 2026
- google/skills_lint.darttheir own version, first seen Oct 30, 2026
- lycorp-jp/sim-usetheir own version, first seen Oct 21, 2026
- on-par/software-factorytheir own version, first seen Oct 20, 2026
Other versions9 projects use a version of this.
Word for word · 0
None found.
Edited · 0
None found.
Wrote their own version · 9
- lycorp-jp/sim-use★ 1.5ksame name
- danielvm-git/bigpowers★ 292same name
- Azure/azure-sdk-tools★ 134same name
- BariBariGood/manzanas★ 22same name
- google/skills_lint.dart★ 8same name
- on-par/software-factory★ 6same name
- robertn702/reasoning-router★ 4same name
- PyLops/pylops-agent-skills★ 1same name
- rmatx/multi-agent-cs-crew★ 0same name
How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.
In this repository9 other files it gives its agents.
- AGENTS.mdAGENTS.md
- CLAUDE.mdCLAUDE.md
- audit-reviewabilitySkill
- cloudflare-observability-reviewSkill
- golden-truthSkill
- improvements-pipelineSkill
- live-drift-resolutionSkill
- retrieval-system-auditSkill
- truth-maintenanceSkill
The file
We split this skill into the rules it gives an agent
12 rules. 0 are common, 7 are distinctive, and 5 are very specific to this project.
0
Common
Most of the 8 edited versions we read keep these.
7
Distinctive
This author's own choices. Worth a look.
5
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- could be automatic: a setting could enforce it
- Very specific: line 8
Every instrument asks one question at some layer: does an agent driving this MCP end to end give a correct, current, non-fabricated answer?
- Distinctive: line 31
Call the result as soon as the evidence meets the stop rule.
- Distinctive: line 62
Do not test or evaluate with Haiku or GPT-6 Luna.
- Distinctive: line 160
Before reading any aggregate, assert the expected job count.
- Very specific: line 178
Read wrong counts before correct counts.
- Distinctive: line 212
A reported regression is an outcome, not a cause: Before you special-case a dropped keyword or rank, check whether a threshold (for example …
- Very specific: line 222
Findings: Use improvements-pipeline.
- Very specific: line 225
Goldens: Every judge-facing change goes through golden-truth with live evidence and a root cause.
- Very specific: line 238
After case edits, npm run eval:qa:lint -- --since <ref> --stale is green and npm run eval:qa:register re-stamped the clusters.
- Distinctive: line 246
When two instruments disagree, the one closer to the headline wins.
- Distinctive: line 247
No per-question tuning. No schemas or judge code copied from prior-art repos.
- Distinctive: line 249
Never print or commit secrets.
“Common” is measured: kept by at least 6 in 10 of the 8 edited versions of this file we could read. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Run evals
Every instrument asks one question at some layer: does an agent driving this MCP end to end give a
correct, current, non-fabricated answer? The instruments and their contracts live in
eval/EVALS.md (map), eval/README.md
(routing), and eval/qa/README.md (QA flags, identity pins, judge
contract). This skill decides what to run, how much, and what to do with the result.
1. Decide before you spend
The size of the work is what to keep small: how long it takes, what it costs, and where it runs. Routing, plan, lint, and self-test checks are free: run them whenever their inputs changed. Before any paid call or long agent task, write four lines in the round ledger or pull request:
- Decision. What changes with the result: ship or revert, switch a model, file a finding.
- Smallest run. The fewest cases, arms, and clients that can change that decision. Reuse a stored result when its case, model, rubric, pack, and source identities match. A new session is not a reason to collect again.
- Stop rule, cap, and estimate. The observation that ends the run for each outcome, the
dollar cap (
--max-budget-usd) or call count, and the expected time and cost. - Review coverage. Which rows get transcript review: safety rows, failures, and decision-changing disagreements first, plus a fixed sample of agreements.
Then hold to it:
- Call the result as soon as the evidence meets the stop rule. Do not buy a confirmation round, a second look, or a re-judge that cannot change the decision.
- A no-switch or inconclusive result finishes the round. Never expand collection or review to get a preferred answer.
- A changed scope or stop rule is a named amendment: keep the old results, failures, and denominator, and report the new sample separately.
- A paid run needs the owner's spending authorization. One call-count or dollar authorization covers one method run; a re-run needs its own, even when it repairs the method.
- Get an independent review of the four lines only when the owner asks for one, or when the run is
a multi-arm comparison or a source-addition lane
(
references/source-addition.md). Reuse that review while its inputs hold. A missed review blocks launch; it does not invalidate data already collected.research/audits/2026-07-11-eval-round-orchestration.mdrecords what such reviews catch.
Cost counts. Record cost per answer beside correctness for every arm. Prefer the more correct path only when the alternative is far less correct: new wrong answers, fabrication, or a failed safety or trap behavior. A path that is much cheaper and only slightly less correct can be the better product, because two calls for the price of one leave room for clarification and refinement. State both numbers and make the call; scores are evidence, not the decision.
2. Clients, models, and billing
| Client under test | CLI and model ID | Use |
| Claude Sonnet 5.5 | claude --model claude-sonnet-5-5 | Default set |
| GPT-6.1 Terra | codex -m gpt-6.1-terra | Default set |
| Grok 4.7 | grok --model grok-4.7 | Default set |
| Claude Opus 5.5 | claude --model claude-opus-5-5 | Add only where a stronger client can change the decision |
| GPT-6.1 Sol | codex -m gpt-6.1-sol | Add only where a stronger client can change the decision |
- Do not test or evaluate with Haiku or GPT-6 Luna. They are uncommon among users and too inconsistent to weigh.
- One client is enough when the decision does not depend on the client. Add clients only for a client-dependent decision.
- Run here, on
hc. Development tests and evals use the localwrangler devserver (npm run dev:eval) and the subscription CLIs (Claude Code, Codex, Grok) driven throughhc, before any API-billed path (Cloudflare AI Gateway, provider APIs). Use API billing only when no CLI can do the job, and say why in the four lines of each plan. Two lanes qualify. The Playground lane measures the product route, which itself calls the Gateway. The Gateway second judge is the opt-in diagnostic ineval/qa/gateway-judge.md: it needs an independent OpenAI judge model, and that contract documents why the Codex CLI transport is unsupported for it. Neither exception extends to other API judging or collection. - The QA runner and the discovery agent arm spawn headless
claude -pthemselves. Start the runner as anhcchild (hc run --bg --name <lane> ...) andhc waitfor it.--modelselects the Claude client. - Codex and Grok have no committed runner. Run them as
hclanes with the launch lines in.agents/model-roster.md, register the loopback MCP URL withcodex mcp addorgrok mcp add, and have each lane write its answers to a file. Report the result as a client diagnostic with its own denominator; never merge it with QA-runner results.
Three roles stay separate. The orchestrating agent runs this skill and never answers cases
itself. The answering agent and the judge are spawned by the runner. Their defaults
(AGENT_MODEL in eval/qa/run-qa.mjs, JUDGE_MODEL in eval/qa/judge.mjs) are a measurement
contract: they move only by a recorded eval decision, never by the repo-work routing in
AGENTS.md. The judge stays on its pinned model so that verdicts remain comparable across runs. Report the answering model, judge model, rubric, pack, case count, and results stamp
with every QA number.
3. Pick the instrument
| What changed or what you ask | Run |
| Scoring, catalog, or manifest | npm run eval:routing -- --gate (free, seconds, always) |
| Retrieval work (decomposition, vocabulary) | Routing; the extended lane is the target, the legacy lane the non-regression gate |
| Skill routing or skill store | Routing; read the skills lane against its floor |
| Upstream drift refresh landed | Routing gate; refresh improvements/ statuses |
| Tool descriptions, MCP instructions, nudges | Small QA sample plus plan regrade; routing math does not move |
| Answer quality before and after a big change | QA battery on matched IDs (headline, paid) |
| Executor, adapter, or envelope | QA live lane (--cases eval/qa/corpus/live/live-cases.json) |
| Search behavior of agents | Agentic lane (paid, Claude Code Workflow tool, live server) |
| Tool-plan question on a stored run | npm run eval:plan -- <results> (free, offline) |
| Stale-gospel gate fired | truth-maintenance owns the queue; each case goes through golden-truth |
| Playground model or prompt | eval/playground/README.md (paid, API-billed) |
Open .agents/rounds/<YYYY-MM-DD>-<slug>.md for a round with paid instruments or several lanes;
a single free check needs no ledger. Use truth-maintenance only for an explicitly broad round.
4. Free checks
npm run eval:selftest # grader math and frozen live-contract pins
npm run eval:compile # corpus -> eval/routing-cases.json
npm run eval:routing -- --gate # exit 1 on a breach or a changed denominator
npm run eval:qa:compile # only when QA cases changed
npm run eval:qa:lint -- --stale # add --since <base-ref> after editing judge-facing gospelWithout --since, the gospel-change lane skips locally while CI enforces it. Generated files
(routing-cases.json, qa/cases.json, qa/sample.json, qa/lifecycle-registry.json,
plan/op-classes.json) come from these scripts; CI byte-pins them. The two live contracts are
frozen: change them only with a provenance note, a contract-version bump, and a digest update.
npm run eval:qa:selftest is paid (seven judge calls) although it needs no server. Run it only
when the rubric, prompt, evidence pack, or judge adapter changed, under its own authorization.
5. Paid runs
- Server. Reuse a terminal already running
npm run dev:eval(hc ls, thenhc read <t>for its port). If none exists, start your own:hc run --bg --name dev-eval npm run dev:eval -- --port <port>, thenhc wait dev-eval --output Ready. Stop only a server you started. Usedev:eval, notdev: without its--host localhostevery request returns 401. Readiness. A plain GET on
/mcpproves nothing. Expect200from a real initialize:curl -s -o /dev/null -w '%{http_code}' -X POST "http://localhost:${PORT}/mcp" \ -H 'content-type: application/json' -H 'accept: application/json, text/event-stream' \ -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-03-26","capabilities":{},"clientInfo":{"name":"probe","version":"0"}}}'- Pins and cap. Compute the identity pins in the collection shell as
eval/qa/README.mddescribes, and pass exactly one--max-budget-usd. A changed pin stops the method; a new pin needs a new authorization. Never resume a stopped artifact under the same authorization. - Estimate cost from stored runs, not from a per-case figure in a README: read
meta.totalCostUsdineval/qa/results/*.jsonfor the same lane, model, rubric, and pack, and quote the observed range and median. Run.
# QA sample (variant A = the shipped search tool); add the pin flags from eval/qa/README.md node eval/qa/run-qa.mjs --variant A --sample 30 --port "$PORT" --max-budget-usd "$MAX_BUDGET_USD" ... # smallest run: --ids a,b,c collect only: --no-judge other client: --model <id> # live lane: --cases eval/qa/corpus/live/live-cases.json instead of --variant/--sample npm run eval:plan -- eval/qa/results/<stamp>-variantA.json # free regrade of the stored runThe agentic lane needs the Claude Code Workflow tool; follow
eval/agentic/README.md. Report QA by the five-track contract ineval/qa/README.md, and never emulate a retry with an ad hoc rerun.
Before reading any aggregate, assert the expected job count. A harness that filters failed jobs shrinks the denominator silently. A short run forbids an aggregate or a baseline delta; a paired per-row comparison on the completed jobs stays valid. When attributing a delta across several landed changes, name the cases each change can affect before the run, or claim only a diagnostic. The noise floor bounds what variance can explain; it is not a significance threshold.
6. Read the result
Gates first. eval/gates.json owns the legacy, skills, and holdout thresholds. On a failure,
a regression is fixed or reverted. Re-baseline only when the numbers moved for a legitimate, named
reason (drift, a deliberate policy change): update gates.json in the same commit, check per-case
hit-to-miss regressions (zero is the standard), and fill its provenance fields.
QA verdicts are evidence, not ground truth. Review the declared rows against the full
transcript and the exact golden. caseInput on each row holds the judge input; a review packet
needs the question, golden, trap and freshness tags, as-of metadata, and transcript. Truncated
execute results end in a --- SOURCE BASIS --- block.
- Read
wrongcounts beforecorrectcounts. Compare runs only on the same case IDs, model, rubric, and pack; the sampler can change membership when the corpus grows. - Untagged cases are judged corpus-blind, so a grounded specific that the judge cannot see can be graded as fabrication. Resolve a decision-bearing dispute from the transcript.
- A verdict is one sample. A flip does not show which grade is right; keep both observations.
Re-judge (
eval/qa/re-judge.mjs, one--max-budget-usd) only when it can settle the decision. - For freshness cases, a sourced difference from the golden snapshot is fine; a confident unsourced contradiction is not.
- A
wrongwhose rationale cites a must-avoid item phrased as "beyond corpus support" is a suspect judge artifact until verified live. - The paired verdict (
npm run eval:qa:paired) is experimental and never a ship gate.
7. Triage reviewed failures
Classify each reviewed failure and surprising pass. Disclose unreviewed rows; they are not passes.
| Root cause | How to recognize | Where it goes |
| Judge artifact | Live re-execution contradicts the verdict | Round record; rubric note if the mode is new |
| Agent failure | Tool use or synthesis is wrong in the transcript | Round record; .agents/TODO.md only for a pattern |
| Own-repo gap (scoring, catalog, executor, adapters) | Search buried the right entry, or a normalizer misread a payload | .agents/TODO.md, never improvements/ |
| Eval-side gap (stale golden, mislabeled case) | The golden disagrees with the service's own live answer | .agents/TODO.md; fix through golden-truth |
| Upstream data or content gap | Correct agent and plumbing still cannot answer from what the service returns | improvements/ finding |
| Upstream semantics or spec gap | The service's self-description misleads any consumer | improvements/ finding |
| Corpus-coverage diagnostic | The truth lives with another canonical owner and no tested surface undertakes to host it | Keep truth in the golden; record a coverage result, not a Docs issue |
- Bar for acting. The same failure in two unrelated cases, a contract mismatch, a reproducible infrastructure bug, or trace evidence that the model was asked the wrong thing. A single-case finding goes on a named monitor-only list in the round record.
- No overfitting. Fix a general mechanism or leave the case red with a note. No
query-to-service maps and no per-question vocabulary. Case IDs, question phrases, and golden
fragments stay in the round record, out of descriptions, keywords, scorer constants, and prompts.
The same bar binds docs scoping changes (
improvements-pipeline). When the first failed boundary is an upstream gap, file or update the finding before you propose local wording. - A reported regression is an outcome, not a cause. Before you special-case a dropped keyword or rank, check whether a threshold (for example the document-frequency cut) is doing its job.
- Before a wording fix, read what the model was already told at the failure moment, nearest
first: guard and
codemode.*error strings (src/executor/providers.ts), truncation footers (src/policy/truncate.ts), adapter hints (src/adapters/),searchnextSteps, tool descriptions andSERVER_INSTRUCTIONS(src/mcp/tools.ts), then catalog descriptions (generated; changescripts/). If the prose already teaches the behavior, prefer a mechanism (a fail-loud guard, an error that carries the fix) over more words. If it is absent or wrong, change the one surface nearest the failure. Measure before and after on the same instrument and revert a prose edit with no measured shift. A catalog description change re-runs the routing gate. - Findings. Use
improvements-pipeline. A judge's opinion never moves a finding pastproposed;verifiedneeds your own live re-execution. Update an existing finding before filing a near-duplicate. - Goldens. Every judge-facing change goes through
golden-truthwith live evidence and a root cause. A desired score is never a reason.
8. Close
Results JSON is local-only evidence under ignored results/ directories. Record in the ledger or
pull request: the decision and the evidence that met the stop rule, result stamps, models, cost,
review coverage, retained failures, and limits. Keep dated evidence in research/audits/ or
eval/qa/reviewed/ only when a current artifact needs it
(retention). Then check:
- Gate verdict recorded;
gates.jsonre-baselined in-commit if the movement was legitimate. - Verified findings filed or explicitly deferred; own-repo fixes are
.agents/TODO.mdentries. - After case edits,
npm run eval:qa:lint -- --since <ref> --staleis green andnpm run eval:qa:registerre-stamped the clusters. - A requested independent review finished and every finding was reconciled.
Hard rules
- One headline (QA end to end) and three routing gates; everything else is diagnostic. Lanes never merge, denominators never change quietly, and no view becomes a gate without a recorded decision.
- When two instruments disagree, the one closer to the headline wins.
- No per-question tuning. No schemas or judge code copied from prior-art repos.
- Freshness-sensitive truth is graded as behavior, never as pinned values in a gate.
- Never print or commit secrets. Paid Lumenloop research stays off; eval runs use free operations.
- Sandbox and network posture is never relaxed for eval convenience.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

