eval-foundations
Skill4 starsChanged 37 days agoMITView on GitHub
The principles behind every LLM judgment and eval, for designing or revising one.
In this repository53 other files it gives its agents.
- llms-full.txtllms.txt
- llms.txtllms.txt
- agent-sendSkill
- agent-stocktakeSkill
- author-calibrated-evalSkill
- authorship-strategySkill
- collect-contextSkill
- context-syncSkill
- decision-historySkill
- detail-loopSkill
- generation-auditSkill
- harness-boundarySkill
- harness-stocktakeSkill
- harness-syncSkill
- headline-craftSkill
- hf-syncSkill
- hook-stocktakeSkill
- implementation-chainSkill
- jev-judgment-designSkill
- jsonld-knowledge-graphSkill
- learn-evalSkill
- llm-as-judgeSkill
- llms-txt-writerSkill
- mono-figureSkill
- persona-read-loopSkill
- plugin-directory-submitSkill
- prompt-perturbSkill
- prose-translationSkill
- public-commentSkill
- readme-writerSkill
- release-doiSkill
- repair-disciplineSkill
- repo-asset-stocktakeSkill
- review-to-lintSkill
- rfc-writerSkill
- rules-distillSkill
- rules-stocktakeSkill
- search-firstSkill
- session-judgment-miningSkill
- skill-complySkill
- skill-healthSkill
- skill-stocktakeSkill
- skill-writerSkill
- social-postSkill
- spawn-sessionSkill
- task-stocktakeSkill
- task-triageSkill
- verbal-dispute-checkSkill
- verify-bootstrapSkill
- widen-the-askSkill
- wiki-harvestSkill
- wiki-querySkill
- x-draftSkill
The file
We split this skill into the rules it gives an agent
2 rules. About 0 are common, 2 are distinctive, and 0 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈2
Distinctive · estimate
This author's own choices. Worth a look.
0
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 12
Read this skill when you design a judgment or revise one: a judge, an eval, a review gate, a reader loop.
- Distinctive: line 43
Fixing the procedure is itself a countermeasure to wobble: Run every evaluation on the fixed procedure, rather than having the model …
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Eval Foundations — Principles Behind Every Judgment
Read this skill when you design a judgment or revise one: a judge, an eval, a review gate, a reader loop. A run of an existing judgment needs only the skill that runs it; that skill holds everything the run requires.
The principles
- Reach the verdict by holistic judgment, not by summing rubric points. A measure that becomes the target breaks down (Goodhart). Holistic judgment is what to aim for.
- Treat noise and bias separately. Repeating the same judge reduces only the wobble of the moment (occasion noise). The judge's stable habits, and a shift that lands on every judge in the same direction, survive repetition.
- Mix judges by lineage rather than by number (for example, Claude and Codex). Keep a disagreement between lineages as a split instead of averaging it. Habits that all lineages share still survive the mix; only outside feedback shows them.
- A judgment that guards the floor may be a panel; a judgment that picks the ceiling stays with one eye. Guarding the floor means finding gaps or a dominant No. Picking the ceiling means choosing which one is best.
- The overall judgment and the per-criterion evaluation bleed into each other. Decide in advance which one comes first and what each judge is shown.
- Keep the loop that fixes the work apart from the judgment that evaluates it. A judgment inside the loop is learned by the loop and pushed on, so place a judgment that does not see the loop's results outside the loop.
- Use a wobble check only to learn how large the wobble is, and keep the verdict of the first run. Attach the items that wobbled widely to the verdict.
- Fixing the procedure is itself a countermeasure to wobble. Run every evaluation on the fixed procedure, rather than having the model reassemble the flow each time.
- Leave details that go stale when the model changes to the model at hand, rather than writing them into the skill — counts, prompt wording, specific model names.
The judge's context
The judge does not hold the writer's context. Judge within the writer's context only when acceptance rests with another eye: a judge in a separate context, or the author.
Who owns what
| What | Owner |
| The principles and the judge's-context rule above (why, and what to do in general) | this skill |
| The form of a judge: binary checks as evidence, one named verdict, no aggregation, the dominant No, refutation questions, code-checkable facts first, the four stages | skill llm-as-judge |
| A skill's own output and sequence: what it produces, in which order, who receives what | that skill |
| A reason measured in one domain (for example, how README readers behave across rounds) | that skill's MAINTENANCE.md, or the reference its judge reads every run |
A skill that runs a judge points to skill llm-as-judge, which points here. A skill with no judge
of its own (a reader loop, a loop design review, the author's own reading) points here directly.
Point to one owner per rule, by skill name, and keep only the instruction for its own run.
What a skill fixes and what it leaves to the model
Fix the structure: the stages, who is shown what, the verdict vocabulary, the output shape. Write how to choose a count or a model (for example, by how deeply a reader role reads) rather than the count or the model itself. This holds for a skill's body, its presets, its references and an agent's body.
Two things are written down as they are. A fixed prompt that a script hands to a reader, so answers
can be compared across rounds, is part of the instrument (principle 8); keep it in a preset or
reference, outside the skill's body. A script's default value and an agent's model: frontmatter
are wiring.
Beside this file
provisional.md— the decisions held provisionally, each with what would replace it and where to change it.MAINTENANCE.md, when present — the open questions this skill leaves undecided, and where the principles came from.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

