legal-ai-engineering-lab / evals
AGENTS.md2 starsChanged 3 months agoMITView on GitHub
These rules apply when writing or changing labelled cases, expected answers, or recorded results. AGENTS.md at the repository root governs everything else. - The expected answer must come from a source independent of the code being measured — a careful reading of the document, a worked example, the rule in the practice direction. An expectation produced by running the code and deciding the output looked reasonable can never disagree with the code. It passes by construction, and it is worse…
Where files are headingNew AGENTS.md files have more rules about keeping secrets out of the code, and fewer about code style.
Of the AGENTS.md files we read, the 33,315 first seen in the last six months, against the older ones: the kind of rule that gained most share, and the kind that lost most. About files like this in general, not this one, and an estimate, since a kind is a pattern in the words.
In this repository21 other files it gives its agents.
- .agents/AGENTS.mdAGENTS.md
- AGENTS.mdAGENTS.md
- .claude/rules/agents.mdAGENTS.md
- tests/AGENTS.mdAGENTS.md
- CLAUDE.mdCLAUDE.md
- architecture-reviewSkill
- code-reviewSkill
- evaluation-reviewSkill
- explainSkill
- interview-reviewSkill
- learning-reviewSkill
- security-reviewSkill
- start-projectSkill
- architecture-reviewSkill
- code-reviewSkill
- evaluation-reviewSkill
- explainSkill
- interview-reviewSkill
- learning-reviewSkill
- security-reviewSkill
- start-projectSkill
The file
We split this AGENTS.md into the rules it gives an agent
1 rule. About 0 are common, 1 is distinctive, and 0 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈1
Distinctive · estimate
This author's own choices. Worth a look.
0
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 13
Write the label before running anything: If output has already been seen, say so in the record rather than pretending otherwise.
≈ “Common” is an estimate: at least 1 in 5 of the 42,283 AGENTS.md files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
<!-- claude-rule-paths: evals/**, experiments/*/cases.json, experiments/*/results/** -->Working in evals/ and on evaluation cases
These rules apply when writing or changing labelled cases, expected answers, or recorded
results. AGENTS.md at the repository root governs everything else.
- The expected answer must come from a source independent of the code being measured — a careful reading of the document, a worked example, the rule in the practice direction. An expectation produced by running the code and deciding the output looked reasonable can never disagree with the code. It passes by construction, and it is worse than no case at all because it is mistaken for evidence.
- Write the label before running anything. If output has already been seen, say so in the record rather than pretending otherwise; a label written afterwards is a different kind of evidence and should not be presented as the same kind.
- A case set of only straightforward cases reports a flattering number that predicts nothing. Include at least one case where the correct answer is "this document does not say", one near-miss that should not be extracted, and one genuinely ambiguous to a careful reader.
- Choose the metric from what the wrong answer costs, not by default. Record which failure was chosen to tolerate and why — that sentence is most of what makes a result defensible to a lawyer.
- A set you wrote measures behaviour on structures you chose, not how often they occur. Prevalence is a fact about the real documents and cannot be recovered from invented ones, so a figure from a synthetic set holds "on cases of this kind" and never "in production".
- Never round a result upward, infer a number that was not measured, or describe an unmeasured behaviour as validated. Record what the result does not establish.
- A recorded model response kept as a fixture is stale, not circular. It measures one run, on one date, with one prompt and one model version. Re-record before making any claim about current behaviour.
evaluation-review carries the full method; evals/README.md carries the conventions.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

