sorted / evals
CLAUDE.md0 starsChanged 45 days agoMITView on GitHub
How well the model reads photos, measured through the app's own parser and rules. docs/EVALS.md has the results and what they mean; this is how to change them without fooling yourself.
In this repository11 other files it gives its agents.
- AGENTS.mdAGENTS.md
- CLAUDE.mdCLAUDE.md
- content/CLAUDE.mdCLAUDE.md
- scripts/CLAUDE.mdCLAUDE.md
- seed/CLAUDE.mdCLAUDE.md
- static/CLAUDE.mdCLAUDE.md
- checkSkill
- evalSkill
- rebrandSkill
- resetSkill
- screenshotsSkill
The file
We split this CLAUDE.md into the rules it gives an agent
1 rule. About 0 are common, 0 are distinctive, and 1 is very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈0
Distinctive · estimate
This author's own choices. Worth a look.
1
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 19
A run is a record: Don't edit one; make a new one.
≈ “Common” is an estimate: at least 1 in 5 of the 26,980 CLAUDE.md files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
evals/
How well the model reads photos, measured through the app's own parser and rules. docs/EVALS.md has the results and what they mean; this is how to change them without fooling yourself.
| Path | What it is |
prompts/vN.txt | Every prompt version, exactly as the model saw it. The newest is the one in classifier.PROMPT |
splits/dev.txt, holdout.txt, real.txt | Which photos are in each set |
labels_real.json | Labels for the 20 real photos, with a link to each one's source. The photos aren't in the repository |
runs/<date>/vN.json | What the model replied to each photo, with the prompt's hash and each photo's fingerprint |
run.py | Runs the model (Apple silicon only). /eval wraps it |
score.py | Scores runs with no model, so any machine can re-score them |
Rules
- Tune on
dev. Look atholdoutonce per candidate you'd ship. A prompt tuned against the holdout set has nothing left to be measured on. - The real photos are a check, not a target. Generated photos are cleaner than real ones. Report the real set alongside, and don't tune on it.
- Never change a label to match the model. Labels change only when a person looks at the photo and finds the label wrong. That moves every score, so re-score and say so in
docs/EVALS.md. The edit hook blocks agents from changing labels. - A run is a record. Don't edit one; make a new one. Scores come from the raw replies, so a parser or rules change is re-scored on old runs for free: run
score.pyagain. - A prompt change needs a new run before it ships.
tests/test_units.pyfails unless the prompt inclassifier.pyis the newest file here and has a run on every published dev and holdout photo. - Decisions and hazards beat size. A size error changes no decision (decision 2), while a missed hazard sends a crew to paint tins. Weigh a change by "Decision" and "Hazards caught" first.
- The sets are small. On 15 real fly-tips, one photo is 7 points. Treat a difference of one or two photos as noise.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.