eval-brief
Skill3 starsChanged 18 days agoMITView on GitHub
Design evals as the definition of done for an AI surface. Use when the user says eval, golden set, quality bar, LLM-as-judge, acceptance criteria for an agent, or "how do we know this is good enough to ship.
In this repository24 other files it gives its agents.
- AGENTS.mdAGENTS.md
- docs/agents.mdAGENTS.md
- company-product-signalsSkill
- company-researcherSkill
- company-strategySkill
- pm-os-onboardingSkill
- brainstormingSkill
- continual-learningSkill
- decision-loggerSkill
- exec-communicatorSkill
- experiment-designerSkill
- experiment-writeupSkill
- knowledge-updaterSkill
- launch-postSkill
- launch-readinessSkill
- meeting-to-actionsSkill
- one-pagerSkill
- pptx-creatorSkill
- prd-writerSkill
- stakeholder-updateSkill
- strategy-connectorSkill
- what-ifSkill
- working-backwardsSkill
- writing-clearlySkill
The file
We split this skill into the rules it gives an agent
7 rules. About 0 are common, 7 are distinctive, and 0 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈7
Distinctive · estimate
This author's own choices. Worth a look.
0
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 15
After a production miss (add the miss to the set)
- Distinctive: line 25
Default set for most copilots:
- Distinctive: line 34
Add latency/cost if the job is interactive or loops.
- Distinctive: line 55
If PM and eng disagree on labels, the eval is the product conversation.
- Distinctive: line 65
Do not wait for a platform.
- Distinctive: line 67
When production fails, add a case the same day.
- Distinctive: line 68
Never use "helpfulness" as the only dimension.
≈ “Common” is an estimate: at least 1 in 5 of the 350,495 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Eval Brief
Turn "the AI should be good" into a testable quality bar. For AI surfaces, evals are acceptance criteria — not a later QA step.
When to use
- New AI feature or agent job
- Prompt, model, tool, or retrieval change
- "What does done mean?" on an LLM surface
- After a production miss (add the miss to the set)
Process
1. Name the jobs
List end-to-end jobs, not model capabilities. One row per job: user, done looks like, forbidden.
2. Pick 3–5 dimensions
Default set for most copilots:
| Dimension | Why |
| Faithfulness | Did it invent facts? |
| Boundary | Did it take a forbidden action? |
| Usefulness | Did it complete the job? |
| Failure honesty | Did it admit unknowns? |
Add latency/cost if the job is interactive or loops.
3. Build a small set from reality
Start with ~20 cases:
- Golden (ideal)
- Edge (missing evidence, conflict)
- Adversarial ("just make up a number")
- At least one real production miss, as soon as you have one
A pass is evidence about known failures. It is not a claim of general quality.
4. Choose judges you can afford
| Judge | Use for |
| Heuristic / regex / tool-trace | Boundaries, schema, forbidden tools |
| LLM-as-judge | Faithfulness, tone — calibrate on human labels |
| Human | Disagreements and new failure modes |
If PM and eng disagree on labels, the eval is the product conversation. Resolve it before scaling the judge.
5. Write the brief
Save to knowledge/products/<product>/evals.md. If the file does not exist, copy knowledge/products/_template/evals.md.
Also log a decision if this eval gates a ship or blocks autonomy.
Rules
- Do not wait for a platform. A spreadsheet is a valid v1.
- Re-run on every prompt/model/tool/context change.
- When production fails, add a case the same day. That is the improvement loop.
- Never use "helpfulness" as the only dimension.
- Claims in launches and sales must match the last eval run (date + conditions).
Output
A filled evals.md plus, if asked, a 10-case starter table the user can label this week.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.