agentic-evals
Skill4 starsChanged 5 months agoMITView on GitHub
Design evaluation contracts and test plans for agentic systems. Create deterministic tests, trajectory evals, quality dimensions, gold-set criteria, and CI gates before or after implementation. Use when asked for tests first, an eval plan, success criteria, non-deterministic testing, LLM-as-judge setup, or retrofitting evals. Do NOT use for routine test failure debugging, simple unit-test additions with established patterns, or merely running tests.
In this repository3 other files it gives its agents.
The file
We split this skill into the rules it gives an agent
3 rules. About 0 are common, 3 are distinctive, and 0 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈3
Distinctive · estimate
This author's own choices. Worth a look.
0
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 14
Use this skill to define how an agentic feature will be judged before treating it as correct.
- Distinctive: line 29
Use trajectory evals when tool choice, ordering, or recovery path matters.
- Distinctive: line 30
Use LLM-as-judge only with a task-specific rubric and a small gold set.
≈ “Common” is an estimate: at least 1 in 5 of the 350,495 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Agentic Evals
Use this skill to define how an agentic feature will be judged before treating it as correct. Keep deterministic tests separate from probabilistic evals.
Workflow
- Identify the target:
SPEC.md, existing code, agent behavior, tool contract, or production workflow. - Split behavior into:
- Deterministic: pure logic, schemas, permissions, parsing, API contracts.
- Non-deterministic: planning, tool choice, summaries, ranking, judgment, multi-step recovery.
- For deterministic behavior, write Arrange-Act-Assert test cases and expected failure modes.
- For non-deterministic behavior, define evals across task completion, correctness, efficiency, and safety.
- Use trajectory evals when tool choice, ordering, or recovery path matters.
- Use LLM-as-judge only with a task-specific rubric and a small gold set; prefer pairwise comparison when comparing versions.
- Generate
EVAL_PLAN.mdfromassets/templates/EVAL_PLAN.md.
Gate Design
- Level 1: fast PR checks such as schemas, fixtures, tool contract tests.
- Level 2: deeper nightly or pre-release evals with repeated trials.
- Level 3: deployment gate using acceptance thresholds and manual review for high-impact changes.
References
Read ../agentic-engineering-sdlc/references/day4_security_and_evaluation.md
for deeper evaluation, judge calibration, and trajectory guidance.
Done Criteria
- Success criteria are measurable.
- Deterministic tests and non-deterministic evals are separated.
- Trial count, thresholds, and gold-set source are documented.
- CI gate ownership and failure response are clear.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.