eval-agent
Skill1 starsChanged 5 months agoMITView on GitHub
Regression-test an LLM or agent workflow against a fixtures set — run each case, compare output to the expected/baseline, and flag drift, refusals, tool-call errors, and cost/latency changes. The test harness for prompt and agent changes where a normal unit test can't assert exact strings. Trigger phrases "eval this agent", "test the prompt", "did my prompt change break anything".
In this repository16 other files it gives its agents.
- catchupSkill
- claude-mdSkill
- context-budgetSkill
- db-querySkill
- debug-fixSkill
- explainSkill
- fix-issueSkill
- incident-ruleSkill
- pr-reviewSkill
- refactorSkill
- secrets-checkSkill
- setup-projectSkill
- shipSkill
- tddSkill
- test-writerSkill
- trace-agentSkill
The file
We split this skill into the rules it gives an agent
4 rules. About 0 are common, 1 is distinctive, and 3 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈1
Distinctive · estimate
This author's own choices. Worth a look.
3
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 19
Find the harness: Look for an existing eval set: evals/, tests/evals/, a cases.jsonl / fixtures/, or a promptfoo / langsmith / pytest eval …
- Very specific: line 21
If none exists, STOP and offer to scaffold: a cases.jsonl (each line = input
- Very specific: line 25
Confirm provider + model: Read the code for the actual client (anthropic / openai / a LangChain wrapper) and the exact model id — do not …
- Distinctive: line 46
Offer to update the baseline: only if the new behavior is intended and Eugen confirms.
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
eval-agent
Trigger: "eval this agent" / "test the prompt" / "check the agent didn't regress"
Standing rules: ../_conventions.md.
An LLM change can't be judged by exact-string assertions — you judge it against cases. Run the eval set, compare to baseline, report drift honestly. Never claim "no regression" without having actually run the cases. If there's no eval set yet, say so and offer to scaffold one — don't fake a pass.
Steps
- Find the harness. Look for an existing eval set:
evals/,tests/evals/, acases.jsonl/fixtures/, or a promptfoo /langsmith/ pytest eval file. If none exists, STOP and offer to scaffold: acases.jsonl(each line = input- expected-behavior + optional gold output) plus a runner. Don't invent results from a set that doesn't exist.
- Confirm provider + model. Read the code for the actual client
(
anthropic/openai/ a LangChain wrapper) and the exact model id — do not assume. If it's Anthropic/Claude, consult theclaude-apireference for the current model id and pricing before reasoning about cost. - Run the cases against the CURRENT prompt/agent. Capture per case: output, tool calls made, tokens in/out, latency, any refusal or error. Keys come from env vars only — if a key is missing, STOP and name the env var to set. Never hardcode.
- Compare to baseline. Diff each case against the stored expected behavior:
- Behavioral pass/fail (did it do the right thing?), not string-equality.
- New refusals or truncations that weren't there before.
- Tool-call changes (wrong tool, missing call, malformed args).
- Cost / latency delta vs baseline (flag if materially up).
- Report (terse table): N cases · X pass / Y fail · new refusals · tool
errors · cost delta. List each FAILING case as
input → expected → actual. End with the single most important regression to fix. If everything passed, say so plainly — and only because you ran it. - Offer to update the baseline only if the new behavior is intended and Eugen confirms. Never silently overwrite the gold set.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

