agent-eval
Skill0 starsChanged 8 months agoMITView on GitHub
Reproducible evaluation harness for coding agents, prompts, and skills — head-to-head comparison with pass rate, cost, time, and consistency metrics captured in git worktrees. Use when comparing coding agents (Claude Code, Aider, Codex), regression-testing your own skills after changes, measuring variance across repeated runs of the same prompt, validating a model upgrade before adopting it, or A/B testing two versions of a prompt. Trigger on "compare Claude Code vs aider", "is my skill still passing", "run regression evals on this skill", "how much variance does this prompt have", "did the model upgrade regress anything".
36 other projects use a version of this, 18 of them edited.
Adopted by
Most recently taken up by
- AndresArreguy/ecc-pi-portedited, first seen Nov 16, 2026
- Rojios/everything-claude-codeedited, first seen Nov 8, 2026
- ahmadvahidi/skill-atlasedited, first seen Nov 8, 2026
- danielrosehill/AI-Evaltheir own version, first seen Nov 8, 2026
- timurgaleev/vibestacktheir own version, first seen Nov 8, 2026
- pankaj-mahaur/Agent-Daemonedited, first seen Nov 6, 2026
Used with
Projects with it also use backend-patterns and api-design.
Other files people use alongside this one.
Files found in the same public projects as a version of this one, more often than they turn up anywhere else. Each project counts once, large collections count for less, and this file's own repository is left out.
HABUBUSSSkillbackend-patterns
Backend architecture patterns, API design, database optimization, and server-side best practices for Node.js, Express, and Next.js API routes.
Also in 3 projects that use this
MicrosoftSkillapi-design
Guide for designing and documenting RESTful APIs. Use when asked to design an API, create endpoints, or document an API.
Also in 3 projects that use this
Other versions36 projects use a version of this.
Word for word · 0
None found.
Edited · 18
- loulanyue/awesome-claude-notes★ 27228% same
- Fmarzochi/EGC★ 6126% same
- gongyijie85/dsh-ecc★ 827% same
- ghvinashvili/specagent★ 725% same
- AndresArreguy/ecc-pi-port★ 127% same
- get-proofpilot/claude-skills-catalog★ 128% same
- NISTALTALSON/essential-claude-skills★ 127% same
- Rojios/everything-claude-code★ 128% same
- SamuelNDCE/claude-super-skill-library★ 128% same
- TironFox/everything-claude-code★ 128% same
8 more not listed
Wrote their own version · 18
- colbymchenry/codegraph★ 77ksame name
- GoogleCloudPlatform/professional-services★ 3.1ksame name
- alejandro-ao/skills★ 13same name
- timurgaleev/vibestack★ 8same name
- I-Onlabs/claude-code-skills★ 4same name
- OpenxIntelligence/Coding-Agent-Skills★ 4same name
- WOWCharlotte/harness-skills★ 4same name
- danielrosehill/AI-Eval★ 3same name
- lagrahhn/codeGraph-branch★ 3same name
- ReStartAllKill/skills★ 2same name
8 more not listed
How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.
In this repository2 other files it gives its agents.
- CLAUDE.mdCLAUDE.md
- review-codeSkill
The file
We split this skill into the rules it gives an agent
6 rules. 0 are common, 6 are distinctive, and 0 are very specific to this project.
0
Common
Most of the 32 edited versions we read keep these.
6
Distinctive
This author's own choices. Worth a look.
0
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 23
If what you want is a one-off "does this code work", use the normal test runner.
- Distinctive: line 35
If agent-eval is not available, the skill's workflow is still useful as a hand-run protocol — the structured task definitions and judge …
- Distinctive: line 41
Every eval is a YAML file naming the task, the repo, the files under test, the prompt to hand to the agent, and the judge criteria.
- Distinctive: line 101
Always run at least 3 trials.
- Distinctive: line 137
Use these as a *secondary* signal — to catch cases where the agent passed tests by hardcoding or faking.
- Distinctive: line 148
Use only when deterministic judges aren't available — e.g.
“Common” is measured: kept by at least 6 in 10 of the 32 edited versions of this file we could read. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Agent Eval
Version: 1.0.0
A disciplined way to answer "is this agent/skill/prompt actually good?" with numbers instead of vibes. Runs tasks in isolated git worktrees, collects pass rate / cost / time / consistency, and produces a comparison report.
What this is for
The original use case is head-to-head agent comparison — Claude Code vs Aider vs Codex on your own codebase. But the same harness covers four situations that aren't about picking a tool:
| Situation | Why the harness helps |
| Regression-test your skills | You edited a skill and want to prove it still passes on the cases it used to. Run the skill's evals/evals.json under agent-eval with N=3 and compare against the previous version. |
| Variance analysis | A prompt works great sometimes and fails other times. Run it 10 times, see the consistency score, decide if the variance is acceptable or if the prompt needs tightening. |
| Validate a model upgrade | A new model just dropped (Claude 4.6, Sonnet 4.5, etc.). Re-run your regression suite against old and new, confirm nothing regressed before switching defaults. |
| A/B test prompts | You rewrote a system prompt. Run both variants against the same task set and compare pass rate and consistency to decide which ships. |
If what you want is a one-off "does this code work", use the normal test runner. Agent-eval is for measuring agents, prompts, and skills as systems — the kind of work where a single run is uninformative because the system is non-deterministic.
Installation
Agent-eval is an external CLI. Install it from source after reviewing the code — it executes agent processes against your files, so know what you're running.
# Clone and install (or whatever the install method is for the fork you trust)
git clone https://github.com/joaquinhuigomez/agent-eval ~/dev/personal/agent-eval
cd ~/dev/personal/agent-eval && make installIf agent-eval is not available, the skill's workflow is still useful as a hand-run protocol — the structured task definitions and judge criteria work as a mental framework even without the automation.
Core concepts
Task definitions (YAML)
Every eval is a YAML file naming the task, the repo, the files under test, the prompt to hand to the agent, and the judge criteria.
name: add-retry-logic
description: Add exponential backoff retry to the HTTP client
repo: ./my-project
commit: "abc1234" # pin to a specific commit for reproducibility
files:
- src/http_client.py
prompt: |
Add retry logic with exponential backoff to all HTTP requests.
Max 3 retries. Initial delay 1s, max delay 30s.
judge:
- type: pytest
command: pytest tests/test_http_client.py -v
- type: grep
pattern: "exponential|backoff|retry"
files: src/http_client.pyPinning the commit is the non-negotiable part. If you re-run this task next month and the main branch has moved on, you're measuring a different problem.
Git worktree isolation
Each run spawns a fresh git worktree from the pinned commit. No Docker, no container setup — just native git isolation. The agent can't corrupt your working copy and can't see other agents' output from the same session. When the run finishes, the worktree is either kept for inspection or cleaned up.
The four metrics
| Metric | What it measures | When it's the decisive one |
| Pass rate | Did the agent produce code that satisfies the judge? | Always — this is the ground truth. |
| Cost | API spend per task (when the agent reports it) | Cheaper agent at 90% pass rate may beat a premium agent at 95%. |
| Time | Wall-clock seconds from prompt to judged result | Interactive coding workflows — slow agents break flow even when correct. |
| Consistency | Pass rate across N repeated runs of the same task | The most underrated metric. A 3/3 pass beats a 2/3 every time; non-determinism is a bug. |
Workflow
1. Write tasks
mkdir tasks
# One YAML per task. Aim for 3-5 tasks that represent your real workload.
# Toy examples measure toy performance.Tasks should be at the granularity you actually delegate to an agent — a feature, a refactor, a bug fix — not "rename a variable" or "write Fibonacci". Include at least one task per type of work you care about (new code, refactor, bug, test-writing).
2. Run
# Single task, single agent, 3 trials
agent-eval run --task tasks/add-retry.yaml --agent claude-code --runs 3
# Head-to-head
agent-eval run --task tasks/add-retry.yaml \
--agent claude-code \
--agent aider \
--runs 3Always run at least 3 trials. Agents are non-deterministic — a single pass or fail tells you nothing about consistency.
3. Report
agent-eval report --format tableTask: add-retry-logic (3 runs each)
┌──────────────┬───────────┬────────┬────────┬─────────────┐
│ Agent │ Pass Rate │ Cost │ Time │ Consistency │
├──────────────┼───────────┼────────┼────────┼─────────────┤
│ claude-code │ 3/3 │ $0.12 │ 45s │ 100% │
│ aider │ 2/3 │ $0.08 │ 38s │ 67% │
└──────────────┴───────────┴────────┴────────┴─────────────┘Pass rate is the first column but not the only one. A 95% agent at 10x the cost is often the wrong answer.
Judge types
Code-based (deterministic)
The most trustworthy judges are tests and builds. They don't lie and they don't add variance.
judge:
- type: pytest
command: pytest tests/ -v
- type: command
command: npm run buildPattern-based
Use these as a secondary signal — to catch cases where the agent passed tests by hardcoding or faking.
judge:
- type: grep
pattern: "class.*Retry|exponential_backoff"
files: src/**/*.pyModel-based (LLM-as-judge)
Use only when deterministic judges aren't available — e.g. judging explanation quality or documentation clarity. LLM judges add noise; never rely on them alone.
judge:
- type: llm
prompt: |
Does this implementation correctly handle exponential backoff?
Check for: max retries, increasing delays, jitter.Rule of thumb: every task should have at least one deterministic judge, even if it also has an LLM judge. If you can't write a deterministic judge for a task, the task is probably too fuzzy to evaluate.
Evaluating your own skills
Claude skills ship with evals/evals.json (reasoning cases) and evals/trigger-evals.json (activation discrimination). Agent-eval can run the reasoning cases through the skill under test and measure pass rate across N trials.
Pattern for regression-testing a skill you just edited:
name: skill-regression-prompt-v2
description: Verify /prompt still passes its own reasoning evals after iteration 2 changes
repo: ~/.claude
files:
- skills/prompt/SKILL.md
prompt: |
Load the skill at ~/.claude/skills/prompt/SKILL.md.
Run each case in ~/.claude/skills/prompt/evals/evals.json.
For each case, invoke the skill with the prompt and judge whether the
response matches the expected_output description.
Return a pass/fail per case and a summary.
judge:
- type: llm
prompt: |
Check the skill achieved >= 8/10 pass on its reasoning evals.
Any regression from the previous version is a failure.The previous result is the baseline. If pass rate drops or consistency drops, the edit regressed something.
Best practices
- 3–5 tasks minimum drawn from real workload, not synthetic examples. Toy tasks measure toy performance.
- Pin commits in every task YAML. Otherwise "reproducible" is a lie.
- At least one deterministic judge per task. LLM-only judges are too noisy to gate decisions on.
- Run at least N=3 trials to capture variance. N=1 tells you nothing about consistency — the most important long-term metric.
- Track cost alongside pass rate. A correct-but-expensive agent may still be the wrong choice for your budget.
- Version your task YAML. They are test fixtures; commit them to the repo they test.
- Re-run after every model/tool upgrade. Model drift is real; baselines go stale in weeks, not months.
When NOT to use agent-eval
- Single deterministic tests — use the normal test runner. Agent-eval's overhead isn't worth it for code you wrote by hand.
- One-off comparison that doesn't need to be reproducible — just try both tools informally and decide.
- Tasks that can't be judged without a human — skill at explanation, code review subtleties, UX judgment. Don't force an LLM judge where a human is required.
Related
- Skill evaluation bar: every shipped skill under
~/.claude/skills/hasevals/evals.jsonandevals/trigger-evals.json. Agent-eval operationalizes those. skill-stocktake: periodic quality audit of skills (uses subagent judgment). Agent-eval is the quantitative counterpart.- Upstream reference: github.com/joaquinhuigomez/agent-eval
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

