agent-eval-harness
Skill4 starsChanged 4 months agoMITView on GitHub
Builds and runs a mini eval for any skill or agent task — 3–5 test prompts, an expected-behavior rubric, a pass threshold set before running, then a scored verdict table. Use when the user says "eval this skill", "does this actually work", "test the agent on this", "build an eval", or before trusting any new skill, prompt change, or model swap. Don't use for choosing which model tier to run once the eval exists — that's model-cost-optimizer.
1 other project uses a version of this.
Adopted by
Most recently taken up by
- OneWave-AI/claude-skillstheir own version, first seen Sep 21, 2026
Other versions1 project uses a version of this.
Word for word · 0
None found.
Edited · 0
None found.
Wrote their own version · 1
- OneWave-AI/claude-skills★ 378same name
How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.
In this repository74 other files it gives its agents.
- context-budget-managerSkill
- incident-rollbackSkill
- memory-hygieneSkill
- model-cost-optimizerSkill
- prompt-debuggerSkill
- skill-author-coachSkill
- third-party-skill-vetterSkill
- api-endpoint-testerSkill
- bug-triage-protocolSkill
- changelog-release-notesSkill
- docs-readme-writerSkill
- env-secrets-hygieneSkill
- github-pr-reviewSkill
- git-ship-flowSkill
- lovable-brief-writerSkill
- supabase-migration-writerSkill
- vercel-deploy-checkSkill
- cold-outreach-sequencerSkill
- competitor-pricing-scanSkill
- contract-redline-scanSkill
- customer-support-inboxSkill
- invoice-runnerSkill
- kpi-dashboard-weeklySkill
- lead-research-dossierSkill
- payment-reconciliationSkill
- pipeline-trackerSkill
- proposal-builderSkill
- stripe-payment-linkSkill
- testimonial-harvesterSkill
- weekly-revenue-reportSkill
- brand-voice-guardianSkill
- case-study-builderSkill
- content-repurposerSkill
- deep-research-briefSkill
- email-nurture-sequenceSkill
- landing-page-copywriterSkill
- linkedin-content-packSkill
- newsletter-assemblerSkill
- seo-blog-writerSkill
- social-listening-reportSkill
- x-thread-builderSkill
- youtube-script-writerSkill
- budget-guardrailsSkill
- ceo-operating-doctrineSkill
- delegation-protocolSkill
- editor-personaSkill
- heartbeat-standupSkill
- hiring-briefSkill
- ops-agent-personaSkill
- researcher-personaSkill
- sales-agent-personaSkill
- support-agent-personaSkill
- weekly-retroSkill
- writer-personaSkill
- calendar-conciergeSkill
- decision-journalSkill
- email-triageSkill
- file-organizerSkill
- habit-trackerSkill
- inbox-zero-sprintSkill
- The first 60 shown.
The file
We split this skill into the rules it gives an agent
1 rule. About 0 are common, 1 is distinctive, and 0 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈1
Distinctive · estimate
This author's own choices. Worth a look.
0
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 20
Before and after any prompt, skill, or model change — the eval is the before/after instrument.
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Agent Eval Harness
"It seemed to work when I tried it once" is not evidence. This skill builds the smallest eval that still means something: 3–5 realistic test prompts, a rubric written before any run, a pass threshold you commit to in advance, and a verdict table you can re-run after every change. This is the course's core discipline — nothing about an agent is "good" until it passes YOUR eval.
When to Use
- A new skill just shipped (including from
skill-author-coach) and needs proof it works. - Before and after any prompt, skill, or model change — the eval is the before/after instrument.
- The user asks "is the agent good at X?" and the honest answer is "let's measure."
- A recurring job is being handed to a cheaper model or a Paperclip agent and needs an acceptance bar.
- Not for: picking the cheapest passing model (
model-cost-optimizer— it consumes this harness), or diagnosing WHY a failing output fails (prompt-debugger).
Quick Reference
| Action | Command / Call |
| Create eval file | Write evals/<task-name>.md (cases + rubric + threshold) |
| Run one case | Give the agent the test prompt verbatim in a FRESH session |
| Score a case | Judge output against rubric criteria: 2 = meets, 1 = partial, 0 = miss |
| Compute result | score = points earned / points possible per case; case passes at ≥80% |
| Verdict | PASS if passing cases ≥ threshold set in step 3, else FAIL |
| Re-run | Same eval file, verbatim, after every skill/prompt/model change |
Procedure
- Define the job in one sentence — "Given [input], the agent should produce [output] that [quality bar]." If you can't write this sentence, you're not ready to eval — go clarify the job first.
- Write 3–5 test prompts — realistic, varied, and fixed before any run: at least one easy/typical case, one hard case (messy input, ambiguity), and one boundary case (input the skill should REFUSE or route elsewhere — the Don't-use-for clause, tested). Write them exactly as a user would phrase them, not as ideal specs.
- Write the rubric and threshold BEFORE running — per case, list 3–5 checkable criteria (present/absent facts, format matches template, correct refusal, no fabrication). Score each criterion 0/1/2. Commit the pass threshold now — default: 4 of 5 cases pass (or 3 of 3–4), each case needing ≥80% of rubric points. Writing the rubric after seeing outputs is grading your own homework.
- Run each case in a fresh session — one prompt per clean context so cases can't contaminate each other. Capture the full output verbatim into the eval file. Same model, same skill version, same settings across all cases.
- Score against the rubric only — go criterion by criterion, cite the line of output that earns each point. Charm, length, and confidence earn nothing that isn't in the rubric. Note every failure's symptom in one line — this feeds
prompt-debugger. - Render the verdict table — fill the template below. Verdict is mechanical: passing cases vs. committed threshold. No "close enough" upgrades after the fact.
- File it for re-use — save prompts + rubric + results in
evals/<task-name>.md. This exact eval re-runs after every future change; that's the whole point. If the verdict is FAIL, hand the failing cases toprompt-debugger— do not ship.
Output Template
## Eval: <task/skill name> — <date>
Model: <model> | Skill version: <x.y.z> | Threshold: <n>/<total> cases
| # | Case (short) | Rubric points | Score | P/F | Failure symptom |
|---|---|---|---|---|---|
| 1 | typical: <...> | 8 | 7 (88%) | PASS | — |
| 2 | hard: <...> | 10 | 5 (50%) | FAIL | <one line> |
| 3 | boundary/refusal: <...> | 4 | 4 (100%) | PASS | — |
VERDICT: PASS | FAIL (<passing>/<total> vs threshold <n>)
Next action: <ship | prompt-debugger on cases X,Y | re-run after fix>
Eval file: evals/<task-name>.mdPitfalls
- Rubric written after the outputs — you unconsciously grade toward what the agent produced. Recovery: throw the scores out, rewrite the rubric blind against the job definition from step 1, re-score the captured outputs fresh.
- All-easy test set — 5/5 pass because every case is the happy path. Recovery: require the hard case and the refusal/boundary case in every eval; if the skill has never failed a case, the eval is too soft to be load-bearing.
- Contaminated sessions — running all cases in one conversation lets case 1's context prop up case 3's answer. Recovery: invalidate the run, re-run one case per fresh session.
- Moving the threshold post-hoc — "3/5 is probably fine" after committing to 4/5. Recovery: the threshold from step 3 stands; if it was genuinely wrong, change it in the eval file with a dated note, then re-run the FULL eval under the new threshold.
- Eval drift — the skill changes for weeks but the eval never re-runs. Recovery: re-run the saved eval after every skill/prompt/model change; a stale green result is a red result.
- Unscoreable criteria — rubric lines like "output is high quality" that two graders would score differently. Recovery: rewrite each criterion as a check a stranger could apply ("names all 3 stakeholders", "under 200 words", "declines and points to X"); if you can't make it checkable, it doesn't belong in the rubric.
Verification
- Rubric and threshold exist in the eval file with a timestamp BEFORE any recorded run
- 3–5 cases including at least one hard case and one boundary/refusal case
- Every score cites the output line that earned or lost the points
- Each case ran in its own fresh session under identical model/skill/settings
- Verdict follows mechanically from committed threshold — no post-hoc adjustment
- Every rubric criterion is checkable by a stranger — no "high quality" lines
- Eval file saved to
evals/<task-name>.mdand re-runnable by a fresh agent verbatim
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

