ai-eval-harness
Skill0 starsChanged 4 months agoMITView on GitHub
Designs and runs evaluation harnesses for AI features and for agent skills themselves. Use when shipping LLM features or measuring skill quality. Emits EVAL_PLAN / EVAL_REPORT. Never claims production quality from a handful of vibes-only prompts.
In this repository27 other files it gives its agents.
- adr-enforcerSkill
- api-contract-guardianSkill
- change-impact-analyzerSkill
- code-reviewerSkill
- context-loaderSkill
- defect-analystSkill
- delivery-plannerSkill
- engineering-mentorSkill
- engineering-osSkill
- feature-implementerSkill
- guardrails-configuratorSkill
- incident-commanderSkill
- loop-engineerSkill
- migration-surgeonSkill
- model-routerSkill
- observability-designerSkill
- performance-auditorSkill
- postmortem-writerSkill
- pr-generatorSkill
- quality-gateSkill
- refactoring-advisorSkill
- requirements-grillSkill
- security-auditorSkill
- spec-validatorSkill
- tdd-executorSkill
- test-strategy-designerSkill
- threat-modelerSkill
The file
We split this skill into the rules it gives an agent
4 rules. About 0 are common, 3 are distinctive, and 1 is very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈3
Distinctive · estimate
This author's own choices. Worth a look.
1
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 55
Run if possible → EVAL_REPORT with per-case results.
- Distinctive: line 96
Never invent pass rates.
- Distinctive: line 97
Never rely solely on a single cherry-picked demo prompt.
- Distinctive: line 98
Never log secrets from model outputs.
≈ “Common” is an estimate: at least 1 in 5 of the 350,495 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Purpose
Define datasets, metrics, graders, and regression gates for AI systems (product features or this skill portfolio).
When to Use / When NOT to Use
Use when: LLM/agent features; prompt/skill changes; “are we better?”; offline evals; skill portfolio regression.
Do not use when: non-AI CRUD with no model calls; pure infra without generative behavior.
Preconditions
Clear target behavior and some way to run the system/skill under test (or design-only mode if runtime unavailable).
Inputs / Outputs
Inputs: feature/skill under test, CONTEXT_PACK, success criteria, existing evals if any.
Outputs: EVAL_PLAN then EVAL_REPORT when executed.
Upstream / Downstream
Upstream: spec-validator, test-strategy-designer, engineering-os, skill authors.
Downstream: feature-implementer, quality-gate, pr-generator, delivery-planner.
Core Principles
- Evals need fixed cases + graded outcomes — not vibes alone.
- Split: unit-ish (deterministic checks), scenario, adversarial/safety, human review sample.
- Measure regressions; version datasets.
- Cost/latency are metrics too.
- For skills: trigger accuracy, artifact validity, stop-condition obedience.
- Report confidence intervals qualitatively when N is small.
- Never fabricate scores.
Process
Modes
product— AI feature in the appskill— evaluate an Agent Skill / portfolio path
- Define behaviors and failure modes to catch.
- Build/extend dataset cases (inputs, expected traits, forbidden traits).
- Choose graders: exact, rubric LLM-as-judge (with caution), programmatic assertions.
- Set pass thresholds and CI gating vs offline-only.
- Run if possible → EVAL_REPORT with per-case results.
- Recommend next dataset expansions.
Evidence Requirements
Case IDs, commands, raw graded outputs summarized (redact PII). N stated.
Stop Conditions / Failure Modes
| Condition | Action |
| No cases defined | Block “quality” claims |
| Runtime unavailable | Ship EVAL_PLAN only; Decision ProceedWithConditions |
| Safety cases failing | Block ship for product mode |
Severity + Confidence
Safety/eval failures on HighRisk AI → Critical/High for release.
Output Contract
## EVAL_PLAN
Mode: product | skill
Behaviors: ...
Dataset: ...
Graders: ...
Thresholds: ...
## EVAL_REPORT (if run)
N: ...
Results summary: ...
Failures: ...
Decision: Proceed | ProceedWithConditions | Revise | BlockHandoffs
quality-gate, feature-implementer, test-strategy-designer, security-auditor (prompt injection / data leak).
Never
- Never invent pass rates.
- Never rely solely on a single cherry-picked demo prompt.
- Never log secrets from model outputs.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.