Evaluation Frameworks
Skill0 starsChanged 4 months agoMITView on GitHub
Measure whether an LLM system actually works. Auto-load when building or improving evals, golden sets, or benchmarks for an LLM feature; when comparing prompts, models, or parameters; when someone claims an AI change "seems better"; when calibrating an LLM-as-judge; or before shipping any prompt/model change to an LLM system. An eval is to an LLM system what a test is to code. NOT for writing conventional code tests (validation-and-testing) and NOT for fixing malformed outputs (llm-integration-reliability).
In this repository25 other files it gives its agents.
- Agent EngineeringSkill
- API and Interface DesignSkill
- Architecture AnalysisSkill
- Build and ReleaseSkill
- Campaign PlanningSkill
- Change ControlSkill
- Codebase OnboardingSkill
- Code Review PlaybookSkill
- Configuration ManagementSkill
- Debugging PlaybookSkill
- Dependency ManagementSkill
- Documentation PracticesSkill
- Engineering MinimalismSkill
- Failure ArchaeologySkill
- LLM Integration ReliabilitySkill
- LLM System DesignSkill
- Observability and DiagnosticsSkill
- Performance EngineeringSkill
- Prompt and Context EngineeringSkill
- Proof and AnalysisSkill
- Refactoring PlaybookSkill
- Research MethodologySkill
- Security Review PlaybookSkill
- System DesignSkill
- Validation And TestingSkill
The file
We split this skill into the rules it gives an agent
2 rules. About 0 are common, 1 is distinctive, and 1 is very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈1
Distinctive · estimate
This author's own choices. Worth a look.
1
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 76
Report honestly: Every result carries: the N, the set version/date, the single variable changed, both scores, and an epistemic label if the …
- Distinctive: line 151
Every claimed improvement cites: baseline score, new score, the single variable changed, and the set version.
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Evaluation Frameworks
Purpose
LLM systems fail silently and improve anecdotally: a prompt tweak that fixes the case in front of you can break ten cases you did not look at. This skill is the measurement discipline — build the golden set before tuning, baseline before improving, change one variable at a time, and never trust a judge you have not calibrated.
Metadata
- Prerequisites:
llm-system-design(the output contract being evaluated should already be designed);validation-and-testing(the risk-first testing mindset this skill extends to LLM behavior). - Related Skills:
llm-integration-reliability(judge ground-truth injection mechanics),prompt-and-context-engineering(the thing most often being compared),research-methodology(evidence discipline and epistemic labels for reporting results). - Owns: evaluation methodology for LLM systems; golden sets; LLM-judge calibration; regression evals; sample-size honesty.
When to Use / When NOT to Use
Use when:
- Building or changing any prompt, model choice, temperature, or pipeline step in an LLM system.
- Someone (including you) says an AI change "seems better" — that sentence is this skill's trigger.
- Designing an LLM-as-judge or grader and deciding whether to trust its verdicts.
- Choosing between models or providers for a task.
Do NOT use (load the sibling instead):
- Testing deterministic code →
validation-and-testing. - Outputs are malformed/unparseable (a reliability defect, not a quality question) →
llm-integration-reliability. - Deciding what the LLM's job should be in the first place →
llm-system-design. - Exploratory prototyping where you have not committed to shipping → defer evals, but write down that you deferred them (see "when NOT to build eval infrastructure" below).
Definitions & Mental Model
- Golden set: a fixed collection of inputs with known-correct (or human-labeled) expected outputs, used to score the system.
- Baseline: the measured score of the current system before any change. No baseline → no claim of improvement.
- LLM-as-judge: using a model to score free-text outputs against a rubric. Powerful, but itself an LLM system that must be calibrated before trusted.
- Regression eval: the golden-set run wired into the change workflow so a prompt/model change cannot ship without a score.
- Calibration: measuring the judge's agreement with human labels on a sample before using its verdicts.
Mental model: an eval is to an LLM system what a test suite is to code — with one difference: code tests are binary and deterministic; evals are statistical. That means sample size matters (10 examples distinguish nothing), variance matters (run twice, compare the spread before crediting a 2% gain), and the honest unit of reporting is "X/N with these labels, on this dated set" — never "it's better now." Everything else in this skill is the transfer of validation-and-testing's risk-first discipline into statistical territory.
Core Methodology
- Define what "works" means, mechanically, before tuning anything. One sentence per behavior: "Given a support ticket, the system outputs the correct queue label." If you cannot state it, you cannot measure it — go back to
llm-system-designstep 2. - Build the golden set BEFORE improving the system. Composition rule of thumb:
- Real cases (the bulk): sampled from production/history, with correct outputs labeled by a human.
- Adversarial cases: inputs designed to fool the system (ambiguous, misleading, prompt-injection-shaped).
- Edge cases: empty, maximal-length, wrong-language, malformed — the same awkward-input discipline
validation-and-testingdemands of parsers. - Size: enough that a meaningful difference is visible above noise. Tens distinguish gross failures; low hundreds distinguish real improvements. State the N in every report; never imply significance a small N cannot carry.
- Freeze and date-stamp the set. Changing the set and the system simultaneously destroys comparability.
- Select the metric by output type (decision tree):
What shape is the system's output?
├─ Enum / classification (queue names, scores from a fixed set)
│ → exact-match accuracy per class; report the confusion pattern,
│ not just the total (one bad class hides in a good average).
├─ Structured output (JSON with fields)
│ → per-field assertions: pass/fail checks on required fields,
│ value ranges, cross-field consistency. Deterministic — no judge.
├─ Extraction / short factual answer
│ → normalized exact match or contains-match against labeled truth.
└─ Free text (summaries, explanations, reviews)
→ rubric-based LLM-as-judge — BUT:
├─ inject the ground truth into the judge's prompt (the evidence,
│ the source, the reference answer) so the judge scores against
│ supplied facts, never memory (mechanics:
│ llm-integration-reliability step 6)
├─ enum-constrain the verdict ({"verdict":"pass|partial|fail"})
└─ CALIBRATE before trusting (step 5)- Baseline first, then change ONE variable. Measure the current system on the golden set and record the score. Then change exactly one thing — the prompt, OR the model, OR the temperature, OR a pipeline step — and re-measure. Two changes at once means the result attributes to neither (the same one-variable law as
debugging-playbook). Keep a plain log: date, variable changed, score, N. - Calibrate any LLM-as-judge before trusting it. Sample 20–50 judge verdicts, label the same items yourself (or with a human), and compute agreement. High agreement → trust with periodic spot-checks. Low agreement → fix the rubric/evidence injection, or fall back to human labels. An uncalibrated judge is a random-number generator with confidence. Re-calibrate whenever the judge's prompt or model changes.
- Wire the eval into the change workflow. A prompt or model change ships only with a before/after score on the frozen set — the LLM analog of "no fix without a test" (
change-controlstep 8). Prompts are code; their regression suite is the eval. - Report honestly. Every result carries: the N, the set version/date, the single variable changed, both scores, and an epistemic label if the margin is within noise ("no detectable difference" is a valid, publishable result — see
research-methodologyfor label discipline). - Know when NOT to build eval infrastructure. One-off scripts, throwaway prototypes, and exploration sprints do not need golden sets — but write the deferral down ("no evals; do not ship without adding them"), so the prototype cannot silently become production. Proportionality is
engineering-minimalism's law; it applies to evals too.
Eval readiness checklist
- "Works" defined mechanically per behavior
- Golden set exists: real + adversarial + edge cases, human-labeled, frozen, date-stamped, N recorded
- Metric matches output type (deterministic checks wherever possible; judge only for free text)
- Judge (if any): ground truth injected, verdict enum-constrained, calibrated against human labels
- Baseline score recorded before the first change
- Change log: one variable per entry, before/after scores
- Eval runs as a gate on prompt/model changes
- Reports state N and set version; noise-level differences reported as "no detectable difference"
Discovery & Audit Commands
Audit an existing codebase's evaluation posture:
# Is there any eval asset at all? (absence is the finding)
find . -type d -iname "*eval*" -o -type d -iname "*golden*" -o -type d -iname "*bench*" 2>/dev/null | grep -v node_modules
grep -rln -iE "golden|eval_set|test_cases|rubric" --include="*.py" --include="*.ts" --include="*.json" . | grep -v node_modules | head
# Are prompts versioned/diffable? (prompts-as-code prerequisite)
git log --oneline -- "*prompt*" "*PROMPT*" 2>/dev/null | head
# Is there a judge, and is its verdict enum-constrained?
grep -rn -iE "judge|grade|score|verdict|rubric" --include="*.py" --include="*.ts" . | grep -v node_modules | head -20
# Any record of before/after comparisons?
grep -rn -iE "baseline|accuracy|pass_rate" --include="*.md" --include="*.py" . | grep -v node_modules | headFailure Modes & Anti-patterns
| Symptom | Mistake | Correction |
| "The new prompt seems better" ships | No golden set, no baseline | Freeze a set, measure before and after (steps 2, 4) |
| Improvement on the demo case, regressions everywhere else | Tuned against the case in front of you | The golden set is the target, never a single example |
| Prompt AND model changed, gain credited to the prompt | Two variables at once | One variable per comparison (step 4) |
| Judge passes everything | Uncalibrated LLM-as-judge, vague rubric | Calibrate against human labels; enum verdicts; inject ground truth (steps 3, 5) |
| Judge scores against its own memory of the topic | Ground truth not injected | Evidence in the judge's prompt (llm-integration-reliability step 6) |
| 7/10 vs 8/10 declared a win | Sample-size dishonesty | State the N; tens detect gross failure only; report "no detectable difference" |
| Scores not comparable across weeks | Golden set silently edited | Freeze + version + date-stamp the set; changing it resets baselines |
| One awful class hidden by a good average | Aggregate-only reporting | Per-class/confusion reporting for classification |
| Free-text output "checked" by eyeballing | No metric for the output type | Decision tree in step 3; rubric judge with calibration |
| Prototype became production with zero evals | Deferral never written down | "No evals; do not ship without adding them" recorded at deferral time (step 8) |
Worked Example
Task: improve the summary quality of a "summarize this incident report" feature.
- Definition: a good summary names the affected system, the impact, and the current status, in ≤3 sentences, with no invented facts.
- Golden set: 60 historical incident reports; a human writes reference summaries and labels the three required elements for each; plus 8 adversarial reports (buried impact statements, contradictory updates) and 6 edge cases (one-line reports, non-English, 10-page reports). Frozen as
eval/incidents-v1, dated. - Metric: structured checks where possible — length ≤3 sentences (deterministic), three required elements present (judge with the source report injected, verdict per element
{"present":"yes|no"}), no invented facts (judge with source injected,{"faithful":"yes|no"}). - Calibration: 30 judge verdicts spot-labeled by the engineer; agreement 27/30 → trusted with monthly spot-checks.
- Baseline: current prompt scores 41/60 all-elements-present, 55/60 faithful.
- One variable: restructure the prompt to demand the three elements explicitly (see
prompt-and-context-engineering). Re-run: 52/60 present, 54/60 faithful. The faithfulness dip (55→54) is within single-item noise on N=60 — recorded as "no detectable change." - Gate: the eval command is now part of the PR checklist for any change under
prompts/.
Repository Examples
Limited repository evidence available (as of 2026-07-04): none of the investigated repositories carries a golden set, judge calibration, or regression evals — evaluation is the least-evidenced discipline in the source corpus, which is precisely why this skill leans on the generic worked example above.
The closest real artifact: NYTW (~/NYTW) implements an LLM-as-judge with the right per-verdict mechanics — grading prompts inject the actual git diff + codebase evidence + user answer and demand strict enum JSON {"score":"correct|partial|incorrect","explanation"} validated on parse (as of 2026-07-04). What it lacks is the framework around the judge: no golden set of answer/score pairs, and no recorded calibration of the grader against human labels — the natural first application of this skill.
Validation Exercise (runnable in any repository with an LLM feature): pick one LLM behavior; write 20 input/expected-output pairs from real usage; measure the current system and record X/20 with today's date; change one variable; re-measure. If you cannot complete this exercise, the missing piece (undefined behavior, no labels, unparseable outputs) is itself the finding — route to llm-system-design or llm-integration-reliability accordingly.
Validation Criteria
You applied this skill correctly when:
- A frozen, dated, human-labeled golden set exists with real + adversarial + edge cases and a stated N.
- The metric matches the output type, and everything checkable deterministically is checked deterministically.
- Any judge is calibrated (agreement number on record) and receives injected ground truth with enum verdicts.
- Every claimed improvement cites: baseline score, new score, the single variable changed, and the set version.
- A prompt/model change cannot ship without the eval score attached.
- At least one report in the log says "no detectable difference" — evidence the noise floor is respected.
Provenance & Maintenance
- Sources:
~/NYTW(judge mechanics) — investigated 2026-07-04; the framework methodology is general practice, not derived from the example repos (their absence of evals is the documented evidence gap). Owner context: LLM output reliability is the owner's hardest problem; evals are its measurement arm. Skill authored 2026-07-06. - Assumptions: golden-set size guidance (tens vs low hundreds) is a practical heuristic, not a statistical guarantee — for high-stakes decisions compute a real confidence interval. NYTW facts are point-in-time.
- Re-verification commands:
grep -rn "correct|partial|incorrect" ~/NYTW/quiz --include="*.js" | head find ~/NYTW -iname "*golden*" -o -iname "*eval*" | head # expect: nothing (the gap) - Likely to drift: NYTW may gain a golden set (upgrade the example when it does); judge-calibration best practice evolves with model capability.
- Maintenance checklist:
- Re-run re-verification; re-stamp the Repository Examples section.
- If any owner repo gains an eval set, replace the Validation Exercise's prominence with the real case study.
- Confirm cross-referenced skills still exist under their directory names.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

