ai-eval-scorer
Skill0 starsChanged 6 months agoMITView on GitHub
Take an AI feature spec and produce a full eval harness - extracted behaviors, golden + edge + adversarial test cases, a judge rubric, and a scorecard. Use before shipping any LLM-powered feature, when "it works on my examples" isn't enough, or when stakeholders ask "how do you know it's good". Triggers on phrases like "build evals for this feature", "score this AI feature", "test cases for LLM feature", "eval rubric", "AI quality scorecard", "оцени AI-фичу", "сделай эвалы".
In this repository6 other files it gives its agents.
- deep-market-researchSkill
- feedback-triageSkill
- prd-builderSkill
- prd-criticSkill
- product-trio-sessionSkill
- synthetic-user-interviewSkill
The file
We split this skill into the rules it gives an agent
9 rules. About 0 are common, 9 are distinctive, and 0 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈9
Distinctive · estimate
This author's own choices. Worth a look.
0
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 18
Before shipping any LLM-powered feature to users
- Distinctive: line 19
When migrating models (Sonnet 4.6 → Sonnet 4.7, OpenAI → Claude) and need a regression check
- Distinctive: line 20
When stakeholders ask "how do you know it's good"
- Distinctive: line 47
Read the spec and pull out the discrete behaviors the feature is claiming:
- Distinctive: line 92
If able to execute:
- Distinctive: line 93
Run each case through the feature
- Distinctive: line 97
If not able to execute:
- Distinctive: line 139
Hidden rubric: if the judge's reasoning isn't surfaced, you can't trust the score.
- Distinctive: line 141
Skipping the "untestable" flag: if the spec has untestable claims, the spec is broken.
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
AI Eval Scorer
Purpose
Most AI features ship with vibes-based quality. The PM tries 5 examples, says "looks good", and ships. Two weeks later, customers find the broken cases.
This skill replaces vibes with a structured eval harness: explicit behaviors the feature should support, test cases per behavior (golden, edge, adversarial), a judge rubric, and a scorecard you can rerun on every model change.
This is the signature skill for PMs who want to ship AI features with the same rigor backend teams use for APIs.
When to use
- Before shipping any LLM-powered feature to users
- When migrating models (Sonnet 4.6 → Sonnet 4.7, OpenAI → Claude) and need a regression check
- When stakeholders ask "how do you know it's good"
- For features where quality is the entire product (chatbot, summarizer, generator)
- For prompt iterations: compare old vs new prompt against the same suite
When NOT to use
- For non-AI features (use regular acceptance criteria)
- For one-off internal prototypes nobody depends on
- For features where the "right answer" is genuinely unknowable (creative ideation, exploratory chat)
- Without a real spec - garbage in, useless evals out
Inputs
Required:
- Feature spec or natural-language description of what the feature does
- Examples of expected behavior if available
Strongly improves quality:
- Target user and use case
- Known edge cases the team is already worried about
- Constraints (length, tone, formatting, prohibited outputs)
- Comparable existing features (for behavioral reference)
Process
Phase 1: Extract behaviors
Read the spec and pull out the discrete behaviors the feature is claiming:
- Functional behaviors ("summarizes the input in under 200 words")
- Stylistic behaviors ("matches the brand voice")
- Safety behaviors ("refuses to give medical advice")
- Constraints ("never outputs more than 3 paragraphs")
Each behavior is a testable claim. If a behavior cannot be tested by inspecting the output, mark it untestable and flag for spec revision.
Phase 2: Generate test cases
For each behavior, generate three case types:
Golden cases (5-10 per behavior)
- Clear, representative inputs
- The "happy path"
- These define what good looks like
Edge cases (5-10 per behavior)
- Boundary conditions (very short, very long, mixed languages, ambiguous intent)
- Unusual but valid inputs
- These catch the failures that survive into prod
Adversarial cases (3-5 per behavior)
- Prompt injection attempts
- Inputs designed to elicit prohibited behavior
- Edge cases the team would lose money on
- These catch the failures that show up in headlines
Each case has: input, expected behavior, scoring criteria, severity if failed.
Phase 3: Build the judge rubric
For each case, define how to score the model's output:
- Binary criteria ("Did it refuse the medical question? Y/N")
- Scaled criteria (1-5) with explicit anchors:
- 1: catastrophic failure
- 3: acceptable but not great
- 5: ideal output
- Reasoning required: judge must explain the score, not just give a number
Rubric is part of the skill output. The user takes it to a judge model (Claude, GPT-4, human reviewers, or a panel).
Phase 4: Run the cases (optional, if model access available)
If able to execute:
- Run each case through the feature
- Score with an LLM judge using the rubric
- Aggregate per behavior and overall
If not able to execute:
- Output the runnable suite as a JSONL or markdown table
- Hand off to the eng team to wire into CI
Phase 5: Scorecard
# Eval Scorecard: [Feature]
## Overall: X / Y cases passed (Z% golden, W% edge, V% adversarial)
## Per-behavior scores
| Behavior | Golden | Edge | Adversarial | Verdict |
## Failures by severity
- Critical (ship-blocking)
- Major (must-fix before scale)
- Minor (track, fix in next iteration)
## Qualitative observations
[3-5 themes the judge noticed across cases]
## Recommendations
- Ship / Iterate / Block
- Specific prompt or model changes to try
- Cases to add to regression suiteOutput
The skill produces three artifacts:
- Behaviors list (markdown)
- Test case suite (markdown table or JSONL, runnable)
- Judge rubric (markdown, model-agnostic)
Plus the scorecard if cases were actually run.
Common failure modes to avoid
- Vague behaviors: "responds helpfully" is not testable. "Answers the user's question within 3 sentences" is.
- No adversarial cases: the team will find them, then customers will. Better here.
- Hidden rubric: if the judge's reasoning isn't surfaced, you can't trust the score.
- Scoring once and forgetting: this suite is a regression net, rerun it on every model change.
- Skipping the "untestable" flag: if the spec has untestable claims, the spec is broken.
Why this skill exists
AI quality is the new frontier for PMs and most don't have a process for it yet. The teams shipping good AI products treat evals like unit tests: boring, mandatory, and the reason nothing catches fire on launch day.
This skill is the PM-side starting kit. Engineers will build the harness; PMs need to bring the rubric.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

