eval-engineering-lite
Skill0 starsChanged 48 days agoMITView on GitHub
Set up a small repeatable AI evaluation with baseline, provenance and a release decision; use when model or retrieval behavior changes.
In this repository9 other files it gives its agents.
- content-qaSkill
- evidence-contentSkill
- failure-analyzerSkill
- faq-builderSkill
- meeting-to-actionsSkill
- project-handoffSkill
- requirement-clarifierSkill
- research-synthesizerSkill
- workflow-plannerSkill
The file
We split this skill into the rules it gives an agent
3 rules. About 0 are common, 2 are distinctive, and 1 is very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈2
Distinctive · estimate
This author's own choices. Worth a look.
1
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 14
Use when changing prompts, RAG, agents, model configuration or semantic decisions.
- Distinctive: line 28
Run deterministic schema/rule checks first, then real provider output only if configured.
- Very specific: line 42
Run python3 tests/validate.py from the repository root to check this skill's contract and example provenance.
≈ “Common” is an estimate: at least 1 in 5 of the 350,495 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Eval Engineering Lite
Purpose
Separate deterministic test health from probabilistic model quality.
When to Use
Use when changing prompts, RAG, agents, model configuration or semantic decisions.
Inputs
journey, risk_level, cases: [{id,input,expected,ground_truth_status}], baseline, optional traces[].
Outputs
rubric, case_results, regressions, failure_taxonomy, release_gate.
Workflow
- Define success, must-not-happen conditions and critical cases before editing behavior.
- Preserve the baseline run and case provenance; synthetic cases remain
synthetic_unverified. - Run deterministic schema/rule checks first, then real provider output only if configured.
- Compare candidate versus baseline on the same cases, including high-risk failures and traces.
- Classify failures and mark the release
PASS,FAILorBLOCKEDwith evidence.
Failure Handling
An unavailable model, missing independent ground truth or failed judge is BLOCKED/NOT_RUN, never PASS. Do not lower a rubric or erase a hard case to improve a score.
Example
synthetic_unverified: two fictional RAG cases validate fixture shape; without a provider, report MODEL_QUALITY=NOT_RUN.
Tests
Run python3 tests/validate.py from the repository root to check this skill's contract and example provenance. This is a structural test, not model-quality evidence.
Limitations
This lite skill is not a full production eval platform; human-reviewed Golden Sets and domain owners are required for high-risk claims.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.