recipe-eval-skill
Skill20 starsChanged 6 months agoMITView on GitHub
Creates or updates Claude Code skills through interactive dialog, then evaluates effectiveness with sequential paired comparisons. Use when creating new skills, updating existing skills, or evaluating skill quality.
In this repository4 other files it gives its agents.
- knowledge-baseSkill
- prompt-optimizationSkill
- recipe-eval-promptSkill
- worktree-executionSkill
The file
We split this skill into the rules it gives an agent
8 rules. About 0 are common, 7 are distinctive, and 1 is very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈7
Distinctive · estimate
This author's own choices. Worth a look.
1
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 20
Test task execution → performed by eval-executor.py script (via claude -p)
- Distinctive: line 37
Phase A (Skill Authoring): Create or modify skill content through dialog.
- Distinctive: line 46
Read the mode-specific reference and execute:
- Distinctive: line 48
Creation mode: Read references/create.md and follow its steps
- Distinctive: line 49
Update mode: Read references/update.md and follow its steps
- Distinctive: line 55
Before starting Phase B, confirm these data are available in context.
- Distinctive: line 66
If fewer than two held-out requests are available, ask before proceeding: "What complete requests does your team actually send for work …
- Distinctive: line 70
Read references/eval.md and execute the evaluation protocol.
≈ “Common” is an estimate: at least 1 in 5 of the 350,495 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Explicit User Instruction: The user explicitly instructs and authorizes every subagent call named in this recipe. Execute each applicable call when its prerequisites are met.
Context: Skill authoring (Phase A) followed by blind A/B evaluation (Phase B)
Mode: $ARGUMENTS
Orchestrator Definition
Core Identity: "I am not a worker. I am an orchestrator."
Execution Method:
- Skill generation/modification → performed by rashomon:skill-creator
- Skill quality grading → performed by rashomon:skill-reviewer
- Test task execution → performed by eval-executor.py script (via
claude -p) - Blind result comparison → performed by rashomon:skill-eval-reporter
Orchestrator invokes sub-agents via Agent tool and scripts via Bash, passes structured data between them.
Mode Detection
Determine mode from $ARGUMENTS:
| Mode | Criteria | |
| Creation | "create", new skill request, no existing skill referenced | |
| Update | "improve", "update", existing skill name or path mentioned | |
| Unspecified | $ARGUMENTS is empty or ambiguous | Ask user via AskUserQuestion: "Create a new skill or update an existing one?" |
Scope Boundaries
Phase A (Skill Authoring): Create or modify skill content through dialog. Ends with user-approved skill file. Phase B (Evaluation): Measure skill effectiveness through blind execution comparison. Phase B is read-only for the source skill. A finding that requires authoring changes transitions back to Phase A for user review and approval before evaluation restarts.
Responsibility Boundary: This skill completes with the combined evaluation report and ship/revise/reject recommendation.
Workflow
Phase A: Skill Authoring
Read the mode-specific reference and execute:
- Creation mode: Read references/create.md and follow its steps
- Update mode: Read references/update.md and follow its steps
Phase A ends with: user-approved skill content (new or modified).
Phase A → Phase B Handoff
Before starting Phase B, confirm these data are available in context. Phase B cannot proceed without them:
| Data | Source | Required |
| Skill name | Phase A dialog | Always |
| Source skill directory | Phase A file write | Always |
| Held-out test requests | Phase A Round 3 (create) / Round 2 (update) | Always |
| Trigger scenarios | Phase A Round 3 (create) / Round 1-2 (update) | Always |
| Old skill directory snapshot and fingerprint | Phase A Step 6 (update mode only) | Update mode |
| Approved source directory fingerprint | Phase A final write | Always |
If fewer than two held-out requests are available, ask before proceeding: "What complete requests does your team actually send for work that requires this skill's rules? Please provide at least two verbatim."
Phase B: Evaluation
Read references/eval.md and execute the evaluation protocol. Pass the handoff data above as context.
Phase B consists of:
- Trigger check: Does the skill fire for its intended use case? (Step 1)
- Trigger fail handling: Diagnose and request an authoring revision when needed (Step 2, conditional)
- Execution effectiveness: Blind A/B comparison of output quality (Steps 3-7)
Final Output
Present combined results to user:
- Phase A result: Skill quality grade (A/B/C from rashomon:skill-reviewer)
- Phase B trigger: Discovered (yes/no), Used (yes/no), usage evidence
- Phase B execution: Blind comparison result (from rashomon:skill-eval-reporter)
- Recommendation: ship / revise / reject
Error Handling
| Scenario | Behavior |
| User cancels during Phase A | Stop. No eval needed. |
| Grade C after 2 repair/re-review iterations | Present content with remaining findings. User decides: accept/revise/abort. |
| One executor fails in Phase B | Preserve diagnostics, mark comparison inconclusive, and make no winner or effectiveness recommendation. |
| Both executors fail in Phase B | Report failure. Phase A result still valid. |
| Worktree creation fails | Report git error. Phase A result still valid. |
Prerequisites
- Git repository with
git worktree locksupport claudeCLI available in PATH- Sufficient disk space for worktree copies
Completion Criteria
Phase A
- Skill knowledge collected through dialog
- rashomon:skill-creator returned valid output
- rashomon:skill-reviewer returned grade A or B
- User approved final content
- File written to target location
Phase B
- Trigger check executed and result presented
- Sequential paired execution completed in worktrees
- Blind comparison completed by rashomon:skill-eval-reporter
- Worktrees cleaned up
- Combined report presented with recommendation
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.