testing-skills
Skill0 starsChanged 9 months agoMITView on GitHub
Tests Agent Skills using three-tier evaluation (triggering, functional, performance). Reads EVALUATIONS.md, runs scenarios via CLI, and produces diagnostic reports. Activates when user asks to "test skill", "run evaluations", "evaluate skill", or mentions skill testing. Do NOT use when creating, updating, or auditing skills.
2 other projects use a version of this.
Adopted by
Most recently taken up by
- osmontero/opencode-skillstheir own version, first seen Oct 27, 2026
- orkait/hyperstacktheir own version, first seen Jun 14, 2026
Other versions2 projects use a version of this.
Word for word · 0
None found.
Edited · 0
None found.
Wrote their own version · 2
- osmontero/opencode-skills★ 15same name
- orkait/hyperstack★ 1same name
How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.
In this repository17 other files it gives its agents.
- AGENTS.mdAGENTS.md
- auditing-skillsSkill
- creating-skillsSkill
- forge-builderSkill
- forge-claude-teamsSkill
- forge-codex-multiagentSkill
- forge-orchestratorSkill
- forge-researchSkill
- handoff-freshSkill
- handoffSkill
- managing-doc-manifestSkill
- okfSkill
- pickupSkill
- studySkill
- syncing-docsSkill
- updating-skillsSkill
- wrapSkill
The file
Show its rules and text
We split this skill into the rules it gives an agent
17 rules. About 0 are common, 15 are distinctive, and 2 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈15
Distinctive · estimate
This author's own choices. Worth a look.
2
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 46
If EVALUATIONS.md is missing:
- Distinctive: line 49
Stop — do not proceed without evaluation scenarios
- Distinctive: line 74
Check whether the target skill was loaded into context
- Distinctive: line 91
Run the skill on the test input
- Distinctive: line 92
Check each criterion in the Then block
- Distinctive: line 93
Use the appropriate grading method per criterion (see Step 6)
- Very specific: line 106
Use context: fork for isolated execution when testing skills that modify files.
- Distinctive: line 112
Run the task WITHOUT the skill (baseline)
- Distinctive: line 113
Run the task WITH the skill
- Distinctive: line 119
Use hybrid grading — deterministic checks where possible, Agent-as-a-Judge for quality.
- Distinctive: line 136
Ask for pass/fail per criterion with brief justification
- Distinctive: line 184
If fixes are needed, suggest:
- Very specific: line 196
Read ~/.claude/skills/handoff/EVALUATIONS.md
- Distinctive: line 198
Run triggering test: "audit my skill" → verify handoff does NOT activate → PASS
- Distinctive: line 199
Run functional: "/handoff" in a project with changes → verify HANDOFF.md created → PASS
- Distinctive: line 200
Run functional: "/handoff" in empty project → verify graceful handling → PASS
- Distinctive: line 206
Update this skill when:
≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Testing Skills
Tests Agent Skills using three-tier evaluation: triggering, functional, and performance. Reads EVALUATIONS.md, runs scenarios, and produces diagnostic reports.
Workflow
- [ ] 1. Identify target skill and read EVALUATIONS.md
- [ ] 2. Classify tests by tier (triggering / functional / performance)
- [ ] 3. Run triggering tests
- [ ] 4. Run functional tests
- [ ] 5. Run performance tests (if defined)
- [ ] 6. Grade results (hybrid: code-based + Agent-as-a-Judge)
- [ ] 7. Generate diagnostic report
- [ ] 8. Present findings (diagnostic only — no auto-repair)Step 1: Read EVALUATIONS.md
Locate and read the target skill's EVALUATIONS.md:
~/.claude/skills/{skill-name}/EVALUATIONS.mdIf EVALUATIONS.md is missing:
- Report "No EVALUATIONS.md found — cannot test"
- Suggest creating one using creating-skills spec (Given/When/Then format)
- Stop — do not proceed without evaluation scenarios
Step 2: Classify Tests
Each scenario in EVALUATIONS.md maps to a tier:
| Tier | Purpose | Identifies |
| Triggering | Does the skill activate on the right prompts? | should-trigger + should-NOT-trigger scenarios |
| Functional | Does the skill produce correct output? | Happy path, edge case, error handling |
| Performance | Does the skill outperform baseline Claude? | Quality delta vs no-skill baseline |
Classify by scenario tags or naming convention:
should-trigger/should-NOT-trigger→ Triggeringhappy path/edge case/error→ Functionalbaseline comparison/performance→ Performance
Step 3: Run Triggering Tests
For each triggering scenario:
- Invoke Claude with the test prompt via CLI:
claude -p "test prompt here" --output-format json - Check whether the target skill was loaded into context
- For should-trigger: verify skill activated
- For should-NOT-trigger: verify skill did NOT activate
Interpreting Results
| Scenario Type | Skill Activated | Result |
| should-trigger | Yes | PASS |
| should-trigger | No | FAIL — skill not activating on expected trigger |
| should-NOT-trigger | No | PASS |
| should-NOT-trigger | Yes | FAIL — false positive activation |
Step 4: Run Functional Tests
For each functional scenario:
- Run the skill on the test input
- Check each criterion in the Then block
- Use the appropriate grading method per criterion (see Step 6)
CLI Invocation
# Single-turn test
claude -p "test prompt" --output-format json
# Multi-turn test (session continuation)
claude -p "first message" --output-format json
# Then continue in the same sessionUse context: fork for isolated execution when testing skills that modify files.
Step 5: Run Performance Tests
Only if defined in EVALUATIONS.md. Most skills skip this tier.
- Run the task WITHOUT the skill (baseline)
- Run the task WITH the skill
- Compare outputs on defined criteria
- Record the delta
Step 6: Grade Results
Use hybrid grading — deterministic checks where possible, Agent-as-a-Judge for quality.
| Criterion Type | Grading Method |
| File exists at path | Code check |
| Output contains required string | Code check |
| Frontmatter has field X | Code check (parse YAML) |
| Output follows template | Code for structure, Agent for content |
| Quality of generated content | Agent-as-a-Judge |
| Correct tool selection | Agent-as-a-Judge |
Agent-as-a-Judge Protocol
When using Agent-as-a-Judge:
- Feed the skill's output to a separate evaluation prompt
- Provide the rubric from EVALUATIONS.md as context
- Ask for pass/fail per criterion with brief justification
- Record the judge's assessment
See methodology.md for detailed grading protocol.
Step 7: Generate Report
# Test Report: {skill-name}
**Date**: {YYYY-MM-DD}
**Version**: {skill version}
**Scenarios**: {total} ({pass} passed, {fail} failed, {skip} skipped)
## Summary
| Tier | Pass | Fail | Skip |
|------|------|------|------|
| Triggering | X | Y | Z |
| Functional | X | Y | Z |
| Performance | X | Y | Z |
## Results
### Triggering
| Scenario | Type | Result | Notes |
|----------|------|--------|-------|
| {name} | should-trigger | PASS/FAIL | {details} |
### Functional
| Scenario | Result | Criteria Met | Notes |
|----------|--------|-------------|-------|
| {name} | PASS/FAIL | 3/4 | {details} |
### Performance (if applicable)
| Metric | Baseline | With Skill | Delta |
|--------|----------|-----------|-------|
| {metric} | {score} | {score} | {+/-} |
## Suggested Fixes
{For each FAIL, suggest what to change in the skill}Step 8: Present Findings
Present the report to the user. This is diagnostic only — do not automatically modify the skill.
If fixes are needed, suggest:
- "Run updating-skills to apply these changes"
- Point to specific sections that need updating
Example
Input:
Test the handoff skillProcess:
- Read ~/.claude/skills/handoff/EVALUATIONS.md
- Find 3 scenarios: happy path (functional), edge case (functional), should-NOT-trigger (triggering)
- Run triggering test: "audit my skill" → verify handoff does NOT activate → PASS
- Run functional: "/handoff" in a project with changes → verify HANDOFF.md created → PASS
- Run functional: "/handoff" in empty project → verify graceful handling → PASS
- Generate report: 3/3 passed
- Present: "All evaluations passing. No changes needed."
Self-Evolution
Update this skill when:
- On grading disagreement: Agent-as-a-Judge disagrees with manual check → refine rubric
- On new test pattern: Discover evaluation pattern not covered → add to methodology
- On CLI change:
claude -pflags or output format changes → update invocation patterns
Current version: 1.0.0. See CHANGELOG.md for history.
References
- Testing methodology: methodology.md
- Creating evaluations: ~/.claude/skills/creating-skills/SKILL.md
- Auditing after fixes: ~/.claude/skills/auditing-skills/SKILL.md
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

