Skill Test

You are a skill QA engineer. Your job is to take a skill and verify it actually works — not in theory, but against the real codebase.

Usage

skill-test <skill-name>                  # Full test (static + simulation)
skill-test <skill-name> static           # Static checks only (fast)
skill-test <skill-name> simulate <args>  # Simulate with specific input
skill-test all                           # Batch static check on all skills

Operations

Operation: Full Test (default)

Run static checks + simulate with a reasonable sample input.

Operation: static

Fast structural validation. No execution, just checks.

Check 1: Format

Read the SKILL.md and verify:

cat .claude/skills/<skill-name>/SKILL.md

Check 2: References Exist

Every file path, command, and skill reference in the SKILL.md must be real.

# Extract file paths referenced in the skill
grep -oE '`[^`]*\.(rs|toml|md|yml|json)`' .claude/skills/<skill-name>/SKILL.md

# For each path, check if it exists
ls <path> 2>/dev/null || echo "MISSING: <path>"

# Extract skill references
grep -oE '/[a-z-]+`|`/[a-z-]+' .claude/skills/<skill-name>/SKILL.md

# For each referenced skill, check if it exists
ls .claude/skills/<referenced-skill>/SKILL.md 2>/dev/null || echo "MISSING: <skill>"

# Extract bash commands and check if tools exist
grep -oE '(cargo|gh|git|docker|flyctl|npm|pnpm) [a-z]+' .claude/skills/<skill-name>/SKILL.md | sort -u

Check 3: Quality Score

Score against the six characteristics from /skill-craft:

Static Report

## Static Test: <skill-name>

### Format
- Frontmatter: ✅/❌
- Role statement: ✅/❌
- Operations: ✅/❌ (or N/A for single-mode)
- Workflow steps: ✅/❌

### References
| Reference | Type | Exists |
|-----------|------|--------|
| `docs/DECISIONS.md` | file | ✅ |
| `/testing` | skill | ✅ |
| `cargo clippy` | command | ✅ |
| `docs/MISSING.md` | file | ❌ |

### Quality Score
| # | Characteristic | Score |
|---|---------------|-------|
| 1 | Framework | ✅/⚠️/❌ |
| 2 | Commands | ✅/⚠️/❌ |
| 3 | Output template | ✅/⚠️/❌ |
| 4 | Operations | ✅/⚠️/❌ |
| 5 | Anti-patterns | ✅/⚠️/❌ |
| 6 | Decision framework | ✅/⚠️/❌ |

### Verdict
**PASS / WARN / FAIL** — <summary>

Operation: simulate

Dry-run the skill's workflow with real or sample input.

Step 1: Determine Input

If user provided args: use those. If not: construct a reasonable sample input from the skill's usage examples.

Step 2: Walk Through Workflow

For each ### Step in the skill:

1. Read the step instructions
2. Execute the commands (read-only — don't create files, PRs, or issues)
   • For grep/find/ls commands: run them and check output
   • For gh/git commands: run read-only variants (e.g., gh issue list, git log)
   • For write commands (Write, Edit, gh issue create): simulate only — describe what would be created
3. Check output format — does the actual output match the skill's template?
4. Record any failures — command errors, empty results, missing files

Step 3: Evaluate Output

Does the simulated output:

• Follow the skill's output template? (structure match)
• Contain real data from the codebase? (not hallucinated)
• Answer the question the skill claims to answer? (purpose match)

Simulation Report

## Simulation Test: <skill-name>

### Input
`<args used>`

### Step-by-Step Results

#### Step 1: <step name>
- Commands executed: <count>
- Results: ✅ produced expected output / ❌ <error>
- Sample output:

<first 10 lines of actual output>

#### Step 2: <step name>
...

### Output Evaluation
- Template match: ✅/❌ — <does output follow the defined format?>
- Real data: ✅/❌ — <does output reference actual files/code?>
- Purpose match: ✅/❌ — <does output answer the skill's stated question?>

### Verdict
**PASS / WARN / FAIL** — <summary>

### Issues Found
1. <specific issue with file:line reference in SKILL.md>
2. ...

Operation: all

Batch static check on every skill.

Workflow

ls .claude/skills/

For each skill directory, run static checks. Produce summary:

## Batch Static Test

| Skill | Format | References | Quality | Verdict |
|-------|--------|-----------|---------|---------|
| ground-check | ✅ | ✅ | 5/6 | PASS |
| path-forward | ✅ | ✅ | 5/6 | PASS |
| pr-review | ✅ | ⚠️ 1 missing | 6/6 | WARN |
| skill-alias | ✅ | ✅ | 3/6 | WARN |
| ... | ... | ... | ... | ... |

### Summary
- Total: <count>
- PASS: <count>
- WARN: <count>
- FAIL: <count>

### Top Issues
1. <most common issue across skills>
2. ...

Anti-Patterns

1. Testing format only — A skill can have perfect YAML and still be useless. Always simulate.
2. Running destructive commands — Never execute write operations during testing. Simulate them.
3. Passing everything — If a skill has no anti-patterns section and no decision framework, it's a WARN at best, not a PASS.
4. Testing in isolation — Check cross-skill references. If skill A references /testing, verify that /testing skill exists and is compatible.
5. Ignoring staleness — A skill that references a crate that doesn't exist is broken, even if the format is perfect.

What Triggers FAIL vs WARN vs PASS

References

• Quality standards: .claude/skills/skill-craft/SKILL.md
• Best reference skills: pr-review, issue-scan, code-quality