agentleFS
Sign inStart free
d-oitd-oit/rust-2026-template/.agents/skills/skill-evaluator/SKILL.md

skill-evaluator

Skill15 starsChanged 5 months agoMITView on GitHub

Reusable skill for evaluating other skills with structure checks, eval coverage review, and real usage spot checks. Use when you need to check a skill, add evals, benchmark a skill, validate outputs against assertions, or compare current skill behavior against a baseline.

22 other projects use a version of this.

Adopted by

Most recently taken up by

The file

We split this skill into the rules it gives an agent

7 rules. 0 are common, 6 are distinctive, and 1 is very specific to this project.

  • 0

    Common

    Most of the 17 edited versions we read keep these.

  • 6

    Distinctive

    This author's own choices. Worth a look.

  • 1

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    Read evals/evals.json if present and assess whether each case is realistic.

    line 61
  2. Distinctive:

    no evidence path for pass/fail

    line 74
  3. Distinctive:

    Run at least one representative prompt from the eval set or create a focused ad hoc prompt.

    line 78
  4. Distinctive:

    read only the files the skill itself points to

    line 83
  5. Distinctive:

    When useful, rerun the same prompt without the skill or against a snapshot of the older skill.

    line 89
  6. Distinctive:

    Prefer assertions that can be checked directly.

    line 108
  7. Distinctive:

    Every pass or fail must include evidence.

    line 122

“Common” is measured: kept by at least 6 in 10 of the 17 edited versions of this file we could read. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Skill Evaluator

Evaluate local skills with a repeatable loop: inspect structure, read eval definitions, run one or more realistic prompts, then score the output with explicit assertions and evidence.

Required Inputs

At minimum, identify:

SKILL_PATH: absolute or workspace-relative path to the skill directory
GOAL: structure check / eval review / live run / baseline comparison

Evaluation Workflow

1. Structure Check

Confirm the skill directory is sane before judging outputs.

Expected layout:

skill-name/
  SKILL.md
  evals/evals.json  # recommended
  references/       # recommended
  scripts/          # optional but useful

Flag these issues explicitly:

  • missing SKILL.md
  • nested duplicate directory like skill-name/skill-name/
  • evals/ exists but evals/evals.json is missing or invalid JSON
  • eval cases missing id, prompt, or expected_output

2. Quantitative Evaluation

Consult .agents/metrics.jsonl to assess skill performance over time. Analyze trends in:

  • Success rate
  • Human intervention rate
  • Token usage and duration

3. Eval Review

Read evals/evals.json if present and assess whether each case is realistic.

Good evals include:

  • a real user prompt
  • a short success definition
  • optional input files
  • assertions that are concrete and checkable

Weak evals include:

  • vague prompts
  • purely subjective assertions
  • no evidence path for pass/fail

4. Live Run

Run at least one representative prompt from the eval set or create a focused ad hoc prompt.

For each live run:

  • load the target skill
  • read only the files the skill itself points to
  • produce the answer or output
  • grade against assertions with evidence

5. Baseline Comparison

When useful, rerun the same prompt without the skill or against a snapshot of the older skill.

Compare:

  • pass rate
  • missing details
  • format compliance
  • time or token cost if available

6. Verdict

End with one of:

  • PASS — structure is sound and live output meets assertions
  • NEEDS_WORK — usable, but structure gaps or output gaps remain
  • FAIL — skill is broken, misleading, or missing core pieces

Assertion Rules

Prefer assertions that can be checked directly.

Good:

  • The answer cites the exact minimum cover dimensions
  • The output includes all 7 scoring dimensions
  • evals.json contains at least 2 cases

Bad:

  • The output is good
  • The skill feels smart
  • The answer is polished

Every pass or fail must include evidence.

Output Format

## Eval Report: <skill-name>
- Goal: <goal>
- Structure: PASS/NEEDS_WORK/FAIL
- Live run: PASS/NEEDS_WORK/FAIL
- Baseline: not run / summary

### Assertion Results
- PASS: <assertion> — <evidence>
- FAIL: <assertion> — <evidence>

### Issues
- <issue>

### Next Fixes
1. <fix>
2. <fix>

### Verdict
PASS | NEEDS_WORK | FAIL — <one-sentence summary>

References

  • ../skill-creator/references/evaluating-skills.md — the canonical eval workflow and grading guidance. This is a sibling copy removed to stop the two drifting.

Rationalizations

RationalizationReality
"The skill works fine, no need to evaluate."Unvalidated skills drift. Regular evaluation catches regressions early.
"One eval case is enough."One case can't cover edge scenarios. Aim for 3+ diverse evals per skill.
"I'll use subjective grading."Subjective grading is unreliable. Use concrete, checkable assertions.

Red Flags

  • Evaluating a skill without a clear GOAL (structure check / eval review / live run)
  • Using vague assertions like "the output is good"
  • Not comparing against a baseline when measuring improvement

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

Check your setup, free