agentleFS
Sign inStart free
thenavidmthenavidm/skill-builder/skills/skill-eval/SKILL.md

skill-eval

Skill0 starsChanged 29 days agoMITView on GitHub

Runs a skill's tasks with and without it in a real AI tool, grades every output and opens a review page. Use for "does this skill actually help", or to test one.

  • Installs packages

21 other projects use a version of this.

Adopted by

Most recently taken up by

Used with

Projects with it also use mutation-test, postmortem and 5 more.

Other files people use alongside this one.

Files found in the same public projects as a version of this one, more often than they turn up anywhere else. Each project counts once, large collections count for less, and this file's own repository is left out.
  • squall-chuaSkill

    mutation-test

    Break the source on purpose, find the mutants the tests fail to catch, and report each survivor with the behaviour nobody is checking.

    Also in 3 projects that use this

  • Hongyu-yuSkill

    postmortem

    Use when analyzing failures, outages, incidents, or negative outcomes, conducting blameless postmortems, documenting root causes with 5 Whys or fishbone diagrams, identifying corrective actions with owners and timelines, learning from near-misses, establishing prevention strategies, or when user mentions postmortem, incident review, failure analysis, RCA, lessons learned, or after-action review.

    Also in 3 projects that use this

  • nicolasestremSkill

    brainstorm

    Structured idea generation and solution exploration with systematic analysis.

    Also in 3 projects that use this

  • MicrosoftSkill

    handoff

    Compact the current conversation into a handoff document for another agent to pick up.

    Also in 5 projects that use this

  • NVIDIASkill

    research

    Before designing a feature, research how comparable libraries solve the problem, check best practices, known pitfalls, and upstream changes. Use when starting a new feature, investigating an unfamiliar area, or the user asks to research something.

    Also in 3 projects that use this

  • SentrySkill

    review

    Fixture review skill.

    Also in 3 projects that use this

The file

We split this skill into the rules it gives an agent

1 rule. 0 are common, 1 is distinctive, and 0 are very specific to this project.

  • 0

    Common

    Most of the 19 edited versions we read keep these.

  • 1

    Distinctive

    This author's own choices. Worth a look.

  • 0

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Run the same tasks with the skill and without it, grade each run, and show the difference.

    line 8

“Common” is measured: kept by at least 6 in 10 of the 19 edited versions of this file we could read. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Does the skill help?

Run the same tasks with the skill and without it, grade each run, and show the difference. A skill that doesn't beat the plain model on its own tasks is weight in every conversation.

The engine runs as npx -y @thenavidm/skill-builder@latest. Where skill-builder is installed (npm install -g @thenavidm/skill-builder), run skill-builder instead: it's the same engine, without the download.

1. The tasks

The skill's evals/evals.json holds them. If it has none, write it with them first: one task per thing the skill promises, real input files in evals/files/, and assertions that check outcomes. The format and what makes a test prove something are in the eval format.

2. Run

Say how many runs it will take (tasks x 2 x runs), then:

npx -y @thenavidm/skill-builder@latest eval <skill> --tool <their main tool> --runs 3

Each run gets its own throwaway folder and may write only there. Nothing switches permission checks off. Where no AI tool can be run headless (in claude.ai, say), the skill-executor agent runs each task in a fresh context and the skill-grader agent grades it, so the numbers come the same way.

3. Read the result with them

  • With and without: the share of assertions passed in each, with the spread across runs, and the difference.
  • The skill never ran in some runs with it: those measure the model, not the skill. Usually a description problem: use skill-description.
  • Passed as often without the skill: those assertions prove nothing. Sharpen them.
  • Blocked: the run stopped on a permission headless mode can't ask for; it's left out of the numbers.
  • Time and tokens: a skill that helps but costs much more is worth knowing about.

For the patterns the averages hide, start the skill-analyzer agent on the results.

Open the review page it wrote (review.html) and ask them to look at the outputs. Their notes, saved as feedback.json, are what skill-improve fixes next.

4. In every tool

To see the same tasks in each AI tool they have installed, use skill-bench.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

Check your setup, free