agentleFS
alvin-reyesalvin-reyes/better-agentic-ide/.claude/skills/bmad-eval/SKILL.md

bmad-eval

Skill43 starsChanged 31 days agoMITView on GitHub

Runs the evals of a BMad skill through the agent harness the user works in and reports what they show: baseline against the bare model, variant against a stripped or prior version, quality against a rubric, and trigger accuracy of the description with an optional optimization loop. Use when the user asks to evaluate, benchmark, test the triggers of, grade, or compare versions of a skill, or when the bmad-toolsmith skill asks for an eval of a skill it built.

2 other projects use a version of this, 2 of them edited.

Adopted by

Most recently taken up by

The file

We split this skill into the rules it gives an agent

4 rules. About 0 are common, 0 are distinctive, and 4 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈0

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 4

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    Read config with node {project-root}/_bmad/ade-runtime.mjs resolve_config --project-root {project-root} --key core.output_folder --key …

    line 27
  2. Very specific:

    Verify <skill-path>/SKILL.md exists.

    line 28
  3. Very specific:

    When --runs is greater than one, uv run {skill-root}/scripts/aggregate_benchmark.py --baseline <run-dir>/<config-a> --variant …

    line 58
  4. Very specific:

    When a run comes back weak and the user wants the skill improved from it, follow references/self-improvement.md.

    line 60

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Skill Eval Runner

You run a skill's evals and report what they say. Cite specific findings, flag evals that pass for trivial reasons, and never widen a tolerance to make a run look like it succeeded. No model or harness is named in this skill: the harness a project's evals run through is recorded in this skill's customization at the first run.

The four modes

ModeQuestion it answersScript / reference
baselineDoes the skill beat the bare model on the same input?references/eval-format.md, scripts/run_evals.py
variantDoes a section earn its place, or does a stripped version do as well?references/eval-format.md, scripts/run_evals.py
qualityDoes the output meet the named rubric?references/grader.md, references/eval-format.md
triggerDoes the description fire on the right queries and stay quiet on the rest?references/harness.md, scripts/run_triggers.py

Baseline runs every case with the skill staged and with nothing staged. Variant runs the skill against the one at --variant-path. Quality grades one config against its rubric. Trigger measures real firing; references/description-optimization.md improves the description across rounds.

Case format

A case is input + rubric + optional state_prefix + optional fixture files; the state_prefix is a bracketed prime prepended to the input that places the skill mid-workflow in one shot. Cases live in <skill>/evals/cases.json, trigger queries as {query, should_trigger} in <skill>/evals/triggers.json. The full shape and the strong-versus-weak expectation taxonomy are in references/eval-format.md. A skill-creator evals.json converts with uv run {skill-root}/scripts/convert_cases.py <evals.json> --output <cases.json> (--to skill-creator reverses it).

On activation

  1. Read config with node {project-root}/_bmad/ade-runtime.mjs resolve_config --project-root {project-root} --key core.output_folder --key core.communication_language --key core.user_name and customization with node {project-root}/_bmad/ade-runtime.mjs resolve_customization --skill {skill-root} --project-root {project-root} --key workflow. Absent keys are fine. {reports_folder} is {workflow.reports_folder} with {output_folder} filled in; set {communication_language} and {user_name} from what comes back.
  2. Verify <skill-path>/SKILL.md exists; halt with a clear error if not.
  3. When workflow.harness.command came back empty, work out the harness facts for the CLI you are running in per references/harness.md, prove them on one case, and record them by invoking the bmad-customize skill (install: npx skills add bmad-code-org/BMAD-METHOD --skill bmad-customize); the runner reads them from there. When nobody is at the keyboard, stop and show the table to record instead.
  4. Find the cases file: the one the user named, then <skill-path>/evals/, then <project-root>/evals/<skill-name>/, then anywhere under <project-root>/evals/. If nothing is found, halt and say so; the runner does not invent cases.
  5. Confirm the run summary unless the user asked for a non-interactive run, then execute. The summary names the skill, cases, modes and output dir, shows the harness command, and says that it runs with permission prompts off and is not contained unless the command wraps it in a sandbox. A non-interactive run needs the harness already recorded.

Run execution

The user asks for a run in <mode> mode on <skill> (the directory holding SKILL.md), with a variant skill for variant mode. The project root is the first ancestor of the skill holding _bmad/ or .git/; the output dir is {reports_folder} when BMad is set up, else ~/bmad-evals/. Every run is its own timestamped folder there, so runs never collide. Each case runs in a clean room, a temporary folder outside the project with a copy of the skill staged into its workspace and an environment built from scratch, so host config, prior runs, the project's instruction files and its installed skills cannot bias the result. The runner reads the harness from customization itself; in a project without BMad, pass --harness <json> with the same keys.

Baseline, variant and quality:

uv run {skill-root}/scripts/run_evals.py \
  --cases <cases-file> --skill-path <skill> --output-dir <dir> \
  --mode quality|baseline|variant [--variant-path <skill>] \
  --label <skill-name>-<mode> [--runs N]

--runs defaults to 1 for a quick look; a result counts as settled only at 3 or more. Case subsets, timeouts and workers are in the script docstrings. Exit 0 is a complete run; 1 means a case errored or timed out or the command was not found, 3 that no harness is recorded; a trigger query failing its threshold is a result, not an error.

Trigger:

node {project-root}/_bmad/ade-runtime.mjs run_triggers --skill-root {skill-root} \
  --skill-path <skill> --queries <queries-file> --output-dir <dir> \
  [--runs-per-query N]

For quality, spawn the grader in references/grader.md per case with its rubric, transcript path, artifacts dir (the case's cwd/) and grading_path of <case-folder>/grading.json. Relay its rubric feedback. If a grader errors, mark the case grading_error, never a default verdict.

When --runs is greater than one, uv run {skill-root}/scripts/aggregate_benchmark.py --baseline <run-dir>/<config-a> --variant <run-dir>/<config-b> gives mean, sample standard deviation, min, max and delta (--runs <run-dir>/<config> for one config).

When a run comes back weak and the user wants the skill improved from it, follow references/self-improvement.md.

Artifacts

Each run is a dated folder under the output dir, with run.json naming the harness command; each case folder, and each trigger attempt, holds its prompt, the transcript (what the harness printed), stderr.txt, cwd/ (the workspace after the run), timing.json (written the moment the invocation ends) and grading.json when quality ran. Never delete, overwrite or rotate a run folder. Tell the user where it is when you finish.

Outcomes

  • The run reflects the skill in a clean working directory, not the host shell.
  • Failures cite specific expectations with evidence; a superficial pass is flagged.
  • A result from fewer than 3 runs per case (or per trigger query) is reported as a quick look, never as settled.
  • A baseline the skill no longer wins points to retiring the skill, not patching it.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.