agentleFS
abhaylaabhayla/claude-best-practices/.claude/skills/skill-evaluator/SKILL.md

skill-evaluator

Skill9 starsChanged 7 months agoMITView on GitHub

Evaluate Claude Code skills for trigger reliability, output quality, and cross-skill conflicts. Use when testing a new or updated skill before promotion, auditing existing skills for trigger conflicts, or verifying a skill adds value over the agent's baseline. Invoked automatically by /writing-skills during authoring, or standalone for auditing existing skills. For structural pattern-FILE validation (frontmatter, portability, registry sync) use /pattern-quality instead.

22 other projects use a version of this.

Adopted by

Most recently taken up by

Used with

Projects with it also use skill-creator.

Other files people use alongside this one.

Files found in the same public projects as a version of this one, more often than they turn up anywhere else. Each project counts once, large collections count for less, and this file's own repository is left out.
  • SentrySkill

    skill-creator

    Create new agent skills following the Agent Skills specification. Use when asked to "create a skill", "add a new skill", "write a skill", "make a skill", "build a skill", or scaffold a new skill with SKILL.md. Guides through requirements, writing, registration, and verification.

    Also in 5 projects that use this

The file

Show its rules and text

We split this skill into the rules it gives an agent

75 rules. 0 are common, 60 are distinctive, and 15 are very specific to this project.

  • 0

    Common

    Most of the 17 edited versions we read keep these.

  • 60

    Distinctive

    This author's own choices. Worth a look.

  • 15

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Run these checks BEFORE evaluation.

    line 49
  2. Distinctive:

    Verify these fields against platform constraints:

    line 66
  3. Distinctive:

    Every skill must front-load its runtime needs (owner mandate 2026-07-26): DECLARE them, VERIFY them in a STEP 0 preflight while the user is …

    line 112
  4. Distinctive:

    VERIFY them in a STEP 0 preflight while the user is present at invocation, and HARD-STOP with a complete missing-items list on gaps — never …

    line 113
  5. Distinctive:

    If ANY check in 0.1-0.5 fails, report it in the evaluation output as a

    line 128
  6. Very specific:

    If --baseline: snapshot current version (cp -r <skill-path> <workspace>/skill-snapshot/)

    line 145
  7. Distinctive:

    NOT obviously irrelevant queries — those test nothing

    line 168
  8. Distinctive:

    Run each query 3 times (model nondeterminism), each in clean context (subagent)

    line 173
  9. Distinctive:

    Should-trigger threshold: ≥80% trigger rate

    line 175
  10. Distinctive:

    Should-not-trigger threshold: ≤20% trigger rate

    line 176
  11. Distinctive:

    Run should-trigger queries with ALL installed skills active

    line 181
  12. Distinctive:

    Verify the target skill activates, not a competitor

    line 182
  13. Distinctive:

    Bidirectional signposting: when a conflict is detected, check if BOTH skills' descriptions mention the other with clear boundary language.

    line 184
  14. Distinctive:

    Near-duplicate resolution: if overlap is HIGH, check whether the target skill delegates to the neighbor (orchestrator wrapper pattern) or …

    line 185
  15. Distinctive:

    Run old version's trigger queries against new description

    line 190
  16. Distinctive:

    No regressions: every old query must still trigger

    line 191
  17. Distinctive:

    If stuck after 3 incremental tweaks, try structurally different framing

    line 197
  18. Distinctive:

    Verify description stays under 1024 characters

    line 200
  19. Distinctive:

    Run 3 representative prompts WITHOUT the skill (subagent, clean context)

    line 211
  20. Distinctive:

    Run same prompts WITH the skill

    line 212
  21. Distinctive:

    If with-skill doesn't meaningfully improve: flag "skill may not be adding value"

    line 213
  22. Very specific:

    If evals/evals.json exists in skill directory: load it

    line 217
  23. Distinctive:

    Start with 2-3 cases; expand after first results

    line 219
  24. Very specific:

    If --baseline: also run against old version

    line 225
  25. Distinctive:

    If the skill targets multiple model tiers, run scenarios across available models (e.g., Haiku, Sonnet, Opus).

    line 230
  26. Distinctive:

    When model matrix is not feasible (time/cost), note "Tested on \<model\> only" in the evaluation report.

    line 234
  27. Distinctive:

    If the skill has dual-mode operation, verify skip conditions work correctly.

    line 261
  28. Distinctive:

    Test A — Mode Detection: Verify the skill detects its runtime environment correctly.

    line 267
  29. Very specific:

    Run with .claude/skills/learn-n-improve/SKILL.md present → should enter FULL mode (write to learnings.json)

    line 268
  30. Distinctive:

    Run without learn-n-improve → should enter STANDALONE mode (score with haiku subagent)

    line 269
  31. Very specific:

    Check that references/CHANGELOG.jsonl is created if missing

    line 270
  32. Distinctive:

    Test B — Gap Detection with Novel Knowledge: Run the skill with a scenario that surfaces a domain-specific pattern, edge case, or gotcha …

    line 272
  33. Distinctive:

    Verify the presentation includes: what was found, target file, proposed action, and score

    line 285
  34. Distinctive:

    Test E — Post-Update Integrity: After approved updates:

    line 287
  35. Very specific:

    Verify patch version bumped in SKILL.md frontmatter

    line 288
  36. Very specific:

    Verify CHANGELOG.jsonl has a new entry with CREATE action

    line 289
  37. Distinctive:

    Verify two-tier structure maintained (Consolidated Principles + Active Observations)

    line 290
  38. Distinctive:

    Verify no-gap scenario: when all knowledge is already covered, skill reports "References are up to date"

    line 291
  39. Very specific:

    Add results to the evaluation report under the REFERENCE SELF-UPDATE section.

    line 301
  40. Distinctive:

    Write assertions AFTER first run — you don't know "good" until the skill runs

    line 305
  41. Distinctive:

    Remove assertions that always pass in both configs (inflating pass rate)

    line 323
  42. Distinctive:

    Check time/token outliers: read execution transcript for bottleneck

    line 327
  43. Distinctive:

    Run each query with all skills active

    line 341
  44. Very specific:

    Conductor arms pinned to the target cheap model: Every worker Agent() dispatch uses model: <--model, default sonnet> — never the T0 …

    line 364
  45. Very specific:

    After returning the verdict, append to EVAL-LEARNINGS.md (standalone AND delegated runs — this replaces the batch-only capture in …

    line 471
  46. Distinctive:

    Use the existing EVAL-LEARNINGS.md entry format (verdict / caught / missed / proposed improvements).

    line 482
  47. Very specific:

    Always read EVAL-LEARNINGS.md before evaluating (Step 0.0) and capture lessons after (Step 8) — Why: recorded misses that recur because the …

    line 491
  48. Distinctive:

    Always run each eval in clean context (subagent isolation) — Why: leftover state produces unreliable results

    line 492
  49. Distinctive:

    Always test both trigger AND output for full mode — Why: a skill that triggers correctly but produces bad output is broken

    line 493
  50. Distinctive:

    Always check cross-skill conflicts, not just target in isolation — Why: isolated testing misses real activation conflicts

    line 494
  51. Distinctive:

    Always run skill necessity baseline for new skills — Why: the most important question is "should this skill exist?"

    line 495
  52. Distinctive:

    Always provide evidence for every assertion grade — Why: "looks good" is not a grade

    line 496
  53. Distinctive:

    Always include fix-type-mapped recommendations with FIX/FAIL — Why: a verdict without actionable fixes wastes the caller's time

    line 497
  54. Distinctive:

    Always test with 5 fresh queries after description optimization — Why: optimized descriptions can overfit to the eval set

    line 498
  55. Distinctive:

    Always review assertions during grading — Why: broken assertions waste iterations

    line 499
  56. Distinctive:

    When delegated from writing-skills: skip human review, return verdict directly — Why: writing-skills manages the approval gate

    line 500
  57. Distinctive:

    Always validate name constraints (64 chars, lowercase/hyphens, no reserved words) in pre-flight — Why: platform rejects non-conforming names

    line 501
  58. Distinctive:

    Always check description is third-person in pre-flight — Why: first/second person causes discovery problems per platform docs

    line 502
  59. Very specific:

    Always check reference depth ≤1 from SKILL.md in pre-flight — Why: Claude partially reads deeply nested files

    line 503
  60. Distinctive:

    Always run the prerequisites-contract check (0.5) — Why: a skill that discovers a missing tool, credential, or user input mid-run stalls …

    line 504
  61. Very specific:

    Always run all 10 pre-flight checks (0.4) for skills with references/ — Why: the protocol file, CHANGELOG, entry format, mode detection, …

    line 505
  62. Very specific:

    Always run Tests A-E (3.4b) for skills with references/ during output evaluation — Why: structural presence is not enough.

    line 506
  63. Distinctive:

    Always test both FULL and STANDALONE modes when learn-n-improve is available in the project — Why: downstream projects may or may not have …

    line 507
  64. Distinctive:

    Always freeze trap-mode cases before dispatching any arm — Why: cases edited after seeing results are no longer a certification, they're a …

    line 508
  65. Very specific:

    Always pin trap-mode arms to the target --model (never the T0 frontier model) — Why: the whole point is proving the asset works under the …

    line 509
  66. Distinctive:

    Always run the trap-mode fingerprint gate before judging — Why: a misfiled answer graded against the wrong case produces a meaningless …

    line 510
  67. Distinctive:

    Always name specific missed case IDs in a NOT-CERTIFIED trap verdict — Why: "62% catch-rate" gives the repair loop nothing to fix

    line 511
  68. Distinctive:

    MUST NOT report PASS without running all evaluations for the mode — report only what was tested

    line 515
  69. Distinctive:

    MUST NOT skip cross-skill conflict check — isolated testing misses real conflicts

    line 516
  70. Distinctive:

    MUST NOT grade without evidence — every assertion needs specific output reference

    line 517
  71. Distinctive:

    MUST NOT suggest fixes contradicting writing-skills best practices — fixes must align with authoring guidance

    line 518
  72. Distinctive:

    MUST NOT add specific keywords from failed queries to description — that's overfitting.

    line 519
  73. Distinctive:

    MUST NOT block on human review when delegated from writing-skills — return verdict immediately

    line 520
  74. Distinctive:

    MUST NOT weaken a trap case, the rubric, or the pass bar to force a CERTIFIED verdict — repair the ASSET only, never the exam

    line 521
  75. Distinctive:

    MUST NOT exceed 2 repair rounds in trap mode — report NOT-CERTIFIED honestly rather than iterating indefinitely

    line 522

“Common” is measured: kept by at least 6 in 10 of the 17 edited versions of this file we could read. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Skill Evaluator — Evaluate Skill Quality

Structured evaluation of skill trigger reliability, output quality, and cross-skill compatibility. Produces a graded verdict (PASS/FIX/FAIL) with evidence and recommended fixes.

Request: $ARGUMENTS


STEP 0: Pre-Flight Checks

0.0 Read Eval Learnings (self-learning loop, entry gate)

Read EVAL-LEARNINGS.md in this skill's directory before evaluating. It records what past evaluations MISSED (found manually afterward) and the improvements proposed from those misses. Apply every pending "proposed evaluator improvement" as an extra ad-hoc check during THIS run — a recorded miss that recurs because the log was never read is a process failure, not an evaluator gap. This applies to standalone runs too, not only EVAL-WORKFLOW batch runs.

Run these checks BEFORE evaluation. They catch the issues most commonly missed during Skills #1-5 evaluation batch (registry drift, missing frontmatter, structural defects). Each check is pass/fail with specific evidence.

0.1 Registry Sync

Compare registry/patterns.json entry against the skill's SKILL.md:

CheckHowFail Condition
Description matchCompare registry description vs frontmatter descriptionEmpty registry description, or semantic mismatch
Hash freshnessCompute SHA-256 of SKILL.md, compare to registry hashHash differs
Version consistencyCompare registry version vs frontmatter version vs latest changelog entryAny mismatch
Dependency completenessScan body for Agent("...agent"), Skill("...skill"), /skill-name references → verify each in registry dependenciesReferenced but not listed

0.2 Frontmatter Completeness

Verify these fields against platform constraints:

FieldValidation Rules
nameRequired. Max 64 chars. Lowercase letters/numbers/hyphens only. No XML tags. No reserved words (anthropic, claude). Prefer gerund form (processing-pdfs). MUST match directory name
descriptionRequired. Non-empty. Max 1024 chars. No XML tags. Must be third-person ("Processes..." not "I process..." or "You can..."). Should start with verb. Must describe what AND when to use
typeRequired. Must be workflow or reference
triggersRequired. ≥3 entries (BLOCKING if missing — skill is invisible to natural language)
allowed-toolsScan body for Agent(), Skill(), Write, Edit, Bash usage. Flag under-declared or over-declared tools
versionRequired. SemVer format

0.3 Structural Integrity

CheckHowFail Condition
Code fence balanceCount ``` openings vs closings per sectionUnequal (orphaned fence)
Numbered-list continuityScan for numbered items (1., 2., ...) between ## headersOrphaned items not part of a contiguous list
Step cross-referencesFind all "Step N" mentions, verify each N exists as ## STEP N:Reference to nonexistent step
Skill name referencesFind all /skill-name mentions, verify each exists in .claude/skills/Dead reference
Agent referencesFind all *-agent mentions, verify each exists in .claude/agents/Dead reference
Placeholder markersScan for <!-- TODO -->, <!-- FIXME -->, <!-- PLACEHOLDER -->Any found
Preamble constraintsFor workflow skills, verify critical constraints appear in BOTH preamble (top) AND CRITICAL RULES section (bottom)Missing from either location
Reference depthScan all Read: / See: / link references in SKILL.md and referenced filesAny file that references another file (depth >1 from SKILL.md)
Reference TOCCheck reference files >100 lines for a table of contents section at topMissing TOC in long reference file

0.4 Reference Self-Update Mechanism

For skills that have a references/ directory, verify the full self-update protocol is present and correctly configured:

#CheckHowFail ConditionSeverity
1Protocol file existsCheck for self-update-protocol.md in the skill's references/ dirMissingMAJOR
2Protocol content validVerify protocol contains all 7 sections: N.1 (mode detection), N.2 (scan), N.3 (admission gate), N.4 (entry format), N.5 (scoring), N.6 (consolidation), N.7 (version bump)Missing sectionsMAJOR
3Reference Completeness Check stepScan for a step that reads the self-update protocol fileSkill has references/ but no self-update stepMAJOR
4CHANGELOG.jsonl existsCheck for references/CHANGELOG.jsonlMissing audit sidecarMINOR
5Entry format matches specVerify protocol's entry format has all 13 fields (ID, State, Temporal, Scope, Date, Context, Observation, Application, Confidence, Applied-In, Source, Supersedes, Tags)Missing fieldsMAJOR
6Mode detection presentProtocol includes FULL vs STANDALONE mode detection based on learn-n-improve presenceNo mode detectionMAJOR
7Admission gate presentProtocol has explicit exclusion criteria (generic knowledge, temp workarounds, user-specific data, etc.)No admission gateMAJOR
8Scoring mechanism presentSTANDALONE mode has independent scoring (haiku subagent or equivalent); FULL mode routes through learnings.jsonNo quality gate beyond user approvalMAJOR
9Two-tier file structureProtocol defines Consolidated Principles + Active Observations sectionsMissing two-tier structureMINOR
10User approval gateProtocol requires user approval before any reference writesAuto-writes without approvalCRITICAL

Overall severity: Skills with references/ that lack the self-update mechanism entirely get a MAJOR finding. Missing user approval gate is CRITICAL.

0.5 Prerequisites & Preflight Contract

Every skill must front-load its runtime needs (owner mandate 2026-07-26): DECLARE them, VERIFY them in a STEP 0 preflight while the user is present at invocation, and HARD-STOP with a complete missing-items list on gaps — never stall mid-run waiting for an absent user, never improvise an undeclared fallback.

CheckHowFail ConditionSeverity
Prerequisites declaredScan for a ## Prerequisites section (or an explicit Prerequisites: none line)Neither presentMAJOR
Preflight gate presentIf any prerequisite is declared, verify the first executable step is a ## STEP 0: Preflight (or equivalently named preflight step) covering every declared itemDeclared prerequisites with no preflight stepMAJOR
User inputs front-loadedScan steps after the preflight for mid-run user prompts requesting inputs that were knowable at invocationMid-run prompt for a collectable inputMAJOR
No undeclared fallbackAny fallback/alternative path in the body is declared in the Prerequisites section and needs no mid-run user inputImprovised fallback on a missing prerequisiteMAJOR

Grandfather rule: skills predating this contract (created before 2026-07-26 without a ## Prerequisites section) get a WARN — report the gap, do not fail solely on it — until the hub-wide compliance sweep enrolls them. NEW skills and renamed/rewritten skills FAIL.

If ANY check in 0.1-0.5 fails, report it in the evaluation output as a PRE-FLIGHT FAILURE and include it in the fix recommendations. Do not skip these checks — they caught issues in 5/5 evaluated skills.


STEP 1: Parse Mode and Load Skill

ModeWhat It Tests
triggerDescription activation, trigger rate, cross-skill conflicts
outputOutput quality: scenarios, stress test, assertions, benchmarks
fullBoth trigger + output, aggregated report
conflictsCross-skill conflict scan across ALL installed skills
trapFable-independence certification: does the asset still produce disciplined, correct behavior when a CHEAP model drives it? Planted-defect exam + blind judging
  1. Load target skill's SKILL.md, extract: name, description, triggers, allowed-tools
  2. If --baseline: snapshot current version (cp -r <skill-path> <workspace>/skill-snapshot/)
  3. Detect invocation context:
    • Standalone: include human review (Step 5)
    • Delegated from writing-skills: skip human review, return verdict directly

STEP 2: Trigger Evaluation (modes: trigger, full)

Read: references/description-optimization.md for description writing principles, eval query design, and optimization loop methodology.

2.1 Generate 20 Eval Queries

10 should-trigger queries:

  • Vary phrasing (formal, casual, typos/abbreviations)
  • Vary explicitness (names domain directly vs describes the need)
  • Vary detail (terse one-liners vs context-heavy with file paths)
  • Vary complexity (single-step vs multi-step where skill is buried in chain)
  • Most valuable: queries where the skill helps but connection isn't obvious

10 should-not-trigger queries:

  • Near-misses that share keywords but need a different skill
  • NOT obviously irrelevant queries — those test nothing
  • Include realistic context (file paths, personal context, casual language)

2.2 Test Activation

  • Run each query 3 times (model nondeterminism), each in clean context (subagent)
  • Compute trigger rate per query (triggers / runs)
  • Should-trigger threshold: ≥80% trigger rate
  • Should-not-trigger threshold: ≤20% trigger rate
  • Note: agents may not consult skills for simple tasks — this is expected

2.3 Cross-Skill Conflict Check

  • Run should-trigger queries with ALL installed skills active
  • Verify the target skill activates, not a competitor
  • Report: competing skill name + the conflicting query + trigger rates
  • Bidirectional signposting: when a conflict is detected, check if BOTH skills' descriptions mention the other with clear boundary language. If not, flag as "missing reciprocal boundary"
  • Near-duplicate resolution: if overlap is HIGH, check whether the target skill delegates to the neighbor (orchestrator wrapper pattern) or reimplements its logic (true duplicate). Read the skill body and any dispatched agents to determine this
  • Skill-vs-rule overlap: compare skill steps against auto-loaded rules in .claude/rules/ for redundancy (e.g., a skill that restates a globally-loaded workflow rule)

2.4 Trigger Regression (--baseline only)

  • Run old version's trigger queries against new description
  • No regressions: every old query must still trigger

2.5 Description Optimization (if trigger verdict is FIX)

  1. Identify failing queries from train set (60% of queries)
  2. Revise description — generalize from the category, don't add specific keywords
  3. If stuck after 3 incremental tweaks, try structurally different framing
  4. Validate against held-out set (40% of queries)
  5. Final: test with 5 fresh queries never seen during optimization
  6. Verify description stays under 1024 characters

STEP 3: Output Evaluation (modes: output, full)

Read: references/eval-driven-iteration.md for test case design, assertion writing, grading methodology, and iteration patterns.

3.1 Skill Necessity Baseline

  • Run 3 representative prompts WITHOUT the skill (subagent, clean context)
  • Run same prompts WITH the skill
  • If with-skill doesn't meaningfully improve: flag "skill may not be adding value"

3.2 Design or Load Test Cases

  • If evals/evals.json exists in skill directory: load it
  • Otherwise: generate 5 scenarios (3 happy-path + 2 edge-case)
  • Start with 2-3 cases; expand after first results

3.3 Run Scenarios

  • Each run in clean context (subagent isolation — no leftover state)
  • With-skill run for each test case
  • If --baseline: also run against old version
  • Capture outputs + timing ({total_tokens, duration_ms})

3.3b Model Matrix (optional but recommended)

If the skill targets multiple model tiers, run scenarios across available models (e.g., Haiku, Sonnet, Opus). Flag divergent results — a skill that works on Opus but fails on Haiku needs more explicit guidance.

When model matrix is not feasible (time/cost), note "Tested on <model> only" in the evaluation report. If the skill requires MCP tools not available in the eval context, document as "untestable dependency" in the report — do not report false FAIL.

3.4 Stress Test

10 adversarial inputs — pick most realistic for the skill's domain:

#CategoryWhat It Exposes
1Ambiguous requestRouting/classification failures
2Off-topic queryMissing scope guard
3Conflicting constraintsSilent data loss
4Oversized inputTruncation, timeout
5Minimal/empty inputCrash or null proceeding
6Partial prerequisiteBroken state from incomplete setup
7Repeated invocationDuplicate output, state corruption
8Malformed formatUnhelpful error or silent wrong behavior
9Stale contextActing on outdated data
10Adversarial phrasingTrigger misfire

Score each: CRITICAL (wrong output/data loss) / MAJOR (partial/misleading) / MINOR (cosmetic) / PASS (handled correctly).

Delegation edge cases (for skills that call other skills/agents):

  • What happens when the delegate returns empty/zero results?
  • Does the skill handle standalone vs pipeline mode differently? Test both paths.
  • If the skill has dual-mode operation, verify skip conditions work correctly.

3.4b Reference Self-Update Behavior (skills with references/ only)

For skills that have a references/ directory and a self-update-protocol.md file within it, test the full self-update mechanism across both runtime modes.

Test A — Mode Detection: Verify the skill detects its runtime environment correctly.

  • Run with .claude/skills/learn-n-improve/SKILL.md present → should enter FULL mode (write to learnings.json)
  • Run without learn-n-improve → should enter STANDALONE mode (score with haiku subagent)
  • Check that references/CHANGELOG.jsonl is created if missing

Test B — Gap Detection with Novel Knowledge: Run the skill with a scenario that surfaces a domain-specific pattern, edge case, or gotcha NOT in any reference file.

  • Skill should identify ≥1 new knowledge item
  • Entry should use the structured 13-field format (ID, State, Type, Confidence, etc.)
  • Confidence should be ≥0.70 for persisted entries

Test C — Admission Gate + Scoring: Inject both valid and invalid knowledge:

  • Valid: domain-specific gotcha discovered during execution → should pass admission gate
  • Invalid: generic knowledge ("use try/catch"), user-specific path, ephemeral debug info → should be filtered
  • In STANDALONE mode: verify haiku subagent scoring produces KEEP/REVIEW/DISCARD verdicts
  • In FULL mode: verify entries are written to learnings.json, NOT directly to references

Test D — User Approval Gate: The skill MUST present findings and wait for user approval before any reference writes.

  • Auto-writing without approval is a CRITICAL failure
  • Verify the presentation includes: what was found, target file, proposed action, and score

Test E — Post-Update Integrity: After approved updates:

  • Verify patch version bumped in SKILL.md frontmatter
  • Verify CHANGELOG.jsonl has a new entry with CREATE action
  • Verify two-tier structure maintained (Consolidated Principles + Active Observations)
  • Verify no-gap scenario: when all knowledge is already covered, skill reports "References are up to date"
TestPass CriteriaFailure Severity
A: Mode detectionCorrect mode selected based on learn-n-improve presenceMAJOR
B: Gap detection≥1 new entry with valid 13-field format and confidence ≥0.70MAJOR
C: Admission + scoringGeneric content filtered; valid content scored independentlyMAJOR
D: User approval gateFindings presented, writes blocked until approvalCRITICAL
E: Post-update integrityVersion bumped, CHANGELOG updated, two-tier structure intactMINOR

Add results to the evaluation report under the REFERENCE SELF-UPDATE section.

3.5 Write and Grade Assertions

  • Write assertions AFTER first run — you don't know "good" until the skill runs
  • Good: "output is valid JSON", "chart has labeled axes", "report has ≥3 recs"
  • Weak: "output is good" (vague), "uses exact phrase X" (brittle)
  • Grade each PASS/FAIL with specific evidence (quote output, reference files)
  • Review assertions themselves: too easy (always pass)? too hard? unverifiable?

3.6 Aggregate Benchmarks

{
  "with_skill": {"pass_rate": 0.83, "tokens_mean": 3800, "time_mean_s": 45},
  "without_skill": {"pass_rate": 0.33, "tokens_mean": 2100, "time_mean_s": 32},
  "delta": {"pass_rate": "+0.50", "tokens": "+1700", "time_s": "+13"}
}

3.7 Pattern Analysis

  • Remove assertions that always pass in both configs (inflating pass rate)
  • Investigate always-fail in both (broken assertion or impossible task)
  • Study skill-only-pass (where skill adds value — understand WHY)
  • For inconsistent results (high stddev): tighten ambiguous instructions
  • Check time/token outliers: read execution transcript for bottleneck

3.8 Blind Comparison (--baseline only)

  • Present both outputs to a judge without revealing which is old vs new
  • Score holistic qualities: organization, formatting, usability, polish
  • Complements assertion grading — two outputs can pass all assertions but differ in quality

STEP 4: Conflicts Mode (mode: conflicts)

  1. Scan all installed skills in .claude/skills/ and core/.claude/skills/
  2. For each skill, generate 3 representative should-trigger queries
  3. Run each query with all skills active
  4. Report any query that activates the wrong skill
  5. Output: conflict matrix showing which skills compete

STEP T: Trap Mode (mode: trap)

Read: references/trap-test-protocol.md for the full procedure — case authoring, conductor discipline, blind judging, the rubric, and the repair loop. This section is only the orchestration summary; the reference is the source of truth.

CLI: /skill-evaluator trap <asset-path> [--model sonnet] [--cases <path>] [--bar all|0.9]. <asset-path> is any hub asset (skill, plugin manual, rule) whose value claim includes working under a cheaper/other model. --model defaults to sonnet (the cheap target being certified). --cases overrides the default <asset>/evals/trap-cases/ location. --bar defaults to all (every case must be caught); 0.9 accepts a documented ≥90% catch-rate for large case sets.

Non-negotiables (detailed procedure in the reference file):

  1. Cases FROZEN before any arm runs. Author + freeze traps/replays/probes to <asset>/evals/trap-cases/ (one file per case) BEFORE the certification run starts. No case may be added, removed, or edited after dispatch begins.
  2. Conductor arms pinned to the target cheap model. Every worker Agent() dispatch uses model: <--model, default sonnet> — never the T0 frontier model. The asset under test (skill, manual, rule) is given to the arm exactly as a downstream cheap-model user would receive it.
  3. Judge = opus, fresh context, anonymized + shuffled answers, fixed rubric. A separate judge dispatch per answer, blind to which case/arm produced it, scored against the frozen rubric (see reference §4). Mix in ≥1 calibration case with a known score; a miscalibrated judge batch is void and must be re-run.
  4. Verdict is CERTIFIED only when every trap case is caught, or the documented --bar is met. Anything short is NOT-CERTIFIED — name the specific missed case IDs, never a bare percentage.
  5. Misses drive a bounded repair loop. For each miss, procedurally rewrite the FAILING SECTION of the asset (never the case, never the rubric), re-run ONLY the missed cases, max 2 rounds, then re-freeze. If still not certified after 2 rounds, report NOT-CERTIFIED honestly — do not loosen the bar to force a pass.

Output: a TRAP CERTIFICATION report (per-case table, catch-rate, misses, judge calibration result, verdict) appended to the caller's evaluation report (STEP 6) or written standalone when trap is the only mode requested.


STEP 5: Human Review (standalone only)

When invoked standalone (not delegated from writing-skills):

  • Present outputs alongside grades for each test case
  • Record specific, actionable feedback per case
  • Empty feedback = output passed review
  • Feedback feeds into iteration

When delegated from writing-skills: SKIP this step — return verdict directly. Writing-skills handles the single user approval gate.


STEP 6: Produce Evaluation Report

Locked output format:

SKILL EVALUATION REPORT: <skill-name>
=====================================
Mode: <trigger|output|full|conflicts|trap>
Iteration: <N>

SKILL NECESSITY
  Without skill: <pass_rate>% | With skill: <pass_rate>%
  Delta: +<N>% — <adds clear value | marginal | not adding value>

TRIGGER EVALUATION
  Should-trigger:    N/10 activated (N% rate)
  Should-not:        N/10 correctly ignored
  Cross-skill:       N conflicts [list]
  Regressions:       N/N (--baseline only)
  Fresh validation:  N/5 passed
  Trigger verdict:   PASS | FIX | FAIL

OUTPUT EVALUATION
  Scenarios:         N/5 passed
  Stress test:       N% (N CRIT, N MAJOR, N MINOR)
  Assertions:        N/N (N%)
  Baseline delta:    +N% pass rate, +N tokens, +Ns
  Output verdict:    PASS | FIX | FAIL

REFERENCE SELF-UPDATE (skills with references/ only)
  Pre-flight (0.4):  <N/10 checks passed>
  Protocol file:     <present + valid | missing | incomplete>
  CHANGELOG.jsonl:   <present | missing>
  Test A (mode):     <PASS — detected FULL/STANDALONE correctly | MAJOR>
  Test B (gaps):     <PASS — N entries, valid format, confidence ≥0.70 | MAJOR>
  Test C (gate+score): <PASS — generic filtered, valid scored | MAJOR>
  Test D (approval): <PASS — waited for user | CRITICAL — auto-wrote>
  Test E (integrity): <PASS — version bumped, CHANGELOG updated | MINOR>
  Self-update verdict: <PASS | FIX | FAIL>

MODEL COVERAGE
  Tested on:         <model list or "single model: X">
  Divergent results: <list or "N/A">

TRAP CERTIFICATION (mode: trap only)
  Target model:      <--model, e.g. sonnet>
  Cases:              N traps / N replays / N probes (frozen <path>)
  Catch-rate:         N/N (N%)
  Missed cases:       <case IDs, or "none">
  Judge calibration:  <PASS | discarded+re-run — N deviations>
  Repair rounds:      <0-2>
  Trap verdict:       CERTIFIED | NOT-CERTIFIED

OVERALL VERDICT: PASS | FIX | FAIL
Blocking issues: <list or "none">
Recommended fixes: <prioritized, mapped to failure types>

STEP 7: Return Verdict

VerdictCriteriaWhat Caller Should Do
PASSTrigger ≥80%, stress ≥90%, no conflicts, no regressions, skill adds valueProceed
FIXMinor issues below thresholdFix specific issues, re-run eval
FAILCritical failures, design gaps, skill adds no valueMajor rework

Trap mode returns its own verdict (see STEP T): CERTIFIED or NOT-CERTIFIED — never a percentage in place of naming missed case IDs.


STEP 8: Capture Eval Lesson (self-learning loop, exit gate — MANDATORY)

After returning the verdict, append to EVAL-LEARNINGS.md (standalone AND delegated runs — this replaces the batch-only capture in EVAL-WORKFLOW.md for single evals):

Event this runAction
Something was found OUTSIDE the standard checks (ad-hoc, prompted, or manual)Append it + a "proposed evaluator improvement" checkbox
A pending proposed improvement from 0.0 caught a real issue again (2+ evidence)Promote: edit THIS SKILL.md's check tables, bump minor version, tick the checkbox as applied
A standard check produced a false positive/negativeAppend it — check-calibration lesson
Clean run, all findings from standard checksAppend the one-line header only (skill, date, verdict) — per EVAL-WORKFLOW rule 8, absence of findings is data

Use the existing EVAL-LEARNINGS.md entry format (verdict / caught / missed / proposed improvements). Why this exists: the log had entries but no read gate and no standalone-run capture — misses recorded in April were still pending in August because nothing forced a re-read.


MUST DO

  • Always read EVAL-LEARNINGS.md before evaluating (Step 0.0) and capture lessons after (Step 8) — Why: recorded misses that recur because the log was never read are process failures; the log is only useful if it gates every run
  • Always run each eval in clean context (subagent isolation) — Why: leftover state produces unreliable results
  • Always test both trigger AND output for full mode — Why: a skill that triggers correctly but produces bad output is broken
  • Always check cross-skill conflicts, not just target in isolation — Why: isolated testing misses real activation conflicts
  • Always run skill necessity baseline for new skills — Why: the most important question is "should this skill exist?"
  • Always provide evidence for every assertion grade — Why: "looks good" is not a grade
  • Always include fix-type-mapped recommendations with FIX/FAIL — Why: a verdict without actionable fixes wastes the caller's time
  • Always test with 5 fresh queries after description optimization — Why: optimized descriptions can overfit to the eval set
  • Always review assertions during grading — Why: broken assertions waste iterations
  • When delegated from writing-skills: skip human review, return verdict directly — Why: writing-skills manages the approval gate
  • Always validate name constraints (64 chars, lowercase/hyphens, no reserved words) in pre-flight — Why: platform rejects non-conforming names
  • Always check description is third-person in pre-flight — Why: first/second person causes discovery problems per platform docs
  • Always check reference depth ≤1 from SKILL.md in pre-flight — Why: Claude partially reads deeply nested files
  • Always run the prerequisites-contract check (0.5) — Why: a skill that discovers a missing tool, credential, or user input mid-run stalls unattended sessions; a NEW skill without the declare→verify→hard-stop contract must not pass evaluation
  • Always run all 10 pre-flight checks (0.4) for skills with references/ — Why: the protocol file, CHANGELOG, entry format, mode detection, admission gate, scoring, two-tier structure, and approval gate must all be present for the self-update mechanism to work
  • Always run Tests A-E (3.4b) for skills with references/ during output evaluation — Why: structural presence is not enough; mode detection, gap detection, scoring, approval gating, and post-update integrity must all work at runtime
  • Always test both FULL and STANDALONE modes when learn-n-improve is available in the project — Why: downstream projects may or may not have learn-n-improve; skills must work correctly in both environments
  • Always freeze trap-mode cases before dispatching any arm — Why: cases edited after seeing results are no longer a certification, they're a curve-fit
  • Always pin trap-mode arms to the target --model (never the T0 frontier model) — Why: the whole point is proving the asset works under the CHEAP model a downstream user will actually run
  • Always run the trap-mode fingerprint gate before judging — Why: a misfiled answer graded against the wrong case produces a meaningless certification
  • Always name specific missed case IDs in a NOT-CERTIFIED trap verdict — Why: "62% catch-rate" gives the repair loop nothing to fix

MUST NOT DO

  • MUST NOT report PASS without running all evaluations for the mode — report only what was tested
  • MUST NOT skip cross-skill conflict check — isolated testing misses real conflicts
  • MUST NOT grade without evidence — every assertion needs specific output reference
  • MUST NOT suggest fixes contradicting writing-skills best practices — fixes must align with authoring guidance
  • MUST NOT add specific keywords from failed queries to description — that's overfitting; generalize from the category
  • MUST NOT block on human review when delegated from writing-skills — return verdict immediately
  • MUST NOT weaken a trap case, the rubric, or the pass bar to force a CERTIFIED verdict — repair the ASSET only, never the exam
  • MUST NOT exceed 2 repair rounds in trap mode — report NOT-CERTIFIED honestly rather than iterating indefinitely

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.