ywc-toolkit-eval
Skill8 starsChanged 5 months agoMITView on GitHub
(ywc) Use when periodically evaluating or scoring the quality of existing ywc-* skills or agents in this toolkit, running the evaluate→improve cycle, or auditing the skill/agent catalog for activation accuracy, structure compliance, and behavioral efficacy. Triggers: "skill 평가", "agent 평가", "toolkit 평가해줘", "skill 점수 매겨줘", "평가 사이클", "evaluate skills", "score the agents", "skill quality eval", "스킬 평가", "에이전트 평가", "スキル評価", "エージェント評価", "evaluation cycle", "toolkit 품질 점검". Do not use for authoring or restructuring a skill (use ywc-skill-author), for reviewing application/product code (use ywc-impl-review), for spec quality review (use ywc-spec-validate), or for one-off structural lint that scripts/validate.sh already covers.
In this repository4 other files it gives its agents.
- AGENTS.mdAGENTS.md
- codex/AGENTS.mdAGENTS.md
- CLAUDE.mdCLAUDE.md
- graftSkill
The file
We split this skill into the rules it gives an agent
3 rules. About 0 are common, 3 are distinctive, and 0 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈3
Distinctive · estimate
This author's own choices. Worth a look.
0
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 29
When tempted to bypass a rule, check this table first:
- Distinctive: line 99
Run the deterministic scorer:
- Distinctive: line 167
Before declaring an evaluation cycle complete, verify:
≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
ywc-toolkit-eval
Announce at start: "I'm using the ywc-toolkit-eval skill to score the toolkit's skills and agents and drive the evaluate→improve cycle."
Graded quality-evaluation harness for the toolkit's own Claude Code ywc-* skills and custom agents. Where ywc-skill-author defines the binary compliance rules for building a skill, this skill produces a graded scorecard (0–5 per axis, weighted to 100) for existing Claude Code skills and agents, ranks the weakest items, and persists score history so quality trends are visible release over release. Input: a target root (claude-code/skills, claude-code/agents, or both). Output: evals/scorecard.md (per-item axis scores + prioritized backlog) and an appended evals/history.json row. Codex skill/agent evaluation is owned by .codex/skills/ywc-codex-toolkit-eval.
Internal-only skill (locale-exempt):
ywc-toolkit-evallives under.claude/skills/as a toolkit-maintenance tool and is not distributed underclaude-code/skills/. It is therefore exempt from the en/ja/ko README locale-set requirement thatscripts/validate.shenforces on distributed skills.
The harness is two-tier by design, because not every quality axis can be measured deterministically:
| Tier | How | Axes | CI |
| Mechanical | scripts/score.py (deterministic, no model) | Structure compliance, token economy, integrity, tool-grant minimality, output-contract presence, description-collision heuristic | Regression-blocking |
| Judgment | Agent judge pass (Workflow / Task subagents) | Activation precision/recall, behavioral efficacy, role-boundary clarity, dispatch accuracy, prompt quality | On-demand, non-blocking |
Rationalization Defense
When tempted to bypass a rule, check this table first:
| Excuse | Reality |
| "Mechanical score is green, the skill is good" | Mechanical axes are ~40% of the weight. A skill can pass every structural check and still never activate (S1) or fail its own workflow (S3). Never report a total from the mechanical tier alone. |
| "This skill scored 5/5 last quarter, skip re-eval" | Sibling skills change descriptions; a collision can appear without this skill changing one line. Activation accuracy is a property of the whole catalog, not one file. Always re-score against the current siblings. |
| "The judge agreed with my guess, so it's right" | A single judge over a single phrasing is one data point. Activation precision needs the full positive/negative/collision case set, not one confirming example. |
| "Score went down but the skill is actually better now" | If the rubric disagrees with reality, fix the rubric in the same PR and say so — do not silently override the number. An unexplained manual score is indistinguishable from gaming. |
| "I'll evaluate skills now, agents later" | Agents are dispatched BY skills. An unscored agent is an unmeasured edge of the same graph. Score both roots in one cycle or the catalog-fit axis (S6/A1) is blind. |
| "Backlog is obvious, skip the prioritized list" | Without the ranked backlog the cycle has no next action and degrades into a one-off audit. The ranked list IS the deliverable that makes it a cycle. |
| "Low score on a rarely-used skill, ignore it" | A rarely-used skill with an over-broad description steals activations from its siblings every turn (Tier-1 cost is paid every conversation). Low usage ≠ low blast radius. |
Violating the letter of these rules is violating the spirit. A scorecard that flatters the toolkit is worse than no scorecard — it certifies decay as health.
Arguments
Two distinct surfaces: the deterministic score.py flags, and the skill/orchestrator-level arguments this skill interprets but never passes to the script.
score.py flags (consumed by scripts/score.py):
| Parameter | Format | Example | Description |
--target | --target <root> | --target claude-code/skills | Root to evaluate: claude-code/skills, claude-code/agents, or all (default). |
--item | --item <name> | --item ywc-commit | Score a single skill/agent instead of the whole root. Rejected when combined with --ci. |
--format | --format <fmt> | --format markdown | Output format: json (default) or markdown. |
--ci | flag | Mechanical-only; compares against the committed evals/history.mechanical.json baseline and exits non-zero on any per-axis mechanical regression. Used by the CI gate. |
Skill / orchestrator arguments (interpreted by this skill; not score.py flags):
| Parameter | Format | Example | Description |
--mode | --mode <m> | --mode mechanical | mechanical (script only), judge (model pass only), or full (both, default). |
--advisor-budget | --advisor-budget <n> | --advisor-budget 3 | Max Opus judge escalations for genuinely ambiguous activation/boundary calls. Default 5. |
Scoring Model
Each item is scored on six axes, each 0–5, combined by fixed weights into a /100 total. Skills and agents use different axis sets (an agent has no README locale set; a skill has no tool-grant). The full 0–5 banding per axis lives in the references — do not inline the rubric tables (they are >30-line static content, Tier 3 per ywc-skill-author A14).
Skill axes — see references/skill-rubric.md for the 0–5 banding:
| Axis | Weight | Tier |
| S1 Activation accuracy (trigger precision/recall + collision) | 30 | collision = mechanical, precision/recall = judgment |
| S2 Structure compliance (10-rule mechanical subset of ywc-skill-author A1–A14; A5/A10/A12/A13 out of mechanical scope) | 15 | mechanical |
| S3 Behavioral efficacy (does following SKILL.md produce the right outcome) | 20 | judgment |
| S4 Token economy (Tier-1 leanness, ≤500 body, Tier-3 extraction) | 10 | mechanical |
| S5 Consistency & integrity (locale set, pointer resolution, dangling refs) | 15 | mechanical |
| S6 Catalog fit (redundancy / gap vs siblings) | 10 | judgment |
Agent axes — see references/agent-rubric.md for the 0–5 banding:
| Axis | Weight | Tier |
| A1 Role-boundary clarity (crisp, non-overlapping scope) | 20 | judgment |
| A2 Dispatch accuracy (orchestrator picks it at the right time) | 25 | collision = mechanical, rest = judgment |
| A3 Tool-grant minimality (least privilege) | 15 | mechanical |
| A4 Output-contract compliance (Status/Next-action shape) | 15 | mechanical |
| A5 Model-tier appropriateness (Opus/Sonnet/Haiku fit) | 15 | mechanical heuristic + judgment |
| A6 System-prompt quality (clarity, anti-rationalization, no vague language) | 10 | judgment |
The activation methodology (how positive/negative/collision cases yield precision and recall) is in references/trigger-eval-method.md. The exact scorecard layout and history.json schema are in references/scorecard-format.md.
Workflow
Step 1: Inventory Targets
Resolve --target to a concrete item list. For all, score claude-code/skills and claude-code/agents only. Do not score codex/skills or codex/agents; use .codex/skills/ywc-codex-toolkit-eval for Codex evaluation. Read evals/history.json (if present) to obtain the previous scores for regression comparison.
Step 2: Mechanical Tier — scripts/score.py
Run the deterministic scorer:
python3 .claude/skills/ywc-toolkit-eval/scripts/score.py --target <root> --format jsonIt walks each item, parses frontmatter and body, and emits per-axis mechanical sub-scores plus the raw signals (body line count, locale-set completeness, dangling-reference list, sibling description n-gram overlap pairs, agent tool list, contract-marker presence, trigger-case coverage, prose_lint).
prose_lint reports no-op exhortations ("write clean code") and non-directive phrasing ("X is recommended" where the body should command). It is warning-only: it never gates an axis or the CI baseline — treat its hits as backlog candidates, not deductions. Lines carrying a concrete anchor and lines that quote guidance (tables, block quotes, fenced code) are excluded by design, so a low hit count means the bodies are clean rather than that the check is off. Do not hand-compute these — read them from the script output. Stop and fix the script if a signal is obviously wrong rather than overriding it in prose.
Step 3: Judgment Tier — Agent Judge Pass
Skip this step when --mode mechanical or --ci. Otherwise spawn parallel judge subagents (Task tool, model: sonnet; escalate only genuinely ambiguous calls to model: opus within --advisor-budget). One judge per item per judgment axis:
- Activation judge (S1 / A2) — given the item's
descriptionand the case set inevals/trigger-cases.json, predict activation for each positive / negative / collision prompt; compute precision & recall. A collision case that the item wins when a sibling should own it is a false positive. Read the mechanicalsignals.coverage.sufficientfirst: when it isfalse(the item is below the ≥3-positive / ≥2-collision floor), return S1/A2 as"unmeasured"with a one-line reason instead of a fabricated precision/recall — never carry forward a prior run's score. An unmeasured item enters the prioritized backlog regardless of its total, because its heaviest axis is unmeasurable until the fixture is backfilled. The floor counts only independently-sourced cases (source: session-trace/user-prompt): a case authored from the item's own description is judged by a judge reading that same description, so it cannot fail.positives_total/collisions_totalreport every case regardless of provenance — a wide gap between the two pairs is the backlog item, not a licence to score anyway.- Run the activation judge 3 times on the same input and take the majority verdict. The judge is an LLM and a single pass is one data point; without an isolated execution harness, repetition is the only available defense against nondeterminism. If the three runs disagree, do not average them — record "descriptions genuinely ambiguous" and treat the spread itself as the S1 signal (see references/trigger-eval-method.md Determinism Note).
- Behavioral judge (S3) — prefer measurement over reading. When the item has fixtures under
evals/fixtures/, do not judge: runscripts/runner.pyover them and take S3 from the observed reliability (passes / trialsat 6 trials), recordings3_source: "runner". Band 4 is unreachable at 6 trials — 5/6 lands in band 3 and 6/6 in band 5 — so a measured 4 is never a valid result (see references/skill-rubric.md S3). When the item has no fixture, fall back to reading the body — if an agent followed only this body on the canonical scenario, would the output satisfy the stated purpose? — score the gap citing the under-specified step, tag it(read-only), and records3_source: "read-only". A read 4 and a measured 4 are different claims; never merge them. If a fixture exists but every trial errored, S3 is"unmeasured", not0. Runner results never enteraxes.S3— they reach the scorecard and backlog only, so--cistays deterministic. - Boundary / fit judge (S6 / A1) — compare the item's responsibility against its nearest siblings; flag overlap (two skills that would both fire) or gap (a real need no skill owns).
- Prompt-quality judge (A6) — score persona clarity, anti-rationalization coverage, and vague-language density in the agent body.
Each judge returns a 0–5 score with a one-line justification and a file:line citation. A score without a citation is rejected — re-run that judge.
Step 4: Aggregate the Scorecard
Combine mechanical and judgment sub-scores per the weights above into a /100 total per item. Write evals/scorecard.md using the layout in references/scorecard-format.md: one row per item, per-axis scores, total, and the weakest-axis tag.
Do not hand-compute the totals or the run statistics. score.py owns those rules: item_total(axes, weights) returns None when any axis is unmeasured, and build_history_row(scored, weights) produces the history.json entry with items.<name>: null for unmeasured items, unmeasured[], measured, s3_source, and a mean_total / below_threshold computed over measured items only. Deriving these in prose is how the two documents drift apart.
Step 5: Prioritized Improvement Backlog
Rank items ascending by total. Break ties by the activation axis (S1 / A2) — a skill that mis-fires hurts the whole catalog, so it outranks an internally-weak but correctly-scoped one. Emit the top N as a numbered backlog, each line naming the item, its weakest axis, the file:line evidence, and the concrete fix. The fix for a structural axis points at ywc-skill-author; the fix for an activation axis names the specific trigger/anti-trigger edit.
Step 6: Persist History
Append one row to evals/history.json: timestamp, per-item totals, and catalog-level aggregates (mean total, count below threshold). This is what turns a one-off audit into a trend. Never overwrite prior rows.
Output Format
# Toolkit Scorecard — 2026-06-12 (illustrative example — numbers are not current)
## claude-code/skills (38 items, mean 82/100)
| Item | S1 | S2 | S3 | S4 | S5 | S6 | Total | Weakest |
|------|----|----|----|----|----|----|-------|---------|
| ywc-commit | 5 | 5 | 4 | 5 | 5 | 4 | 94 | S3 |
| ywc-tech-research | 3 | 4 | 3 | 3 | 4 | 2 | 64 | S6 |
| ywc-sample-nofx | 5 | 5 | ? | 5 | 5 | 4 | — | S3 |
| ywc-sample-mech | 5 | 5 | · | 5 | 5 | · | — | — |
Notation: `?` = cannot be measured (no fixture — backfill it) · `·` = not measured
this run (judgment tier skipped — re-run `--mode full`) · `—` = no total, because
an axis is `?` or `·`. `?` and `·` are never merged: one is coverage debt, the
other is just a cheaper run mode. An item with a missing axis has at most 80
weight available, so any `/100` it reported would understate itself — it records
`items.<name>: null` and is excluded from `mean_total`, but still enters the
backlog. Full legend: [references/scorecard-format.md](references/scorecard-format.md).
## Prioritized Backlog
1. ywc-tech-research (64) — S6 catalog fit: description overlaps ywc-product-review
on "compare options"; SKILL.md:1. Fix: add "Do not use for product/business
comparison (use ywc-product-review)" anti-trigger.
2. ywc-sample-nofx (—) — S3 unmeasurable: no fixture under evals/fixtures/.
Fix: add one happy_path and one negative case, then re-run.Validation
Before declaring an evaluation cycle complete, verify:
- Every item in the resolved target list has a row (no silent skips) — count rows == count items.
- No total is derived from the mechanical tier alone unless
--mode mechanicalwas explicit. - Every judgment-axis score carries a file:line citation.
- The prioritized backlog has at least one concrete, actionable fix per listed item.
evals/history.jsongained exactly one new row; no prior row was mutated.- Any item whose mechanical
signals.coverage.sufficientisfalsehas its S1 (skill) or A2 (agent) recorded as "unmeasured" — not a number, and not a value carried forward from a prior run. Every unmeasured item appears in the prioritized backlog regardless of its total. - The activation judge ran 3 times per item and the reported score is the majority verdict; a three-way disagreement was recorded as "descriptions genuinely ambiguous" rather than averaged.
- In
--cimode, exit code reflects regression (non-zero if any per-axis mechanical score dropped vs the committedhistory.mechanical.jsonbaseline).
Common Mistakes
- Reporting mechanical-only totals as the final score — the script is ~40% of the weight. Always note when the judgment tier was skipped, and never present the partial total as the item's quality.
- Scoring a skill in isolation — S1/S6 are relative to siblings. Re-derive collisions against the current catalog every run; a description that was unique last quarter may collide today.
- Letting the rubric and reality drift silently — if a high-quality item scores low, the rubric is wrong; fix the rubric (in references) and record it, do not patch the number by hand.
- Treating the scorecard as the deliverable — the prioritized backlog is the deliverable. A scorecard with no ranked next action is a dashboard nobody acts on.
Integration
- Upstream:
ywc-skill-author— its A1–A14 rules ARE the S2 sub-rubric; this skill scores compliance, it does not redefine the rules. Reference by name; never duplicate the rule text. - Downstream: feed the prioritized backlog back into
ywc-skill-author(for structural/authoring fixes) or a normal edit pass (for description/trigger fixes), then re-run this skill to confirm the score moved. - Pairs with:
scripts/validate.sh(binary structural gate — this skill is the graded layer on top) andywc-impl-review(which reviews application code; this skill reviews the toolkit's own skills/agents).
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

