Skill Creator

Take skills from "drafted" to "proven" through rigorous multi-layer evaluation.

Installation

bash $HOME/.golems/skills/golem-powers/skill-creator/scripts/install.sh

The installer is idempotent — safe to re-run. It:

1. Symlinks the skill into ~/.claude/skills/skill-creator (golem-powers convention)
2. Ensures $SKILL_CREATOR_REPO/.claude/agents/, $SKILL_CREATOR_REPO/.codex/agents/, and $SKILL_CREATOR_REPO/scripts/ exist as the project-scope home
3. Symlinks the session-miner agent definition into the project-scope agents dir (gates the sub-agent so only skillCreator-family sessions can spawn it)
4. Symlinks the parser into the project-scope scripts dir
5. Symlinks the skill-creator agent (agents/skill-creator.md) into ~/.claude/agents/skill-creator.md — global scope, so any repo's Claude can dispatch Agent(subagent_type="skill-creator"). A hand-placed regular file there is moved to ~/.claude/agents/.skill-creator.md.bak-<YYYYMMDD> first; skill-creator-backup.md is reported and never touched

Every target is under $HOME, so HOME=<dir> bash scripts/install.sh installs into a scratch home (the tests in tests/ do this). Project-scope steps run only when $HOME/Gits/skill-creator is itself a Git checkout. If it is absent or a plain directory, the installer skips those steps and still installs the global skill and agent.

Run bash scripts/install.sh --dry-run to preview without changes. bash scripts/install.sh --help for usage.

Core Loop: DRAFT → EVAL → RED → GREEN → SMOKE → SHIP

Every skill change goes through this pipeline. No exceptions.

1. DRAFT — Understand the Skill

• Read the SKILL.md or MCP tool definition completely
• brain_search for: prior evals, user complaints, known issues, related skills
• Identify: what does this skill claim to do? What's the baseline without it?
• Document intent, inputs, outputs, edge cases
• Classify: capability uplift (may become obsolete) vs encoded preference (durable)
• Load first, then follow (references/placement-vs-wording.md):
  • If the user never says the trigger words, the skill needs placement to load: a ROUTE row, or a hook for a deterministic input shape (a link, a file type). Description wording alone barely moved it in E2/E3 (cheaper-tier 0/30); results are model-specific (skill study 2026-11-02: on the Claude lead tier, /large-plan loaded 10/30 with the old ROUTE row and 30/30 with R3's precisely rewritten row).
  • Once the text is loaded, precise, case-specific wording is the lever. Measure it per case.

2. EVAL DESIGN — Create Structured Test Cases

Design eval cases as structured JSON in evals/evals.json:

{
  "id": 1,
  "name": "descriptive-kebab-name",
  "category": "compliance|structure|quality",
  "description": "What this eval tests",
  "prompt": "The input scenario given to the agent",
  "assertions": [
    {
      "type": "tool_usage|content|negative",
      "name": "assertion-name",
      "description": "What correct behavior looks like"
    }
  ]
}

Cover: happy path, edge cases, failure modes, interaction with other skills. See references/scoring-rubric.md for scoring methodology.

3. RED — Baseline Measurement (without skill)

• Run each eval WITHOUT the skill loaded
• Score baseline performance across all eval cases
• Document baseline scores
• Gate: If baseline >70%, the skill may not be adding value — flag it

4. GREEN — With-Skill Measurement

• Run each eval WITH the skill loaded
• Score delta between with_skill and without_skill
• Gate: Delta <10% = marginal, consider retiring. Delta >30% = clearly valuable.
• Maximum 3 iterations before flagging for human review

5. SMOKE — Live Agent Testing

Two tiers based on skill importance:

Tier A: Static Smoke (all skills)

• Review evals.json assertions manually
• Verify no contradictions between skill instructions and assertions
• Check that strongest discriminators are tested

Tier B: Live Agent Smoke (flagship skills only)

• Run live-eval-runner.sh for top 3 discriminator evals
• Agent routing: bounded Claude in-process/headless tests → claude.subagent.cheap; visible or decision-grade Claude tests → claude.judgment; code tests → codex.implement; audits → Cursor
• Compare captured output against assertions
• Gate: Live delta must be within 15% of static delta
• Results stored in evals/results/live-{date}.json and brain_store'd

See workflows/live-eval.md for the full live eval workflow.

6. SHIP — Package and Document

• Ensure every skill ships with its evals (no eval = no ship)
• brain_store eval results, delta scores, issues found
• Update skill metadata (version, last-eval-date, compliance-score)
• Follow workflows/create-skill.md and workflows/audit-skill.md for SKILL.md structure compliance before shipping structural changes.
• REGISTER a NEW skill — committed ≠ installed. A new golem-powers skill is invisible to every agent until it's symlinked into ~/.claude/skills/<name> (never ~/.claude/commands/; run golem-install, which auto-discovers + links every golem-powers/*/ dir; or symlink the one new skill). Then VERIFY it appears in the available-skills list — committing/merging does NOT register it. (2026-05-30: /weave was committed + merged but unusable by any agent until the symlinks existed. See workflows/create-skill.md "Final Step: REGISTER".)

Declaring Required Environment Variables

Add optional requires: frontmatter only when a skill cannot do its job at all without an environment variable:

requires:
  - FILE_HOST_TOKEN

List variable names (^[A-Z][A-Z0-9_]*$), never secret values. The declaration and runtime behavior are two halves of one contract: lint proves the metadata is well-formed, but the skill body must fail loud when a requirement is missing. Carry this one-line instruction in the body:

If FILE_HOST_TOKEN is unset, tell the user and stop. Do not guess, improvise a fallback, or report partial success.

Eval Methodology

with_skill vs without_skill comparison is MANDATORY. No exceptions.

Eval Result Provenance — Validity Gate

No provenance = no eval. Every static or live eval result MUST record one provenance entry per agent or eval arm with:

model_effective must be the concrete runtime ID, not a floating alias or friendly display label, and uses an ID token with no whitespace. Observation sources begin with the exact case-sensitive labels shown above; append details after :, -, —, or ( when useful.

JSON records use eval-result-provenance.schema.json. Markdown records use an ## Eval Provenance table with these fields plus an agent_or_arm column. Examples inside fenced code blocks are not evidence.

When runtime observation was impossible, the corresponding value and source MUST say NOT DETERMINED; omission, inference, and silent fallback are invalid. This is valid honesty but the checker marks the result NON_COMPARABLE; it cannot support a cross-arm/model delta claim. A NON_COMPARABLE record that contains any score, delta, or positive comparability claim is mechanically INVALID, not merely discouraged by prose. Pre-contract historical results may use provenance: alias-only and are also non-comparable; an alias-only record that carries a score, delta, or positive comparability claim is likewise INVALID. New results may not use that historical escape.

Before publishing or citing a score/delta, run the checker with --require-comparable. Exit 0 means comparable evidence, exit 3 means honest retained history that cannot carry the claim, and exit 1 means invalid input. Run the mechanical provenance check in workflows/audit-skill.md before scoring. An invalid result is not evidence and receives no score, delta, or verdict.

The requested alias is insufficient: cmuxlayer spawn_agent.model has a confirmed defect that silently drops non-alias models while reporting them honored. A requested model records intent, not runtime outcome—the same false surface class as delivered:true or exit 0 without live verification.

Behavioral Compliance Scoring

Scoring Thresholds

Agent Routing for Live Evals

Resolve model/alias with node scripts/model-roles.mjs <role> --field model|alias (one field) from the golems checkout. Keep substitutions in commands; choose effort per /large-plan phase with a why and pass explicitly at dispatch. Never use the mechanical candidate before its bench.

Workflows

References

Integration with Other Skills

• /cmux-agents — spawning live eval agents in cmux panes
• /never-fabricate — Read() every file before reporting results
• /pr-loop — shipping skill changes through the full PR lifecycle

Packaged Sub-Agents (skill-creator family only)

These sub-agents ship in BOTH Claude and Codex formats. They are invokable only from sessions with cwd inside $SKILL_CREATOR_REPO/ (i.e., skillcreatorClaude, skillcreatorCodex, or a repoGolem session rooted there). Sessions in other repos cannot spawn them directly — they must dispatch a skillCreator first.

Canonical source-of-truth lives in this skill's agents/ dir. scripts/install.sh symlinks them into the project repo's project-scope dirs (.claude/agents/ for Claude, .codex/agents/ for Codex). The dual-format packaging means the same sub-agent is available regardless of whether the parent is Claude Code or Codex CLI.

The skill-creator agent itself (agents/skill-creator.md, Claude format only) is the exception to this scope: scripts/install.sh links it into the global ~/.claude/agents/, because it is the agent dispatched from any repo.

Both formats invoke the same deterministic parser at scripts/session-miner.py. The Claude format uses subagent_type="session-miner" via the Agent tool; the Codex format is referenced by name in natural-language prompts (session_miner, mine X to Y).

For when to choose sub-agent vs skill: see references/subagent-vs-skill.md.

Critical Rules

1. Never claim "tests pass" without running them. Read actual output.
2. Every skill ships with evals. No eval = no ship.
3. brain_search BEFORE starting work — someone may have already evaluated it.
4. brain_store AFTER completing work — results, delta scores, decisions.
5. Max 3 iterations per skill before flagging for human review.
6. Read() every file you cite. No citing from compacted summaries.
7. Sequential skill compounding — build skills one at a time, each learning from the last.