llm-language:refine-skills — Skill Auditor & Refiner

Overview

A meta-skill that applies llm-language's self-refine methodology at the skill level. Scans user-owned skills, scores them on 10 quality dimensions (see references/skill-quality-rubric.md), and applies non-destructive fixes (catalog in references/skill-fixer-patterns.md).

Core pattern: Generate → Critique → Refine, but for skills instead of prompts.

Complements the Phase 7 light-touch auto-audit in the main llm-language pipeline. Phase 7 detects candidates during any /llm-language invocation; this skill runs the full deep audit on demand.

When to Use

Explicitly invoke via /llm-language:refine-skills when:

• You want a full audit of all your personal skills
• You've been using skills designed for Opus 4.6 and want Opus 4.7 migration
• Phase 7 auto-audit flagged multiple refinement candidates
• You're adopting a new skill pattern (e.g., effort parameter, adaptive thinking) and want to backport

Optional arguments:

• /llm-language:refine-skills — audit all user-owned skills
• /llm-language:refine-skills <skill-name> — audit a specific skill
• /llm-language:refine-skills --scope=user — only user-level
• /llm-language:refine-skills --scope=project — only project-level
• /llm-language:refine-skills rollback <skill-name> — restore from backup

Safety Guarantees

1. Scoped to user-owned skills — never touches marketplace-owned plugins or Claude Code built-ins
2. Never modifies llm-language itself — recursive self-modification loop is blocked
3. Backup before every auto-fix — .backup-<timestamp>.md sibling file, kept 30 days
4. Audit trail — every fix logged to ~/.claude/projects/<project>/memory/skill-refinement-audit.md
5. Dry-run mode — --dry-run flag shows what WOULD change without writing
6. Approval gate — risky fixes (restructure, rewrite) require explicit user approval
7. Git-awareness — if skill is in a git repo with uncommitted changes, prompts to stash first

Pipeline

Phase 1: Discovery

1. Glob user-owned skills:
   • ~/.claude/skills/*/SKILL.md (user-level)
   • ./.claude/skills/*/SKILL.md (current project)
   • ./packages/**/.claude/skills/*/SKILL.md (monorepo)
2. Exclude:
   • ~/.claude/plugins/marketplaces/** unless repo is owned by user (detect via git remote)
   • ~/.claude/plugins/cache/** (always)
   • llm-language's own skills
   • Claude Code bundled skills
3. Build audit targets list with scope classification

Phase 2: Quick Scan (parallel)

For each target, run cheap checks (regex-based, <500 tokens per skill):

• File line count
• Description length
• Presence of deprecated patterns (temperature/top_p/top_k/budget_tokens)
• Presence of ultrathink literal
• Emphasis marker count
• Frontmatter completeness check

Output: preliminary quality estimate (0-10 scale) + flagged issues.

Phase 3: Deep Audit (selective)

For skills scoring < 8.5 in quick scan, OR user-requested deep audit:

• Read full content
• Apply the 10-dimension rubric from references/skill-quality-rubric.md
• Use WebSearch if needed to verify technical claims
• Generate weighted score + per-dim evidence + fix proposals

Runs in parallel via Agent dispatches (batch of 3-5 skills per round) to keep total time bounded.

Phase 4: Fix Categorization

For each issue found:

1. Classify fix type: SAFE / RISKY / FORBIDDEN (see references/skill-fixer-patterns.md)
2. Estimate impact: quality delta per dimension
3. Estimate risk: could this break existing behavior?

Phase 5: Application

Auto-apply SAFE fixes:

1. Create backup: <skill-dir>/.backup-<timestamp>.md
2. Apply fix via Edit tool
3. Validate: re-read file, confirm diff is as expected
4. Log to skill-refinement-audit.md

Propose RISKY fixes to user:

1. Show diff
2. Ask via AskUserQuestion
3. If approved: apply with backup + log
4. If rejected: log "proposed but declined"

Never touch FORBIDDEN patterns — only report in summary.

Phase 6: Report

Generate a summary:

★ refine-skills — Audit Report ──────────────────
Scanned: {N} skills
Refinement candidates: {N with score < 8.0}
Auto-fixes applied: {N}
Risky fixes proposed: {N}
Read-only findings (marketplace): {N}
Time: {duration}
Audit trail: ~/.claude/projects/<project>/memory/skill-refinement-audit.md
──────────────────────────────────────────────────

## Per-Skill Summary

### ✅ {skill-name} — 9.2/10 (no action needed)

### 🔧 {skill-name} — 7.4/10 → 8.7/10 (auto-refined)
Fixes applied:
- F1: Replaced ultrathink literal with effort=xhigh
- F3: Stripped 5 emphasis markers
- F5: Added effort+allowed-tools frontmatter

### 🛠️ {skill-name} — 6.8/10 (risky fixes proposed — awaiting approval)
Proposed:
- R1: Move lines 180-420 to references/lookup.md (progressive disclosure)
- R2: Rewrite ambiguous section 3 with concrete examples

### 📢 {skill-name} — 5.2/10 (marketplace-owned, read-only)
Flagged issues (report upstream):
- F1: Uses ultrathink literal (3 occurrences)
- F2: Uses budget_tokens (2 occurrences)
Consider opening PR to: {upstream-url}

Phase 7: ROSETTA + Auto-Memory Writeback

1. Update ROSETTA with:
   • Skills user OWNS and their quality trajectory
   • Anti-patterns observed across user's skills (if a fix repeats 3+ times, note as pattern)
2. Write audit trail to auto-memory (type: reference)
3. Update Meta-Reasoner bandit if audit improved outcomes

Fix Catalog (summary)

See references/skill-fixer-patterns.md for full catalog. Quick reference:

SAFE (auto-apply):

• F1: deprecated thinking → effort
• F2: strip deprecated sampling params
• F3: strip emphasis markers
• F4: de-duplicate directives
• F5: add missing frontmatter
• F6: truncate over-long description
• F7: remove manual CoT triggers
• F8: fully-qualify skill refs
• F9: add missing anti-pattern section
• F10: fix YAML syntax

RISKY (propose-only):

• R1: restructure for progressive disclosure
• R2: rewrite vague instructions
• R3: add output format spec
• R4: split multi-purpose skill
• R5: update outdated citations
• R6: consolidate duplicate skills

FORBIDDEN:

• X1: third-party plugin modification
• X2: self-modification
• X3: Claude Code built-in modification
• X4: change skill's core purpose
• X5: remove user-specific content

Invocation Examples

Full audit (all user-owned)

/llm-language:refine-skills

Audit single skill

/llm-language:refine-skills solana-dev

Dry-run (see what would change, no writes)

/llm-language:refine-skills --dry-run

Rollback a previous fix

/llm-language:refine-skills rollback solana-dev

Restores from the most recent .backup-<timestamp>.md (or user-specified timestamp).

Audit focused on Opus 4.7 migration

/llm-language:refine-skills --filter=opus-4-7-incompatible

Only shows skills with deprecated param issues or model-compat problems.

Integration with Main Pipeline

The main llm-language skill runs a lightweight Phase 7 Skill Audit during every /llm-language invocation:

• Only evaluates skills that are RELEVANT to the current task (from Phase 1 registry)
• Applies only F1, F2, F3 (deprecated syntax, emphasis strip) if issues found
• Flags other issues to Summary Banner as refinement candidates
• Logs to ROSETTA patterns

To run the full deep audit on all skills, invoke this sub-skill explicitly.

Token Cost Estimate

Use effort: xhigh by default (Opus 4.7 guidance for agentic work).

Anti-Patterns

Failure Modes

If a fix application fails:

1. Restore from backup (created pre-fix)
2. Log failure to audit trail with error details
3. Skip that skill, continue with others
4. Report failures in final summary
5. Exit non-zero (so harness knows something went wrong)

References

• references/skill-quality-rubric.md — 10-dimension scoring rubric
• references/skill-fixer-patterns.md — full catalog of safe/risky/forbidden fixes
• Main SKILL: llm-language/skills/llm-language/SKILL.md — Phase 7 description
• Main rubric: llm-language/skills/llm-language/references/scoring-rubric.md