tune-skills-and-agents
Skill0 starsChanged 5 months agoMITView on GitHub
Analyze, test, and improve skills, subagents, and their context layout — what lives HOT vs COLD, whether a grep/file index earns its cost, why a subagent won't read its references, cutting per-turn token bloat, and A/B testing a prompt or rule change.
In this repository63 other files it gives its agents.
- .claude/skills/nestjs-best-practices/AGENTS.mdAGENTS.md
- accessibility-a11ySkill
- architecture-blueprint-generatorSkill
- backend-delivery-loopSkill
- backend-designSkill
- backend-dev-guidelinesSkill
- backend-testingSkill
- backend-to-frontend-handoff-docsSkill
- building-componentsSkill
- bullmq-specialistSkill
- canvas-designSkill
- code-patternsSkill
- components-composition-patternsSkill
- conventional-commitSkill
- core-web-vitalsSkill
- create-svgSkill
- css-architectureSkill
- cypress-authorSkill
- cypress-docsSkill
- cypress-explainSkill
- delivery-loop-retroSkill
- diagnoseSkill
- fastify-best-practicesSkill
- figma-code-connectSkill
- frontend-delivery-loopSkill
- frontend-fundamentalsSkill
- fullstack-delivery-loopSkill
- grill-meSkill
- hunt-race-conditionSkill
- impeccableSkill
- mermaid-diagramsSkill
- nestjs-best-practicesSkill
- nextjs-best-practicesSkill
- nextjs-performanceSkill
- nodejs-backend-patternsSkill
- owasp-security-reviewSkill
- performance-hunterSkill
- playwright-best-practicesSkill
- postgresql-code-reviewSkill
- postgresql-optimizationSkill
- postmanSkill
- prisma-postgresSkill
- pro-state-machineSkill
- race-conditionSkill
- react-best-practicesSkill
- react-performanceSkill
- react-querySkill
- redis-coreSkill
- redis-query-engineSkill
- redis-securitySkill
- refactorSkill
- review-panelSkill
- run-delivery-chainSkill
- spec-drivenSkill
- sql-optimizationSkill
- system-designSkill
- testing-strategySkill
- ui-ux-pro-maxSkill
- vercel-react-view-transitionsSkill
- viteSkill
- The first 60 shown.
The file
We split this skill into the rules it gives an agent
6 rules. About 0 are common, 4 are distinctive, and 2 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈4
Distinctive · estimate
This author's own choices. Worth a look.
2
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 15
Every piece of context lives in one of two states.
- Distinctive: line 35
An index inside HOT content is pointless: If the whole file is already in context, there is nothing to "grep to" — the agent has every line.
- Distinctive: line 48
The silent-skip failure is the dangerous one: If a critical fact lives *only* in a cold reference and the model's training is stale/wrong, …
- Very specific: line 76
Characterize: Read the target (skill, agent .md, CLAUDE.md, the doc).
- Distinctive: line 84
A/B test it: when the effect is non-obvious or the user wants proof.
- Very specific: line 94
Clean up: If you ran subagent tests, scrub the memory they generated (references/memory-hygiene.md) so it can't contaminate later work.
≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Tuning skills, subagents, and their context layout
This skill is about making skills and subagents cheaper per use and more correct, and about not getting fooled by plausible-but-wrong intuitions while doing it. The guidance here is empirically grounded — the headline verdicts below came from A/B tests, and a few of them overturned the "obvious" design. When you apply this skill, you measure before you presume.
The one mental model: hot vs cold
Every piece of context lives in one of two states. Internalize this — every decision flows from it.
| state | where it lives | cost |
| HOT | CLAUDE.md, an agent's .md body, a skills:-preloaded SKILL.md, the system prompt | in context every turn / every invocation — paid N times |
| COLD | references/*, a blueprint doc, any file reached by Read/Grep | paid only when read — a tool call away |
The whole game is putting the right things in the right state. Hot is a standing tax; cold is pay-per-use. Optimizing is mostly moving rarely-needed detail from hot to cold, and keeping hot to {what's needed almost every time + what must never be skipped}.
Hard-won verdicts (measured, often counter-intuitive)
Lead with these. Several contradict the natural guess — that's exactly why they're worth stating.
- A grep/anchor index over a small, well-structured doc gives ~no retrieval benefit.
Markdown headers are already grep targets; an agent greps
## 7.or a keyword on its own. Adding<!-- tag -->anchors + a token index measured worse (more lines read, same tool calls) on docs in the hundreds of lines. Don't add index machinery by reflex. →references/file-indexes.md - An index inside HOT content is pointless. If the whole file is already in context, there is nothing to "grep to" — the agent has every line. Anchors in a hot file are dead weight.
- The real token win is the TRIM, not the index. Moving mid-depth prose out of an
always-loaded file (CLAUDE.md / agent body) into a cold reference is what actually saves
tokens. The index that often accompanies it is usually ceremony. →
references/hot-vs-cold.md - Cold references are read reluctantly. On a neutral prompt, a subagent will answer from
its hot context + training and not open a relevant cold reference — even one engineered to
be needed. It reads when: the fact is clearly project-specific or version-sensitive (a
ground truth it knows it lacks), when it senses it can't recall a precise value, or when
the prompt induces it. →
references/ab-test-harness.md - The silent-skip failure is the dangerous one. If a critical fact lives only in a cold
reference and the model's training is stale/wrong, the agent never looks and answers
confidently wrong — silently. Fixing this is a rules problem, and the fix is layered and
measurable. →
references/subagent-verification-rules.md - Provenance forcing is the cheapest robust fix. Requiring an agent to state where a
specific claim came from converts a silent confident-wrong answer into a visibly-flagged
estimate, even when it still doesn't read the reference. It fired reliably across tests where
the read-trigger only fired sometimes. →
references/subagent-verification-rules.md - Measure, don't presume — and clean up after yourself. The A/B harness below is how every
verdict above was earned. Agent memory contaminates repeat tests; clean it between runs. →
references/ab-test-harness.md,references/memory-hygiene.md
When to reach for what
| The user wants to… | Do this | Reference |
| Decide CLAUDE.md vs reference; cut a long hot file | Hot/cold triage + trim | references/hot-vs-cold.md |
| Add/judge a grep index, anchors, table-of-contents | Apply the index cost test (usually: don't) | references/file-indexes.md |
| Know if a prompt/rule/skill change actually helped | Run the A/B harness with transcript instrumentation | references/ab-test-harness.md |
| Fix a subagent that won't read its docs / answers stale | Add the verification + provenance rules | references/subagent-verification-rules.md |
| Run tests on subagents without poisoning future runs | Clean agent-memory / reflection_store / index | references/memory-hygiene.md |
Core workflow
Whatever the specific ask, the shape is the same: characterize → hypothesize → A/B → keep what wins.
- Characterize. Read the target (skill, agent
.md, CLAUDE.md, the doc). For every chunk, ask: hot or cold? Used almost-every-time or occasionally? Load-bearing (must-never-skip) or optional depth? This classification is most of the analysis. - Hypothesize a change, and predict its effect in hot/cold terms. "Move §X to a reference" → saves hot tokens. "Add an anchor index" → predict ~no retrieval gain on a small doc (verdict 1); say so. "Add a verify rule" → predict it fires for project/version-specific facts.
- A/B test it when the effect is non-obvious or the user wants proof. Same prompt, change
only the one variable (the rule, the skill, the doc layout) → clean causal attribution.
Instrument via the transcript, not self-report: count tool calls, lines read (input
proxy), correctness, and provenance honesty (claimed source vs actual tool call). Full
protocol in
references/ab-test-harness.md. - Keep what wins, revert what doesn't, and say what you measured. Don't ship ceremony. If the index didn't help, drop it; if the trim saved tokens, keep it; if a rule fired only partially, report the limit honestly rather than overclaiming.
- Clean up. If you ran subagent tests, scrub the memory they generated
(
references/memory-hygiene.md) so it can't contaminate later work.
Anti-patterns this skill exists to stop
- Index-by-reflex. Adding anchors/TOC/grep-tokens to every doc "for navigability." Measure first; on small docs it's cost without benefit (verdict 1).
- Hoarding hot. Letting CLAUDE.md / an agent body accrete mid-depth prose that's needed 5% of the time. That prose is a per-turn tax. Push it cold.
- Burying must-apply rules cold. A non-negotiable, divergent-from-default, or version-pinned fact placed only in a reference will be silently skipped (verdict 5). Critical → hot, or guard it with a verify/provenance rule.
- Presuming instead of measuring. "This is obviously better" is how the grep-index almost shipped as dogma. Run the A/B; let the transcript decide.
- Leaving test memory behind. Subagent runs write
agent-memory+reflection_store+ index entries that re-inject into later runs. Always clean.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

