agentleFS
jgamaraalvjgamaraalv/delivery-loop/.claude/skills/tune-skills-and-agents/SKILL.md

tune-skills-and-agents

Skill0 starsChanged 5 months agoMITView on GitHub

Analyze, test, and improve skills, subagents, and their context layout — what lives HOT vs COLD, whether a grep/file index earns its cost, why a subagent won't read its references, cutting per-turn token bloat, and A/B testing a prompt or rule change.

The file

We split this skill into the rules it gives an agent

6 rules. About 0 are common, 4 are distinctive, and 2 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈4

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 2

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Every piece of context lives in one of two states.

    line 15
  2. Distinctive:

    An index inside HOT content is pointless: If the whole file is already in context, there is nothing to "grep to" — the agent has every line.

    line 35
  3. Distinctive:

    The silent-skip failure is the dangerous one: If a critical fact lives *only* in a cold reference and the model's training is stale/wrong, …

    line 48
  4. Very specific:

    Characterize: Read the target (skill, agent .md, CLAUDE.md, the doc).

    line 76
  5. Distinctive:

    A/B test it: when the effect is non-obvious or the user wants proof.

    line 84
  6. Very specific:

    Clean up: If you ran subagent tests, scrub the memory they generated (references/memory-hygiene.md) so it can't contaminate later work.

    line 94

≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Tuning skills, subagents, and their context layout

This skill is about making skills and subagents cheaper per use and more correct, and about not getting fooled by plausible-but-wrong intuitions while doing it. The guidance here is empirically grounded — the headline verdicts below came from A/B tests, and a few of them overturned the "obvious" design. When you apply this skill, you measure before you presume.

The one mental model: hot vs cold

Every piece of context lives in one of two states. Internalize this — every decision flows from it.

statewhere it livescost
HOTCLAUDE.md, an agent's .md body, a skills:-preloaded SKILL.md, the system promptin context every turn / every invocation — paid N times
COLDreferences/*, a blueprint doc, any file reached by Read/Greppaid only when read — a tool call away

The whole game is putting the right things in the right state. Hot is a standing tax; cold is pay-per-use. Optimizing is mostly moving rarely-needed detail from hot to cold, and keeping hot to {what's needed almost every time + what must never be skipped}.

Hard-won verdicts (measured, often counter-intuitive)

Lead with these. Several contradict the natural guess — that's exactly why they're worth stating.

  1. A grep/anchor index over a small, well-structured doc gives ~no retrieval benefit. Markdown headers are already grep targets; an agent greps ## 7. or a keyword on its own. Adding <!-- tag --> anchors + a token index measured worse (more lines read, same tool calls) on docs in the hundreds of lines. Don't add index machinery by reflex. → references/file-indexes.md
  2. An index inside HOT content is pointless. If the whole file is already in context, there is nothing to "grep to" — the agent has every line. Anchors in a hot file are dead weight.
  3. The real token win is the TRIM, not the index. Moving mid-depth prose out of an always-loaded file (CLAUDE.md / agent body) into a cold reference is what actually saves tokens. The index that often accompanies it is usually ceremony. → references/hot-vs-cold.md
  4. Cold references are read reluctantly. On a neutral prompt, a subagent will answer from its hot context + training and not open a relevant cold reference — even one engineered to be needed. It reads when: the fact is clearly project-specific or version-sensitive (a ground truth it knows it lacks), when it senses it can't recall a precise value, or when the prompt induces it. → references/ab-test-harness.md
  5. The silent-skip failure is the dangerous one. If a critical fact lives only in a cold reference and the model's training is stale/wrong, the agent never looks and answers confidently wrong — silently. Fixing this is a rules problem, and the fix is layered and measurable. → references/subagent-verification-rules.md
  6. Provenance forcing is the cheapest robust fix. Requiring an agent to state where a specific claim came from converts a silent confident-wrong answer into a visibly-flagged estimate, even when it still doesn't read the reference. It fired reliably across tests where the read-trigger only fired sometimes. → references/subagent-verification-rules.md
  7. Measure, don't presume — and clean up after yourself. The A/B harness below is how every verdict above was earned. Agent memory contaminates repeat tests; clean it between runs. → references/ab-test-harness.md, references/memory-hygiene.md

When to reach for what

The user wants to…Do thisReference
Decide CLAUDE.md vs reference; cut a long hot fileHot/cold triage + trimreferences/hot-vs-cold.md
Add/judge a grep index, anchors, table-of-contentsApply the index cost test (usually: don't)references/file-indexes.md
Know if a prompt/rule/skill change actually helpedRun the A/B harness with transcript instrumentationreferences/ab-test-harness.md
Fix a subagent that won't read its docs / answers staleAdd the verification + provenance rulesreferences/subagent-verification-rules.md
Run tests on subagents without poisoning future runsClean agent-memory / reflection_store / indexreferences/memory-hygiene.md

Core workflow

Whatever the specific ask, the shape is the same: characterize → hypothesize → A/B → keep what wins.

  1. Characterize. Read the target (skill, agent .md, CLAUDE.md, the doc). For every chunk, ask: hot or cold? Used almost-every-time or occasionally? Load-bearing (must-never-skip) or optional depth? This classification is most of the analysis.
  2. Hypothesize a change, and predict its effect in hot/cold terms. "Move §X to a reference" → saves hot tokens. "Add an anchor index" → predict ~no retrieval gain on a small doc (verdict 1); say so. "Add a verify rule" → predict it fires for project/version-specific facts.
  3. A/B test it when the effect is non-obvious or the user wants proof. Same prompt, change only the one variable (the rule, the skill, the doc layout) → clean causal attribution. Instrument via the transcript, not self-report: count tool calls, lines read (input proxy), correctness, and provenance honesty (claimed source vs actual tool call). Full protocol in references/ab-test-harness.md.
  4. Keep what wins, revert what doesn't, and say what you measured. Don't ship ceremony. If the index didn't help, drop it; if the trim saved tokens, keep it; if a rule fired only partially, report the limit honestly rather than overclaiming.
  5. Clean up. If you ran subagent tests, scrub the memory they generated (references/memory-hygiene.md) so it can't contaminate later work.

Anti-patterns this skill exists to stop

  • Index-by-reflex. Adding anchors/TOC/grep-tokens to every doc "for navigability." Measure first; on small docs it's cost without benefit (verdict 1).
  • Hoarding hot. Letting CLAUDE.md / an agent body accrete mid-depth prose that's needed 5% of the time. That prose is a per-turn tax. Push it cold.
  • Burying must-apply rules cold. A non-negotiable, divergent-from-default, or version-pinned fact placed only in a reference will be silently skipped (verdict 5). Critical → hot, or guard it with a verify/provenance rule.
  • Presuming instead of measuring. "This is obviously better" is how the grep-index almost shipped as dogma. Run the A/B; let the transcript decide.
  • Leaving test memory behind. Subagent runs write agent-memory + reflection_store + index entries that re-inject into later runs. Always clean.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.