agentleFS
netanelavrnetanelavr/dotcursor/skills/kit-evolve/SKILL.md

kit-evolve

Skill6 starsChanged 27 days agoMITView on GitHub

Grow and prune the harness from repeated corrections. Propose one skill or rule edit at a time, and keep or delete text only after a measured with/without replay of saved tasks. Use for periodic harness maintenance, after sessions with repeated steering, or after a model change.

The file

Show its rules and text

We split this skill into the rules it gives an agent

13 rules. About 0 are common, 8 are distinctive, and 5 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈8

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 5

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    Not a rule from one correction (kit-capture-lesson).

    line 13
  2. Very specific:

    Not catalog hygiene: overlap, stale names, drift (kit-audit-skills).

    line 14
  3. Very specific:

    Not trimming always-on context by host (kit-optimize-context).

    line 15
  4. Very specific:

    Not proving a code change works (kit-prove).

    line 16
  5. Distinctive:

    Not auto-editing. The agent proposes and measures.

    line 17
  6. Distinctive:

    Ask two questions. Usually both are no.

    line 30
  7. Distinctive:

    Check the graveyard first: a buried idea returns only revised, under a new name.

    line 44
  8. Distinctive:

    Default to a scoped rule. A new always-on skill is the last resort: its description costs context on every turn, and the agent still has to …

    line 59
  9. Distinctive:

    KEEP only when all four hold:

    line 74
  10. Distinctive:

    No task collapses: a big drop on any one task → rerun that task once.

    line 77
  11. Distinctive:

    No task that never failed at base starts failing

    line 78
  12. Distinctive:

    Never add or delete text the agent never saw.

    line 99
  13. Very specific:

    Always-on text is a budget: every line is paid on every turn.

    line 100

≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Evolve

Measure harness changes instead of trusting them. One success proves little: the same agent passes a task once and fails it the next time. Collect corrections from real sessions, test one edit at a time on replayed tasks with and without it, and keep it only on a real gain. Test existing skills and rules the same way before deleting them.

What this is NOT

  • Not a rule from one correction (kit-capture-lesson). A single miss goes there; this skill needs repeats and replays.
  • Not catalog hygiene: overlap, stale names, drift (kit-audit-skills).
  • Not trimming always-on context by host (kit-optimize-context).
  • Not proving a code change works (kit-prove). This measures whether harness text changes agent behavior.
  • Not auto-editing. The agent proposes and measures; you approve the plan. Nothing lands without your yes.

The evidence store

Lives in the consuming repo, not in the kit: .kit-evolve/

  • lessons.md - one line per correction. Its header holds the format: # Lessons - YYYY-MM-DD | task | did: ... | should: .... kit-work-loop and kit-capture-lesson append lines during the session. Duplicates stay; repeats are the signal.
  • tasks/<slug>/ - a replayable task: request.txt (the user's words, verbatim), base (commit where it was broken), check.sh (exits 0 when solved), fix.patch (proof it can be solved).
  • journal.md - every cycle and verdict, append-only.
  • graveyard/<slug>.md - killed candidates with the reason, so the same idea is not proposed again unchanged.

1. Collect (during and at the end of a session)

Ask two questions. Usually both are no; then record nothing.

  1. Did the user correct the agent, even once? → one lesson line per correction that has no line yet.
  2. Did something go from broken to working? → offer to save a replay task.

A task is a ruler only if its check separates broken from fixed. Before keeping one, run check.sh in three states: broken must fail; broken + fix must pass; broken + fix minus the fix's test edits must still pass (a replaying agent will not rewrite your test). Fail any state → quarantine the task, repair it once, delete it on a second fail.

Collection never edits skills or rules. That starts at step 2.

2. Find ONE theme

Bar: one lesson logged at least 3 times, or one cause behind at least 2 failing tasks, and no live skill or rule already covers it.

  • Below the bar → stop. Record the cycle as barren. Run no replays.
  • Check the graveyard first: a buried idea returns only revised, under a new name.
  • One theme per cycle. Never batch.

3. Choose the surface

Measured findings behind this table: auto-loaded rules did most of the work. Skills helped only when offered next to the files the task touches. Skills offered on every turn were rarely opened.

Where the fix livesSurface
One folder or areaRule with globs scoped there - it loads itself when the agent works on those files
Must fire no matter whatAlways-on rule, kept short
Spread repo-wide, a paced workflowSkill whose description names the areas and files it serves
One fact the agent breaks anywhereOne line in AGENTS.md, not a skill
Must run mechanically, every timeA hook or CI check, not text

Default to a scoped rule. A new always-on skill is the last resort: its description costs context on every turn, and the agent still has to choose to open it.

4. Propose the one edit

Draft the exact diff for the candidate, under a new name. Show it with the measurement plan: the tasks, the runs per arm, the expected cost. Stop. Wait for yes.

5. Measure with/without

  • Screen: replay the theme's tasks 2 times with the harness as-is (base) and 2 times with the candidate (cand), interleaved. No gain, or the candidate never reached the agent → kill it here; the confirm is not paid for.
  • Confirm: replay every valid task 3 times per arm.
  • Each run starts from a clean state (fresh clone or reset). The task's check.sh decides pass/fail - never your read of the transcript.
  • Record exposure per run: rule loaded, skill listed, skill opened. Credit or blame the candidate only on tasks where it reached the agent.

6. Verdict (from the numbers, not vibes)

KEEP only when all four hold:

  • Net win: runs gained minus runs lost >= 2 (a net of 1 is luck at k=3)
  • No task collapses: a big drop on any one task → rerun that task once; kill if the drop repeats
  • No task that never failed at base starts failing
  • The candidate fired at least once (rule loaded or skill opened)

Anything else → KILL. Bury it in the graveyard with the gates it failed. A KEEP moves the candidate live, and its scores become the new baseline. Journal either verdict. After 2 cycles in a row without a KEEP, stop the loop.

7. Prune (monthly, and after every model change)

Same measurement in reverse. A stronger model may already do what an item teaches, so old verdicts expire.

  1. Plan: the items to test (after a model change: all of them, largest always-on first), their tasks, runs, cost. Stop. Wait for approval.
  2. One item at a time: replay its tasks 3 times with the harness and 3 times without it.
  3. Verdicts:
    • REJECT (keep it): removing it lost runs, collapsed a task, or broke a never-failing task.
    • UNMEASURED (keep it): it never fired in the base runs. Text for a rare case looks like dead weight until the case arrives. Never delete what you never saw fire.
    • ACCEPT (delete it): it fired, and removing it cost nothing measurable. Bury it in the graveyard.
  4. Cheaper moves first, each its own measurement: narrow an always-on item used in one folder to globs; merge overlapping items.
  5. Journal every verdict, and the model it ran on.

Constraints

  • One edit per cycle. The check decides, not impression.
  • Never add or delete text the agent never saw. Unknown exposure blocks the verdict - a tooling failure is not dead weight.
  • Always-on text is a budget: every line is paid on every turn. Prefer globs. If the always-on total grows past a screenful, prune before adding.
  • Generic only: no employer names, private hosts, or work-only tools (AGENTS.md).

Quality checklist

  • Theme cleared the repeat bar and is not already covered
  • Graveyard checked; candidate has a new name
  • Surface chosen from the table (scoped rule by default)
  • Every replay task's check validated in all three states
  • Both arms replayed k times, interleaved, clean state per run
  • Exposure recorded; the verdict reads only exposed tasks
  • KEEP passed all four gates; KILL buried with reasons
  • Journal appended

Hand back

  • Verdict, and the diff that landed (or the graveyard path)
  • Sweep numbers: passes per arm per task, and where the item fired
  • Journal path
  • Next: the next theme, the prune due date, or "harness quiet - nothing to evolve"

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.