agentleFS
kadeskikadeski/skills/.claude/skills/hillclimb/SKILL.md

hillclimb

Skill1 starsChanged 3 days agoMITView on GitHub

Improve one skill in this repo against its eval cases in evals/<skill>/, one change per round, keeping a change only when train and test scores rise beyond noise. Use when asked to hillclimb, tune or improve a skill with its evals, or to cut a skill's cost at the same score.

15 other projects use a version of this.

Adopted by

Most recently taken up by

Used with

Projects with it also use babysit-pr, retro and 1 more.

Other files people use alongside this one.

Files found in the same public projects as a version of this one, more often than they turn up anywhere else. Each project counts once, large collections count for less, and this file's own repository is left out.
  • owidSkill

    babysit-pr

    Spawn a background agent that babysits an open PR — triggers a Codex review, watches CI, judges and fixes valid findings, replies to and resolves review threads, and loops to a cap. Never merges. Trigger when the user asks to "babysit the PR", "ask for codex review(s) and watch CI", or after pushing a substantial chunk to a PR branch.

    Also in 3 projects that use this

  • AWSSkill

    retro

    Conduct a retrospective on a coding session.

    Also in 4 projects that use this

  • GoogleSkill

    grill-me

    A relentless interview to sharpen a plan or design.

    Also in 4 projects that use this

The file

We split this skill into the rules it gives an agent

9 rules. 0 are common, 4 are distinctive, and 5 are very specific to this project.

  • 0

    Common

    Most of the 14 edited versions we read keep these.

  • 4

    Distinctive

    This author's own choices. Worth a look.

  • 5

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    Never put eval content into the skill: No case topics, numbers, file names, wording or answers in SKILL.md.

    line 14
  2. Distinctive:

    Do not edit cases or graders mid-climb: If one is wrong, stop and say so: changing the ruler resets the baseline.

    line 16
  3. Very specific:

    Prefer making an existing instruction clearer or removing one over adding text.

    line 17
  4. Distinctive:

    Check the ruler before climbing:

    line 23
  5. Distinctive:

    If the suite is above 0.95, report that it is saturated and stop: it needs harder cases, not a better skill.

    line 26
  6. Very specific:

    Read the train misses: grades.json in evals/results/<run>/train/*/run*/, then the run's files under home/ and its trace.jsonl to see why.

    line 31
  7. Very specific:

    Keep the edit if compare says Keep.

    line 35
  8. Very specific:

    Add an entry to evals/<skill>/hillclimb-log.md: date, root cause, the edit in one line, train and test before and after with bands, kept or …

    line 36
  9. Distinctive:

    Stop after the round limit (default 5), or after 3 rounds in a row with nothing kept.

    line 38

“Common” is measured: kept by at least 6 in 10 of the 14 edited versions of this file we could read. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Hillclimb

Improve plugins/<plugin>/skills/<skill>/SKILL.md against evals/<skill>/. Read evals/README.md first for how runs and grading work. If it says the skill's evals are paused, stop and say so: a run would cost money and grade nothing.

Rules that keep the score honest

  • Never read the test set. Not evals/<skill>/test/, not evals/results/*/test/. run.py prints only the test aggregate; that is all you get.
  • Never put eval content into the skill. No case topics, numbers, file names, wording or answers in SKILL.md. A change must be a general rule a new learner on a new topic would benefit from.
  • One change per round, aimed at a root cause, so a score change can be traced to it. Rewording without a reason is not a change.
  • Do not edit cases or graders mid-climb. If one is wrong, stop and say so: changing the ruler resets the baseline.
  • Prefer making an existing instruction clearer or removing one over adding text. SKILL.md length is a cost every run pays.

Setup

  1. Goal: the user's, or score by default. For cost: same score with a cheaper --model or a shorter skill.
  2. Baseline: the newest full run of the current skill in evals/results/ (the skill_sha256 in its summary.json, a hash of every file in the skill folder, matches the current one), else evals/run.py run <skill>.
  3. Check the ruler before climbing:
    • evals/run.py rejudge <baseline>. If more than 1 in 10 verdicts flip, the claims are too vague: report which, and stop.
    • A case that scores 1.0 on every run has no headroom; one that swings widely between runs is noise. Note both.
    • If the suite is above 0.95, report that it is saturated and stop: it needs harder cases, not a better skill.
    • With fewer than 5 test cases, a real fix rarely moves test beyond noise, so most rounds will end Unclear. Say so before starting.

Each round

  1. Read the train misses: grades.json in evals/results/<run>/train/*/run*/, then the run's files under home/ and its trace.jsonl to see why. Look across cases for the same cause.
  2. Name the root cause in one sentence, as a gap or ambiguity in SKILL.md, not as a fact about a case.
  3. Make one edit to SKILL.md.
  4. evals/run.py run <skill>, then evals/run.py compare <previous kept run> <new run>.
  5. Keep the edit if compare says Keep. If it says Unclear, keep only when the edit is plainly general; three Unclear keeps in a row is overfitting: revert to the last clear Keep. Otherwise git checkout plugins/<plugin>/skills/<skill>/SKILL.md.
  6. Add an entry to evals/<skill>/hillclimb-log.md: date, root cause, the edit in one line, train and test before and after with bands, kept or reverted.

Stop after the round limit (default 5), or after 3 rounds in a row with nothing kept. On a stall, edit nothing: sort each remaining train miss into skill gap, bad case, bad grader or noise, with the evidence, and report.

Finish

Report the start and end scores for train and test with their bands, and each kept edit. Recommend merging only if test rose beyond noise. Merging a SKILL.md change means bumping the version in each of the plugin's manifests and running scripts/check.sh. Do not commit unless asked.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.