agentleFS
syangasyanga/astack/skills/hillclimb-council/SKILL.md

hillclimb-council

Skill0 starsChanged 57 days agoMITView on GitHub

Hillclimb a quality that models can judge but no command can measure, such as UI design or writing, scored by a blind council.

The file

We split this skill into the rules it gives an agent

7 rules. About 0 are common, 7 are distinctive, and 0 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈7

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 0

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Follow hillclimb with a council vote as its metric.

    line 9
  2. Distinctive:

    Add a scripted layout check, such as no horizontal overflow or overlapping controls, to the correctness checks.

    line 18
  3. Distinctive:

    Ask the user, with the question tool, to approve the brief and, for visual work, the screen list.

    line 20
  4. Distinctive:

    Every listed flaw's quote is present in one other ballot version's text, the same version for all.

    line 31
  5. Distinctive:

    If every introduced flaw points to the same part, remove that part.

    line 40
  6. Distinctive:

    After a round that keeps nothing, pivot per hillclimb step 6 and still target queued flaws: a bundle of other queued flaws, or a larger …

    line 44
  7. Distinctive:

    Add these to the council review section that council puts in the PR:

    line 63

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Hillclimb council

Follow hillclimb with a council vote as its metric. The sections below replace the hillclimb steps they name. Hillclimb owns the rest.

Set up the vote

This replaces the workload, repro, and metric in hillclimb step 1, and the measurement harness in step 2.

  1. Write the judge brief from the request: what the judges assess, for example the design quality of these screens. List the fixed constraints, such as brand colors or placeholder test data, only from the user's limits, the project instructions, and the test setup. Tell the judges not to count them as flaws. Define a major flaw as one that hurts the audience's understanding, next action, or trust, and every other flaw as minor. Describe what a 4, 6, 8, 9, and 10 look like by their major and minor flaws, so fresh judges use the same scale each round.
  2. Choose the input, and add any reference examples the user supplies, such as competitor screenshots. For text, the judges read the files. For visual work:
    • Write a capture script that takes the same screenshots every round, for example through t3-preview. Freeze animations and the page clock, and seed test data from a script, so an untouched screen captures identically every round. Capture versions and screens concurrently, with a limit on open pages, and confirm that a capture at that concurrency repeats byte for byte. Give the script a screen filter for checking a fix, and use a full capture for every vote and correctness check. Save the text each screenshot shows outside the ballot. Before the baseline vote, confirm that each screen's saved text holds the text its screenshot shows and no more, such as no text from a closed disclosure.
    • Add a scripted layout check, such as no horizontal overflow or overlapping controls, to the correctness checks. Run it over every page at each supported width, including pages the screenshots omit.
  3. Write a quote check, a script that lists, per member, the flaws whose quote is absent from a given version's text, by default the text of the version each flaw describes. A quote of n words is present when some run of consecutive words in that text differs from it, ignoring case and punctuation, by at most n/5 words changed, added, or removed, rounded down. Count each character of a script written without spaces, such as Chinese, as a word. A quote matches a text exactly when it appears there, ignoring case and punctuation, with no word changed, added, or removed. A quote is also absent when it matches another ballot version's text exactly but matches the given version's text only within that allowance. A flaw with no quote counts as absent. Give the check two options: limit each flaw to the screen or file it names in every text it compares, and read text that is not a ballot version, such as the state after some of a bundle's parts. The check reads the files for text work and the saved screen text for visual work, and skips versions with neither, such as reference screenshots.
  4. Ask the user, with the question tool, to approve the brief and, for visual work, the screen list. When the user delegated this pick, record both with the task's other decisions instead. Then freeze the brief and references.
  5. Run a baseline vote, per the vote steps, on the starting version and the references alone. Its scores are the baseline, and its flaws, less those the quote check lists per vote step 7, fill the first flaw queue. Look up each flaw the quote check lists, with and without the limit option, in the input. If any quote does appear, fix the saved text or the check, and rerun only the check. If a member lists flaws and the quote check lists all of them, change how vote step 4 asks for quotes, leaving the approved brief unchanged. From each judge's time per subdirectory, estimate its time on a round's ballot, which holds up to two more subdirectories than the baseline's. If a judge times out or its estimate passes half the 30-minute limit in vote step 3, skip its retries and change the input format to a smaller one, such as a lower image size or a more compact file type. If a judge fails after its retries, fix its command, keeping its model and effort. After a change to how vote step 4 asks for quotes, to the input format, or to a judge command, rerun the baseline, at most twice. Only the last baseline vote counts toward the stop rules. If the content itself would have to change, or the second rerun still fails, ask the user. Then freeze the input format, the judge commands, any capture script, and the quote check. Base round 1 on a bundle from the last baseline vote's queue per vote step 7.

The target defaults to 9 out of 10 and the budget to 10 rounds. Judges re-anchor their scale each vote, so measure the target on the baseline scale. In a vote, a member's baseline-scale score for the current best is its baseline score for the starting version plus its score for the current best minus its score for the starting version. After a keep, the current best is the version just kept. A member without a baseline score uses its starting-version score from the first vote in which it scores.

Vote each round

  1. Fix broken inputs, such as an empty file or a blank screenshot, before the vote. For visual work, capture the new, current best, and starting versions in one run against the same backend and seeded data. Instead of capturing the current best or the starting version, you may copy an earlier full capture of identical source files that passed the correctness checks, taken with the frozen capture script and seed and with the same backend commit and browser version. Record each capture's source commit, backend commit, and browser version. For visual work, find regressions only in the captures and through the correctness checks. Fix visible regressions, such as overflowing content, and confirm that screens the change does not touch match the current best's. If the current best's capture is a copy and an untouched screen differs from it, capture every version again in one run. In a bundle, fix each regression inside the part that caused it, so the work builds after each part. For visual work, when the parts' diffs do not show which part caused a regression, capture the state after each part in one run.
  2. Make a new ballot directory with mktemp -d outside any Git checkout. Put each version in its own randomly named subdirectory: the new version, the current best, each reference, and, after the first keep, the starting version. When versions have byte-identical files or captures, put only one of them in the ballot and give the others its scores and flaws. Make a second ballot, for a confirm vote, with each vote on a new version or a trim. A confirm vote and a vote on a simplification get none. Give the second ballot its own mktemp -d directory and new random subdirectory names. Record in the label map, before starting judges, which ballot is the confirm ballot. Keep the label map, prompts, and judge outputs outside the ballot directory.
  3. Even in T3 Code, start judges through their CLIs instead of delegate_task. Start a fresh judge for each of council's members in the ballot directory, with that member's model and effort and the read-only invocation in provider execution. When a vote has a confirm ballot, start the judges for both ballots at the same time. Add --skip-git-repo-check for Codex. Allow each judge 30 minutes. For a judge command that prints progress while it runs, count 10 minutes without a new line as a failure, on every attempt. Set any host command timeout longer than the judge's limit. These limits replace the nine-minute default in provider execution. Outside the votes that setup step 5 runs, a judge quotes another version when all of these hold for one subdirectory, with each flaw limited to the screen or file it names:

    • The quote check lists at least two of the flaws the judge names there, and more than half of them.
    • Every listed flaw's quote is present in one other ballot version's text, the same version for all.
    • For visual work, none of those quotes matches exactly the text that subdirectory's screenshots show.

    If a judge fails, times out, lacks a score for any subdirectory, or quotes another version, retry that member up to twice, each time with a fresh judge in the same ballot directory and a 60-minute limit. If its last attempt still quotes another version, discard that member's scores and flaws for the vote, so the member has no score. Keep the other members' scores.

  4. Give each judge the brief with no repository paths. Tell it to read only the ballot directory and to judge only by the brief, ignoring any other style instructions it holds. Ask, for each subdirectory, for a score out of 10 to one decimal place and a list of up to 8 flaws, each marked major or minor, with major flaws first and each group ranked by damage. Each flaw names the file in that subdirectory where it appears, such as a screenshot, quotes one unbroken span of the text at that spot or nearest it, and gives the change that would fix it. The judges must not learn the target, earlier scores, the round number, or which subdirectory holds which version.
  5. A vote favors the new version when no member scores it below the current best and at least one scores it above, by any margin. Keep it when every member's gain is at least 0.5. When the vote favors it but some member's gain is below 0.5, keep it only if its confirm vote, the second ballot from vote step 2, favors it too. When the vote it confirms is inconclusive or needs no confirm vote, stop the confirm ballot's judges and treat the confirm vote as never run. A confirm vote counts as after the vote it confirms, whatever order they finish in. This keep rule replaces the noise threshold in hillclimb steps 5 and 8. In hillclimb step 6, keep a simplification unless some member scores it more than 0.3 below the current best.
  6. When vote step 5's rule, after any confirm vote, does not keep a round's new version, trim it if the round has had no trim, all the round's votes were conclusive, and the round's bundle has more than one part. Otherwise revert the round. To trim:
    • Run the quote check on the new version's major flaws from the round's last vote, limiting each flaw to the screen or file it names. Treat that screen or file as empty in any state that lacks it. A flaw is introduced when its quote is present in the new version and absent from the current best. If no flaw is introduced, revert the round.
    • Apply the parts one at a time, in order, to a copy of the current best, and run the quote check the same way after each. A flaw points to the first part after which its quote is present. Use the files for text work. For visual work, capture the state after each part in one run with the capture script, and use each state's saved screen text. If a state fails to capture, or the text after the last part differs from the new version's saved text, capture every state and the new version again in one run, once. Revert the round if any state fails to build, or if either problem repeats.
    • If every introduced flaw points to the same part, remove that part. Otherwise revert the round.
    • Build the rest from the current best and the other parts. If they do not apply, revert the round. Otherwise capture the rest as in vote step 1 and run the correctness checks, without changing its content. Revert the round if the rest is byte-identical to the current best, fails a check, shows a visible regression, or has a broken input that only a content change would fix. Otherwise run vote steps 2 to 5 on the rest as a trim vote, and revert the round unless vote step 5's rule keeps the rest.
    • Record the trim and its votes in the round's row and experiment record.
  7. After the keep decision, rebuild the flaw queue from the current best's flaws in the round's last vote, drop the flaws the quote check lists, and mark the flaws already attempted, except as the inconclusive rule below says. If the queue is empty, follow the empty-queue stop. Otherwise base the next round on a bundle of fixes for three queued flaws, or every queued flaw if fewer. Count each flaw once, even when several judges name it, and prefer unattempted flaws, then major flaws, then flaws that several judges name. Take each member's baseline-scale score for the current best from the round's last vote, or from a Stop rerun of that vote if one ran, did not end the run, and the member scored in it. When some member's score reaches the target, rank flaws named by a member below the target after unattempted flaws and ahead of major flaws. The bundle is hillclimb's one change. Put each fix in its own part, such as a commit or patch, so a trim can remove it alone. Fixes that edit the same text or depend on each other share one part. Give each part its own implementer and worktree from the current best, with the part's flaws and the brief's fixed constraints, then apply the parts in order, confirming that the work builds after each. If a part does not apply, rebase it in its worktree onto the parts before it. Give each build that runs beside another build or a capture its own ports and backend. When commits are in scope, a kept round keeps one commit per part. For visual work, also give each implementer the capture script, have it check each fix with the screen filter, and have it fix broken inputs and visible regressions per vote step 1.
  8. After a round that keeps nothing, pivot per hillclimb step 6 and still target queued flaws: a bundle of other queued flaws, or a larger change for the same flaws that restructures the screens or files that hold them. Build it per vote step 7.

If a member still has no score after its retries, record the vote as inconclusive and continue. In a round, keep nothing, mark as attempted only the flaws whose part a trim removed, if any, and skip the vote step 8 pivot. The round still counts against the budget. If the user lets an inconclusive baseline vote stand, start round 1 from the other members' flaws.

Stop

This replaces hillclimb step 7. After any requested minimum rounds, stop at the first of:

  • Every member's baseline-scale score for the current best reaches the target, and every member scores the current best above every reference. Check this with the baseline vote's scores, and after each round's keep decision with that round's last vote. Each time it holds, rerun that vote outside the budget: the baseline vote, whose new scores replace the baseline for each member that scores in it, or a vote on the current best, the starting version, and the references. Stop only if it holds there too. While a rerun after a round runs, build the next round's bundle from that round's last vote in worktrees apart from the run's checkout. If this or another stop holds, abandon that build and record it in the trail as not voted. Otherwise, rebuild the bundle when the rerun changes vote step 7's selection, then apply it to the run's checkout and start its votes. When it and the budget stop fall on the same round, report the target stop.
  • The budget runs out.
  • The flaw queue is empty after the baseline vote or a round's keep decision, even before any requested minimum rounds. Rerun that vote as in the target stop, and rebuild the queue from its flaws as the first sentence of vote step 7 does. When the target stop also holds, run one rerun for both, and report the target stop if it holds there. Stop if the rerun is conclusive and the queue is still empty, and report that the judges named no flaw that passed the quote check. If the rerun is inconclusive, ask the user.
  • Two votes in a row, counting confirm and trim votes, end without a score from the same member, even before any requested minimum rounds.

At any stop, the last conclusive vote replaces the frozen measurement in hillclimb step 8's recheck, so rerun only the correctness checks there, then finish that step. If no vote is conclusive, report the recheck as unavailable. After a budget stop, ask the user whether to extend the budget. If the user extends it, continue the loop on the same branch and repeat hillclimb step 8 at the next stop.

If hillclimb step 8 finds a moved base, then instead of reusing the last conclusive vote, the recheck runs vote steps 1 to 4 with the rebased current best as the new version and the new base as the current best, plus the starting version and the references. For visual work, capture every version fresh, against a backend on which the new base passes its correctness checks. Vote step 5's rule, with its confirm vote, serves only as hillclimb step 8's noise test, and the recheck keeps or reverts nothing and runs no trim. If the recheck is inconclusive, ask the user before opening the PR. Later rounds use the new base as the starting version. Each member's baseline score for the new base is its baseline-scale score for the new base in the recheck's first vote.

Report

Add these to the council review section that council puts in the PR:

  • The brief.
  • The score table. Write each cell as <new> vs <best>, followed after the first keep by the starting version's score and then any reference scores, such as 8.3 vs 7.9, start 6.4, ref 8.8. Write a round's votes in one cell in the order they count, per vote step 5, and prefix the trim vote, not its confirm vote, with trim, such as 8.3 vs 7.9 / 7.8 vs 7.9 / trim 8.2 vs 7.9 / 8.1 vs 7.9. For a version left out as identical, write same as <version> in place of its score, such as 8.3 vs 7.9, start same as best. Add a row per member with its baseline-scale score for the current best after each round, and a row that names the flaws each round targeted and any part a trim removed.
  • Each judge failure, including failures in unused confirm votes before you stopped their judges: the member, the vote, the attempts, and the error or the other version the judge quoted.
  • For visual work, before and after screenshots attached per open-pr.

In hillclimb's reply, give scores from the recheck instead of deltas: the accepted version, the starting version, any new base, and each member's baseline-scale score for the accepted version. Write a version left out as identical as same as <version>, as in the score table. Still list any dropped change.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.