agentleFS
oaustegardoaustegard/claude-skills/evolving-with-swarms/SKILL.md

evolving-with-swarms

Skill398 starsChanged 8 months agoMITView on GitHub

Improves a program against an automatic scorer by running rounds of cheap Haiku 5.5 swarms, where each proposer suggests one short idea and one writer per swarm turns the ideas into a new program that is scored and kept in an archive, all metered to a dollar cap through headless claude -p. Use for "evolve this program against my evaluator", "optimize this function's score cheaply", "OpenEvolve-style search on a budget", "AlphaEvolve-style construction search with Haiku", "run a swarm to maximize this score", "search for a better heuristic or construction", or any program whose quality a script can score, when the budget is small. Ships the driver, a gateway with a dollar cap and a usage-window pause, a sandbox hook for model-written code, an example problem and an offline selftest; runs from Claude Code on the web or the desktop.

The file

We split this skill into the rules it gives an agent

2 rules. About 0 are common, 2 are distinctive, and 0 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈2

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 0

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Use the swarm when the budget is small or the goal is "good, cheaply".

    line 36
  2. Distinctive:

    No sandbox only under the earned exception above.

    line 94

≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Evolving with swarms

Each round, scripts/evolve.py runs K swarms in parallel. A swarm is N proposer calls, each reading one parent program and one focus x thinking-style prompt and replying with an idea of at most 45 words, then one writer call that turns the ideas and the parents into a complete program. The program is scored by your evaluator as soon as it exists and joins the archive the next round draws parents from: refine one of the best two, combine three of the best ten, explore two from anywhere, start fresh from the initial program. Every model call goes through headless claude -p, is priced at list rates from its modelUsage, and counts against a dollar cap.

The design is a text-only version of SwarmEvolve (Jiang and Karniadakis, arXiv 2611.08483), whose agents think inside an open model's activations, which an API model does not expose.

When it pays

Measured 2026-11-09 against stock OpenEvolve on Sonnet 5.5, three seeds per arm, three problems (references/evidence-2026-11.md):

swarm (defaults)Sonnet loop, best effort per problem
$ to 99% of reference, circle packing$0.013–0.019$0.32–0.44
$ to 99%, hexagon packing$0.035–0.060$0.53–0.58
$ to 99%, Heilbronn triangle$0.09–0.50$0.77–1.02
best at $0.25above every paired Sonnet seed on all three
final score at $3swarm stops at 60 min, about $1the same finals, in 7–42 minutes

Use the swarm when the budget is small or the goal is "good, cheaply". When the last 0.1% matters, the money is there, and wall-clock time counts, a Sonnet loop at the problem's best effort level gets to the same place faster.

When NOT to use this skill

situationuse instead
hundreds of failing tests to drive to greenswarming-to-green
choosing a model or effort for one subagent or a cascadeagent-routing
many independent tasks with checkable outputs, not one program to improvedispatching-subagents (headless runner)
quality only a person or a judge model can rateno scorer, no search: write the scorer first, or use a judged loop
claude.ai chat, where there is no claude binaryorchestrating-chat-agents

Abandon a run mid-way when:

  • status shows rejected_above_max climbing or a best score above anything the problem allows: the scorer has a hole and the search is now optimising the hole;
  • the best score is flat for ten rounds and rounds.jsonl shows every refine swarm parenting the same program: the archive has converged, and more rounds buy nothing;
  • last_quota.seven_day passes about 0.40: the gateway stops at 0.45 anyway.

Earned exceptions

rulebanned whenearned when
writer at effort medium—effort low on a problem already known to be easy (circle packing: low reached 0.999 cheaper)
N=8 proposers per swarmthe problem is new and N=0 has not been triedan N=0 control on this problem lost to N=8 (on circle packing the writer alone nearly tied)
no sandbox prefixthe evaluator imports model-written code on a machine with secrets or networka throwaway container with no credentials, and the user agreed

Setup

  1. Make a problem directory. Copy assets/example/ and replace its four files:
    • problem.md: the task in plain words, including the reference value if known;
    • initial_program.py: a valid, weak starting program;
    • evaluator.py: evaluate(program_path) -> dict of numbers and strings, with the score under score_key (default combined_score) and an error key on failure;
    • config.toml: entry function name, the interface text the models see, budgets.
  2. Score the initial program by hand and read the dict:
    python3 -c "import importlib.util as u; s=u.spec_from_file_location('e','evaluator.py'); e=u.module_from_spec(s); s.loader.exec_module(e); print(e.evaluate('initial_program.py'))"
    A score of 0 with no error, or a missing score_key, means the contract is wrong; fix it before spending anything.
  3. Set score_max from a bound you can prove (a known optimum, an area or volume bound), slightly above it. Scores above it count as 0 and are flagged. Leave it unset only when no bound exists, and then re-check the best program independently.
  4. Choose the sandbox. The evaluator imports and runs model-written code.
    • Claude Code on the web with the claude-workspace hub: the hub jail.
      [sandbox]
      prefix = ["sh", "/home/user/claude-workspace/scripts/jail.sh", "-C", "{run_dir}", "--"]
      chown_uid = 65534
      and put --run-dir under /tmp/jail/<name>/ (mkdir and chmod 755 it first). Launch through a script file: the hub's jail_route guard refuses a Bash command that names /tmp/jail/ next to python3.
    • Desktop with Docker: prefix = ["docker", "run", "--rm", "--network=none", "-v", "{run_dir}:{run_dir}", "python:3.12-slim"] (add numpy or what the evaluator needs to the image).
    • No sandbox only under the earned exception above.
  5. Check the CLI: which claude (desktop: the app's bundled CLI may not be on PATH; set EVOLVE_CLAUDE_BIN). Then run the offline selftest:
    python3 scripts/test_evolve.py      # three "ok" lines, about 3 s, no model calls

Run

S=/mnt/skills/user/evolving-with-swarms/scripts
python3 $S/evolve.py run problem/config.toml --run-dir RUN --seed 1 --cap 1.0 --wall 60 > RUN.log 2>&1

Launch it with Bash run_in_background: true and end the turn. The completion line is done -> RUN/summary.json; relaunching the same command resumes from RUN/archive.json and keeps the spent dollars. Check in with python3 $S/evolve.py status RUN. In Claude Code on the web a background task is stopped after two hours: keep --wall under 110 minutes per launch, or relaunch on the completion notice.

knob ([swarm], [budget])defaultmeasured
K, N4, 8K=2,N=16 and K=8,N=4 did no better on circle packing; N=0 nearly tied
proposer_effortlowthe floor; Haiku still thinks on about a third of calls
writer_effortmediumlow helped on circle packing, hurt on hexagon and Heilbronn
writer_modelhaikusonnet (one seed): 99% at $0.11 on circle packing, between swarm and loop
cap_usd, wall_minutes1.0, 60the swarm spent $0.9–1.4 an hour on 4 vCPUs

On a new problem, run the default and an N=0 control at the same cap before a long run.

Reading results

RUN/summary.json: programs, best_score, spent, usd_to (dollars when a program first scored 0.99, 0.999, 0.9999), calls_by_role, rejected_above_max, duplicates, no_code, last_quota. RUN/best_program.py is the top program. events.jsonl has one line per evaluation with the spend at that moment, which is the curve to plot; rounds.jsonl has each swarm's parents and ideas.

Common failure modes

  • best_score far above the reference, or rejected_above_max > 0 → the scorer accepts something it should not. Read the offending program in RUN/r*/s*/prog.py, fix the evaluator or tighten score_max, and start a new run: the archive is poisoned.
  • Every refine swarm in rounds.jsonl parents the same broken program → a scorer hole made it the leader; same remedy. Diagnosed 2026-11-09: a hexagon run spent its last 40 minutes refining a program the scorer had wrongly rated 3.9e9.
  • no_code climbing → the writer's reply has no fenced block defining entry. Check that entry in the config matches the interface text and the initial program.
  • duplicates close to the number of writer calls → the writer returns its parent unchanged; usually a writer that cannot see why the score is low. Add the useful metrics to show_metrics, or raise writer_effort.
  • Evaluation errors with rc=125 or rc=2 → the jail could not isolate (125) or nobody cannot enter the run directory (2). Put the run dir under /tmp/jail/<name>/ and set chown_uid = 65534.
  • Spend per program well above a cent → the gateway's ledger shows the cost by role; proposers were 30–39% of the spend in the 2026-11 ledgers, the writer the rest. Each claude -p call carries about 1,370 tokens of the CLI's own, so many tiny calls cost more than their prompts suggest. Fewer proposers (or N=0) and the writer's effort are the levers.
  • pauses.jsonl appears and the run slows → the account's five-hour window passed 90% (non-Haiku calls) or 97% (Haiku). The gateway waits for the reset and leaves the wait out of the run clock; nothing to do unless it is urgent.
  • A run stops with stopped_by starting quota: → seven-day utilisation reached 0.45. Stop launching runs this week, or raise week_stop in gateway.py deliberately.

Verification

python3 $S/test_evolve.py                         # offline: run, resume, status, score bound, sandbox prefix
python3 $S/evolve.py status RUN                   # programs > 1, spent <= cap + one round, best_score <= score_max

Re-score RUN/best_program.py with the evaluator yourself (in the sandbox) and compare it with best_score. Programs that optimise with randomness or a time budget re-score lower; two of three hexagon programs did on 2026-11-09. A bad success is a high score the evaluator should not have given: check the best program against an independent condition (a stricter geometric test, a known bound) before reporting a record.

Diagnosed failures

  • 2026-11-09, hexagon packing. OpenEvolve's stock hexagon evaluator scored an outer side of 1e-9 at 3.9e9: an absolute tolerance on an unnormalised cross product. Signal: a score nine orders above the reference. Fixed by an area bound; score_max exists for this.
  • 2026-11-09, cost accounting. claude -p wrote prompt cache at the 1-hour TTL (2x input) until FORCE_PROMPT_CACHING_5M=1; the gateway sets it. Signal: cacheCreationEphemeral1hInputTokens non-zero in the ledger.
  • 2026-11-09, effort. MAX_THINKING_TOKENS=0 and alwaysThinkingEnabled: false did not stop Haiku 5.5 from thinking, and an output cap below its need failed the call while billing it. Effort low is the floor; do not set max_tokens tight.
  • 2026-11-09, usage window. Ten parallel runs used most of a five-hour window in about 70 minutes and the old gateway aborted runs at 90%. It now pauses and resumes instead.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.