agentleFS
MiaoDXMiaoDX/roboclaws/skills/eval-evolution/SKILL.md

eval-evolution

Skill7 starsChanged 53 days agoMITView on GitHub

Prepare, run, review, and explicitly promote bounded Skill or existing-MCP Eval Evolution campaigns through the canonical eval facade.

The file

We split this skill into the rules it gives an agent

4 rules. About 0 are common, 3 are distinctive, and 1 is very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈3

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 1

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    Use this Skill for maintainer-owned optimization campaigns driven by frozen eval evidence.

    line 8
  2. Distinctive:

    Do not mix Skill and MCP changes.

    line 28
  3. Distinctive:

    Run the frozen campaign only after provider/runtime readiness is established:

    line 38
  4. Distinctive:

    Stop before creating that approval unless a maintainer has reviewed the exact candidate, paired training, sealed confirmation, limitations, …

    line 79

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Eval Evolution

Use this Skill for maintainer-owned optimization campaigns driven by frozen eval evidence. roboclaws.evals remains the control plane. The optimizer and robot trials are distinct OpenAI Agents SDK agents; a provider profile selects their models and never changes the agent engine.

Prepare

Freeze an eval_evolution_campaign_v1 JSON manifest before the first optimizer call. Bind one target kind and target, the baseline commit and target digest, mutable paths, optimizer and robot provider/model identities, paired training suites, the orchestrator-only sealed confirmation reference, quality and minimum-improvement policy, runtime identity, and explicit turn/trial/token/ cost/time/retry ceilings. The budget block also declares positive optimizer_call_tokens, optimizer_call_cost_usd, robot_attempt_tokens, and robot_attempt_cost_usd reservations. Each reservation is a frozen maximum for one optimizer run or one robot attempt, not a post-hoc usage target.

Skill campaigns target exactly one skills/<name>/SKILL.md. Keep static-full as the baseline. no-skill is a non-promotable negative control. Do not mix Skill and MCP changes.

Run

Inspect the blocked preflight without provider execution:

just agent::eval evolve campaign=<campaign.json>

Run the frozen campaign only after provider/runtime readiness is established:

just agent::eval evolve campaign=<campaign.json> live_execution=run

The optimizer can only read the declared target, read sanitized feedback, and submit one hypothesis plus patch. It has no shell, filesystem, git, network, eval-launch, commit, or publication tool. Host-owned validation creates one content-addressed full baseline snapshot and runs deterministic gates before paired robot trials.

Review

Reject missing or mismatched identity, incomplete paired evidence, privacy or checker failures, trajectory/terminal violations, quality regressions, neutral rewrites, and no-skill. Efficiency ranks only candidates that meet the frozen quality and minimum-improvement rule. At most one training winner reaches one sealed confirmation. Its evidence never returns to the optimizer.

The orchestrator reserves the full optimizer-run or robot-attempt token and cost maximum before provider execution and passes that limit into the Agents SDK model settings. Insufficient remaining campaign capacity is inconclusive without starting the provider. A model without catalog output pricing is also blocked before execution because its cost ceiling cannot be derived safely. Behavior/provider failures do not retry. A separately recorded classified infrastructure attempt is allowed only when the frozen campaign permits one retry.

Promote

Promotion requires an accepted selection report and a digest-bound eval_evolution_promotion_manifest_v1 with maintainer_approved=true:

just agent::eval evolve-promote \
  report=<selection-report.json> \
  manifest=<maintainer-approved.json> \
  live_execution=run

Stop before creating that approval unless a maintainer has reviewed the exact candidate, paired training, sealed confirmation, limitations, and digests. Promotion applies only the reviewed patch. It does not commit, change defaults, or publish a baseline/catalog artifact.

MCP description candidates additionally require the Phase 2 structural text-only proof. Never run MCP behavior candidates live until the Phase 3 malicious isolation proof has passed on the selected placement.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.