eval-evolution
Skill7 starsChanged 53 days agoMITView on GitHub
Prepare, run, review, and explicitly promote bounded Skill or existing-MCP Eval Evolution campaigns through the canonical eval facade.
In this repository8 other files it gives its agents.
- AGENTS.mdAGENTS.md
- CLAUDE.mdCLAUDE.md
- eval-harnessSkill
- household-worldSkill
- raw-fpv-visual-labelerSkill
- report-performance-analysisSkill
- scene-gaussian-map-alignmentSkill
- visual-result-showcaseSkill
The file
We split this skill into the rules it gives an agent
4 rules. About 0 are common, 3 are distinctive, and 1 is very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈3
Distinctive · estimate
This author's own choices. Worth a look.
1
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 8
Use this Skill for maintainer-owned optimization campaigns driven by frozen eval evidence.
- Distinctive: line 28
Do not mix Skill and MCP changes.
- Distinctive: line 38
Run the frozen campaign only after provider/runtime readiness is established:
- Distinctive: line 79
Stop before creating that approval unless a maintainer has reviewed the exact candidate, paired training, sealed confirmation, limitations, …
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Eval Evolution
Use this Skill for maintainer-owned optimization campaigns driven by frozen eval
evidence. roboclaws.evals remains the control plane. The optimizer and robot
trials are distinct OpenAI Agents SDK agents; a provider profile selects their
models and never changes the agent engine.
Prepare
Freeze an eval_evolution_campaign_v1 JSON manifest before the first optimizer
call. Bind one target kind and target, the baseline commit and target digest,
mutable paths, optimizer and robot provider/model identities, paired training
suites, the orchestrator-only sealed confirmation reference, quality and
minimum-improvement policy, runtime identity, and explicit turn/trial/token/
cost/time/retry ceilings.
The budget block also declares positive optimizer_call_tokens,
optimizer_call_cost_usd, robot_attempt_tokens, and
robot_attempt_cost_usd reservations. Each reservation is a frozen maximum for
one optimizer run or one robot attempt, not a post-hoc usage target.
Skill campaigns target exactly one skills/<name>/SKILL.md. Keep
static-full as the baseline. no-skill is a non-promotable negative control.
Do not mix Skill and MCP changes.
Run
Inspect the blocked preflight without provider execution:
just agent::eval evolve campaign=<campaign.json>Run the frozen campaign only after provider/runtime readiness is established:
just agent::eval evolve campaign=<campaign.json> live_execution=runThe optimizer can only read the declared target, read sanitized feedback, and submit one hypothesis plus patch. It has no shell, filesystem, git, network, eval-launch, commit, or publication tool. Host-owned validation creates one content-addressed full baseline snapshot and runs deterministic gates before paired robot trials.
Review
Reject missing or mismatched identity, incomplete paired evidence, privacy or
checker failures, trajectory/terminal violations, quality regressions, neutral
rewrites, and no-skill. Efficiency ranks only candidates that meet the frozen
quality and minimum-improvement rule. At most one training winner reaches one
sealed confirmation. Its evidence never returns to the optimizer.
The orchestrator reserves the full optimizer-run or robot-attempt token and cost
maximum before provider execution and passes that limit into the Agents SDK
model settings. Insufficient remaining campaign capacity is inconclusive
without starting the provider. A model without catalog output pricing is also
blocked before execution because its cost ceiling cannot be derived safely.
Behavior/provider failures do not retry.
A separately recorded classified infrastructure attempt is allowed only when
the frozen campaign permits one retry.
Promote
Promotion requires an accepted selection report and a digest-bound
eval_evolution_promotion_manifest_v1 with maintainer_approved=true:
just agent::eval evolve-promote \
report=<selection-report.json> \
manifest=<maintainer-approved.json> \
live_execution=runStop before creating that approval unless a maintainer has reviewed the exact candidate, paired training, sealed confirmation, limitations, and digests. Promotion applies only the reviewed patch. It does not commit, change defaults, or publish a baseline/catalog artifact.
MCP description candidates additionally require the Phase 2 structural text-only proof. Never run MCP behavior candidates live until the Phase 3 malicious isolation proof has passed on the selected placement.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

