agentleFS
Sign inStart free
ravindu57ravindu57/SCML/.claude/skills/run-eval/SKILL.md

run-eval

Skill0 starsChanged 26 days agoApache-2.0View on GitHub

Reproducible benchmark run into a new versioned result file — never over a committed baseline.

  • Reads credentials

7 other projects use a version of this.

Adopted by

Most recently taken up by

The file

We split this skill into the rules it gives an agent

5 rules. 0 are common, 0 are distinctive, and 5 are very specific to this project.

  • 0

    Common

    Most of the 7 edited versions we read keep these.

  • 0

    Distinctive

    This author's own choices. Worth a look.

  • 5

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    Output path: Always --out benchmarks/results/v2 (create it).

    line 14
  2. Very specific:

    Run. For long runs use run_in_background and wait for the completion notification.

    line 15
  3. Very specific:

    Write a sidecar <same-name>.run.json with everything from step 2, start/end time and the exit code.

    line 16
  4. Very specific:

    Run the eval-analyst agent on the new JSON.

    line 18
  5. Very specific:

    Do not edit any frozen file or docs/EVAL_RESULTS.md here.

    line 19

“Common” is measured: kept by at least 6 in 10 of the 7 edited versions of this file we could read. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Testbed and arguments: $ARGUMENTS

Testbeds: injecagent, memory_poisoning (via python -m benchmarks.cli), memory_poisoning_harm (python -m benchmarks.testbeds.memory_poisoning.harm_report), load (python -m benchmarks.load), soak (python -m benchmarks.soak), agentdojo (python -m benchmarks.testbeds.agentdojo.runner), classifier_eval, affordance_detector. Read benchmarks/README.md for each one's flags.

  1. Cost check. AgentDojo, the LLM classifier and anything with SCANNER_BACKEND=llm call a paid API. Estimate calls × tokens × price, show it, and wait for approval. Never read .env to find a key; ask the owner how the key is provided.
  2. Pin the run. Record: HEAD commit and whether the tree is dirty (git status --short), Python and package versions (pip freeze | sha256sum), every TRUST_MEDIATOR_*/SCANNER_* env var in effect, seeds, model name, and the exact command line.
  3. Output path. Always --out benchmarks/results/v2 (create it). Rename the outputs to <testbed>_<YYYY-MM-DD>[_<variant>].{md,json}. If that name exists, add _2, _3. Never pass --out benchmarks/results — the frozen guard blocks it.
  4. Run. For long runs use run_in_background and wait for the completion notification.
  5. Write a sidecar <same-name>.run.json with everything from step 2, start/end time and the exit code.
  6. Compare against the committed baseline for the same testbed in benchmarks/results/ and report the delta. Always include the no_tool_policy ablation row where the testbed has one, and both storage and harm ASR for memory poisoning.
  7. Run the eval-analyst agent on the new JSON. Report anything it cannot reproduce.
  8. Do not edit any frozen file or docs/EVAL_RESULTS.md here; /kpi-report does that.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

Check your setup, free