agentleFS
open-thoughtsopen-thoughts/OpenThoughts-Agent/.agents/skills/eval-standard-cleanup/SKILL.md

eval-standard-cleanup

Skill309 starsChanged 4 months agoApache-2.0View on GitHub

Consolidate FINISHED standard / lm_eval (evalchemy) math-suite eval jobs — the Delphi #6279 MATH-500 / AIME24 / gsm8k grid launched via eval-standard-launch — into the SCORES.md tracker. Per job: confirm a NON-EMPTY seed42 result (a crash leaves it empty — don't record it as a score), rsync the results_*.json into the local per-model dir, write scalar partials, parse the scalars (MATH-500 acc×100 / AIME24 10-seed mean±sd / gsm8k strict+flex / Raw), and flip the SCORES.md row 🚀 eval submitted → ✅ done. The artifact is SCALAR SCORES IN A TRACKER — HF-upload-only, NEVER DB. Use when asked to consolidate / harvest finished standard math-eval jobs or fill the scaling-laws score grid. DISTINCT from eval-agentic-cleanup (the Harbor-trace + Supabase DB path).

The file

We split this skill into the rules it gives an agent

1 rule. About 0 are common, 0 are distinctive, and 1 is very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈0

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 1

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    Keep the rsynced JSONs in <basename>/ as the archive.

    line 56

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

eval-standard-cleanup

Post-run consolidation for STANDARD evals (lm_eval / evalchemy): the Delphi #6279 RL-scaling-laws math suite — MATH-500 (1 seed) + AIME24 (10-seed mean±sd) + gsm8k (strict+flex) — launched by the eval-standard-launch skill on Leonardo. The deliverable is scalar scores in a tracker: no HF trace upload, no model upload, NEVER a DB registration (see §5). For the agentic Harbor-trace + Supabase path, use eval-agentic-cleanup instead.

0. Reference files (source of truth — derive the flow from these)

Local notes dir: /Users/benjaminfeuer/Documents/experiments/active/delphi/rl-scaling-laws-6279/

  • eval-standard-launch skill — the launch + delphi_eval.sbatch + the per-model output layout it produces (…/experiments/delphi-eval/<RUN_NAME>/seed{42..51}/). Read it for what's already on the cluster.
  • main_sft_evals/SCORES.md — the master tracker for the 54-run main grid; its header describes the consolidation flow (per-model dirs main_sft_evals/<basename>/, partials main_sft_evals/.partial/).
  • EVAL_CONVENTION.md §3.4 / §5.2-E — the per-cluster naming, the idempotent-skip seed42 dir, the exact scalar-parse method, and the "COMPLETED is NOT proof — verify numeric scores" rule.
  • The earlier cold-start grid is the same flow but a different tracker (eval/SCORES.md, artifacts eval/<RUN>/) — don't cross-file; the 54-run main grid lands in main_sft_evals/.

1. Per finished job — verify a real, non-empty result (do NOT record a crash as a score)

A SLURM COMPLETED state is necessary but not sufficient: evalchemy catches engine errors, exits 0, and writes an empty results: {} JSON (silent drop). For each job:

  1. Confirm terminal state via sacct -j <jobid> --format=State -n -P | head -1 (the source of truth, not log-tailing).
  2. Confirm the job produced a NON-EMPTY seed42 result — the §3.4 idempotent-skip dir (…/delphi-eval/<RUN_NAME>/seed42/) must exist and its results_*.json must carry numeric scores (a crashed/timed-out run leaves an empty or missing seed42). An empty/missing seed42 → leave the row pending and note it; do not fabricate a score. (If gsm8k failed after MATH-500 wrote seed42, seed42 exists but is partial — note it; deleting seed42 + re-running is a launcher action, not a cleanup one.)

2. Consolidate — rsync results into the local per-model dir + write scalar partials

Pull only the per-task results_*.json (the examples payloads live inside the JSON, ~1-2 MB each; skip the multi-MB trace subdirs). For the main grid the destination is main_sft_evals/<basename>/:

RUN=delphi-9e19-p33m67-k0p20-lr83-a002-magpie_lr1e5-sft   # = the SFT model basename, maps 1:1 to the row
DEST=/Users/benjaminfeuer/Documents/experiments/active/delphi/rl-scaling-laws-6279/main_sft_evals/$RUN
mkdir -p "$DEST"
rsync -avz --prune-empty-dirs --include='*/' --include='results_*.json' --exclude='*' \
  Leonardo:/leonardo_work/AIFAC_5C0_290/bfeuer00/experiments/delphi-eval/$RUN/ "$DEST/"

This pulls the MATH-500 seed42 JSON, the gsm8k seed42 JSON (written by the plain lm_eval step), and the AIME24 seed42..seed51 JSONs. Write the extracted scalars to main_sft_evals/.partial/<basename>.json. Keep the rsynced JSONs in <basename>/ as the archive. Parse by loading the JSON and reading only numeric keys — never print the huge examples list.

3. Parse the scalars (the §5.2-E method — round to 1 decimal to match the table)

  • MATH-500 = results["MATH500"]["accuracy"] × 100 (it's a fraction; 0.018 → 1.8). From seed42.
  • AIME24 = results["AIME24"]["accuracy_avg"] × 100, taken over the 10 seed dirs (seed42..seed51); report mean ± sample-stdev of the per-seed values (×100). (e.g. 0.1±0.1.)
  • gsm8k (from the lm_eval output under <basename>/seed42): exact_match,strict-match → gsm8k-S and exact_match,flexible-extract → gsm8k-F, each ×100.
  • Raw = mean(MATH-500, AIME24, gsm8k-strict).
  • Note any format-fail cell (empty \boxed{} / model_answer:"") — that's a real signal, not an eval bug.

The marin compiler marin:experiments/evals/evalchemy_results_compiler.py documents the exact JSON shape (results[task]…) — reuse its extraction shape rather than rolling a new parse. (verify the precise key names against the rsynced JSON on first use; evalchemy versions have drifted.)

4. Update the SCORES.md row — 🚀 eval submitted → ✅ done

Fill that row's MATH-500 | AIME24 (mean±se) | gsm8k-S | gsm8k-F | Raw cells from the partial and flip status from 🚀 eval submitted to ✅ done (keep the eval job id in its column). Edit the single row (rows are independent lines) and preserve the table format exactly. A crashed/empty-seed42 job stays pending with a note (step 1) — never invent numbers.

5. HF-upload only — NEVER DB

These are scalar scores in a tracker, not a model/trace artifact. Do NOT call manual_db_eval_push.py / touch Supabase — that is the eval-agentic-cleanup (agentic / Harbor-trace) path. Results live in this experiment's docs + (optionally) as a training_logs/-style JSON on the model's HF repo; there is no leaderboard row. (Per project_delphi_sft_hf_only_no_db.)

6. Disk / hygiene

The local artifacts are small (~1-2 MB JSONs); no large delete is needed. On the cluster, leave the delphi-eval/<RUN_NAME>/ dirs in place — they back the §3.4 idempotent skip, so a re-harvest or re-submit stays safe. No find / du / rglob on GPFS; use canonical paths.


Launcher: eval-standard-launch. Model-publishing cleanups: rl-agentic-job-cleanup / sft-job-cleanup. Per-cluster particulars (ssh, the step-ca cert refresh for rsync, the /leonardo_work/AIFAC_5C0_290/bfeuer00/… paths) → .agents/ops/<cluster>/.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.