eval-standard-cleanup
Skill309 starsChanged 4 months agoApache-2.0View on GitHub
Consolidate FINISHED standard / lm_eval (evalchemy) math-suite eval jobs — the Delphi #6279 MATH-500 / AIME24 / gsm8k grid launched via eval-standard-launch — into the SCORES.md tracker. Per job: confirm a NON-EMPTY seed42 result (a crash leaves it empty — don't record it as a score), rsync the results_*.json into the local per-model dir, write scalar partials, parse the scalars (MATH-500 acc×100 / AIME24 10-seed mean±sd / gsm8k strict+flex / Raw), and flip the SCORES.md row 🚀 eval submitted → ✅ done. The artifact is SCALAR SCORES IN A TRACKER — HF-upload-only, NEVER DB. Use when asked to consolidate / harvest finished standard math-eval jobs or fill the scaling-laws score grid. DISTINCT from eval-agentic-cleanup (the Harbor-trace + Supabase DB path).
In this repository50 other files it gives its agents.
- CLAUDE.mdCLAUDE.md
- analyze-datagen-campaign-summarySkill
- analyze-dataset-token-lengthSkill
- analyze-id-eval-rankingSkill
- analyze-job-history-irisSkill
- analyze-rl-behaviorSkill
- analyze-training-run-irisSkill
- build-tpu-image-irisSkill
- code-create-staged-planSkill
- code-execute-staged-planSkill
- commitSkill
- crud-archive-runSkill
- crud-otagent-supabaseSkill
- crud-purge-below-gate-evalsSkill
- crud-purge-stale-eval-placeholdersSkill
- datagen-create-task-datasetSkill
- datagen-job-cleanupSkill
- datagen-launch-irisSkill
- datagen-launchSkill
- datagen-reduce-dataset-snapshotsSkill
- datagen-standard-launchSkill
- debugSkill
- docs-deslopSkill
- eval-agentic-cleanupSkill
- eval-agentic-launch-irisSkill
- eval-agentic-launchSkill
- eval-standard-launchSkill
- file-issueSkill
- monitor-cron-sweep-irisSkill
- monitor-cron-sweepSkill
- monitor-job-tablesSkill
- monitor-restore-irisSkill
- monitor-restoreSkill
- monitor-restore-unifiedSkill
- rl-agentic-job-cleanupSkill
- rl-agentic-launch-irisSkill
- rl-agentic-launch-jupiterSkill
- rl-job-health-deep-diveSkill
- rl-standard-job-cleanupSkill
- rl-standard-launch-leonardoSkill
- serve-model-vibe-testSkill
- sft-cleanup-hf-onlySkill
- sft-job-cleanupSkill
- sft-launchSkill
- supervisor-initSkill
- utils-cleanup-stale-sandboxesSkill
- utils-fix-permissionsSkill
- utils-reclaim-stale-snapshotsSkill
- write-testsSkill
- writing-styleSkill
The file
We split this skill into the rules it gives an agent
1 rule. About 0 are common, 0 are distinctive, and 1 is very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈0
Distinctive · estimate
This author's own choices. Worth a look.
1
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 56
Keep the rsynced JSONs in <basename>/ as the archive.
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
eval-standard-cleanup
Post-run consolidation for STANDARD evals (lm_eval / evalchemy): the Delphi #6279 RL-scaling-laws
math suite — MATH-500 (1 seed) + AIME24 (10-seed mean±sd) + gsm8k (strict+flex) — launched by the
eval-standard-launch skill on Leonardo. The deliverable is scalar scores in a tracker: no HF
trace upload, no model upload, NEVER a DB registration (see §5). For the agentic Harbor-trace +
Supabase path, use eval-agentic-cleanup instead.
0. Reference files (source of truth — derive the flow from these)
Local notes dir: /Users/benjaminfeuer/Documents/experiments/active/delphi/rl-scaling-laws-6279/
eval-standard-launchskill — the launch +delphi_eval.sbatch+ the per-model output layout it produces (…/experiments/delphi-eval/<RUN_NAME>/seed{42..51}/). Read it for what's already on the cluster.main_sft_evals/SCORES.md— the master tracker for the 54-run main grid; its header describes the consolidation flow (per-model dirsmain_sft_evals/<basename>/, partialsmain_sft_evals/.partial/).EVAL_CONVENTION.md§3.4 / §5.2-E — the per-cluster naming, the idempotent-skipseed42dir, the exact scalar-parse method, and the "COMPLETED is NOT proof — verify numeric scores" rule.- The earlier cold-start grid is the same flow but a different tracker (
eval/SCORES.md, artifactseval/<RUN>/) — don't cross-file; the 54-run main grid lands inmain_sft_evals/.
1. Per finished job — verify a real, non-empty result (do NOT record a crash as a score)
A SLURM COMPLETED state is necessary but not sufficient: evalchemy catches engine errors, exits 0,
and writes an empty results: {} JSON (silent drop). For each job:
- Confirm terminal state via
sacct -j <jobid> --format=State -n -P | head -1(the source of truth, not log-tailing). - Confirm the job produced a NON-EMPTY
seed42result — the §3.4 idempotent-skip dir (…/delphi-eval/<RUN_NAME>/seed42/) must exist and itsresults_*.jsonmust carry numeric scores (a crashed/timed-out run leaves an empty or missingseed42). An empty/missing seed42 → leave the row pending and note it; do not fabricate a score. (If gsm8k failed after MATH-500 wrote seed42, seed42 exists but is partial — note it; deleting seed42 + re-running is a launcher action, not a cleanup one.)
2. Consolidate — rsync results into the local per-model dir + write scalar partials
Pull only the per-task results_*.json (the examples payloads live inside the JSON, ~1-2 MB each;
skip the multi-MB trace subdirs). For the main grid the destination is main_sft_evals/<basename>/:
RUN=delphi-9e19-p33m67-k0p20-lr83-a002-magpie_lr1e5-sft # = the SFT model basename, maps 1:1 to the row
DEST=/Users/benjaminfeuer/Documents/experiments/active/delphi/rl-scaling-laws-6279/main_sft_evals/$RUN
mkdir -p "$DEST"
rsync -avz --prune-empty-dirs --include='*/' --include='results_*.json' --exclude='*' \
Leonardo:/leonardo_work/AIFAC_5C0_290/bfeuer00/experiments/delphi-eval/$RUN/ "$DEST/"This pulls the MATH-500 seed42 JSON, the gsm8k seed42 JSON (written by the plain lm_eval step), and
the AIME24 seed42..seed51 JSONs. Write the extracted scalars to main_sft_evals/.partial/<basename>.json.
Keep the rsynced JSONs in <basename>/ as the archive. Parse by loading the JSON and reading only
numeric keys — never print the huge examples list.
3. Parse the scalars (the §5.2-E method — round to 1 decimal to match the table)
- MATH-500 =
results["MATH500"]["accuracy"] × 100(it's a fraction; 0.018 → 1.8). Fromseed42. - AIME24 =
results["AIME24"]["accuracy_avg"] × 100, taken over the 10 seed dirs (seed42..seed51); report mean ± sample-stdev of the per-seed values (×100). (e.g.0.1±0.1.) - gsm8k (from the
lm_evaloutput under<basename>/seed42):exact_match,strict-match→ gsm8k-S andexact_match,flexible-extract→ gsm8k-F, each ×100. - Raw =
mean(MATH-500, AIME24, gsm8k-strict). - Note any format-fail cell (empty
\boxed{}/model_answer:"") — that's a real signal, not an eval bug.
The marin compiler marin:experiments/evals/evalchemy_results_compiler.py documents the exact JSON shape
(results[task]…) — reuse its extraction shape rather than rolling a new parse. (verify the precise key
names against the rsynced JSON on first use; evalchemy versions have drifted.)
4. Update the SCORES.md row — 🚀 eval submitted → ✅ done
Fill that row's MATH-500 | AIME24 (mean±se) | gsm8k-S | gsm8k-F | Raw cells from the partial and flip
status from 🚀 eval submitted to ✅ done (keep the eval job id in its column). Edit the single row
(rows are independent lines) and preserve the table format exactly. A crashed/empty-seed42 job stays
pending with a note (step 1) — never invent numbers.
5. HF-upload only — NEVER DB
These are scalar scores in a tracker, not a model/trace artifact. Do NOT call
manual_db_eval_push.py / touch Supabase — that is the eval-agentic-cleanup (agentic / Harbor-trace)
path. Results live in this experiment's docs + (optionally) as a training_logs/-style JSON on the model's
HF repo; there is no leaderboard row. (Per project_delphi_sft_hf_only_no_db.)
6. Disk / hygiene
The local artifacts are small (~1-2 MB JSONs); no large delete is needed. On the cluster, leave the
delphi-eval/<RUN_NAME>/ dirs in place — they back the §3.4 idempotent skip, so a re-harvest or re-submit
stays safe. No find / du / rglob on GPFS; use canonical paths.
Launcher: eval-standard-launch. Model-publishing cleanups: rl-agentic-job-cleanup /
sft-job-cleanup. Per-cluster particulars (ssh, the step-ca cert refresh for rsync, the
/leonardo_work/AIFAC_5C0_290/bfeuer00/… paths) → .agents/ops/<cluster>/.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

