eval-standard-launch
Skill309 starsChanged 4 months agoApache-2.0View on GitHub
Launch the fixed Delphi #6279 RL-scaling-laws downstream MATH eval suite (MATH-500 / AIME24 / gsm8k via evalchemy + lm_eval) on CINECA Leonardo, for completed SFT / RL / base checkpoints. Covers finding which cells are newly-completed-but-uneval'd, the offline pre-download, the delphi_eval.sbatch invocation + RUN_NAME/STAGE convention, the load-bearing gotchas (chat-template override, 4k context, TP per head-count, MATH500/gsm8k split), and the SCORES.md tracker update. HF-upload-only — NEVER DB-register. Use when asked to eval Delphi #6279 SFT cells / update the scaling-laws score grid. Refs: experiments/active/delphi/rl-scaling-laws-6279/.
- Reads credentials
In this repository50 other files it gives its agents.
- CLAUDE.mdCLAUDE.md
- analyze-datagen-campaign-summarySkill
- analyze-dataset-token-lengthSkill
- analyze-id-eval-rankingSkill
- analyze-job-history-irisSkill
- analyze-rl-behaviorSkill
- analyze-training-run-irisSkill
- build-tpu-image-irisSkill
- code-create-staged-planSkill
- code-execute-staged-planSkill
- commitSkill
- crud-archive-runSkill
- crud-otagent-supabaseSkill
- crud-purge-below-gate-evalsSkill
- crud-purge-stale-eval-placeholdersSkill
- datagen-create-task-datasetSkill
- datagen-job-cleanupSkill
- datagen-launch-irisSkill
- datagen-launchSkill
- datagen-reduce-dataset-snapshotsSkill
- datagen-standard-launchSkill
- debugSkill
- docs-deslopSkill
- eval-agentic-cleanupSkill
- eval-agentic-launch-irisSkill
- eval-agentic-launchSkill
- eval-standard-cleanupSkill
- file-issueSkill
- monitor-cron-sweep-irisSkill
- monitor-cron-sweepSkill
- monitor-job-tablesSkill
- monitor-restore-irisSkill
- monitor-restoreSkill
- monitor-restore-unifiedSkill
- rl-agentic-job-cleanupSkill
- rl-agentic-launch-irisSkill
- rl-agentic-launch-jupiterSkill
- rl-job-health-deep-diveSkill
- rl-standard-job-cleanupSkill
- rl-standard-launch-leonardoSkill
- serve-model-vibe-testSkill
- sft-cleanup-hf-onlySkill
- sft-job-cleanupSkill
- sft-launchSkill
- supervisor-initSkill
- utils-cleanup-stale-sandboxesSkill
- utils-fix-permissionsSkill
- utils-reclaim-stale-snapshotsSkill
- write-testsSkill
- writing-styleSkill
The file
We split this skill into the rules it gives an agent
3 rules. About 0 are common, 1 is distinctive, and 2 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈1
Distinctive · estimate
This author's own choices. Worth a look.
2
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 67
Do NOT use parentheses inside a bash -lc "..." double-quoted string.
- Very specific: line 68
Do NOT use single quotes inside the outer ssh '...' arg (a single quote closes it).
- Distinctive: line 70
Don't disturb the still-PENDING Delphi SFT dependency chain.
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
eval-standard-launch
Downstream math-eval harness for Delphi #6279 RL-scaling-laws (task #215) — NOT the agentic tb2 eval listener. Scores Delphi checkpoints on MATH-500 (1 seed) + AIME24 (10-seed mean±se) + gsm8k (strict+flex) via evalchemy/lm_eval on Leonardo. HF-upload only — NEVER DB-registered (project_delphi_sft_hf_only_no_db).
0. Reference files (read these first — they are the source of truth)
Local notes dir: /Users/benjaminfeuer/Documents/experiments/active/delphi/rl-scaling-laws-6279/
EVAL_CONVENTION.md— the fixed eval protocol (suites, seeds, naming §3.4, chat-template §2.5).delphi_eval.sbatch— the ONE eval job (all gotchas baked in; canonical). Cluster copy under/leonardo_work/AIFAC_5C0_290/bfeuer00/….main_sft_evals/SCORES.md— master tracker for the 54-run main grid (27 midtrained ckpts × 2 cold-starts:magpie_lr1e5math-strong /wc386k_lr1e5math-weak). Each row: HF modellaion/<basename>, eval-job column, status (✅ done / ⏳ pending). The separate earlier cold-start grid lives incoldstart_grid_evals/(own../eval/SCORES.md).SFT_LEONARDO_INSTRUCTIONS.md— the SFT side (how cells get trained + uploaded).
1. What "newly completed" means
A cell is ready to eval when its SFT model is uploaded to HF laion/<basename> (Delphi SFT is HF-only; an uploaded repo = a finished cell). The work = the SCORES.md rows still ⏳ pending whose laion/<basename> repo now exists + is non-empty (check via huggingface_hub/hf; HF_TOKEN from your secrets env — .agents/secret.md). Rows whose model isn't uploaded yet (e.g. most large 1e21/1e22 cells mid-training) are SKIPPED until done.
2. The launch (per cell)
# RUN_NAME = the exact SFT model basename; STAGE = sft (chat-template ON) | rl | base (template OFF)
RUN=delphi-9e19-p33m67-k0p20-lr83-a002-magpie_lr1e5-sft
sbatch --job-name="delphi-eval-$RUN" <leonardo>/delphi_eval.sbatch laion/$RUN $RUN sft- Model-specific
--job-nameis MANDATORY (not the generic default) so squeue +%x-%j.log+meta.envmap jobid→model 1:1. The script self-renames + emits a greppableEVAL_JOBMAPline + writes<OUT>/meta.env. - Job shape: 1 node / 4 GPU (A100 64GB) / 8h /
boost_usr_prod, conda envevalchemy-marin, runs from/leonardo_work/AIFAC_5C0_290/bfeuer00/code/evalchemy-marin. Output →…/experiments/delphi-eval/<RUN_NAME>/.
3. Pre-download is REQUIRED (compute is offline)
delphi_eval.sbatch runs HF_HUB_OFFLINE=1 (Leonardo compute has no internet). Pre-cache each model on the LOGIN node first into HF_HOME=HF_HUB_CACHE=/leonardo_work/AIFAC_5C0_290/bfeuer00/data/hub (login nodes have direct internet). If a login-node snapshot_download risks the ~100s login-killer, use the notes' documented pre-download path (tmux / a small sbatch). The eval sbatch itself needs NO SSH tunnel (offline + pre-cached); only the pre-download touches the network.
4. Load-bearing gotchas (all handled inside the sbatch — know them when debugging)
- HOME is read-only on Leonardo (login AND compute). The sbatch redirects HOME + flashinfer/triton/inductor/vLLM/XDG caches to a writable
…/delphi-eval/.cache/*; without it every vLLM worker diesPermissionError … /.cache/flashinfer. - delphi_v0 chat-template override (sft/rl only): the SFT/RL repos ship a plain 656-char Llama-3 template (the delphi ReasoningTemplate didn't persist into the repo) → evaluating as-is is a train/eval mismatch (empty think channel). The sbatch overrides the cached tokenizer's
chat_templatetoOpenThoughts-Agent/chat_templates/delphi_v0.jinja2(idempotent, leaves.plainbak) before eval.basestage skips this (no template, raw completion). MAX_MODEL_LEN=4096(NOT the marin 32768 default): Delphi ckpts are 4k-cutoff with a malformed llama3 rope_scaling block; vLLM derives 4096 and HARD-rejects 32768. Generation pinnedMAX_GEN_TOKS=3584. Comparability holds within the 4k cohort; flag any model exposing >4k.--max_tokensMUST be pinned to MAX_GEN_TOKS — MATH500/AIME24 are evalchemy chat_benchmarks whose max_new_tokens DEFAULTS to 32768 (not reached by--gen_kwargs max_gen_toks); unset → lm-eval computes4096-32768 = negative→ truncates prompt to empty →decoder prompt cannot be empty.- TP per head-divisibility:
num_attention_heads % TP == 0. The small Delphi Qwen3 (14 heads) → TP=2 (TP=4 hard-fails). Pin TP per model to the largest node-supported divisor of its head count. - MATH500 and gsm8k run as SEPARATE sequential processes (not one
--taskscall): MATH500 is an evalchemy chat_benchmark, gsm8k is lm-eval-native; in one process the second vLLM engine inits while the first is GPU-resident → OOM/WorkerProc fail and gsm8k is silently dropped. gsm8k runs via plainlm_eval(evalchemy double-builds the engine for native tasks → OOM on 64GB). AIME24 is its own pass. --verbosity INFOis required (evalchemygetattr(logging, args.verbosity); default None →AttributeErrorafter full vLLM init).- Idempotent skip:
<OUT>/seed42existing → the cell is skipped. If gsm8k failed after MATH500 wrote seed42, delete<OUT>/seed42(or re-run gsm8k by hand) — the skip keys on that dir.
5. After submit → tracking + consolidation
- Confirm queued:
squeue -u bfeuer00 | grep delphi-eval; collect job ids. - Update
main_sft_evals/SCORES.md: set submitted rows' status →🚀 eval submitted+ put the Leonardo job id in the eval-job column. Do NOT fabricate score cells (leave—); preserve the table format exactly. - On completion, per-model
results_*.jsonrsync intomain_sft_evals/<basename>/, scalar partials intomain_sft_evals/.partial/<basename>.json, and SCORES.md is consolidated from them (MATH-500, AIME24 mean±se, gsm8k strict/flex, Raw). The #6279 deliverable = how MATH-500/AIME24/gsm8k move with (scale × mix) at each of the two starting points (does a strong-vs-weak math start change the midtraining ranking).
5b. Evaluate a (post-RL) checkpoint on the Delphi eval suite (reusable)
The same harness scores ANY standard Delphi checkpoint — base / post-SFT / post-RL — on the fixed suite (EVAL_CONVENTION.md §1.2: MATH500 1-seed + AIME24 10-seed mean±se + gsm8k strict/flex, pass@1, temp 0.7). rl-standard-job-cleanup defers to THIS section as its final step after the post-RL ckpt is HF-uploaded. The only deltas from §2 are the STAGE token and the tracker the result lands in:
- Point it at the HF-uploaded ckpt. The ckpt is
laion/<run_name>-<BEST>-<size>B(the reporl-standard-job-cleanup§6 just published — weights at root). Pre-cache it on the login node first (§3), same as any cell. - STAGE =
rl(chat-template ON — same delphi_v0 override assft; onlybaseskips it):
Auto-TP=2 for the 30-head 9.7B Delphi Qwen3;RUN=<run_name>-<BEST>-<size>B sbatch --job-name="delphi-eval-$RUN" \ /leonardo_work/AIFAC_5C0_290/bfeuer00/experiments/delphi-eval/delphi_eval.sbatch laion/$RUN $RUN rlmax_model_len=4096/max_gen_toks=3584; all §4 gotchas (template override, 4k context, MATH500/gsm8k split, AIME24 10-seed pass) apply unchanged — baked into the canonicaldelphi_eval.sbatch. One node / 4 GPU / 8h /evalchemy-marinenv. - Results land in the RL tracker, not the SFT one. Per-model output is
…/experiments/delphi-eval/<RUN>/seed{42..51}+meta.env; consolidated scores go tomain_rl_evals/SCORES.md(the post-RL tracker — keyed by (scale, mix, start-point) so SFT-vs-RL deltas line up againstmain_sft_evals/), via the same harvest path as §5 /eval-standard-cleanup. HF-upload-only — NEVER DB (the post-RL ckpt has no models DB row and this eval doesn't create one). After submit, add the row tomain_rl_evals/SCORES.mdset to🚀 eval submittedwith the Leonardo job id; harvest pereval-standard-cleanup.
6. Leonardo SSH quoting traps (these bite repeatedly)
- Do NOT use parentheses inside a
bash -lc "..."double-quoted string. - Do NOT use single quotes inside the outer
ssh '...'arg (a single quote closes it). Use escaped double-quotes / heredocs / plain words. - Refresh the step-ca cert if any tunnel op needs it (per CLAUDE.md) — but offline eval sbatch + a login-node pre-download (direct internet) do not need the tunnel.
- Don't disturb the still-PENDING Delphi SFT dependency chain; evals are independent 1-node jobs.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

