rl-standard-launch-leonardo
Skill309 starsChanged 4 months agoApache-2.0View on GitHub
Launch, relaunch, or sweep STANDARD (non-agentic) SkyRL RL on CINECA Leonardo — GRPO on math/reasoning datasets (gsm8k, MATH/aime) and on-policy distillation (OPD, teacher→student) — via raw `sbatch` of the `hpc/skyrl_standard/leonardo/*` run scripts inside the writable apptainer SANDBOX + uv `marin_venv` (NOT `python -m hpc.launch`, NOT a `.sif`, NOT `--rl_use_conda`). Use when asked to run/relaunch a gsm8k or OPD GRPO canary, throughput/accuracy grid, or multi-node RL on Leonardo A100-64GB. Covers the GRPO/OPD knobs, the grid cell structure, the 1-node-vs-multi-node layout, the A100-64GB ceilings, and the no-internet/offline + gcc/HOME/Ray-temp-dir gotchas. For agentic Harbor+Daytona RL, this is the WRONG skill (Daytona needs internet — infeasible on Leonardo).
- Deletes or force-pushes
In this repository50 other files it gives its agents.
- CLAUDE.mdCLAUDE.md
- analyze-datagen-campaign-summarySkill
- analyze-dataset-token-lengthSkill
- analyze-id-eval-rankingSkill
- analyze-job-history-irisSkill
- analyze-rl-behaviorSkill
- analyze-training-run-irisSkill
- build-tpu-image-irisSkill
- code-create-staged-planSkill
- code-execute-staged-planSkill
- commitSkill
- crud-archive-runSkill
- crud-otagent-supabaseSkill
- crud-purge-below-gate-evalsSkill
- crud-purge-stale-eval-placeholdersSkill
- datagen-create-task-datasetSkill
- datagen-job-cleanupSkill
- datagen-launch-irisSkill
- datagen-launchSkill
- datagen-reduce-dataset-snapshotsSkill
- datagen-standard-launchSkill
- debugSkill
- docs-deslopSkill
- eval-agentic-cleanupSkill
- eval-agentic-launch-irisSkill
- eval-agentic-launchSkill
- eval-standard-cleanupSkill
- eval-standard-launchSkill
- file-issueSkill
- monitor-cron-sweep-irisSkill
- monitor-cron-sweepSkill
- monitor-job-tablesSkill
- monitor-restore-irisSkill
- monitor-restoreSkill
- monitor-restore-unifiedSkill
- rl-agentic-job-cleanupSkill
- rl-agentic-launch-irisSkill
- rl-agentic-launch-jupiterSkill
- rl-job-health-deep-diveSkill
- rl-standard-job-cleanupSkill
- serve-model-vibe-testSkill
- sft-cleanup-hf-onlySkill
- sft-job-cleanupSkill
- sft-launchSkill
- supervisor-initSkill
- utils-cleanup-stale-sandboxesSkill
- utils-fix-permissionsSkill
- utils-reclaim-stale-snapshotsSkill
- write-testsSkill
- writing-styleSkill
The file
We split this skill into the rules it gives an agent
3 rules. About 0 are common, 1 is distinctive, and 2 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈1
Distinctive · estimate
This author's own choices. Worth a look.
2
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- could be automatic: a setting could enforce it
- Distinctive: line 71
Run the standard Leonardo preamble (ops.md), then pre-stage offline data:
- Very specific: line 164
Use sbatch_gsm8k_grid_multinode.sh, sbatch_math_grid_multinode.sh, or sbatch_opd_qwen3.sh.
- Very specific: line 183
Resume: run scripts set resume_mode=null (fresh) per cell to avoid cross-cell stale-global_step resume.
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
⚠ Do not add comments to YAMLs. Report your recommendations directly to the supervisor.
rl-standard-launch-leonardo
⚠ VERIFY checkpoint/export paths resolve to
$WORK($CHECKPOINTS_DIR), NOT$SF/$SCRATCH_FAST— scratch is 1 TB/over-quota; a ckpt write failsOSError [Errno 122] Disk quota exceededmid-run (NOT an OOM). See.agents/ops/leonardo/ops.md"WRITE-PATH MANDATE".
Standard non-agentic SkyRL RL on Leonardo: GRPO on local math/reasoning parquet and on-policy distillation (OPD). Compute nodes are offline: no Harbor, Daytona, terminal_bench, or proxyserver.
Authoritative source docs (this skill distills them — read for full numbers):
notes/RL/gsm8k_grid_leonardo/—grid.md(throughput),accuracy_grid.md(pass@8 to convergence),grid_experiment_log.md(methodology),scripts/.notes/RL/opd_grid_leonardo/—leonardo_opd_qwen3_plan.md,grid.md,throughput_grid.md.- Leonardo access boilerplate (ssh/2FA, preamble, code/data paths, step-ca cert,
login-node killer) →
.agents/ops/leonardo/ops.md+CLAUDE.md.
Launch with an
sbatchwrapper inhpc/skyrl_standard/leonardo/, nothpc.launch. It uses a writable sandbox directory and external uv venv, then calls the SkyRL entrypoint directly.
1. Cluster + env facts
A100-64GB, 4 GPUs/node, x86_64, SLURM. Account AIFAC_5C0_290, partition
boost_usr_prod. QOS: boost_qos_dbg (≤30 min, ≤2 nodes) or normal
(more nodes; 24h max).
- MarinSkyRL =
marin-community/MarinSkyRLmain @ 9bb6d5eat$WORK/code/MarinSkyRL($WORK = /leonardo_work/AIFAC_5C0_290/bfeuer00). Container--pwd=MarinSkyRL/skyrl-train→ SkyRL fixes go to MarinSkyRLmain. - Image = writable sandbox dir
$SF/marinskyrl_sandbox($SF = /leonardo_scratch/fast/AIFAC_5C0_290/bfeuer00), fromdocker://anyscale/ray:2.51.1-slim-py312-cu128. Binary/usr/bin/singularity(SingularityPRO 4.3.1; noapptaineron PATH). - uv, not conda: venv
$SF/marin_venv(uv sync --extra vllm→ torch 2.8.0+cu128, vLLM 0.11.0, flash-attn 2.8.3).$VENV_PY=$VENV/bin/python.
Standing gotchas
- No compute-node internet:
HF_HUB_OFFLINE=1,TRANSFORMERS_OFFLINE=1,WANDB_MODE=offline,HF_HOME/HF_HUB_CACHE=$WORK/data/hub. Pre-stage model + parquet on the LOGIN node first. - gcc for Triton JIT — the ray base image ships no compiler. Wrapper binds
host miniforge
$WORK/miniforge3/envs/otagent/binonto PATH + exportsCC/CXX(gcc 14.3.0).RAY_USAGE_STATS_ENABLED=0. - HOME is read-only in-container: set
HOME=$SF/canary_home,ckpt_path/export_pathat writable$SF. The/leonardo/homeROFileNotFoundError/Read-only file system/Tracebacklines (tvm_ffi dlpack, vLLM telemetry) are benign engine-init noise — ignore them.
2. Pre-launch (login node, tmux)
Run the standard Leonardo preamble (ops.md), then pre-stage offline data:
ssh Leonardo # step-ca cert; 2FA once (ops.md)
cd /leonardo_work/AIFAC_5C0_290/bfeuer00/code/MarinSkyRL && GIT_TERMINAL_PROMPT=0 git pull
# Pre-stage on the LOGIN node (compute has no internet):
hf download Qwen/Qwen2.5-1.5B-Instruct # → $WORK/data/hub
# gsm8k parquet → $WORK/data/gsm8k/{train,validation}.parquet (MarinSkyRL examples/gsm8k/gsm8k_dataset.py)
# MATH: hpc/skyrl_standard/leonardo/math_dataset.py → $WORK/data/math/Edit code locally, commit/push, and git pull on Leonardo; never patch remote files.
3. Launch — single node
cd /leonardo_work/AIFAC_5C0_290/bfeuer00/code/OpenThoughts-Agent/hpc/skyrl_standard/leonardo
sbatch sbatch_gsm8k_canary.sh # bare canary: 1 node × 4 A100, ≤30 minThe sbatch sets DATA_DIR/MODEL_PATH/NUM_GPUS=4/CKPT_DIR + offline env,
then singularity exec --nv --no-home --bind /leonardo_work,/leonardo_scratch
--pwd $MARIN $SANDBOX bash run_gsm8k_canary.sh, which calls
$VENV_PY -m skyrl_train.entrypoints.main_base with the GRPO knobs.
Canary GRPO config (run_gsm8k_canary.sh, Qwen2.5-1.5B-Instruct):
advantage_estimator=grpo, strategy=fsdp2, colocate_all=true,
backend=vllm, run_engines_locally=true, weight_sync_backend=nccl,
async_engine=true, 4 engines × TP1, use_kl_loss=false, lr=1e-6,
n_samples_per_prompt=4, train_batch_size=32, max_prompt_length=512,
max_generate_length=512, gpu_memory_utilization=0.70, env_class=gsm8k,
epochs=1, logger=console (offline). Reference: job 44478923 COMPLETED,
233-step epoch, 9.58 s/step, reward 0.14→0.64, pass@4 0.78.
Grid-cell overrides
run_gsm8k_canary.sh ends in "$@" (trailing hydra overrides, last-wins), but
sbatch_gsm8k_canary.sh does NOT forward "$@" — for grid cells use
sbatch_gsm8k_grid.sh (passthrough + fresh per-cell CKPT_DIR, rm -rf'd
before launch):
sbatch --job-name=grid_cudagraph sbatch_gsm8k_grid.sh generator.enforce_eager=false
sbatch --job-name=grid_tbs128 sbatch_gsm8k_grid.sh trainer.train_batch_size=128 trainer.policy_mini_batch_size=128--job-name=grid_<cell> is load-bearing: the script derives
CKPT_DIR=$SF/grid_ckpts/${SLURM_JOB_NAME#grid_} from it. Per-cell scripts in
notes/RL/gsm8k_grid_leonardo/scripts/run_<cell>.sh; launchers
launch_throughput_grid.sh/launch_accuracy_grid.sh + catalogs *_grid_cells.txt
in hpc/skyrl_standard/leonardo/.
4. Grid structure (one-factor-at-a-time off the base)
- Throughput grid (
grid.md, 18 cells, maximize sec/step / eff tok/s): variestrain_batch_size(32→512),n_samples_per_prompt(4→16),gpu_memory_utilization(0.70→0.85),enforce_eager(CUDA graphs), engine layout (4×TP1 vs 2×TP2 vs 1×TP4),micro_*_batch_size_per_gpu,reshard_after_forward,colocate_all. Winners:enforce_eager=false= −29% sec/step (always on); 4×TP1 > 2×TP2 > 1×TP4; colocated > disaggregated at 4 GPU. Base width is gen-bound (cudagraph fixes it); past ~tbs128 it'spolicy_traincompute-bound; never memory-bound (KV <11%). - Accuracy grid (
accuracy_grid.md, 20 cells, maximize pass@8 to convergence off the throughput winnercombo_C): varies lr (dominant lever; GRPO knee 1e-5, 3e-7 undertrains, 3e-5 unstable),n_samples(n8 winner),max_generate_length(gen1024 winner),use_kl_loss/kl_loss_coef, rollout temp,eps_clip_high(DAPO), entropy bonus (use_entropy_loss=true, entropy_loss_coef=0.01= anti-collapse winner), reward shaping. Best:combo_acc(lr1e-5 + n8 + gen1024) → pass@8 ~0.97 but entropy collapses;combo_acc_stab(+ entbonus) holds ~0.95–0.98 WITHOUT collapse.
gsm8k: short CoT (~245–268 tokens), exact-match ±1 reward, and lr knee 1e-5. MATH/aime needs
max_generate_length=4096; 32B OOMs on a single 4×A100-64GB node, so use multi-node.
5. OPD — on-policy distillation (teacher→student)
Student (Qwen3-1.7B) generates; per-token reward = −KL(student‖teacher) over the
student's tokens. Entrypoint
examples.on_policy_distillation_logits.main_on_policy_distill_logits (NOT
main_base, NOT the agentic main_tbench_opd_logits which needs Daytona). Knobs:
advantage_estimator=no_op, policy_loss_type=importance_sampling,
use_kl_in_reward=true, use_kl_loss=false; the FSDP ref worker is loaded with
the teacher + a separate vLLM-served teacher supplies top-K logprobs
(teacher.top_k_logprobs).
sbatch sbatch_opd_qwen3.sh # smoke defaults (2 nodes, ≤90 min)
sbatch --job-name=opd_q3_full --time=08:00:00 sbatch_opd_qwen3.sh \
MAX_STEPS=60 EPOCHS=2 TRAIN_BATCH_SIZE=64 MINI_BATCH_SIZE=64 N_SAMPLES=8 MAX_GEN_LEN=1024 TOPK=128Layout (2 nodes × 4 A100-64GB): student colocated (FSDP2 ↔ 4× vLLM TP1) on node-0; teacher Qwen3-32B TP2 (32B bf16 ≈ 64 GB > one 64 GB card) on its own Ray PACK PG on node-1; 2 GPUs spare. Shared tokenizer → retokenization is a no-op (Qwen3-1.7B is the nearest size to a nonexistent 1.5B).
OPD is teacher-score-bound (90–97% of each step). The speed lever is top_k; the lr knee is 3e-5.
Recommended OPD: lr=3e-5, top_k=64, n_samples=8, gen=1024, teacher TP2, cudagraph off.
6. Multi-node
Use sbatch_gsm8k_grid_multinode.sh, sbatch_math_grid_multinode.sh, or sbatch_opd_qwen3.sh. These start a
Ray head on node 0, attach workers, and launch the trainer with RAY_ADDRESS. Keep these gotchas:
- InfiniBand
ib0pinned:NCCL_SOCKET_IFNAME=ib0,GLOO_SOCKET_IFNAME=ib0; head IP resolved fromib0(not theeno*mgmt addr). - Ray
--temp-dir=/tmp(not Lustre scratch): the AF_UNIX plasma-store socket path cannot exceed 107 bytes; the Lustre scratch root is already ~55 chars → a temp-dir there overflows. (verifyRAY_TMPin the script before relaunch.) - gsm8k/1.5B multi-node generator scaling does NOT help (train-bound, not gen-bound). Multi-node pays off only for big models (≥32B, single-node-OOM) or genuinely gen-bound small-model long-CoT.
7. Monitoring + completion
- Monitor detached: poll
%x_%j.outfor the per-stepWANDB_MIRROR kind=train step=N metrics={...}JSON lines (offline → stdout). Watchtiming/{step,generate,policy_train,sync_weights}, GRPO reward +policy/policy_entropy(collapse guard, mandatory), grad_norm. OPD:distill/token_kl_mean(should DECREASE),teacher/chosen_logprob_mean, entropy. Sweep cadence →monitor-cron-sweep. - Resume: run scripts set
resume_mode=null(fresh) per cell to avoid cross-cell stale-global_stepresume. Genuine resume:resume_mode=latest+ keepckpt_pathstable; clean re-run:rm -rfthe ckpt dir first.- ⚠ DESTRUCTIVE — pass
RESUME_MODE/DATA_DIRvia--export, NOT positionally.sbatch_delphi_math_rl_multinode.shreads them from the environment; its positional parser only stripsMODEL_PATH/RUN_NAME/STAGE/DATASET/THINK/THINK_MODE/DELPHI_TEMPLATE. PositionalKEY=valtokens leak to hydra →Could not override 'RESUME_MODE'→ head FAILS. An unset/invalidRESUME_MODEis a HARDexit 1(you MUST pass it explicitly):sbatch --export=ALL,DATA_DIR=<path>,RESUME_MODE=latest sbatch_delphi_math_rl_multinode.sh <positional… only>(fresh cell:…,RESUME_MODE=null …).
- ⚠ DESTRUCTIVE — pass
- 24h wall:
boost_usr_prodcaps at23:59:00; OPD full (~24 min/step) fits only ~18–20 steps/slot → ckpt every few steps and chain--dependency=afterany:. - Completion →
rl-standard-job-cleanupfor upload, optional registration, metrics, and cleanup. Measurement runs with throwaway checkpoints only clean disk.
8. Guardrails
- Launch via
sbatch hpc/skyrl_standard/leonardo/sbatch_*.sh, NOTpython -m hpc.launch, NOT a.sif, NOT--rl_use_conda. - Fully offline — pre-stage model + parquet on the login node; the
/leonardo/homeROFileNotFoundError/Tracebacklines are benign (§1.3). - Grid cells need
sbatch_gsm8k_grid.sh(the"$@"-forwarding wrapper) + a unique--job-name=grid_<cell>→ fresh per-cell ckpt dir; the bare canary sbatch does NOT forward overrides. Never share a ckpt dir across cells. - A100-64GB ceilings:
gpu_memory_utilization≤ 0.85 (≥0.90 OOMs eval); dense ≥32B and MoE 30B-A3B OOM single-node → multi-node/disaggregated. - Never alter hparams mid-series (controlled grid) — flag + propose a separate cell. Entropy/log-ratio/grad-norm are mandatory monitoring columns.
- Multi-node:
ib0NICs + Ray--temp-dirshort path (107-byte AF_UNIX limit). - SkyRL fixes → MarinSkyRL
main, pushed + pulled on Leonardo; never patch remote files. - Agentic RL (Harbor/Daytona/TBench) is INFEASIBLE on Leonardo (no compute-node internet) — different skill.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

