sft-job-cleanup
Skill309 starsChanged 4 months agoApache-2.0View on GitHub
Publish + clean up a finished LLaMA-Factory SFT job on a no-internet HPC cluster (Jupiter/Leonardo): cancel pending retries, drop intermediate checkpoints, HF-upload the model to its configured --hub_model_id, register in Supabase via manual_db_push (--training-type SFT default), and free disk. Covers the 8B path (root safetensors, direct upload), the 32B/ZeRO-3 path (consolidate shards → safetensors first), the Qwen3.5 preprocessor_config copy, the don't-upload-partials policy, and the hf-upload gotchas (tmux not nohup, `hf upload` not `-large-folder`, Leonardo sbatch-tunnel not login node). Use when an SFT fine-tune finishes and needs uploading + registering, or "run the SFT cleanup checklist". Distinct from RL cleanup (rl-agentic-job-cleanup) and datagen cleanup (datagen-job-cleanup).
- Reads credentials
- Deletes or force-pushes
In this repository50 other files it gives its agents.
- CLAUDE.mdCLAUDE.md
- analyze-datagen-campaign-summarySkill
- analyze-dataset-token-lengthSkill
- analyze-id-eval-rankingSkill
- analyze-job-history-irisSkill
- analyze-rl-behaviorSkill
- analyze-training-run-irisSkill
- build-tpu-image-irisSkill
- code-create-staged-planSkill
- code-execute-staged-planSkill
- commitSkill
- crud-archive-runSkill
- crud-otagent-supabaseSkill
- crud-purge-below-gate-evalsSkill
- crud-purge-stale-eval-placeholdersSkill
- datagen-create-task-datasetSkill
- datagen-job-cleanupSkill
- datagen-launch-irisSkill
- datagen-launchSkill
- datagen-reduce-dataset-snapshotsSkill
- datagen-standard-launchSkill
- debugSkill
- docs-deslopSkill
- eval-agentic-cleanupSkill
- eval-agentic-launch-irisSkill
- eval-agentic-launchSkill
- eval-standard-cleanupSkill
- eval-standard-launchSkill
- file-issueSkill
- monitor-cron-sweep-irisSkill
- monitor-cron-sweepSkill
- monitor-job-tablesSkill
- monitor-restore-irisSkill
- monitor-restoreSkill
- monitor-restore-unifiedSkill
- rl-agentic-job-cleanupSkill
- rl-agentic-launch-irisSkill
- rl-agentic-launch-jupiterSkill
- rl-job-health-deep-diveSkill
- rl-standard-job-cleanupSkill
- rl-standard-launch-leonardoSkill
- serve-model-vibe-testSkill
- sft-cleanup-hf-onlySkill
- sft-launchSkill
- supervisor-initSkill
- utils-cleanup-stale-sandboxesSkill
- utils-fix-permissionsSkill
- utils-reclaim-stale-snapshotsSkill
- write-testsSkill
- writing-styleSkill
The file
We split this skill into the rules it gives an agent
3 rules. About 0 are common, 2 are distinctive, and 1 is very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈2
Distinctive · estimate
This author's own choices. Worth a look.
1
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 16
After an SFT job completes on Jupiter or Leonardo, publish the model and clean up.
- Very specific: line 30
Don't upload partials: if training is below 100%, relaunch and auto-resume.
- Distinctive: line 64
Wait for it to finish and verify the repo exists on HF Hub.
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
sft-job-cleanup
After an SFT job completes on Jupiter or Leonardo, publish the model and clean up.
Recognition heuristic — which path? (check the checkpoint root first)
ls $CHECKPOINTS_DIR/<job_name>/ | grep -E 'safetensors|global_step'- Root
model-*.safetensors→ 8B path (including Qwen3.5; no consolidation). global_stepN/pluszero_to_fp32.py, without root safetensors → 32B path (consolidate ZeRO-3 shards).
Cross-cutting upload rules (apply to both paths)
hf upload, NEVERhf upload-large-folder(deprecated stub + deadlocks on HF LFS 429s). Wrap any non-trivial upload intmux, notnohup/disown.--privateis a no-value flag — omit it (default public);--private falseis a CLI parse error.- Jupiter: login node has direct internet →
hf uploadfrom the login node (in tmux) works. - Leonardo: the login node SIGKILLs long processes at ~100s → use the sbatch compute-node + SSH-tunnel upload (
.agents/ops/leonardo/ops.md"Leonardo HF Upload — Use sbatch, NOT the Login Node" /sft-launch). - Don't upload partials: if training is below 100%, relaunch and auto-resume. Salvage-upload only with explicit approval, as
laion/<job_name>-<step>-<size>. - Tokenizer sanity check:
tokenizer_config.jsonextra_special_tokensmust be a dict, not a list; replace a list with{}before upload.python -c "import json;d=json.load(open('<ckpt>/tokenizer_config.json'));assert isinstance(d.get('extra_special_tokens',{}),dict), 'LIST — coerce to {}'"
8B SFT Job Cleanup Checklist
0. Cancel pending retries (so stale restarts don't fire mid-upload):
squeue -u $USER --format='%i %j %T' | grep <job_name> | grep PENDING | awk '{print $1}' | xargs -r scancel1. Remove intermediate checkpoints (don't upload cruft):
rm -rf $CHECKPOINTS_DIR/<job_name>/checkpoint-* $CHECKPOINTS_DIR/<job_name>/.cache1b. Qwen3.5 only — copy preprocessor_config.json from the base model:
cp /path/to/Qwen3.5-9B/preprocessor_config.json $CHECKPOINTS_DIR/<job_name>/ # or the -27B base2. Upload model weights to HuggingFace. Naming: full final upload (training reached 100%) → the configured --hub_model_id from the launch command (laion/<descriptive_name>, NO step/size suffix — do NOT use the job name verbatim). (Partial salvage, only-if-OK'd → laion/<job_name>-<step>-<size>.)
# Jupiter login node (direct internet). On LEONARDO use the §11 sbatch-tunnel — login-node hf upload dies at ~100s.
source ~/secrets.env
tmux new-session -d -s hf_upload_<short> \
"source ~/secrets.env && hf upload <hub_model_id> $CHECKPOINTS_DIR/<job_name> . \
--repo-type=model 2>&1 | tee $CHECKPOINTS_DIR/<job_name>/upload.log"
# tmux attach -t hf_upload_<short> (Ctrl-b d to detach)Wait for it to finish and verify the repo exists on HF Hub.
3. Register in the unified DB (SFT is the DEFAULT --training-type, no flag needed):
python scripts/database/manual_db_push.py \
--hf-model-id <hub_model_id> --base-model <base_model_hf> \
--dataset-name <dataset_name> # comma-separated for multi-dataset → sets dataset_namesSKIP for HF-only series (e.g. Delphi #6279 — YAMLs set enable_db_registration: false; do not register, and do not pass an anchor as --base-model since that auto-creates a base-model row).
4. Clean up the experiments dir — only after 1–3 succeed:
rm -rf $EXPERIMENTS_DIR/<job_name>32B SFT Job Cleanup Checklist (DeepSpeed ZeRO-3 — consolidate first)
For 32B ZeRO-3 SFT without stage3_gather_16bit_weights_on_model_save: true, consolidate shards before upload.
0. Cancel pending retries (same as 8B).
1. Verify training reached 100% — trainer_log.jsonl shows current_steps == total_steps. Default policy: don't salvage partials (relaunch + resume); only proceed if explicitly OK'd as a partial.
2. Consolidate ZeRO-3 shards → fp32 state_dict → safetensors:
python -m hpc.launch --job_type consolidate \
--consolidate_input $CHECKPOINTS_DIR/<job_name> \
--consolidate_output_repo <hub_model_id> \
--consolidate_workdir <writable_workdir>/<job_name> \
--time_limit 02:00:00 --num_nodes 1Produces <workdir>/<job_name>/final_repo/ with root-level weights, tokenizer, and config. Do not rely on its
final HF push; manually upload after final_repo/ is complete.
3. Manually upload from final_repo/ (NOT the original checkpoint dir — it still holds ZeRO-3 shards). Naming same as 8B (full → --consolidate_output_repo/--hub_model_id, no suffix):
# Jupiter login node. On LEONARDO use the §11 sbatch-tunnel (131GB → ~4 min). tmux; hf upload (not -large-folder).
source ~/secrets.env
tmux new-session -d -s hf_upload_<short> \
"source ~/secrets.env && hf upload <hub_model_id> <consolidate_workdir>/<job_name>/final_repo . \
--repo-type=model 2>&1 | tee <consolidate_workdir>/<job_name>/upload.log"4. Register in the unified DB (same as 8B step 3; SFT is the default; skip for HF-only series).
5. Clean up — only after 2–4 succeed, remove BOTH the sharded checkpoint dir AND the consolidate workdir (32B sharded ckpt ~700GB + workdir ~200GB):
rm -rf $CHECKPOINTS_DIR/<job_name> <consolidate_workdir>/<job_name>Launch-side details (preamble, configs, sbatch patching, the no-internet pre-download) live in the
sft-launchskill (per-cluster particulars inops/<cluster>/ops.md §SFT); this skill is the post-run publish + cleanup.
Operating notes
- Run full cleanup after a completed, 100% job: cancel pending chain, drop checkpoints, upload, register, and clean the experiment directory. Flag obvious anomalies; cancellation of running jobs remains user-driven.
- Multi-dataset DB registration: pass the full comma-separated list to
--dataset-namesodataset_namesis populated (not just onedataset_id). Known limitation: the script stores it as a single string and does NOT trigger themultiple_datasetspath (dataset_idends up null) — verify the right field after registering. Single-dataset--dataset-nameworks fine and populatesdataset_id. - Baseline model versioning (Sera/CoderForge): flat monotonic
-v5/-v6/-v7in HF repo names + README iteration tables, NOT nestedv4-v2/v4-v3. In-flight runs keep their existing names; the NEXT retrain uses the new scheme (next Sera = v5, skipping v4 to avoid colliding with existing v4 artifacts).
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

