sft-cleanup-hf-only
Skill309 starsChanged 4 months agoApache-2.0View on GitHub
Clean up a completed NON-AGENTIC / HF-only SFT model — HF upload WITHOUT Supabase DB registration. The counterpart to sft-job-cleanup (which uploads AND registers). Use when a completed SFT cell belongs to an HF-only series (config `enable_db_registration: false`, or a known HF-only series like the Delphi #6279 54-grid): upload the weights to laion/ and STOP — do NOT run manual_db_push. Covers cell→hub_model_id resolution (incl. the Delphi launch_54_map.tsv), the Leonardo sbatch-tunnel upload, and the downstream eval hand-off. For DB-registered models use sft-job-cleanup instead.
In this repository50 other files it gives its agents.
- CLAUDE.mdCLAUDE.md
- analyze-datagen-campaign-summarySkill
- analyze-dataset-token-lengthSkill
- analyze-id-eval-rankingSkill
- analyze-job-history-irisSkill
- analyze-rl-behaviorSkill
- analyze-training-run-irisSkill
- build-tpu-image-irisSkill
- code-create-staged-planSkill
- code-execute-staged-planSkill
- commitSkill
- crud-archive-runSkill
- crud-otagent-supabaseSkill
- crud-purge-below-gate-evalsSkill
- crud-purge-stale-eval-placeholdersSkill
- datagen-create-task-datasetSkill
- datagen-job-cleanupSkill
- datagen-launch-irisSkill
- datagen-launchSkill
- datagen-reduce-dataset-snapshotsSkill
- datagen-standard-launchSkill
- debugSkill
- docs-deslopSkill
- eval-agentic-cleanupSkill
- eval-agentic-launch-irisSkill
- eval-agentic-launchSkill
- eval-standard-cleanupSkill
- eval-standard-launchSkill
- file-issueSkill
- monitor-cron-sweep-irisSkill
- monitor-cron-sweepSkill
- monitor-job-tablesSkill
- monitor-restore-irisSkill
- monitor-restoreSkill
- monitor-restore-unifiedSkill
- rl-agentic-job-cleanupSkill
- rl-agentic-launch-irisSkill
- rl-agentic-launch-jupiterSkill
- rl-job-health-deep-diveSkill
- rl-standard-job-cleanupSkill
- rl-standard-launch-leonardoSkill
- serve-model-vibe-testSkill
- sft-job-cleanupSkill
- sft-launchSkill
- supervisor-initSkill
- utils-cleanup-stale-sandboxesSkill
- utils-fix-permissionsSkill
- utils-reclaim-stale-snapshotsSkill
- write-testsSkill
- writing-styleSkill
The file
We split this skill into the rules it gives an agent
4 rules. About 0 are common, 0 are distinctive, and 4 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈0
Distinctive · estimate
This author's own choices. Worth a look.
4
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 19
If it's a normal registered SFT → use sft-job-cleanup (don't skip the DB step there).
- Very specific: line 22
Confirm HF-only completion: Check enable_db_registration: false (or Delphi delphi-*) and final training progress.
- Very specific: line 31
Leonardo: use the sbatch compute-node + SSH-tunnel upload (.agents/ops/leonardo/ops.md "Leonardo HF Upload".
- Very specific: line 46
HF_HUB_ENABLE_HF_TRANSFERMUST be 0/unset: when pre-caching the -sft repos into $HF_HUB_CACHE for eval — hf_transfer isn't in the …
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
sft-cleanup-hf-only
Publish the completed HF-only SFT model without Supabase registration. Use sft-job-cleanup for registered SFT.
When to use
- A completed SFT cell whose config sets
enable_db_registration: false, OR a known HF-only series (Delphi #6279delphi-*cells). - Discovered during a sweep, or "upload the completed Delphi cells (no DB)".
- If it's a normal registered SFT → use
sft-job-cleanup(don't skip the DB step there).
Procedure (per completed cell)
- Confirm HF-only completion. Check
enable_db_registration: false(or Delphidelphi-*) and final training progress. - Cancel that cell's pending restart chain (
squeue … | grep <job> | grep PENDING | awk … | xargs -r scancel). - Prepare weights: 8B uses root
model.safetensors; 32B/ZeRO-3 needs consolidation. Drop intermediatecheckpoint-*and.cache. - Resolve
hub_model_id:- Delphi: name is
laion/delphi-<base>-<recipe>_lr1e5-sft; the authoritative cell→hub_model_id↔jobid map is/leonardo_work/AIFAC_5C0_290/bfeuer00/experiments/delphi-prepared-tok/launch_54_map.tsv. Resolve from there (don't guess). - Otherwise read the rendered launch config's
hub_model_id. - Qwen3.5: copy
preprocessor_config.jsonfrom the base model into the checkpoint before upload. 4b. Check the tokenizer before upload:extra_special_tokensintokenizer_config.jsonmust be a dict, not a list (python -c "import json;d=json.load(open('<ckpt>/tokenizer_config.json'));assert isinstance(d.get('extra_special_tokens',{}),dict)"). If it is a list, set it to{}and re-save; the RL/SkyRL loader expects a dict.
- Delphi: name is
- Upload to HF (public default):
- Leonardo: use the sbatch compute-node + SSH-tunnel upload (
.agents/ops/leonardo/ops.md"Leonardo HF Upload"; usehf upload, notupload-large-folder). - Jupiter: login node has direct internet →
hf uploadin tmux. source "$DC_AGENT_SECRET_ENV"forHF_TOKEN(env var only — never inline).
- Leonardo: use the sbatch compute-node + SSH-tunnel upload (
- Do not run
manual_db_push.py. - Clean the cell's exp/checkpoint dir after the upload is verified (detached
rmon GPFS; nodu/find). - Verify: the
laion/<name>repo exists withmodel.safetensors(>500MB) + tokenizer/config. Note the cell as done.
Downstream chain (every sweep)
- SFT completes → HF upload (this skill; no DB).
- upload completes →
eval-standard-launchon the series' eval grid for the newly-uploaded cell(s). - eval completes → record scores in the experiment tracker — for Delphi, append the cell's result to
/Users/benjaminfeuer/Documents/experiments/active/delphi/rl-scaling-laws-6279/main_sft_evals/SCORES.md. Pull the score from the completeddelphi-eval/<RUN>/output + the eval job's metrics.
Skip work already done.
Delphi eval traps (carry into the eval hand-off)
HF_HUB_ENABLE_HF_TRANSFERMUST be 0/unset when pre-caching the-sftrepos into$HF_HUB_CACHEfor eval —hf_transferisn't in theevalchemyenv, so=1leaves an EMPTY.incompleteblob (silent no-op). Verify the snapshot hasmodel.safetensors(>500MB) before submitting.- The
-sftrepos ship the chat template as a separatechat_template.jinjafile (NOT intokenizer_config.json'schat_templatekey) — the eval sbatch'sdelphi_v0override reads/replaces that file. - Per-cell TP must divide
num_attention_heads: 9e18 heads=9→TP=1, 2e19 heads=11→TP=1, 3e19 heads=12→TP=2 (the sbatch's hardcoded TP=2 is WRONG for 9e18/2e19 — pass the optional TP_OVERRIDE 4th positional arg).
Full Delphi series data:
project_delphi_sft_hf_only_no_db; upload mechanics:.agents/ops/leonardo/ops.md.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

