datagen-launch
Skill309 starsChanged 4 months agoApache-2.0View on GitHub
Launch a datagen (trace-generation) job on an HPC cluster (Jupiter/Leonardo/Perlmutter) via the OpenThoughts-Agent `hpc.launch --job_type datagen` entrypoint — the cluster-AGNOSTIC general flow: extract tasks from a parquet, then submit a managed vLLM-serve + Harbor/Daytona trace run that uploads the trajectories to an HF repo. Use when asked to start a datagen / trace-generation job, generate agent traces from a task dataset, or advance / start a row in the MiniMax-M2.7 datagen tracker. For the Iris TPU path use **datagen-launch-iris** instead; per-cluster particulars (ssh, paths, conda env, which vllm-serve config matches the cluster's GPUs, JSC pre-download) live in `.agents/ops/<cluster>/`.
- Reads credentials
In this repository50 other files it gives its agents.
- CLAUDE.mdCLAUDE.md
- analyze-datagen-campaign-summarySkill
- analyze-dataset-token-lengthSkill
- analyze-id-eval-rankingSkill
- analyze-job-history-irisSkill
- analyze-rl-behaviorSkill
- analyze-training-run-irisSkill
- build-tpu-image-irisSkill
- code-create-staged-planSkill
- code-execute-staged-planSkill
- commitSkill
- crud-archive-runSkill
- crud-otagent-supabaseSkill
- crud-purge-below-gate-evalsSkill
- crud-purge-stale-eval-placeholdersSkill
- datagen-create-task-datasetSkill
- datagen-job-cleanupSkill
- datagen-launch-irisSkill
- datagen-reduce-dataset-snapshotsSkill
- datagen-standard-launchSkill
- debugSkill
- docs-deslopSkill
- eval-agentic-cleanupSkill
- eval-agentic-launch-irisSkill
- eval-agentic-launchSkill
- eval-standard-cleanupSkill
- eval-standard-launchSkill
- file-issueSkill
- monitor-cron-sweep-irisSkill
- monitor-cron-sweepSkill
- monitor-job-tablesSkill
- monitor-restore-irisSkill
- monitor-restoreSkill
- monitor-restore-unifiedSkill
- rl-agentic-job-cleanupSkill
- rl-agentic-launch-irisSkill
- rl-agentic-launch-jupiterSkill
- rl-job-health-deep-diveSkill
- rl-standard-job-cleanupSkill
- rl-standard-launch-leonardoSkill
- serve-model-vibe-testSkill
- sft-cleanup-hf-onlySkill
- sft-job-cleanupSkill
- sft-launchSkill
- supervisor-initSkill
- utils-cleanup-stale-sandboxesSkill
- utils-fix-permissionsSkill
- utils-reclaim-stale-snapshotsSkill
- write-testsSkill
- writing-styleSkill
The file
We split this skill into the rules it gives an agent
8 rules. About 0 are common, 4 are distinctive, and 4 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈4
Distinctive · estimate
This author's own choices. Worth a look.
4
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 79
After submit, confirm the job placed and is producing real trajectories (a served /v1/models healthcheck alone does NOT prove generation …
- Very specific: line 80
Check the job state (squeue/sacct per .agents/ops/<cluster>/).
- Very specific: line 88
Oversized pre-check: before extracting a row, query the HF dataset-viewer /size API …
- Distinctive: line 92
If a dataset's snapshot footprint blocks future launches, shrink it with datagen-reduce-dataset-snapshots.
- Distinctive: line 96
NEVER launch with the wrong (RL-org) Daytona key — it silently zeroes the run.
- Distinctive: line 97
NEVER raise a Daytona snapshot org cap or downgrade its error to a warning.
- Distinctive: line 98
NEVER change experimental configs / hparams mid-series.
- Very specific: line 99
Keep cluster-specific values (paths, env, configs, ssh) in .agents/ops/<cluster>/, not inlined here.
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
datagen-launch
A datagen job stands up a managed vLLM endpoint serving a teacher model, runs Harbor agent rollouts against a set of tasks in Daytona sandboxes, and uploads the resulting trajectories to an HF dataset repo. There is no model checkpoint — the artifact is the trace dataset.
This skill is cluster-AGNOSTIC — the shared flow, flags, and gotchas. Defer everything cluster-specific (ssh / login node, repo path, conda env, JSC login-node pre-download, which datagen_config (vllm-serve) YAML matches the cluster's GPUs and the model) to .agents/ops/<cluster>/. Do NOT inline cluster-specific values here.
Sibling skills: datagen-launch-iris (Iris TPU path), datagen-job-cleanup (post-run upload + disk cleanup), datagen-reduce-dataset-snapshots (shrink a task dataset's Daytona snapshot count under cap).
Required info (ask if missing)
- Dataset / tasks — an HF parquet repo id to extract tasks from (e.g.
DCAgent/r2egym_sandboxes), or an already-extracted local tasks dir. - Teacher model + operating point — determines the
datagen_config(vllm-serve) YAML and thetrace_harbor_configcontext-length YAML. Pick the cluster-matched config from.agents/ops/<cluster>/; don't change configs unless asked. - Target HF repo —
--trace_target_repo(defaultpenfever/orDCAgent2/org per the series).
The 2-step flow
Step 1 — extract tasks from the parquet (writes one task dir per row):
python -m scripts.datagen.extract_tasks_from_parquet \
--parquet <hf-parquet-repo-or-local> \
--output_dir <tasks dir> \
--on_exist overwriteStep 2 — submit the datagen job:
🚧 SUBMIT FROM THE REPO DIR WITH
DCFTSET — the sbatch WORKDIR guard hard-fails otherwise. Before launching/relaunching,cd <repo> && export DCFT=$PWD(on Jupiter:/e/scratch/jureap59/feuer1/OpenThoughts-Agent). The generateduniversal_tracegen.sbatch/universal_taskgen.sbatchresolveWORKDIRfromDCFT_PRIVATE → DCFT → $PWD; submitted from$HOME/a scratch subdir withDCFTunset, the guard detects the wrong dir (missinghpc/shell_utils/triton_cache.shmarker) andexit 1s immediately withFATAL: WORKDIR=... is not the OpenThoughts-Agent repo root. If you see that FATAL,cdto the repo,export DCFT=$PWD, resubmit.
python -m hpc.launch \
--job_type datagen \
--datagen_config <vllm-serve cfg>.yaml \
--trace_harbor_config hpc/harbor_yaml/datagen/<ctx>.yaml \
--tasks_input_path <tasks dir> \
--trace_target_repo <hf repo> \
--daytona_api_key "$DAYTONA_DATA_API_KEY" \
--num_nodes 1 \
--trace-n-concurrent NCRITICAL gotchas (get these wrong → silent total failure)
--daytona_api_key "$DAYTONA_DATA_API_KEY"is MANDATORY. Must be the datagen-org Daytona key, NOT the default RL-org key. The RL-org key rejects declarative sandbox builds, so every trial instant-fails (job looks "running" but produces zero real trajectories). Source the key from your secrets env (see CLAUDE.md "Datagen Daytona Key (CRITICAL)" /.agents/ops/<cluster>/). Do NOT echo the value. (Exact env-var nameDAYTONA_DATA_API_KEY— verify against your local secrets env / ops doc.)- harbor_config needs
auto_snapshot: true. Attaches the prebuilt Daytona snapshot instead of doing a slow per-trial declarative build. Use ahpc/harbor_yaml/datagen/<ctx>.yamlthat sets it. - Do NOT export
DAYTONA_TARGET. Leaving it set misroutes the run. - Per-job effective concurrency caps at ~100–128 (per-job ceiling, not aggregate contention).
--trace-n-concurrent(and the config'sseqs/max_num_seqs) above ~128 is wasted. Parallel datagen jobs are fine; over-fanning a single job is not. - Daytona snapshot org cap is a HARD, server-side limit (e.g. 40 or 60 per org depending on org). Never raise the cap or convert its
ValueError/SnapshotCapExceededinto a warning. If a dataset overflows: clean stale/MISSING snapshots, shrink the dataset's snapshot footprint with datagen-reduce-dataset-snapshots, or reuse a dataset whose snapshots already exist (0 new). If still blocked, ask the user. Snapshots are keyed by sandbox environment, not dataset — they're shared, so do NOT reclaim them per-run. Clean stale at the cap (autonomous):python scripts/daytona/daytona_snapshot_manager.py --api-key-env DAYTONA_DATA_API_KEY --delete-stale --yes— at the stale threshold defined in.agents/projects/daytona/daytona.md§ "How to clean stale snapshots" (GT — don't restate the value) (audit-only without--delete-stale; deletes only idle/unprotectedharbor__*envs). Full procedure + caveats →.agents/projects/daytona/daytona.md. - FD-exhaustion / libuv SIGABRT at ~1h on boundary-hugging datasets:
model_info.max_input_tokensdoes nothing whenenable_summarize=false(our RL/trace default) — nothing truncates the growing multi-turn prompt →VLLMValidationError: 32769 input tokensoverflow. Lowering the token budget is INERT. Real levers: (a) dropn_concurrent_trials(e.g. 900→500), then (b) disable litellm's async logging worker on the hosted_vllm path (itsclear_queue exceeded max_timebacklog leaks FDs ∝ overflow_rate × concurrency). - Reward-realness caveat: LLM-judge datasets can score all-0.0 if the judge API key is invalid/unplumbed — inspect the reward distribution at consolidation, not just avg turns.
Chunking (long datasets)
For large task sets, split into chunked sub-jobs so each gets its own vLLM endpoint, walltime, and HF _chunk{N} repo (avoids one giant job timing out and stranding traces, keeps trace_jobs/ inode use bounded per job):
--chunk_size <S>→ ~ceil(num_tasks / S)chunks, each a separate job + serve +_chunk{N}HF repo.--chunk_array_max <M>→ at most M chunks run concurrently (rollingafteranydependency chain).- Alternatively, a manual
--dependency "afterany:<jobid>"chain serializes individually-launched jobs. - Route
trace_jobs/off inode-tight scratch via--experiments_dir <path>where required by the cluster (see.agents/ops/<cluster>/); without it datagen can write thousands of per-trial dirs per chunk onto a quota-tight filesystem. - Respect the cluster's QOS max walltime via
--time_limit(see ops doc). - The snapshot org cap still applies across chunks — see the HARD-cap gotcha above.
- Curator sharded datagen: pass
--save-every 700(5th positional arg), not 200 → ~4 checkpoints/12h-job instead of ~14, each checkpoint has a serial GPU-wasting pause.
Verify it's running
After submit, confirm the job placed and is producing real trajectories (a served /v1/models healthcheck alone does NOT prove generation works):
- Check the job state (
squeue/sacctper.agents/ops/<cluster>/). - Tail the run's
_vllm.logfor a successful serve, then confirm trial dirs under<run>/trace_jobs/<inner>/are accumulating multi-step trajectories (avg turns > 1), not 1-turn exception stubs. The real-vs-failed check is detailed in datagen-job-cleanup step 2.
MiniMax-M2.7 tracker workflow (the "datagen #N" queue)
The MiniMax-M2.7 131k series is driven by a canonical tracker that is the source of truth — follow its Launch Command and Process Notes EXACTLY, not the ad-hoc command grab-bag. (Tracker path lives in the reference-minimax-datagen-tracker memory note; verify the current path there.) Key conventions:
- One dataset in flight at a time (datasets strictly sequential).
- Per-dataset autonomous N→N+1 loop: extract → launch chunks → wait for ALL chunks to complete → consolidate the
_chunk{N}repos into the final<slug>-…-tracesrepo (scripts/datagen/join_hf_repos.py) → verify (row count == sum of chunks, non-empty, realness: avg turns > 1, sane exception rate) → DELETE the_chunk{N}repos (only after the consolidated repo is verified) → clean local exp/task dirs (detachedrm) → launch dataset N+1. Advance WITHOUT blocking (on chunk-completion / on the cron). - Oversized pre-check: before extracting a row, query the HF dataset-viewer
/sizeAPI (https://datasets-server.huggingface.co/size?dataset=<repo>); skip rows >~10k rows (each oversized extraction burns ~90–100k GPFS inodes).
On completion
- Upload + verify traces and free disk with datagen-job-cleanup (handles the TIMEOUT-stranded-traces case, the one-level
trace_jobs/nesting, the real-vs-failed sanity check, and safe disk cleanup). - If a dataset's snapshot footprint blocks future launches, shrink it with datagen-reduce-dataset-snapshots.
- Traces → tool-calling SFT: to turn the generated traces into SFT-ready rows (
role: tool+ structuredtool_calls, not the lossy defaultconversationsshape), useharbor traces export --sft-format→.agents/projects/harbor/ops.md§ "SFT-ready traces with tool calling —harbor traces export --sft-format". For byte-exact-from-served-tokens SFT instead, the literal path is § "Literal-token trace datasets" in the same doc.
Guardrails
- NEVER launch with the wrong (RL-org) Daytona key — it silently zeroes the run.
- NEVER raise a Daytona snapshot org cap or downgrade its error to a warning.
- NEVER change experimental configs / hparams mid-series; flag and propose a separate run instead.
- Keep cluster-specific values (paths, env, configs, ssh) in
.agents/ops/<cluster>/, not inlined here.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

