monitor-restore-unified
Skill309 starsChanged 4 months agoApache-2.0View on GitHub
Re-register the every-3-hours UNIFIED OPS TICK cron — the CURRENT operator-owned monitor for the qwen3.5-122b-131k-datagen-opencode campaign (keep-3 datagen with autonomous rescue+refill) AND the delphi midtrain 1e23_p33m67_k0p20 (monitor-only), plus a marin+CoreWeave sweep and standing Daytona snapshot cleanup. The cron is session-only and recurring crons auto-expire after 7 days, so it's routinely lost on a session restart. Use at the start of a new session, after a disconnect/restart, or when the user asks to restore the monitor cron. Supersedes monitor-restore-iris (which is the OLDER datagen-OUT-of-scope variant); use THIS skill when the current session owns the 131k datagen campaign + midtrain. The per-tick sweep methodology lives in monitor-cron-sweep-iris; the table format in monitor-job-tables.
- Reads credentials
In this repository50 other files it gives its agents.
- CLAUDE.mdCLAUDE.md
- analyze-datagen-campaign-summarySkill
- analyze-dataset-token-lengthSkill
- analyze-id-eval-rankingSkill
- analyze-job-history-irisSkill
- analyze-rl-behaviorSkill
- analyze-training-run-irisSkill
- build-tpu-image-irisSkill
- code-create-staged-planSkill
- code-execute-staged-planSkill
- commitSkill
- crud-archive-runSkill
- crud-otagent-supabaseSkill
- crud-purge-below-gate-evalsSkill
- crud-purge-stale-eval-placeholdersSkill
- datagen-create-task-datasetSkill
- datagen-job-cleanupSkill
- datagen-launch-irisSkill
- datagen-launchSkill
- datagen-reduce-dataset-snapshotsSkill
- datagen-standard-launchSkill
- debugSkill
- docs-deslopSkill
- eval-agentic-cleanupSkill
- eval-agentic-launch-irisSkill
- eval-agentic-launchSkill
- eval-standard-cleanupSkill
- eval-standard-launchSkill
- file-issueSkill
- monitor-cron-sweep-irisSkill
- monitor-cron-sweepSkill
- monitor-job-tablesSkill
- monitor-restore-irisSkill
- monitor-restoreSkill
- rl-agentic-job-cleanupSkill
- rl-agentic-launch-irisSkill
- rl-agentic-launch-jupiterSkill
- rl-job-health-deep-diveSkill
- rl-standard-job-cleanupSkill
- rl-standard-launch-leonardoSkill
- serve-model-vibe-testSkill
- sft-cleanup-hf-onlySkill
- sft-job-cleanupSkill
- sft-launchSkill
- supervisor-initSkill
- utils-cleanup-stale-sandboxesSkill
- utils-fix-permissionsSkill
- utils-reclaim-stale-snapshotsSkill
- write-testsSkill
- writing-styleSkill
The file
We split this skill into the rules it gives an agent
3 rules. About 0 are common, 1 is distinctive, and 2 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈1
Distinctive · estimate
This author's own choices. Worth a look.
2
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 32
If both prompts are live, keep only one (CronDelete the other).
- Distinctive: line 35
Start of a new session where the 131k datagen campaign / midtrain are in flight.
- Very specific: line 86
If you change the cadence or scope, update BOTH the cron/prompt above and the live job (delete + recreate), and keep …
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
monitor-restore-unified
Read first: Iris tools catalog (
.agents/ops/iris/ops.md) and ops directory (.agents/ops/iris/—ops.mdfor TPUmarinparticulars,ops.mdfor GPU) carry the binding access/preamble/gotchas + helper-script inventory.
The UNIFIED OPS TICK cron is session-only, expires after seven days, and may be lost on restart. Copy the
canonical prompt below verbatim into CronCreate. For the per-tick method use monitor-cron-sweep-iris; for
tables use monitor-job-tables.
Scope — supersedes monitor-restore-iris
monitor-restore-iris installs the older prompt where datagen is out of scope. This skill owns:
- (B) the
qwen3.5-122b-131k-datagen-opencode-iriscampaign — keep-3 steady-state, with autonomous TERMINAL rescue (→ HF with literals) + refill, and confirmed-wedged kill+refill. - (C) the
delphimidtrain1e23_p33m67_k0p20— monitor-only. - (A) the marin+CoreWeave sweep + native-route check + standing Daytona cleanup.
If both prompts are live, keep only one (CronDelete the other).
When to run
- Start of a new session where the 131k datagen campaign / midtrain are in flight.
- The user says the monitor/cron is gone, down, "not firing," or after a disconnect/restart.
- After ~7 days (expiry).
Steps
- Check if it exists —
CronList. If a recurring prompt begins "UNIFIED OPS TICK", do nothing. If this session owns datagen and a stalemonitor-restore-irisvariant is present,CronDeleteit and install the prompt below. - If absent,
CronCreatewith:cron:23 */3 * * *(every 3 h at :23 — off the :00/:30 marks)recurring:trueprompt: the exact text in the fenced block below.
- Tell the user the new job id, that it is session-only and expires after seven days.
durable: trueis not honored; the cron fires only while the REPL is idle.
Notes
- Liveness tooling (operator directive 2026-07-13, memory [[iris_liveness_tooling_not_logtail]]): judge
liveness/wedge via
scripts/iris/iris_ops.py+ direct iris SQL, progress via GCS artifacts (checkpoint step+ts for training, trial count / output-bucket population for datagen). NEVER diagnose liveness from rawiris job logstail (interleaved multi-rank lines;--no-tailreturns startup lines; a clean preempt emits no terminal log line). - cgroup memory tracking is RETIRED (2026-07-13): midtrain (C) reports child state + checkpoint-step delta only.
- Single-region output migration (OT-Agent
c76dd23a): new datagen launches pass NO--gcs-output-dir, so the launcher's region-pin routes output to a co-located single-region bucket (gs://marin-us-<region>/ot-agent/<job>). Rescue resolves each job's RECORDED output URI viahpc.iris.job_output_resolver— legacy jobs stay on multi-regionmarin-models-{us,eu}. NEVER hardcode an output bucket. - Companion skills: monitor-cron-sweep-iris (tick methodology), monitor-job-tables (datagen box table WITH Mean column), datagen-job-cleanup (idempotent rescue for a TERMINAL/wedged arm — dispatch a subagent armed with it), datagen-launch-iris (refill).
Canonical cron prompt (copy verbatim into CronCreate)
UNIFIED OPS TICK. Source /Users/benjaminfeuer/Documents/secrets.env (never echo secrets). Spawn a general-purpose agent that reads the absolute-path ops docs and reports back; use PATH /Users/benjaminfeuer/miniconda3/envs/otagent/bin for iris+python. Wrap cw kubectl and any recursive gsutil in timeout. Use ABSOLUTE paths everywhere. Format the campaign status as a DATAGEN box-drawing table WITH a Mean-reward column per /Users/benjaminfeuer/Documents/OpenThoughts-Agent/.agents/skills/monitor-job-tables.
LIVENESS TOOLING (operator directive 2026-07-13): judge job liveness/wedge via /Users/benjaminfeuer/Documents/OpenThoughts-Agent/scripts/iris/iris_ops.py + direct iris SQL query, and progress via GCS artifacts (checkpoint step+ts for training, trial count / output-bucket population for datagen). NEVER diagnose liveness from raw `iris job logs` tail (interleaved multi-rank lines; --no-tail returns STARTUP lines not latest; a clean preempt emits no terminal log line). See memory iris_liveness_tooling_not_logtail.
(A) SWEEP both iris clusters (marin GCP + cw-us-east-02a) and the native route (expect 401). Note Daytona snapshot count on the cli org. STANDING CLEANUP: each tick, proactively reclaim idle harbor__ snapshots >120min via /Users/benjaminfeuer/Documents/OpenThoughts-Agent/scripts/daytona/daytona_snapshot_manager.py --api-key-env DAYTONA_API_KEY --stale-days 0.0833 --delete-stale --yes (run from the OT-Agent dir, secrets sourced). Deletes ONLY idle harbor__ env snapshots; the --name-prefix harbor__ default GUARDS base images (daytonaio/sandbox:*, daytona-*, windows-*) — NEVER delete those, never ACTIVE-recent (<120min). Report before/after count.
(B) CAMPAIGN qwen3.5-122b-131k-datagen-opencode-iris — keep-3 steady-state. Box table (arm | dataset | completed/total | Mean | liveness). Confirm each arm RUNNING+serving+advancing (advancing = GCS trial count up vs last tick), single serve dir, no job.py:263.
MEAN COLUMN IS MANDATORY — pull it from the harbor `<done>/<total> Mean: <X>` progress line in the job logs: `/Users/benjaminfeuer/Documents/marin/.venv/bin/iris --cluster=marin job logs /benjaminfeuer/<job> 2>&1 | grep -aoE '[0-9]+/[0-9]+ Mean: [-0-9.]+' | tail -1` (retry on a transient finelog `dns error`/`StatsError`; that same line ALSO gives the freshest completed/total, more current than result.json). This targeted metric grep is NOT the prohibited liveness-by-log-tail — it is REQUIRED every tick. Mark `—` ONLY when the arm genuinely emits no Mean line (no verifier); do NOT default to `—` because result.json lacks a mean field. (result.json carries no mean — that is expected and is not a reason to drop the column.)
>=95% KILL-AND-HARVEST (operator directive 2026-07-14): if a state-3 RUNNING arm is past 95% completed/total on ANY tick, KILL the child (autonomously authorized; child only) and HARVEST it as a terminal arm right now — do NOT await the last stragglers (the long tail is not worth the held v5p-8). Treat it exactly like a >=60% terminal arm: rescue via datagen-job-cleanup + refill. (Applies to normal deterministic-verifier arms; a still-cold-compiling or resume-scanning arm that only shows 95% because its total is not yet known is exempt — require real advancing trials.)
TERMINAL arm (state 4/5/6): if >=60% complete, dispatch a subagent ARMED WITH /Users/benjaminfeuer/Documents/OpenThoughts-Agent/.agents/skills/datagen-job-cleanup (read that SKILL.md, follow its idempotent steps): gs:// rescue of the OUTER recorded output dir (resolve via hpc.iris.job_output_resolver — single-region gs://marin-<region>/ot-agent/<job> for new jobs, multi-region gs://marin-models-{us,eu}/ for legacy; NEVER hardcode) so logs/ literals ride along -> avg_turns realness check -> HF upload via make_and_upload_trace_dataset.py --episodes last with literals AUTO-INCLUDED and --served_model Qwen/Qwen3.5-122B-A10B-FP8 -> verify HF non-empty + Literal yield X/Y (X>0) + count_populated_literal_rows>0 -> report rows+yield. (If <60%, resume instead.) THEN submit a refill (next un-launched tracker dataset) on the newest validated :tpu image --preemptible (no --gcs-output-dir, so the single-region region-pin engages) and let iris schedule; update the tracker with the cleanup (repo+rows) and the refill.
CONFIRMED-WEDGED datagen (state 3 RUNNING but authoritative-state RUNNING AND 0 trials / empty output bucket for hours with no engine-serving marker, OR harbor frozen >=3h fd-monitor-only per datagen-job-cleanup) -> kill the child + refill (autonomously authorized; child only). Distinguish from a healthy cold-compile/resume-scan (engine bringing up / recompiling) — do NOT kill those.
If keep-N<3 for any reason, submit refill(s), let iris schedule — do NOT gate on a capacity guess. Preemptible jobs stay preemptible. The LLM-judge-verified datasets (laion/stackexchange-superuser-sandboxes-verified, laion/stackexchange-tezos-sandboxes-verified — tracker rows 120/121) RUN IN THE NORMAL keep-3 SEQUENCE — do NOT skip/hold them: their per-task task.toml propagates OPENAI_API_KEY into the trial verifier sandbox and the standard launch already passes --secrets-env "$DC_AGENT_SECRET_ENV" (which carries OPENAI_API_KEY), so the litellm judge scores real rewards. See datagen-launch-iris (Prerequisites > LLM-judge datasets) for the mechanism.
(C) MIDTRAIN 1e23_p33m67_k0p20 (you OWN this, monitor-only) — ONE-LINE STATUS via iris_ops.py/SQL (NOT log-tail): child state + live step / newest TEMP checkpoint step+ts (gs://marin-us-east5/tmp/ttl=14d/checkpoints-temp/.../delphi-1e23-p33m67-k0p20-lr0.67-b6607e/checkpoints/step-*/metadata.json) + whether the checkpoint step ADVANCED since last tick. NO memory-bounded field (cgroup tracking RETIRED 2026-07-13). A PENDING/re-placing child (preempt/crash-teardown/tier-monotonicity/capacity) is NORMAL — state it, do NOT flag/escalate/bounce; a stale checkpoint under preempt churn is expected. ONLY act on: a confirmed state-3 RUNNING-but-frozen wedge (authoritative state RUNNING + checkpoint stalled across multiple ticks + pod mismatch) -> Option A bounce the WEDGED CHILD only (never coordinator); or a confirmed cgroup OOM.
Relay a tight A/B/C report (campaign as a box-drawing datagen table with Mean).If you change the cadence or scope, update BOTH the cron/prompt above and the live job (delete + recreate),
and keep monitor-cron-sweep-iris in sync — so this skill stays the canonical copy.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

