agentleFS
open-thoughtsopen-thoughts/OpenThoughts-Agent/.agents/skills/analyze-training-run-iris/SKILL.md

analyze-training-run-iris

Skill309 starsChanged 4 months agoApache-2.0View on GitHub

Detailed health check for a Levanter/executor TRAINING run on the marin Iris cluster (e.g. the delphi midtraining runs) — step progress vs target, loss/throughput, preemption + MAJOR step-gap detection, and checkpoint cadence. Use for an executor coordinator (`<run>-coord`) plus its nested `<run>-coord/checkpoints-<step>-<hash>` training child, which the harbor analyzer (analyze-job-history-iris) does NOT cover (training has no harbor trial sidecars, same as GPU-RL). Reads W&B per-step history + `iris job summary` + GCS checkpoints instead of harbor GCS output.

  • Reads credentials

The file

We split this skill into the rules it gives an agent

1 rule. About 0 are common, 0 are distinctive, and 1 is very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈0

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 1

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    NOT training steps. Do NOT grep step N from the logs for progress.

    line 104

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

analyze-training-run-iris

📍 Iris orientation — read first. Before acting on anything in this skill, read the Iris tools catalog (.agents/ops/iris/ops.md) and the Iris ops directory (.agents/ops/iris/ — the CoreWeave GPU particulars in ops.md, the TPU marin particulars in ops.md). They carry the binding access/preamble/gotchas and the helper-script inventory the steps below rely on.

A Levanter training run launched through the marin executor surfaces as TWO Iris jobs:

  • a tiny CPU coordinator — /<user>/<run>-coord — the executor_main DAG-walker (it submits the training job and then blocks); and
  • its nested training child — /<user>/<run>-coord/checkpoints-<step>-<hash> — the multi-task v5p job where the actual training steps happen (e.g. 8 tasks for a v5p-64).

Health = the CHILD's step progress + the run's preemption/gap history. The harbor analyzer (analyze-job-history-iris) does NOT apply — a training job has no harbor trial sidecars (just like GPU-RL). Use the three sources below instead. W&B is primary for step/loss/throughput; iris job summary is primary for preemptions/liveness; GCS is primary for checkpoint cadence.

Source 1 — iris job summary: preemptions, liveness, per-task state (always available)

IRIS=/Users/benjaminfeuer/Documents/marin/.venv/bin/iris
$IRIS --cluster=marin job summary <child_job_id>

Report preemptions=N failures=N, tasks running/completed (all N tasks should be running together — a v5p job is gang-scheduled), the longest task DURATION, and PEAK MEM. preemptions>0 is expected on a preemptible v5p — each one means iris restarted the slice and Levanter resumed from the last checkpoint (every preemption costs a wall-clock gap: re-place + reload weights + XLA recompile). failures>0, a shrinking task count, or a crash-restart loop is a red flag → read the child logs for the error.

Source 2 — W&B per-step history: step / loss / throughput + MAJOR GAP detection (primary)

The run logs to W&B project delphi-midtraining (entity nyu-dice-lab); the run name is the GCS output-path hash — the last path segment of gs://marin-us-east5/checkpoints/<run>-<hash> (e.g. delphi-1e23-p33m67-k0p20-lr0.67-b6607e → run delphi-1e23-p33m67-k0p20-lr0.67-b6607e). Per-step history is NOT mirrored by mum — query the W&B API directly (needs WANDB_API_KEY from $DC_AGENT_SECRET_ENV; use the otagent python which has wandb):

source "${DC_AGENT_SECRET_ENV:?set DC_AGENT_SECRET_ENV to the secrets file first}"
/Users/benjaminfeuer/miniconda3/envs/otagent/bin/python - <<'PY'
import wandb
ENTITY, PROJECT, RUN = "nyu-dice-lab", "delphi-midtraining", "<run-hash>"   # <-- the ...-b6607e hash
api = wandb.Api()
r = api.run(f"{ENTITY}/{PROJECT}/{RUN}")
total = r.config.get("trainer", {}).get("num_train_steps") or r.config.get("num_train_steps")
h = r.history(keys=["_step", "_timestamp", "_runtime", "train/loss"], pandas=True)
if h is None or len(h) == 0:
    print("state:", r.state, "-> pre-first-step (still setup/HF-download/XLA-compile); no training step yet")
else:
    h = h.dropna(subset=["_step"]).sort_values("_step")
    cur = int(h["_step"].iloc[-1])
    ts = h["_timestamp"].to_numpy()
    deltas = [b - a for a, b in zip(ts[:-1], ts[1:])]
    med = sorted(deltas)[len(deltas)//2] if deltas else 0.0
    thr = max(300.0, 20.0 * med)                       # same MAJOR-GAP rule as compute_time.md
    gaps = [d for d in deltas if d > thr]
    toks = 0
    bs = (r.config.get("trainer", {}) or {}).get("train_batch_size")
    sl = r.config.get("train_seq_len") or (r.config.get("model", {}) or {}).get("max_seq_len")
    if bs and sl and med: toks = bs * sl / med
    loss = h["train/loss"].dropna()
    loss = float(loss.iloc[-1]) if len(loss) else None
    eta_h = ((total - cur) * med / 3600.0) if (total and med) else None
    print(f"state            : {r.state}")
    print(f"step             : {cur}/{total}  ({round(100*cur/total,1) if total else '?'}%)")
    print(f"train/loss       : {loss}")
    print(f"median step dt   : {round(med,2)} s   -> ~{round(toks):,} tok/s" if med else "median step dt: n/a")
    print(f"MAJOR gaps       : count={len(gaps)}  total={round(sum(gaps)/3600,2)} h  (threshold {round(thr)} s)")
    print(f"ETA to {total}   : ~{round(eta_h,1)} h of compute (excludes future preemption gaps)" if eta_h else "ETA: n/a")
PY
  • step cur/total = progress against the K-budget target (e.g. 29,945 for K=0.20). This is the headline "progress in steps" number.
  • MAJOR gaps = consecutive _timestamp deltas exceeding max(300 s, 20 × median step interval) — preemption / idle gaps (the metric the user wants: "note major gaps"). Cross-check count against iris job summary preemptions — they should be the same order (a gap with NO matching preemption is a silent stall worth flagging). Report gap count + total hours.
  • tok/s = batch × seq / median_dt. Compare across ticks; a sudden drop = contention or a bad slice.
  • Empty history = the run is still in setup / HF-weight download / first XLA compile — that is NOT a gap; report it as pre-first-step and move on.

Source 3 — GCS checkpoint cadence: resume safety (always available)

gsutil ls gs://marin-us-east5/checkpoints/<run>-<hash>/ | grep -E 'step-[0-9]+' | tail -5
gsutil ls -l gs://marin-us-east5/checkpoints/<run>-<hash>/step-<latest>/ 2>/dev/null | tail -2   # timestamp

Report the latest persisted step + its timestamp (the checkpointer saves on an interval — e.g. save_interval 10m, keep every 1500). The latest checkpoint lagging a bit behind the W&B step is fine (async save). But no step-* checkpoint long after training started is a red flag — under preemption the run would lose all un-checkpointed progress. Only .executor_info / .executor_status* present (no step-*) = still pre-first-checkpoint (early bring-up).

Don't mistake setup steps for training steps

iris job logs <child> early on shows [iris setup] step N/M lines — those are uv-sync SETUP steps, NOT training steps. Do NOT grep step N from the logs for progress. Use the W&B _step (Source 2) as the authoritative training-step counter; only fall back to Levanter's own in-log training-step line if W&B is unreachable.

The compact cron line (one per training run)

<run> state=running step=<cur>/<total> (X%) loss=<L> ~<T>tok/s preempts=<P> gaps=<G>/<H>h ckpt=step-<C> plus a one-line health read: past setup/compile? step rate sane vs the prior tick? loss finite and trending down (not NaN/spiking)? preemptions resuming cleanly (checkpoint advancing)? ETA to the K-budget target.

Running it / offloading to a subagent

The W&B pull is fast (seconds), unlike the harbor analyzer — you usually do NOT need a subagent. If a run is huge or you are sweeping several, the analyze-job-history-iris foreground-and-wait discipline still applies to any slow gsutil/log reads, but the W&B query itself is quick.

Related skills

  • monitor-cron-sweep-iris — the every-3-hours sweep; its class E (executor/Levanter training) invokes this skill, just as class A/B invoke analyze-job-history-iris.
  • analyze-job-history-iris — the harbor (datagen/eval) analyzer; does NOT apply to training runs.
  • rl-job-health-deep-dive — the GPU-RL equivalent (also no harbor sidecars; uses the finelog).

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.