Training Check

Periodically read WandB metrics during training to catch problems early. Do not wait until training finishes to discover it was a waste of GPU time.

Context: $ARGUMENTS

Constants

• WANDB_ENTITY and WANDB_PROJECT: read from CLAUDE.md or passed as argument (format: entity/project/run_id)
• CHECK_INTERVAL: starts at 10 minutes, then gradually increases if consistently healthy: 10 min → 20 min → 30 min → 60 min (cap)
• REVIEWER_BACKEND = llm-chat — external LLM reviewer via llm-chat MCP, used for ambiguous cases only (model defers to LLM_MODEL env)

When to Use

• After training is confirmed running (session alive, loss decreasing for first few steps)
• Set up via CronCreate to fire periodically during training
• This skill checks training QUALITY, not process HEALTH. Process health checks such as session liveness and GPU utilization should be handled by the launcher or monitor workflow.

Workflow

Step 1: Read WandB Metrics

import wandb
api = wandb.Api()
run = api.run("<entity>/<project>/<run_id>")
history = run.history()

If WandB is unreachable (API error, network issue), fall back to reading the log file directly via SSH:

ssh server "tail -100 /path/to/training.log"

Check these signals:

• Loss trend: Is training loss decreasing over the last N steps?
• Eval metrics: Are evaluation metrics improving (or at least not degrading)?
• NaN / Inf: Any NaN or Inf values in loss or gradients?
• Spikes: Sudden large jumps in loss (>10x normal variance)?
• Learning rate: Is the schedule behaving as expected?
• Gradient norm: Exploding or vanishing?

Step 2: Judgment

Step 3: External Reviewer Judgment (only when unsure)

Only escalate to the external LLM reviewer when the signal is ambiguous. For clearly good or clearly bad signals, act directly. Always ask the external reviewer for strict, high-rigor feedback.

mcp__llm-chat__chat:
  prompt: |
    TRAINING HEALTH CHECK — need your judgment on ambiguous metrics.

    Run: <entity>/<project>/<run_id>
    Current epoch/step: X / Y total
    Training loss (last 10 checkpoints): [values]
    Eval metrics (last 3 evals): [values]
    Baseline reference: [numbers from paper/reproduction]

    What I'm unsure about: [specific concern]

    Please respond with exactly one of:
    - STOP: clearly problematic, should kill training
    - CONTINUE: looks fine, check again next interval
    - WAIT: not enough data to judge, check again sooner

Step 4: Act

Integration with Process Monitoring

Training-check and process monitoring operate at different levels:

Use both together:

• Process monitoring catches crashes and idle GPUs immediately
• Training-check catches subtle quality issues (loss plateau, metric degradation)

Rules

• Do not stop training on first sign of noise — some loss spikes are normal. Look at trends over multiple checkpoints.
• When stopping training, always save the WandB run URL and key metrics as evidence.
• If both WandB and log files are unreachable, report the connectivity issue and try again next interval. Do not assume training is broken.
• Gradually increase check interval when healthy (10 → 20 → 30 → 60 min). Reset to 10 min after any anomaly.
• This skill is meant to be automated via CronCreate — do not ask the user whether to set it up. Just set it.

CronCreate Setup Example

After training is confirmed stable:
  CronCreate (recurring, every 10 minutes initially):
    "Run /training-check for wandb run <entity>/<project>/<run_id>"

As the check interval increases, delete the old CronCreate job and create a new one with the longer interval.