agentleFS
MacroMan5MacroMan5/autotrain-yolo/.claude/skills/monitor-training/SKILL.md

monitor-training

Skill8 starsChanged 8 months agoMITView on GitHub

Set up autonomous training monitoring — creates cron jobs to track long-running training, auto-continue pipeline when training completes.

The file

We split this skill into the rules it gives an agent

10 rules. About 0 are common, 7 are distinctive, and 3 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈7

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 3

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    After launching a training run that will take hours

    line 12
  2. Distinctive:

    When you want to chain multiple pipeline phases autonomously (train → analyze → clean → retrain → tune → export)

    line 13
  3. Very specific:

    Check for active GPU processes: nvidia-smi --query-compute-apps=pid,used_memory --format=csv,noheader

    line 21
  4. Very specific:

    Check for results CSV: tail -5 <training_dir>/results.csv

    line 22
  5. Distinctive:

    Check background task status if task ID is known

    line 23
  6. Very specific:

    Use CronCreate with a 5-minute interval to poll training status:

    line 27
  7. Distinctive:

    When a training phase completes:

    line 88
  8. Distinctive:

    GPU awareness: Before running validation or analysis, check GPU memory.

    line 101
  9. Distinctive:

    Error handling: If a training crashes (OOM, data error), report the error and stop — don't auto-retry without understanding why.

    line 102
  10. Distinctive:

    Plan file: Always reference the plan file for pipeline context.

    line 103

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Monitor Training — Autonomous Training Pipeline

Set up cron-based monitoring for long-running YOLO training sessions. Automatically detects completion, reports results, and continues to the next pipeline phase.

When to Use

  • After launching a training run that will take hours
  • When you want to chain multiple pipeline phases autonomously (train → analyze → clean → retrain → tune → export)
  • To monitor training progress without manual polling

Workflow

1. Identify Training State

Determine what's currently running:

  • Check for active GPU processes: nvidia-smi --query-compute-apps=pid,used_memory --format=csv,noheader
  • Check for results CSV: tail -5 <training_dir>/results.csv
  • Check background task status if task ID is known

2. Create Monitoring Cron Job

Use CronCreate with a 5-minute interval to poll training status:

CronCreate:
  cron: "*/5 * * * *"
  recurring: true
  prompt: <monitoring prompt with next-phase instructions>

The monitoring prompt should include:

  • Where to check: Path to results.csv and/or background task ID
  • What to report: Current epoch, mAP50, mAP50-95, training losses
  • Completion criteria: Task finished, early stopping triggered, or target metric reached
  • Next phase actions: Detailed steps to execute when training completes

3. Monitor Prompt Template

Build the cron prompt with these sections:

1. CHECK STATUS
   - Read results.csv (tail -5) for latest metrics
   - Check if training process is still running
   - Report: epoch, mAP50, mAP50-95, whether still active

2. ON COMPLETION — Execute next phase:
   [Phase-specific instructions]

3. AFTER NEXT PHASE — Chain to following phase:
   - Delete current cron job (CronDelete)
   - Create new cron job for the next long-running phase

4. PLAN REFERENCE
   - Read plan file for full pipeline context

4. Pipeline Chaining Pattern

Each cron job monitors one phase and launches the next:

Cron 1: Monitor baseline training
  → On complete: run analysis + dataset cleanup
  → Launch retrain
  → Delete self, create Cron 2

Cron 2: Monitor retrain
  → On complete: launch model.tune()
  → Delete self, create Cron 3

Cron 3: Monitor tune
  → On complete: launch final training with best params
  → Delete self, create Cron 4

Cron 4: Monitor final training
  → On complete: export model, generate final report
  → Delete self

5. Completion Actions

When a training phase completes:

  1. Report final metrics — epoch, per-class mAP, precision, recall
  2. Save best model — copy best.pt to a named location
  3. Run per-class validation — use a free GPU for detailed evaluation
  4. Update plan file — mark phase complete, record metrics
  5. Update tasks — mark current task complete, create next
  6. Launch next phase — start the next pipeline step
  7. Create new monitor — set up cron for the new phase

Guidelines

  • Cron interval: 5 minutes is good for training. Use 2 minutes for shorter operations like analysis.
  • GPU awareness: Before running validation or analysis, check GPU memory. Use a different GPU than the one training, or wait for training to finish.
  • Error handling: If a training crashes (OOM, data error), report the error and stop — don't auto-retry without understanding why.
  • Plan file: Always reference the plan file for pipeline context. Update it as phases complete.
  • Session lifetime: Cron jobs expire after 7 days or when the session ends. For multi-day pipelines, document the current state in the plan file so a new session can resume.
  • Idempotency: The cron prompt may fire multiple times while a phase is still running. Only trigger next-phase actions on actual completion, not on intermediate checks.

Example

User: "Lance le training et surveille-le automatiquement"

1. Launch training in background
2. Create cron:
   CronCreate("*/5 * * * *", "Check training at <path>/results.csv.
     If still running: report epoch and metrics.
     If complete: run validation, save model, launch next phase...")
3. Training runs autonomously
4. Cron detects completion → runs analysis → launches retrain → creates new cron
5. Pipeline continues without user intervention

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.