monitor-training
Skill8 starsChanged 8 months agoMITView on GitHub
Set up autonomous training monitoring — creates cron jobs to track long-running training, auto-continue pipeline when training completes.
In this repository15 other files it gives its agents.
- CLAUDE.mdCLAUDE.md
- active-learningSkill
- analyze-resultsSkill
- annotateSkill
- auto-experimentSkill
- benchmarkSkill
- compare-modelsSkill
- cvat-deploySkill
- cvat-pullSkill
- cvat-pushSkill
- explain-resultsSkill
- review-annotationsSkill
- review-datasetSkill
- setup-projectSkill
- training-loopSkill
The file
We split this skill into the rules it gives an agent
10 rules. About 0 are common, 7 are distinctive, and 3 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈7
Distinctive · estimate
This author's own choices. Worth a look.
3
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 12
After launching a training run that will take hours
- Distinctive: line 13
When you want to chain multiple pipeline phases autonomously (train → analyze → clean → retrain → tune → export)
- Very specific: line 21
Check for active GPU processes: nvidia-smi --query-compute-apps=pid,used_memory --format=csv,noheader
- Very specific: line 22
Check for results CSV: tail -5 <training_dir>/results.csv
- Distinctive: line 23
Check background task status if task ID is known
- Very specific: line 27
Use CronCreate with a 5-minute interval to poll training status:
- Distinctive: line 88
When a training phase completes:
- Distinctive: line 101
GPU awareness: Before running validation or analysis, check GPU memory.
- Distinctive: line 102
Error handling: If a training crashes (OOM, data error), report the error and stop — don't auto-retry without understanding why.
- Distinctive: line 103
Plan file: Always reference the plan file for pipeline context.
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Monitor Training — Autonomous Training Pipeline
Set up cron-based monitoring for long-running YOLO training sessions. Automatically detects completion, reports results, and continues to the next pipeline phase.
When to Use
- After launching a training run that will take hours
- When you want to chain multiple pipeline phases autonomously (train → analyze → clean → retrain → tune → export)
- To monitor training progress without manual polling
Workflow
1. Identify Training State
Determine what's currently running:
- Check for active GPU processes:
nvidia-smi --query-compute-apps=pid,used_memory --format=csv,noheader - Check for results CSV:
tail -5 <training_dir>/results.csv - Check background task status if task ID is known
2. Create Monitoring Cron Job
Use CronCreate with a 5-minute interval to poll training status:
CronCreate:
cron: "*/5 * * * *"
recurring: true
prompt: <monitoring prompt with next-phase instructions>The monitoring prompt should include:
- Where to check: Path to
results.csvand/or background task ID - What to report: Current epoch, mAP50, mAP50-95, training losses
- Completion criteria: Task finished, early stopping triggered, or target metric reached
- Next phase actions: Detailed steps to execute when training completes
3. Monitor Prompt Template
Build the cron prompt with these sections:
1. CHECK STATUS
- Read results.csv (tail -5) for latest metrics
- Check if training process is still running
- Report: epoch, mAP50, mAP50-95, whether still active
2. ON COMPLETION — Execute next phase:
[Phase-specific instructions]
3. AFTER NEXT PHASE — Chain to following phase:
- Delete current cron job (CronDelete)
- Create new cron job for the next long-running phase
4. PLAN REFERENCE
- Read plan file for full pipeline context4. Pipeline Chaining Pattern
Each cron job monitors one phase and launches the next:
Cron 1: Monitor baseline training
→ On complete: run analysis + dataset cleanup
→ Launch retrain
→ Delete self, create Cron 2
Cron 2: Monitor retrain
→ On complete: launch model.tune()
→ Delete self, create Cron 3
Cron 3: Monitor tune
→ On complete: launch final training with best params
→ Delete self, create Cron 4
Cron 4: Monitor final training
→ On complete: export model, generate final report
→ Delete self5. Completion Actions
When a training phase completes:
- Report final metrics — epoch, per-class mAP, precision, recall
- Save best model — copy
best.ptto a named location - Run per-class validation — use a free GPU for detailed evaluation
- Update plan file — mark phase complete, record metrics
- Update tasks — mark current task complete, create next
- Launch next phase — start the next pipeline step
- Create new monitor — set up cron for the new phase
Guidelines
- Cron interval: 5 minutes is good for training. Use 2 minutes for shorter operations like analysis.
- GPU awareness: Before running validation or analysis, check GPU memory. Use a different GPU than the one training, or wait for training to finish.
- Error handling: If a training crashes (OOM, data error), report the error and stop — don't auto-retry without understanding why.
- Plan file: Always reference the plan file for pipeline context. Update it as phases complete.
- Session lifetime: Cron jobs expire after 7 days or when the session ends. For multi-day pipelines, document the current state in the plan file so a new session can resume.
- Idempotency: The cron prompt may fire multiple times while a phase is still running. Only trigger next-phase actions on actual completion, not on intermediate checks.
Example
User: "Lance le training et surveille-le automatiquement"
1. Launch training in background
2. Create cron:
CronCreate("*/5 * * * *", "Check training at <path>/results.csv.
If still running: report epoch and metrics.
If complete: run validation, save model, launch next phase...")
3. Training runs autonomously
4. Cron detects completion → runs analysis → launches retrain → creates new cron
5. Pipeline continues without user interventionDoes it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

