ml-experiment-tracker
Skill17 starsChanged 5 months agoApache-2.0Archived repositoryView on GitHub
Guides ML experiment logging, versioning, and reproducibility using tools like MLflow, Weights & Biases, and DVC for systematic model development.
1 other project uses a version of this.
Other versions1 project uses a version of this.
Word for word · 1
- organvm-iv-taxis/a-i--skills★ 17100% same
Edited · 0
None found.
Wrote their own version · 0
None found.
How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.
In this repository170 other files it gives its agents.
- AGENTS.mdAGENTS.md
- docs/AGENTS.mdAGENTS.md
- CLAUDE.mdCLAUDE.md
- .github/copilot-instructions.mdCopilot instructions
- algorithmic-artSkill
- audio-engineering-patternsSkill
- canvas-designSkill
- creative-writing-craftSkill
- generative-art-algorithmsSkill
- generative-art-deploymentSkill
- generative-music-composerSkill
- interactive-theatre-designerSkill
- modular-synthesis-philosophySkill
- movement-notation-systemsSkill
- narratological-algorithmsSkill
- natural-center-extractionSkill
- react-three-fiber-patternsSkill
- reality-tv-narrative-analyzerSkill
- script-analysis-dramaturgicalSkill
- theme-factorySkill
- three-js-interactive-builderSkill
- corpus-persona-extractionSkill
- data-ingestion-pipelineSkill
- data-pipeline-architectSkill
- data-storytelling-analystSkill
- market-gap-analysisSkill
- prompt-atom-formalizationSkill
- sql-query-optimizerSkill
- systemic-product-analystSkill
- time-series-analystSkill
- accessibility-patternsSkill
- agent-testing-patternsSkill
- api-design-patternsSkill
- artifacts-builderSkill
- backend-implementation-patternsSkill
- browser-extension-patternsSkill
- cli-tool-designSkill
- code-refactoring-patternsSkill
- code-review-checklistSkill
- coding-standards-enforcerSkill
- community-platform-patternsSkill
- configuration-managementSkill
- continuous-learning-agentSkill
- data-backup-patternsSkill
- database-migration-patternsSkill
- deployment-cicdSkill
- docker-containerizationSkill
- dotfile-systems-architectSkill
- error-handling-logging-patternsSkill
- fastapi-patternsSkill
- feature-workflow-orchestratorSkill
- frontend-design-systemsSkill
- fullstack-starter-packSkill
- gcp-resource-optimizerSkill
- htmx-interaction-patternsSkill
- iterative-code-explorationSkill
- json-schema-designSkill
- mcp-builderSkill
- mcp-server-orchestratorSkill
- mobile-platform-architectSkill
- The first 60 shown.
The file
We split this skill into the rules it gives an agent
1 rule. About 0 are common, 1 is distinctive, and 0 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈1
Distinctive · estimate
This author's own choices. Worth a look.
0
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 25
Every experiment should log:
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
ML Experiment Tracker
This skill provides guidance for systematic machine learning experimentation with proper tracking, versioning, and reproducibility practices.
Core Competencies
- Experiment Tracking: MLflow, Weights & Biases (wandb), Neptune, Comet
- Data Versioning: DVC, Delta Lake, LakeFS
- Model Registry: Version control for trained models
- Reproducibility: Environment, code, data, and hyperparameter tracking
Experiment Tracking Fundamentals
What to Track
Every experiment should log:
| Category | Items | Why |
| Code | Git commit hash, branch, diff | Reproduce exact code state |
| Data | Dataset version, hash, lineage | Know which data was used |
| Environment | Python version, dependencies, hardware | Reproduce runtime |
| Hyperparameters | All config values | Understand what changed |
| Metrics | Loss, accuracy, custom metrics | Compare performance |
| Artifacts | Models, plots, predictions | Preserve outputs |
Experiment Organization
project/
├── experiments/
│ ├── baseline/ # Initial experiments
│ ├── feature-engineering/ # Data improvements
│ ├── architecture/ # Model changes
│ └── hyperparameter/ # Tuning runs
├── data/
│ ├── raw/ # Original data (versioned)
│ ├── processed/ # Cleaned data
│ └── features/ # Feature store
└── models/
├── staging/ # Candidates
└── production/ # Deployed modelsMLflow Patterns
Basic Experiment Logging
import mlflow
# Set experiment (creates if not exists)
mlflow.set_experiment("my-classification-project")
with mlflow.start_run(run_name="baseline-v1"):
# Log parameters
mlflow.log_param("learning_rate", 0.01)
mlflow.log_param("batch_size", 32)
mlflow.log_param("epochs", 100)
# Training loop
for epoch in range(epochs):
train_loss = train_epoch(model, train_loader)
val_loss, val_acc = evaluate(model, val_loader)
# Log metrics with step
mlflow.log_metrics({
"train_loss": train_loss,
"val_loss": val_loss,
"val_accuracy": val_acc
}, step=epoch)
# Log model
mlflow.pytorch.log_model(model, "model")
# Log artifacts (plots, configs)
mlflow.log_artifact("confusion_matrix.png")
mlflow.log_artifact("config.yaml")Model Registry Workflow
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ Training │───▶│ Staging │───▶│ Production │
│ Runs │ │ Review │ │ Deployed │
└──────────────┘ └──────────────┘ └──────────────┘
│ │ │
▼ ▼ ▼
Candidate Validated Monitored
Models Models ModelsStages:
- None: Just logged, not registered
- Staging: Candidate for production
- Production: Active serving
- Archived: Historical reference
Weights & Biases Patterns
Project Structure
import wandb
# Initialize with config
config = {
"learning_rate": 0.01,
"architecture": "ResNet50",
"dataset": "imagenet-subset",
"epochs": 100
}
run = wandb.init(
project="image-classification",
group="architecture-experiments", # Group related runs
tags=["baseline", "resnet"],
config=config,
notes="Testing ResNet50 baseline on subset"
)
# Training with automatic logging
for epoch in range(config["epochs"]):
metrics = train_and_eval(model, train_loader, val_loader)
wandb.log(metrics)
# Log media
wandb.log({"predictions": wandb.Image(pred_grid)})
wandb.log({"confusion_matrix": wandb.plot.confusion_matrix(...)})
wandb.finish()Hyperparameter Sweeps
# sweep_config.yaml
program: train.py
method: bayes # or grid, random
metric:
name: val_accuracy
goal: maximize
parameters:
learning_rate:
distribution: log_uniform_values
min: 0.0001
max: 0.1
batch_size:
values: [16, 32, 64, 128]
optimizer:
values: ["adam", "sgd", "adamw"]
early_terminate:
type: hyperband
min_iter: 10DVC for Data Versioning
Setup and Usage
# Initialize DVC in git repo
dvc init
# Track large files
dvc add data/training.csv
git add data/training.csv.dvc data/.gitignore
git commit -m "Add training data v1"
# Push to remote storage
dvc remote add -d storage s3://bucket/dvc
dvc push
# Create pipeline
dvc run -n preprocess \
-d src/preprocess.py -d data/raw \
-o data/processed \
python src/preprocess.py
# Reproduce pipeline
dvc reproDVC Pipeline Definition
# dvc.yaml
stages:
preprocess:
cmd: python src/preprocess.py
deps:
- src/preprocess.py
- data/raw/
outs:
- data/processed/
train:
cmd: python src/train.py
deps:
- src/train.py
- data/processed/
params:
- train.epochs
- train.learning_rate
outs:
- models/model.pkl
metrics:
- metrics.json:
cache: falseReproducibility Checklist
Code Reproducibility
- Pin git commit for each experiment
- Track uncommitted changes (git diff)
- Version control notebooks (nbstripout)
- Document manual steps
Environment Reproducibility
- Lock dependencies (pip freeze, poetry.lock)
- Specify Python version
- Document CUDA/GPU requirements
- Use containers for full isolation
Data Reproducibility
- Version datasets with DVC or similar
- Document data collection process
- Track preprocessing steps
- Save train/val/test split indices
Training Reproducibility
- Set random seeds (Python, NumPy, PyTorch/TF)
- Log all hyperparameters
- Save model checkpoints
- Document non-deterministic operations
Best Practices
Naming Conventions
experiment: {project}-{objective}
run: {date}-{description}-{variant}
model: {architecture}-{dataset}-{version}
Examples:
experiment: fraud-detection-baseline
run: 2024-01-15-xgboost-tuning-lr001
model: xgboost-transactions-v2.3.1Comparison Dashboards
Track these metrics for model comparison:
- Primary metric (what you optimize)
- Secondary metrics (constraints)
- Resource usage (training time, memory)
- Inference performance (latency, throughput)
Experiment Documentation
Each significant experiment should document:
- Hypothesis: What change and expected outcome
- Method: What was actually done
- Results: Metrics and observations
- Conclusions: What was learned, next steps
References
references/mlflow-setup.md- MLflow installation and configurationreferences/wandb-patterns.md- Advanced W&B features and sweepsreferences/reproducibility-checklist.md- Detailed reproducibility guide
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

