ml-pipeline-workflow
Skill4 starsChanged 38 days agoMITView on GitHub
Build end-to-end MLOps pipelines from data preparation through model training, validation, and production deployment. Use when creating ML pipelines, implementing MLOps practices, or automating model training and deployment workflows.
7 other projects use a version of this, 6 of them edited.
Adopted by
Most recently taken up by
- AtilaMedeiros/skillsedited, first seen Nov 16, 2026
- xfstudio/skillsedited, first seen Oct 13, 2026
- Samueladewole/claude-skillsedited, first seen Oct 13, 2026
- Abdox-menara/guard-skillstheir own version, first seen Aug 24, 2026
- ai4brands-design/claude-skillsedited, first seen Mar 22, 2026
- CodeWithBehnam/cc-docsedited, first seen Mar 15, 2026
Other versions7 projects use a version of this.
Word for word · 0
None found.
Edited · 6
- xfstudio/skills★ 992% same
- ai4brands-design/claude-skills★ 292% same
- bcastelino/agent-skills-kit★ 287% same
- CodeWithBehnam/cc-docs★ 2100% same
- AtilaMedeiros/skills★ 192% same
- Samueladewole/claude-skills★ 0100% same
Wrote their own version · 1
- Abdox-menara/guard-skills★ 0same name
How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.
In this repository37 other files it gives its agents.
- AGENTS.mdAGENTS.md
- CLAUDE.mdCLAUDE.md
- ai-debt-detectorSkill
- airflow-dag-patternsSkill
- async-python-patternsSkill
- backtesting-frameworksSkill
- create-readmeSkill
- data-engineeringSkill
- data-quality-frameworksSkill
- dataset-curationSkill
- dbt-transformation-patternsSkill
- design-doc-mermaidSkill
- embedding-strategiesSkill
- eval-harness-firstSkill
- excalidraw-diagramSkill
- execution-algorithmsSkill
- external-diagram-mermaid-generateSkill
- frontend-designSkill
- hybrid-search-implementationSkill
- improve-codebase-architectureSkill
- llm-evaluationSkill
- market-regime-detectionSkill
- mcp-builderSkill
- mcp-registry-syndicationSkill
- mcprush-gateway-deploymentSkill
- mermaid-diagramsSkill
- orchestratorSkill
- postgresql-table-designSkill
- prompt-engineering-patternsSkill
- python-testing-patternsSkill
- rag-implementationSkill
- regulatory-compliance-auditSkill
- risk-metrics-calculationSkill
- security-reviewSkill
- spark-optimizationSkill
- sql-optimization-patternsSkill
- vector-index-tuningSkill
The file
Show its rules and text
We split this skill into the rules it gives an agent
14 rules. 14 are common, 0 are distinctive, and 0 are very specific to this project.
14
Common
Most of the 6 edited versions we read keep these.
0
Distinctive
This author's own choices. Worth a look.
0
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Common: line 97
Run data quality checks
- Common: line 108
Run validation test suite
- Common: line 131
Use data validation libraries (Great Expectations, TFX)
- Common: line 133
Document feature engineering transformations
- Common: line 139
Use model registries (MLflow, Weights & Biases)
- Common: line 146
Start with shadow deployments
- Common: line 147
Use canary releases for validation
- Common: line 177
Start with the basics and gradually add complexity:
- Common: line 180
Level 2: Add validation and monitoring stages
- Common: line 182
Level 4: Add A/B testing and gradual rollouts
- Common: line 222
Pipeline failures: Check dependencies and data availability
- Common: line 229
Check pipeline logs for each stage
- Common: line 231
Test components in isolation
- Common: line 237
After setting up your pipeline:
“Common” is measured: kept by at least 6 in 10 of the 6 edited versions of this file we could read. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
ML Pipeline Workflow
Complete end-to-end MLOps pipeline orchestration from data preparation through model deployment.
Overview
This skill provides comprehensive guidance for building production ML pipelines that handle the full lifecycle: data ingestion → preparation → training → validation → deployment → monitoring.
When to Use This Skill
- Building new ML pipelines from scratch
- Designing workflow orchestration for ML systems
- Implementing data → model → deployment automation
- Setting up reproducible training workflows
- Creating DAG-based ML orchestration
- Integrating ML components into production systems
What This Skill Provides
Core Capabilities
- Pipeline Architecture
- End-to-end workflow design
- DAG orchestration patterns (Airflow, Dagster, Kubeflow)
- Component dependencies and data flow
- Error handling and retry strategies
- Data Preparation
- Data validation and quality checks
- Feature engineering pipelines
- Data versioning and lineage
- Train/validation/test splitting strategies
- Model Training
- Training job orchestration
- Hyperparameter management
- Experiment tracking integration
- Distributed training patterns
- Model Validation
- Validation frameworks and metrics
- A/B testing infrastructure
- Performance regression detection
- Model comparison workflows
- Deployment Automation
- Model serving patterns
- Canary deployments
- Blue-green deployment strategies
- Rollback mechanisms
Reference Documentation
See the references/ directory for detailed guides:
- data-preparation.md - Data cleaning, validation, and feature engineering
- model-training.md - Training workflows and best practices
- model-validation.md - Validation strategies and metrics
- model-deployment.md - Deployment patterns and serving architectures
Assets and Templates
The assets/ directory contains:
- pipeline-dag.yaml.template - DAG template for workflow orchestration
- training-config.yaml - Training configuration template
- validation-checklist.md - Pre-deployment validation checklist
Usage Patterns
Basic Pipeline Setup
# 1. Define pipeline stages
stages = [
"data_ingestion",
"data_validation",
"feature_engineering",
"model_training",
"model_validation",
"model_deployment"
]
# 2. Configure dependencies
# See assets/pipeline-dag.yaml.template for full exampleProduction Workflow
- Data Preparation Phase
- Ingest raw data from sources
- Run data quality checks
- Apply feature transformations
- Version processed datasets
- Training Phase
- Load versioned training data
- Execute training jobs
- Track experiments and metrics
- Save trained models
- Validation Phase
- Run validation test suite
- Compare against baseline
- Generate performance reports
- Approve for deployment
- Deployment Phase
- Package model artifacts
- Deploy to serving infrastructure
- Configure monitoring
- Validate production traffic
Best Practices
Pipeline Design
- Modularity: Each stage should be independently testable
- Idempotency: Re-running stages should be safe
- Observability: Log metrics at every stage
- Versioning: Track data, code, and model versions
- Failure Handling: Implement retry logic and alerting
Data Management
- Use data validation libraries (Great Expectations, TFX)
- Version datasets with DVC or similar tools
- Document feature engineering transformations
- Maintain data lineage tracking
Model Operations
- Separate training and serving infrastructure
- Use model registries (MLflow, Weights & Biases)
- Implement gradual rollouts for new models
- Monitor model performance drift
- Maintain rollback capabilities
Deployment Strategies
- Start with shadow deployments
- Use canary releases for validation
- Implement A/B testing infrastructure
- Set up automated rollback triggers
- Monitor latency and throughput
Integration Points
Orchestration Tools
- Apache Airflow: DAG-based workflow orchestration
- Dagster: Asset-based pipeline orchestration
- Kubeflow Pipelines: Kubernetes-native ML workflows
- Prefect: Modern dataflow automation
Experiment Tracking
- MLflow for experiment tracking and model registry
- Weights & Biases for visualization and collaboration
- TensorBoard for training metrics
Deployment Platforms
- AWS SageMaker for managed ML infrastructure
- Google Vertex AI for GCP deployments
- Azure ML for Azure cloud
- OCI Data Science for Oracle Cloud Infrastructure deployments
- Kubernetes + KServe for cloud-agnostic serving
Progressive Disclosure
Start with the basics and gradually add complexity:
- Level 1: Simple linear pipeline (data → train → deploy)
- Level 2: Add validation and monitoring stages
- Level 3: Implement hyperparameter tuning
- Level 4: Add A/B testing and gradual rollouts
- Level 5: Multi-model pipelines with ensemble strategies
Common Patterns
Batch Training Pipeline
# See assets/pipeline-dag.yaml.template
stages:
- name: data_preparation
dependencies: []
- name: model_training
dependencies: [data_preparation]
- name: model_evaluation
dependencies: [model_training]
- name: model_deployment
dependencies: [model_evaluation]Real-time Feature Pipeline
# Stream processing for real-time features
# Combined with batch training
# See references/data-preparation.mdContinuous Training
# Automated retraining on schedule
# Triggered by data drift detection
# See references/model-training.mdTroubleshooting
Common Issues
- Pipeline failures: Check dependencies and data availability
- Training instability: Review hyperparameters and data quality
- Deployment issues: Validate model artifacts and serving config
- Performance degradation: Monitor data drift and model metrics
Debugging Steps
- Check pipeline logs for each stage
- Validate input/output data at boundaries
- Test components in isolation
- Review experiment tracking metrics
- Inspect model artifacts and metadata
Next Steps
After setting up your pipeline:
- Explore hyperparameter-tuning skill for optimization
- Learn experiment-tracking-setup for MLflow/W&B
- Review model-deployment-patterns for serving strategies
- Implement monitoring with observability tools
Related Skills
- experiment-tracking-setup: MLflow and Weights & Biases integration
- hyperparameter-tuning: Automated hyperparameter optimization
- model-deployment-patterns: Advanced deployment strategies
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

