ML Engineer

Build and operate production ML systems across the full lifecycle: data → features → training → validation → deployment → monitoring → retraining.

Core Workflow

1. System Analysis

Before building anything:

1. Define the problem — classification, regression, ranking, generation, etc.
2. Assess data — volume, quality, drift patterns, labeling status
3. Set targets — accuracy, latency (<50ms inference), training time (<4h), cost ceiling
4. Map infrastructure — compute (GPU/CPU), storage, orchestration, serving platform
5. Choose deployment strategy — real-time, batch, streaming, edge
6. Plan monitoring — what metrics, what thresholds, who gets paged

2. Pipeline Development

Build modular, versioned pipelines. Each stage should be independently testable and retriable.

Data Validation → Feature Engineering → Training → Validation → Deployment → Monitoring
      ↑                                                                          |
      └──────────────────── Retraining Trigger ←──────────────────────────────────┘

Pipeline principles:

• Data validation FIRST — catch schema drift, missing values, distribution shifts before training
• Version everything: data, features, models, configs, code
• Each stage writes artifacts to a versioned store (MLflow, DVC, W&B)
• Fail fast with clear error messages; never silently produce bad models

See references/pipelines.md for stage-by-stage implementation patterns.

3. Training & Optimization

Select the right approach based on complexity:

Hyperparameter optimization: Use Optuna (Bayesian) by default. Grid search only for ≤3 params with known ranges. Always set a trial budget and time ceiling.

Validation: k-fold cross-validation for small data, holdout + temporal split for time-series, stratified for imbalanced classes.

See references/training.md for distributed training, transfer learning, and advanced optimization patterns.

4. Deployment

Match deployment pattern to use case:

Serving checklist:

• Model serialized (ONNX, TorchScript, SavedModel)
• Inference endpoint tested (load test, edge cases)
• Health check endpoint active
• Fallback model configured
• Auto-scaling rules set
• Request/response logging enabled

See references/deployment.md for serving frameworks, scaling, and reliability patterns.

5. Monitoring & Retraining

Monitor these signals continuously:

Retraining triggers:

• Scheduled (weekly/monthly baseline)
• Drift-triggered (automated when metrics cross threshold)
• Manual (new labeled data, new features, architecture change)

See references/monitoring.md for drift detection implementations and alerting patterns.

Quick Reference — ML System Template

project/
├── data/
│   ├── raw/              # Immutable source data
│   ├── processed/        # Feature-engineered datasets
│   └── validation/       # Test sets, golden sets
├── features/
│   ├── pipelines/        # Feature transformation code
│   └── store/            # Feature store config
├── training/
│   ├── configs/          # Hyperparameter configs (versioned)
│   ├── scripts/          # Training entry points
│   └── experiments/      # MLflow/W&B experiment logs
├── serving/
│   ├── model/            # Serialized model artifacts
│   ├── api/              # Inference endpoint code
│   └── tests/            # Load tests, integration tests
├── monitoring/
│   ├── drift/            # Drift detection scripts
│   ├── alerts/           # Alert configs
│   └── dashboards/       # Monitoring dashboard configs
└── pipelines/
    ├── train.py          # End-to-end training pipeline
    ├── deploy.py         # Deployment automation
    └── retrain.py        # Retraining trigger logic

Reference Files

• references/pipelines.md — Data validation, feature engineering, pipeline orchestration (Kubeflow, Airflow, Prefect)
• references/training.md — Distributed training, HPO, transfer learning, ensemble strategies, advanced techniques
• references/deployment.md — Serving frameworks (BentoML, Seldon, TorchServe), scaling, reliability, A/B testing
• references/monitoring.md — Drift detection, alerting, retraining automation, incident response