ML Engineering Methodology

Machine learning engineering is the bridge between model research and production systems. This methodology covers the engineering disciplines needed to train, evaluate, deploy, and maintain ML models reliably.

The ML Engineer's Domain

Reference Files

Templates

Scripts

Evals

evals/evals.json — output-quality eval manifest for this skill: fine-tuning plan review, eval-set design, quantization decision, deployment plan, regression triage, and training-run reproducibility.

Specialization decision boundary

Start from the required outcome and permitted prediction-time information. Include simpler rules, supervised or zero-shot classifiers where appropriate; classification scores are not automatically calibrated probabilities. A proprietary LLM case study is orientation, not proof of architecture optimality. Compare matched context/policy/budgets, preserve critical regressions and require every registered guardrail before promotion. New helper exit zero is descriptive comparison completion only; model execution and release remain separate evidence.

Route training operations to unsloth, typed decisions to system-one, labels/provenance to training-data-annotation, experimental analysis/calibration to data-scientist, and agent-level evaluation to agent-evals-and-observability.

For tabular foundation-model choice, read references/tabular-foundations.md; for named TabPFN setup/context/inference/fine-tuning, use tabpfn.

Core Principles

Measure before you optimize — Never quantize, prune, or distill a model without first measuring its baseline performance. Optimization without measurement is guessing.

Reproducibility is non-negotiable — Every training run needs a reproducible config: seed, data version, hyperparameters, and evaluation methodology. If you can't reproduce it, you can't ship it.

Baseline first — Before running an expensive fine-tuning run, establish a baseline with the base model. If the base model is already good enough, the fine-tuning budget is better spent elsewhere.

Test at the boundary — Model evaluation is most informative at the edges of the capability distribution, not at the center. Hard examples reveal more than easy ones.

The evaluation set is a liability — Every example in your eval set is a potential test-set leak. Use held-out sets, rotate examples, and periodically audit for contamination with the overlap checker.

When not to use

Do not use this skill for statistical modeling, experimental design, or causal inference — that's the data scientist's discipline. Do not use it to operate a specific inference engine: for llama.cpp installation, model loading, benchmarking, and troubleshooting, load the llama-cpp tool skill instead; for vLLM deployment, model configuration, benchmarking, batching tuning, GPU operation, and upgrade/rollback, load the vllm tool skill instead. This skill provides the methodology (eval-set design, quantization trade-offs, deployment plans, regression triage); the tool skills own the runbooks.