ML Lifecycle

A model in a notebook is a claim. A model in production with a baseline, a monitor, and a rollback is a system. Offline accuracy is not evidence of online value, and the first suspiciously good result is usually leakage rather than skill.

Standing reference: references/data-quality-checklist.md.

Process

1. Measure a non-learned baseline first. Last observed value, majority class, a three-line heuristic. Every subsequent number is a delta against it. If the model is within noise of the baseline, the correct deliverable is the baseline.
2. Split by time when the data has time in it, and hold out a period the model never sees. A random split on temporal data trains on the future and produces a number that does not survive next week.
3. Hunt leakage before celebrating. Any feature computed after the label exists, or read from a table that is updated retroactively, is leakage. Audit each feature by asking what its value would have been at prediction time. A first result far above the baseline is an alarm, not a win.
4. Compute features once. Training and serving read the same code path. Otherwise you ship a model that scores differently in production than in the notebook, and the investigation costs a week. If the paths must differ, measure the skew as a number.
5. Evaluate slices, not just the aggregate. By segment, cohort, volume tier, and on the tail. Report the worst slice in the headline, because the worst slice is what generates complaints and it is invisible in the average.
6. Tie the offline metric to a product metric. State the link explicitly: this ranking metric maps to that conversion outcome because of this mechanism. An offline win with no stated link is a number, not a reason to ship.
7. Roll out behind shadow, then canary. Score live traffic without acting on it, compare against the incumbent on the same requests, then take a percentage with an abort condition written down in advance.
8. Monitor inputs and outputs after launch. Feature distribution drift, prediction distribution drift, and the product metric. A degrading model raises no exception; the only signal is a monitor somebody built.
9. Write the rollback before the launch. Which version, how it is pinned, how long it takes, and what triggers it. Retraining is not a rollback: it takes hours and might not help.

Output

## Model: <name> v<n>

- **Baseline:** <what> — <metric> = <n>
- **Model:** <metric> = <n> (delta <n>), worst slice <name> = <n> (n=<n>)
- **Split:** temporal, train <range>, holdout <range>
- **Leakage audit:** <features checked, what was removed and why>
- **Train/serve parity:** <shared path | measured skew>
- **Product link:** <offline metric> to <product metric> via <mechanism>
- **Rollout:** shadow <duration>, canary <n>%, abort if <condition>
- **Monitors:** input drift <how>, prediction drift <how>, product metric <how>
- **Rollback:** to v<n-1> via <mechanism>, takes <time>, tested on <date>

Verification

• Baseline and model numbers come from the same holdout; both quoted.
• Every holdout row is later in time than every training row.
• Shadow and production scores were compared on the same requests, and the skew is a number.
• Drift monitors were run against deliberately shifted input and fired.
• The rollback was executed once outside production and timed.

Red flags