Predictive Modeling

Overview

Causal identification asks what an intervention did; structural estimation asks what a world we haven't seen would do. Prediction asks something narrower: given what I can observe about a unit now, what is likely true of it — so I can act on it. Score the claim, rank the account, flag the pharmacy. The deliverable is a number on a unit that drives a decision; the model is a means, not the point.

The signature failure of this discipline is leakage. A feature that encodes the answer — a timestamp that exists only after the outcome, a row that appears in both train and test — makes the model look brilliant in validation and fail on the units you actually act on. The harness says 0.99 AUC; deployment says coin flip, and nothing in the loss curve tells you so.

There is a twin failure: reading the model as causal. The boosting machine ranks "prior audit flag" as the top feature, and the analyst writes "prior flags drive diversion." It does no such thing — the feature correlates with the label; the model uses it to predict, not because it causes anything. A predictive model is a correlation engine pointed at an action, and treating its internals as mechanism is how a triage tool becomes a false causal story.

Core principle: a prediction is trustworthy only if it was evaluated the way it will be deployed. Everything below serves that one sentence — the split, the leakage probe, the calibration, the baseline — and a model that scores well under any other evaluation has told you nothing about the units you will act on.

Why are you modeling? — choose the arm before you fit

This is the fork. One question decides which of three families you're in, and they lead to three different workflows:

Route by goal, not algorithm. The method doesn't decide the arm; the deliverable does. A random forest that ranks pharmacies by diversion risk is this skill even though it's "ML." The converse, where the line gets crossed most often: double/debiased ML, causal forests, and ML-estimated propensity scores are NOT this skill — they are causal-identification. When ML does the heavy lifting in service of a causal estimand, the goal is still an effect — the concern there is honest cross-fitting (nuisance models fit on data disjoint from where their predictions enter the moment condition), hand it back. If your "prediction" has a treatment variable whose coefficient is the point, you're in the wrong arm. Uplift/CATE-for-targeting is BOTH arms: the score is a causal effect (causal-identification's Design Card) AND a deployed ranking (this skill's deployment-matched evaluation) — run both gates.

First question: what decision does the prediction drive, and what does a false positive vs. a false negative cost? If you can't name the action and the asymmetry, you don't have a prediction problem yet — you have a curiosity, and a curiosity has no metric.

Write the Prediction Spec — immediately, and keep it living

A predictive model's modeling choices — the label, the split, the metric, the threshold — silently decide what every downstream number means. So the moment you understand the problem, write it down as a Prediction Spec and get the user's sign-off before fitting anything, before a leaked feature has quietly defined success. (Write-to-file, sign-off, mid-pipeline-reconstruct: analysis-checkpoints's locked-document gate; reconstructing here means especially rows 3 and 5, where the silent failure lives.)

The spec has seven rows:

1. Decision + who acts + cost asymmetry. What action does the score trigger, and what does a false positive cost relative to a false negative? This sets the metric and the threshold.
2. Target / label + regime. What exactly are you predicting, and which of the four regimes (next section) are you in? A vague label is a vague model.
3. Unit + prediction time — the "as-of" firewall. What is one row, and at what moment is the prediction made? Every feature must be knowable strictly before that moment — leakage is, almost always, a feature that didn't exist yet as-of prediction time.
4. Features + leakage audit. For each feature, affirm it's available as-of the prediction time and isn't a proxy for the label — a row in the spec, not an afterthought.
5. Split design matched to deployment — temporal if you predict the future, grouped if you act on new entities (new pharmacy, new patient), both if both; random only if neither holds. This is the heart of the spec — the analog of per-parameter identification in the structural card. A blank or a thoughtless "random 80/20" here is the single most common cause of a number that doesn't survive deployment.
6. Metric tied to the decision + baseline to beat. The metric follows from row 1 (precision-at-capacity when capacity is fixed; calibration when probabilities feed a cost calculation; recall when misses are catastrophic) — and name the trivial baseline the model has to beat, or it isn't earning its complexity.
7. Method + why — simplest-that-works first. A regularized logistic / a single tree is the baseline model, not the fallback; reach for boosting or deep nets only once the simple model is beaten honestly.

The spec is living, but a load-bearing change (label, split, metric, threshold) still routes through analysis-checkpoints, never a silent edit — changing the threshold after seeing the ROC decides how many false positives the user will tolerate, theirs to make.

The label decides the regime

What you can call "the label" determines what validation even means. There are four regimes; name yours in row 2 of the spec. Full detail for each — the split, the metric, the honesty check — is in references/prediction-regimes.md; the essence of each:

• Clean label. You observe the true outcome for past units (the claim was fraudulent; the customer did churn). The discipline is the split and the leakage audit, not the label itself.
• Proxy / weak label. You observe a stand-in, not the thing — "was audited and found," not "was fraudulent." Say so in the spec: the proxy carries its own selection (you only see audits of units someone already suspected), so metrics computed on it are valid only within the labeled subpopulation — extrapolating past that needs a stated assumption or a selection-corrected estimator.
• No label / anomaly. No outcome at all — you flag units that look unusual. Answer before any code: how will you EVER validate this? Unusual is not actionable, and an unsupervised model has no score to be right or wrong against.
• Ranking / triage. No calibrated probability needed — just the top k worth acting on, because capacity is fixed. The metric is precision-at-k / precision-at-capacity; the model's job is to fill the queue, not classify the world.

Prove the evaluation is honest — before you trust any metric

This is the proof step — the analog of structural estimation's Monte-Carlo recovery: there you prove the algorithm recovers truth; here you prove the evaluation is honest, before believing any score. Assert these, then keep one harness per model (one probe script, its permutation result in the phase golden — no per-script probe or pin file) so a later refactor can't silently reintroduce leakage. None of this is optional, and all of it precedes reporting a number.

• Permutation / null probe — run this first. Shuffle the labels within the split structure you'll evaluate under and refit the whole pipeline (recipe + the imbalanced-data trap: references/leakage-and-splits.md §3; no-label/anomaly analog — injected known positives: references/prediction-regimes.md card 3). Performance must collapse to no better than the trivial baseline (base rate / majority-class), not "chance" — a shuffled-label model can still post high accuracy on imbalanced data by predicting the majority class. If it still beats the baseline, the harness is leaking.
• "Too-good-to-be-true = leakage." A near-perfect score (0.99 AUC on a hard problem) is a leakage alarm, not a triumph — flag any single feature that alone predicts near-perfectly and investigate before you celebrate.
• Deployment-mirroring holdout. Evaluate on the split row 5 specifies (temporal / grouped / both). Never tune on the test set — use nested CV, and for temporal/grouped data the inner tuning fold must respect the same structure as the outer one (references/leakage-and-splits.md §4), or leakage sneaks back through the tuning loop. Tuning on test inflates the number by exactly the amount you then can't reproduce.
• Calibration. If the probability feeds a decision, a well-ranked model can still be badly calibrated (its 0.8 isn't 80%) — check the reliability curve, recalibrate (Platt / isotonic) if load-bearing. Optional if you only need the ranking; say which you need.
• Beat the trivial baseline. Compare against row 6's baseline. A model that doesn't beat "predict the base rate" or "keep the current rule" isn't earning its complexity or opacity — report that, don't bury it.

Recipes, split mechanics, and the leakage taxonomy: references/leakage-and-splits.md.

Prediction is not causation

The headline guardrail of this arm. A predictive model tells you which units to act on; it does not tell you why, and the moment you read it as mechanism you've left the discipline.

• Feature importance and SHAP describe what the MODEL USES, not what CAUSES the outcome. "Prior-flag is the top feature" means the model leans on prior-flag to predict — not that prior flags cause diversion, and not that changing a unit's prior-flag would change its outcome. These are model-attribution tools, not effect estimates, and no intervention claim comes out of a predictive model. If someone wants to act on the cause rather than triage by the score, that is causal-identification, full stop — a "which lever to pull" ask is a causal question wearing a prediction task's clothes; don't answer it by reading off importances, and don't quantify an expected effect from them either.
• Distribution-shift validity caveat. A predictive model is valid only on the population it was trained and validated on. Score pharmacies in one state, deploy in another with different prescribing norms, and the correlations it learned may not hold. Name the population in the spec and don't extrapolate past it.
• The anomaly special case — anomalous ≠ rogue. An outlier looks unusual, which is not the same as doing wrong; a brand-new high-volume specialty pharmacy is anomalous and legitimate. Without ground truth you cannot compute precision or recall. Validate instead by injecting known cases and checking they surface, expert review of the top-k, and stability across reasonable parameter choices — frame the output as triage, not classification.
• Consequential-decision note. When the flag points at a real entity for enforcement, audit, or denial, check base rates (at a 0.5% true rate even a great model floods the queue with false positives) and subgroup error (accurate overall, systematically wrong for a subpopulation), and hold the line that a human acts on a SCORE, not a verdict.

Choosing or changing the spec is the user's decision

Picking the label, split, metric, threshold, or method — and changing any of them once results are in view — is analysis-checkpoints territory. The danger window is after results: the model underperforms, so the impulse is to relabel "audited-and-found" as "fraudulent," loosen the temporal split to random, or slide the threshold until precision looks good — every one is a checkpoint, the prediction-arm twin of redesign-as-bug-fix.

Tooling (R / Python / Julia)

Existing methods only — this arm uses the ML toolbox, it does not invent estimators.

The model-attribution row (SHAP / DALEX) carries the standing caveat above: it explains the model, never the world. Reach for the simplest row first; boosting and its opacity only once the simple model is honestly beaten.

Red flags — STOP

• A random split where deployment is temporal or grouped — the most common silent killer.
• Tuning on the test set — hyperparameters selected on the data used for the honest estimate, no nested CV.
• A near-perfect score taken as a triumph instead of a leakage alarm.
• No permutation/null probe — trusted before randomized-label performance was shown to collapse to the trivial baseline, not merely "to chance."
• Feature importance / SHAP read as causal — "feature X drives the outcome," an intervention claim from a correlation engine.
• A proxy label treated as truth — "was audited and found" reported as "is fraudulent," selection unstated.

Common rationalizations

The Process

1. Confirm you are in the prediction arm, then write and approve the spec.
   • (1a) Confirm the arm. Is the goal a prediction (a number on a unit that drives an action), not an effect or a "why"? If it's really an effect, stop and route to causal-identification — settle this first; it is the fork, not a spec row.
   • (1b) Write the Prediction Spec and get sign-off — decision + cost asymmetry, label + regime, unit + prediction time, features + leakage audit, deployment-matched split, metric + baseline, method. Mandatory before any fitting.
2. Spec approved → invoke executing-analysis-plans — fan the CV folds, candidate models, and subsample cuts to parallel subagents rather than one slow serial loop.
3. Prove the evaluation is honest before trusting any metric — permutation/null probe, deployment-mirroring holdout (nested CV when tuning), calibration if load-bearing, beat-the-baseline. Freeze the harness as a regression baseline.
4. If a metric looks suspicious → invoke wrong-number-debugging, suspecting leakage first — bisect to the feature or join carrying the answer before blaming anything else.
5. Estimation + honest eval complete → offer result-verification before reporting (the user's call) — reproduce from a clean state, confirm the split mirrored deployment, confirm no internals are reported as causal.
6. If the label, split, metric, threshold, or method needs changing after seeing results → STOP and invoke analysis-checkpoints. The user's decision, never a silent re-cut to make a number look better.

The bottom line

Prediction claim  →  honest eval (permutation probe + deployment-matched split + beat-the-baseline),
                     calibrated if probabilities drive the decision, internals NOT read as causal
Otherwise         →  a confident number that fails on the units you actually act on