agentleFS
0trm0trm/gallop/skills/building-models/SKILL.md

building-models

Skill1 starsChanged 44 days agoMITView on GitHub

Decides whether a forecast a plan rests on, or a repeated, at-volume decision, belongs to a model, which output it needs (a forecast, a ranking, an allocation) and whether targeting needs propensity or uplift, validates the model out of time against the base rate and the naive baseline, and hands the impact claim to an experiment. Use when someone asks for a churn, propensity, LTV, scoring, forecasting, uplift, recommendation or allocation model, when a model's offline accuracy is offered as evidence that something worked, or when deciding who gets an offer, a discount or an intervention. When the model will run inside the product, is not an uplift model and bluML is installed, bluML's framing-models takes it after steps 0 to 2; an uplift model stays here.

The file

We split this skill into the rules it gives an agent

3 rules. About 0 are common, 3 are distinctive, and 0 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈3

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 0

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    When the model will run inside the product, scored on a schedule or served on request and kept live, and bluML is installed, steps 0 to 2 …

    line 103
  2. Distinctive:

    Every check below, run on a real pricing model: reference/worked-example.md.

    line 120
  3. Distinctive:

    When the decision runs continuously, keep a permanent holdout: a small slice never scored, so the model's contribution can be read at any …

    line 171

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Building models

The prediction position on the map: a forecast a plan rests on, or a decision made repeatedly, per user or per day, by a system rather than by a person reading a report. A forecast that sets inventory, a ranking that picks who gets the offer, an allocation that splits a budget. The bucket asks what will happen and who gets what, hands back a forecast, a ranking or an allocation with its uncertainty attached, and fails by breaking the moment you intervene, because the model learned a world in which nobody had acted on it yet.

Two rules arrive with every question routed here and leave with it. The model needs volume, history and a trusted measurement floor. And the model claims nothing about impact until an experiment says so: a churn model that predicts beautifully says nothing about whether the campaign works.

The mechanical part is a fixed set of checks on scores from an out-of-time window: lift at the top k over the base rate, calibration, a leakage screen of the features, Qini when a randomised arm exists, and MASE for a forecast. The checks score a fitted model; they do not fit one. Fit with whatever library is at hand.

0 · Is this a model's decision?

Three things have to be true. Routing checked only the first.

  • A forecast, or a decision made continuously. A forecast of one metric's value in a future window belongs here whether it is made once, for a plan (a hiring plan, a budget, a target), or every period. A ranking or an allocation belongs here only when it is decided continuously. A ship-or-kill call made once needs an effect size, not a forecast: send it back to the causal branch.
  • The floor holds. The outcome the model predicts is trusted in the metric registry. A model fits noise and reports confidence; it amplifies an instrumentation problem rather than revealing it. If the outcome is provisional or disputed, the work belongs to defining-metrics first.
  • Volume and history. For a classifier, a few hundred positive outcomes in the out-of-time window alone, and about a thousand in total; for a forecast, several full seasons. Below that the validation cannot separate the model from the base rate at the operating point, so nothing about the model can be believed, and the verdict is not fundable. Say so and stop. Rolling folds do not create outcomes that do not exist, and "build it thin and validate as data accrues" ships an unvalidated model for a year. The useful object at that size is a rule: a threshold on one or two fields, written down with the metric it will be judged on and the volume at which a model would become checkable. A regional squad rarely has the volume to justify anything heavier, and saying so beats shipping a model nobody can validate. A forecast, made once or every period, meets the same bar: below several full seasons, hand back the seasonal naive, or last period's value when no full season exists, with an interval from its past errors, and the history at which a model would become checkable.

1 · What will happen, or who gets what?

Three outputs, and the request names one.

  • A forecast. A metric's value in a future window. Hands back a point and an interval, conditional on nothing changing.
  • A ranking. Units ordered by an expected outcome, acted on from the top. Hands back the ordered list and the operating point: how far down the list the decision goes.
  • An allocation. A fixed resource split across options. Hands back the policy and its expected value, with the forecast or ranking it rests on.

Watch for the causal question in a model's clothing. "Build a churn model so we can see whether the retention emails work" asks two things. The model predicts who churns; the question is whether the emails change that. Prediction rides on correlation and does not care why. The emails were aimed at the people most likely to leave, so the model learns the campaign's targeting and reports it back as risk. Refuse to read impact from a model. Split the request: the model for targeting stays here, the impact goes to designing-experiments. The same refusal covers "the forecast says we beat plan, so the launch worked": a forecast error is not an effect size.

2 · Propensity or uplift

A propensity model ranks by the chance of the outcome. An uplift model ranks by how much the treatment changes that chance. For a decision about who gets an intervention, the second is the question and the first is the usual answer, which targets the wrong people: the sure things who would have stayed anyway and the lost causes who leave regardless sit at the top of a propensity list, and the persuadables sit in the middle. The four kinds of unit and what each model does with them: reference/propensity-vs-uplift.md.

Uplift needs randomised rows: units where the treatment was assigned at random and the outcome logged. Observational rows cannot supply it; a model fit on them learns whoever chose the targeting. If no randomised data exists, the experiment comes first: route to designing-experiments, randomise the offer across the eligible population with exposure logged, and train the uplift model on that experiment's rows. Propensity is still the right model when the question is who will do something rather than whom to treat: fraud, capacity, support routing.

Handing the build to bluML

When the model will run inside the product, scored on a schedule or served on request and kept live, and bluML is installed, steps 0 to 2 are where this skill ends. They have settled that the question is a model's, which output it needs and, for a ranking or an allocation that picks who gets an intervention, propensity or uplift. An uplift model stays here to the end, because bluML does not validate uplift. Anything else goes to bluML's framing-models, and steps 3, 4 and 6 are bluML's from there: the validation, the forecast bar and drift. Its launch test comes to designing-experiments, as step 5 asks, with the current rule as the control and a permanent holdout when the decision runs continuously. Without bluML, carry on below.

3 · Validate out of time, against the base rate

The order is fixed and the checks come before the number. Mechanics and the leakage taxonomy: reference/validation.md. Every check below, run on a real pricing model: reference/worked-example.md.

  • Split by time. Fit on rows before a cutoff, score rows after it. Random folds on time-ordered data leak the future into the training set and report an accuracy the model will never see in production. Leave a gap the length of the label horizon.
  • Screen for leakage. A feature that separates the outcome near-perfectly on its own is usually the outcome, or something logged after it. Every flagged feature is explained before any number below counts.
  • Beat the base rate at the operating point. The decision acts on the top k. Report precision at k against the base rate, as a lift. A lift of 1.0 is the base rate wearing a model. AUC on its own is not a validation, because no decision acts on the whole ranking.
  • Calibration, if the scores are used as probabilities. Thresholds and expected values need calibrated scores; a ranking does not. Reliability table, Brier score against the base-rate forecast, expected calibration error.
  • Uplift: Qini on the randomised rows, against random targeting. Precision at k means nothing for an uplift model.

The number reported reads "at the top 10% the precision is 31% against a base rate of 8%, a lift of 3.9, out of time on July", never "AUC 0.91".

4 · Forecasts and interventions

A forecast is conditional on nothing changing. It has to beat the seasonal naive: at a MASE of 1 or above, last season's value did as well and is the forecast. Over a multi-step horizon, judge it against the naive made at the cutoff and scored on the same dates. Baselines and intervals: reference/forecast-baselines.md.

A launch inside the horizon is the failure mode in its purest form. The model has never seen one and cannot see through it. Hand back two objects: the forecast under no change, and the launch's effect routed to the causal branch as its own question, with a holdout if the plan is still open. A forecast that "includes the launch" is a guess with an interval drawn round it.

5 · The holdout that measures impact

Offline validation says the ranking beats the base rate. It does not say that acting on the ranking moves the metric. That claim belongs to an experiment: hand to designing-experiments with the model as the treatment and the current rule as the control, never no rule at all, randomised at the unit the decision acts on, exposure logged, the metric the model exists to move as the primary. reading-experiments reads it. That result enters the prior store with design set to experiment. The model's lift and AUC never do; an offline metric is not an experiment result.

When the decision runs continuously, keep a permanent holdout: a small slice never scored, so the model's contribution can be read at any time and drift shows up as the gap closing.

6 · Drift and expiry

What ships changes the data. A model in production is scored on the population it changed, and its calibration decays from the day it goes live. Compare production scores with realised outcomes at the label's horizon, on a schedule. The knowledge entry names the expiry event as a retrain trigger: calibration drift past a stated bound, the holdout gap closing, or a product change to a feature's source. A model without a stated expiry is a stale belief automated at volume.

The verdict

One of four, stated with the evidence beside it:

  • Not a model's decision → route back: a ship-or-kill call made once needs an effect size; an untrusted outcome needs defining-metrics first.
  • Not fundable at this volume → a rule, written down, with the metric it is judged on and the volume at which a model becomes checkable. Not a thinner model with a warning attached: that is the failure this verdict exists to prevent, and the rule's effect is read by a holdout like any other policy. For a forecast, the rule is the seasonal naive, or last period's value, with an interval from its past errors.
  • Validated, impact unmeasured → ship behind a holdout; the impact question goes to designing-experiments. A forecast made once, for a plan, ships nothing: hand back the point and the interval under no change, and writing-reports files it as a knowledge entry, with the actual added once the window closes.
  • Impact measured → writing-reports files the report, the belief and the prior-store record, from the experiment's numbers.

Worked requests

The request as it arrivesVerdictWhy
"Build a churn model so we can see if the retention emails work."splitThe model targets; the emails' effect is an experiment
"Score users by churn risk and send the riskiest 20% the discount."uplift, not propensitySure things and lost causes top a propensity list; randomise the offer first
"Which users get the win-back discount this week?"rankingContinuous, at volume; validate out of time, ship behind a holdout
"Forecast Q4 signups. New pricing lands in November."two objectsA forecast under no change, and the pricing effect as a causal question
"Our churn model has 0.94 AUC on cross-validation. Ship it?"not validatedRandom folds on time-ordered data; rerun out of time, report lift at k
"We have 300 conversions a month. Build a propensity model."not fundableToo few outcomes to separate a model from the base rate; hand back a rule
"Build a demand forecast for the regional warehouse."forecastBeat the seasonal naive or ship the naive; intervals widen with horizon
"The forecast beat plan by 8%, so the campaign worked."refuseA forecast error is not an effect size; route to the causal branch

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.