annatchijova·annatchijova/SKILLS/data-leakage-hunting/SKILL.md

data-leakage-hunting

Skill9 starsChanged 4 months agoLicence not identifiedView on GitHub

Hunt the one bug class in machine learning whose symptom is a better score — information from the evaluation set reaching the model through preprocessing, time, groups, duplicates, features derived from the label, or a test set spent on tuning. Use whenever a dataset is split, a pipeline is fitted, a feature is engineered, a benchmark is reported, or a model performs suspiciously well. Trigger on "data leakage", "target leakage", "train/test contamination", "the accuracy is 99%", "too good to be true", "it works offline but not in production", "why did performance drop after deployment", "should I scale before or after the split", "cross-validation", "random split", "one feature dominates", "duplicate rows", "the benchmark may be in the pretraining data", or any metric that jumped with no change in method. Sibling of invariant-hunting: the invariant is that nothing derived from evaluation data may influence the prediction, and this skill hunts every transition where it silently breaks. It does not tune models.