Multi-Level Aggregation Overfitting in Day-Specific Models

Problem

1st place solutions often use family-level and store-level aggregation features (family lag, store rolling mean, store-family ratio features). However, when applied to day-specific models (separate model per horizon day), these features can cause overfitting: CV improves but LB degrades.

Why It Overfits

In day-specific models, each model trains on ~2.87M samples covering all (store, family) pairs. The model already sees sufficient examples per pair. Family/store level features add redundancy:

1. Leakage via aggregation: Family-level lags are highly correlated with individual store-family lags (especially for families with few stores). The ratio features (sf_to_fam_ratio) may capture noise rather than signal.
2. Cross-pair interference: Day-specific models learn patterns across all pairs simultaneously. Adding aggregated features increases the feature space without adding truly independent information.
3. CV overfitting: The expanding-window CV may not penalize these features enough because the family/store patterns are stable across time, but they don't generalize to the test period's specific dynamics.

Evidence

CV improved by 0.00373 but LB got worse by 0.00050 relative to R11c.

In contrast, YoY features (only 4 features) improved LB by 0.00045 with less CV improvement.

Rule of Thumb

• Unified model: Multi-level aggregation likely helps (model needs hints about hierarchy)
• Day-specific model: Multi-level aggregation may overfit (model already sees all pairs)
• Safer alternative: Use target encoding at family/store level instead of raw aggregation
• Feature budget: Prefer fewer high-signal features (like YoY) over many correlated ones

Related

• yoy-364day-features — the feature that actually worked instead
• controlled-submission-experiment — how we isolated this finding
• ts-day-specific-forecasting — the framework where this applies