Feature Engineering Saturation Detection

Context

Feature engineering always saturates. Every project has a "ceiling" past which new features stop helping. Continuing to optimize past saturation wastes weeks of engineering time. In a recent retail SKU recommendation project, 13 consecutive failed experiments (v31-v43) occurred before recognizing saturation at 76.5% F1 (theoretical ceiling 90.7%). This skill teaches how to detect saturation early and switch paradigms.

The core lesson: optimization without a ceiling is wasted effort. If you don't know your theoretical upper bound, you can't know if you're done.

Guidance

The 4 Saturation Signals

Signal 1: Consecutive F1 Stagnation
  - 3-4+ experiments without improvement
  - Each new feature adds ≤0.1pp
  - Action: Stop adding features, switch paradigm

Signal 2: High Correlation with Existing Features
  - New feature has Spearman correlation >0.7 with existing ones
  - Captures no new information
  - Action: Skip the feature, document why

Signal 3: Distance to Theoretical Upper Bound < 15pp
  - F1-EM (oracle ceiling) - actual F1 < 15pp
  - Historical coverage ceiling < 10pp above actual
  - Action: Optimization ROI is low, consider external data / new paradigm

Signal 4: Improvements Only From Threshold/Window Tuning
  - Last 3+ improvements came from "tweak N", "expand window", "adjust threshold"
  - No new information captured
  - Action: Architectural change needed (model class, data source, real-time signals)

Diagnostic Script

def detect_saturation(experiment_log):
    """Returns saturation status + recommended action"""

    recent = experiment_log.tail(5)

    # Signal 1: Stagnation (3+ experiments without >0.2pp improvement)
    improvements = recent['f1_diff'].tolist()
    stagnant = len([x for x in improvements[-3:] if x > 0.002]) == 0

    # Signal 3: Distance to ceiling
    upper_bound = compute_f1_em_upper_bound()  # F1-EM = oracle ceiling
    distance_to_ceiling = upper_bound - recent['f1'].iloc[-1]

    # Signal 2: Feature correlation
    new_feature_max_corr = check_feature_correlation(recent['new_features'])
    high_corr = new_feature_max_corr > 0.7

    if stagnant and distance_to_ceiling < 0.15:
        return "SATURATED — switch paradigm (external data, real-time signals, new model class)"
    elif high_corr:
        return "FEATURE REDUNDANT — try different angle or skip"
    elif distance_to_ceiling < 0.05:
        return "NEAR CEILING — declare success and ship"
    else:
        return "ACTIVE — keep optimizing features"

Decision Tree

Experiment didn't improve?
├── Yes → Check correlation with existing features
│        ├── max corr > 0.7 → Document as "redundant", skip
│        └── max corr < 0.7 → Check if business meaning makes sense
│                          ├── No → Document as "wrong intuition", skip
│                          └── Yes → Try different aggregation/window
└── No → Compare to theoretical upper bound
         ├── < 5pp gap → Declare success, ship to production
         └── > 15pp gap → Continue feature engineering

Saturation Journal Template

Track WHY each failed experiment failed, not just THAT it failed. Failed experiments are organizational assets.

## v31_error_features
- F1: 73.5% (-0.5pp from v28)
- Hypothesis: Add error analysis features (where model was wrong)
- Implementation: [details]
- Failure mode: FEATURE_REDUNDANT
- Diagnosis: New features correlated 0.85+ with `pair_weeks` and `reorder_ratio`
- Action: Skip feature, document pattern

## v34_n_predictor
- F1: 70.0% (-2.5pp from v32)
- Hypothesis: Predict N (recommendation count) instead of using formula
- Failure mode: WRONG_OBJECTIVE
- Diagnosis: N depends on actual purchases that week; using historical N leaks
- Action: Reverted, kept Optuna formula

When to Switch Paradigm (Beyond Feature Engineering)

Once saturated, try:

Why This Matters

In the retail SKU project, 13 consecutive failed experiments (v31-v43) cost ~3 weeks of engineering time. With this skill, you detect saturation at experiment 5 and redirect to "external data" or "real-time signals" — saving 2+ weeks per project.

When to Apply

When to Use

• You have run 3+ feature engineering experiments
• You have computed theoretical upper bound (F1-EM, RMSE-min, AUC-max, etc.)
• You suspect diminishing returns
• You want to know "should I keep optimizing or move on?"
• Your team is asking "why isn't this working anymore?"

When NOT to Use

• You have only run 1-2 experiments (too early)
• You haven't computed theoretical upper bound yet (compute it first)
• Your metric is highly stochastic (need 5+ experiments for confidence)
• You are still in baseline/baseline+1 phase (no saturation possible yet)
• The task is fundamentally novel (no historical patterns to learn from)

Notes

• Saturation ≠ Failure: Saturation means you hit the ceiling of the current approach. Try a new approach (model class, data source, real-time signal), not a better version of the current one
• Document failures religiously: Failed experiments become valuable organizational assets — root cause analysis prevents the team from re-trying the same dead ends months later
• Theoretical ceiling is essential: Without F1-EM, you can't know you're saturated. Always compute it FIRST before optimizing features
• "I have an idea!" is not a hypothesis: Every new feature needs a business reason. "I think this might help" is not a business reason; "Customer X in segment Y behaves like Z, so feature F should capture this" is
• See also: time-series-walk-forward-validation (for computing upper bound), multi-level-aggregation-overfitting

References

• Related principle: "Work Smart > Hard Work" — recognize when to stop and pivot
• Pairing skill: time-series-walk-forward-validation (provides the upper bound calculation this skill depends on)
• Layer 3 wisdom in cultivating-ml-agent framework: cross-domain principles for ML practitioners