Controlled Submission Experiment

Problem

When multiple changes are bundled into a single submission (new features + new model + new post-processing), it's impossible to tell which change helped or hurt. A submission that scores worse than baseline might actually contain valuable model improvements hidden by a post-processing regression.

Symptoms

• New submission scores worse than baseline despite "better" CV
• Multiple changes made simultaneously (features + model + post-processing)
• Zero ratio in submission changes significantly (>2%) between versions
• Can't explain WHY the score changed in a specific direction

Solution

The Controlled Experiment Pattern

Isolate one variable at a time by creating multiple submissions from the same model output:

Baseline:  Model_A + Postproc_A  → Score_A
New:       Model_B + Postproc_B  → Score_B
Control:   Model_B + Postproc_A  → Score_C  (KEY!)

Interpretation matrix:

Implementation

import pandas as pd
import numpy as np

# Step 1: Load both submission predictions (before post-processing)
r10_raw = pd.read_csv("submission_r10_raw.csv")  # baseline raw predictions
r11b_raw = pd.read_csv("submission_r11b_raw.csv")  # new raw predictions

# Step 2: Identify disputed predictions
merged = r10_raw.merge(r11b_raw, on="id", suffixes=("_r10", "_r11b"))
disputed = merged[
    (merged["sales_r11b"] == 0) & (merged["sales_r10"] > 0)
]
print(f"Disputed rows: {len(disputed)}")
print(f"R10 values in disputed: mean={disputed['sales_r10'].mean():.2f}")

# Step 3: Create controlled submission
# Use Model_B predictions but with Postproc_A logic
controlled = r11b_raw.copy()
# Apply only the baseline post-processing
controlled.loc[controlled["sales"] < 0.1, "sales"] = 0
controlled.to_csv("submission_r11c_controlled.csv", index=False)

Diagnostic: Zero Ratio Comparison

# Quick diagnostic — always check this
for name, df in [("R10", r10), ("R11b", r11b), ("R11c", r11c)]:
    zero_pct = (df["sales"] == 0).mean()
    print(f"{name}: zeros={zero_pct:.2%}, mean={df['sales'].mean():.2f}")

# If zero ratio jumps >2% between versions, investigate immediately

Prevention

1. One variable at a time — never change model AND post-processing in the same submission
2. Always compare zero ratios — a jump >2% signals a post-processing change that may hurt
3. Save raw predictions before post-processing, so you can re-apply different post-processing
4. Name your experiments (R10, R11b, R11c) to track what changed between each

Evidence (Real Experiment)

Kaggle Store Sales (Favorita), April 2026:

Without the controlled experiment, we would have concluded "the improved model is worse." With it, we proved the model was actually +0.00056 better, but post-processing regressed -0.00249.

Notes

• This is the ML equivalent of A/B testing — isolate variables rigorously
• Related to rmsle-zero-threshold-asymmetry — the specific post-processing failure mode
• Related to kaggle-optimal-blending — when blending, also test components in isolation
• For ensemble experiments, use the same pattern: add one model at a time, measure delta