Match Submission Format to Evaluation Metric

Problem

A perfect model can score near-random on the public leaderboard if the submission file format doesn't match the metric's expectations. The most common mistake: submitting 0/1 thresholded labels for a metric that needs continuous probabilities.

Real failure case (2026-06-14, S6E2 Heart Disease):

• Model: AutoGluon best_quality ensemble, OOF AUC 0.95554
• Submission A: submission_autogluon.csv (0/1 thresholded via predictor.predict())
  • Public LB: 0.88403 ❌ (looked like complete model failure)
  • Private LB: 0.88643
• Submission B: submission_autogluon_proba.csv (continuous via predictor.predict_proba())
  • Public LB: 0.95357 ✅
  • Private LB: 0.95510
• Same model, same OOF — only the submission format differed. 0.07 LB drop from thresholding alone.

Context / Trigger Conditions

Use this skill when:

• About to submit the final (or any) prediction to a Kaggle competition
• AutoGluon / sklearn / xgboost default predict() returns hard labels for classification
• Competition metric is ranking-based (see list below)
• CV score is great but LB score is suspiciously low
• You see sample_submission.csv with 0.0/1.0 values (those are the target format, NOT necessarily the submission format)

DO NOT threshold for ranking-based metrics:

• roc_auc, auc — area under ROC; needs continuous scores
• auc_mu — multi-class AUC; needs full probability matrix
• log_loss — penalizes confident wrong answers; needs probabilities
• MAP, NDCG — ranking metrics; need scores
• brier_score — squared error on probabilities
• mean_columnwise_auc — column-wise AUC
• rmse on log-target (RMSLE) — for log-transformed regression, submit log predictions directly

DO threshold (or round) for these metrics:

• accuracy — predicted class label
• f1, precision, recall — predicted class label
• quadratic_kappa — rounded integer (for ordinal)
• mae, rmse on raw target — continuous regression predictions are fine, but rounding is harmless

Solution: Metric → Format Decision Tree

Is metric ranking-based (AUC, log_loss, MAP, NDCG, etc.)?
├── YES → Submit continuous probabilities / scores
│         AutoGluon: predictor.predict_proba(test)
│         sklearn:   model.predict_proba(test)
│         xgboost:   model.predict_proba(test)
│
└── NO (accuracy, f1, kappa, etc.)
    └── Submit class labels (or rounded regression values)
        AutoGluon: predictor.predict(test)
        sklearn:   model.predict(test)

Quick reference per framework

AutoGluon:

# WRONG for AUC:
preds = predictor.predict(test)            # hard labels
# CORRECT for AUC:
preds = predictor.predict_proba(test)     # probability DataFrame
# For multi-class AUC, submit the full matrix; column order matters
# and must match sample_submission.csv column order.

scikit-learn:

# WRONG for AUC:
preds = model.predict(test)                # hard labels
# CORRECT for AUC:
preds = model.predict_proba(test)[:, 1]    # positive class probability

XGBoost / LightGBM / CatBoost:

# For binary AUC:
preds = model.predict_proba(test)[:, 1]
# For multi-class AUC (sklearn API):
preds = model.predict_proba(test)          # full matrix
# For multi-class AUC (learning API):
preds = model.predict(test)                # this IS probabilities in raw API

Check sample_submission.csv carefully

head -3 sample_submission.csv

• Values are all 0 and 1 → metric is accuracy/f1 etc. → submit class labels
• Values are 0.0 and 1.0 with target column → likely a target template (binary outcome), but metric is probably AUC. Submit probabilities anyway.
• Values are floats in [0, 1] → metric is log_loss/probability calibration → submit probabilities
• Values are integers in [0, 9] → quadratic_kappa/ordinal → submit integers
• Values are floats with no obvious range → regression, submit raw predictions
• Values are strings like Low/Medium/High → multi-class classification with accuracy/balanced_accuracy etc. → submit hard labels, NOT probability matrix (S6E4 trap!)

Always cross-check the metric on the competition's Overview page, not the sample file.

S6E4 mirror bug (the inverse of S6E2)

The S6E2 bug was "AUC wanted proba, I sent 0/1". The S6E4 bug is the mirror: accuracy wanted hard labels, I sent a probability matrix.

S6E4 sample_submission.csv:

id,Irrigation_Need
630000,Low
630001,Low

If you submit:

id,High,Low,Medium
630000,0.001,0.999,0.000

Kaggle will reject it (SUBMISSION ERROR) because columns don't match sample.

Fix: use predictor.predict(test) (hard labels) instead of predict_proba() for accuracy-style metrics. Or argmax the probability columns to recover labels.

Both bugs are caught by a 30-line GSD verifier that compares your submission columns to sample_submission.csv columns. See gsd-loop-engineering skill for the full workflow.

Verification

Before submitting:

• ✅ Read the metric name in the competition's "Evaluation" tab
• ✅ Check whether the metric is in the "DO NOT threshold" list above
• ✅ Use predict_proba() (not predict()) when in doubt for classification
• ✅ For multi-class AUC: column order in submission = column order in sample_submission.csv

Sanity check after first submission:

• Public LB score is within 0.01-0.02 of OOF (typical for probability submissions)
• Public LB score is suspiciously low (<0.55 for binary, <1/n_classes for multi-class) → re-check format

Example: S6E2 Heart Disease, what went wrong

# Training (correct)
predictor = TabularPredictor(label='Heart Disease', eval_metric='roc_auc').fit(
    train_data, presets='best_quality', time_limit=900
)
# Note: predictor automatically uses AUC internally, so OOF is reliable

# First submission attempt (WRONG)
preds = predictor.predict(test)            # returns "Presence" / " Absence" labels
# Thresholded to 0/1
submission = pd.DataFrame({'id': test['id'], 'Heart Disease': (preds == 'Presence').astype(int)})
submission.to_csv('submission.csv', index=False)
# → Public LB 0.88403 (looked like total failure)

# Second attempt (CORRECT)
preds_proba = predictor.predict_proba(test)
# preds_proba columns: ['Absence', 'Presence'] (alphabetical)
submission = pd.DataFrame({'id': test['id'], 'Heart Disease': preds_proba['Presence']})
submission.to_csv('submission_proba.csv', index=False)
# → Public LB 0.95357 (matches OOF)

Notes

Why this is a sneaky bug:

• OOF score looks great, code "works" locally
• Local validation passes
• The error only surfaces on the public leaderboard
• A 0/1 submission's AUC is exactly 0.5 (random) when the model is balanced — easy to mistake for a real overfitting problem and start "fixing" the model

Why this happens more with AutoGluon:

• predictor.predict() defaults to class labels
• Other frameworks (sklearn, xgboost) often default to probabilities depending on the API
• Easy to forget the .predict_proba() distinction

Time investment:

• Fix: 30 seconds (change predict → predict_proba, save, resubmit)
• Cost of skipping: 1+ day debugging "why is my model broken"

Related Skills

• kaggle-data-format-first — input data format (2D vs 3D), complementary concern
• kaggle-competition-best-practices — overall submission workflow
• kaggle-optimal-blending — blend probability outputs across models
• cv-lb-gap-acknowledgment — distinguishes submission-format gaps from real CV-LB gaps
• autogluon-first — uses predict_proba() correctly for AUC

References

• S6E2 Heart Disease rerun artifacts: ~/projects/s6e2-autogluon-rerun/
  • submission_autogluon.csv (bad, 0/1) → LB 0.88403
  • submission_autogluon_proba.csv (good, proba) → LB 0.95357
• Kaggle Metrics Documentation
• AutoGluon predict_proba API