AutoGluon-First Strategy

Problem

Manual ML on tabular problems often wastes time:

• Hours tuning GBDT hyperparameters (LightGBM, XGBoost)
• Manual feature engineering trial-and-error
• Manual ensemble design (which models? which weights?)
• Reaches a plateau, then struggles to break through

Reality: AutoGluon's best_quality preset achieves in 5-15 minutes what takes humans days.

Context / Trigger Conditions

Use this skill when:

• Starting any new tabular competition (regression or classification)
• Need a strong baseline in <15 minutes
• Manual GBDT work takes hours but gives similar results
• Want to focus manual effort on complementary techniques (deep learning, external data)
• Unsure which algorithm to use (LightGBM vs XGBoost vs CatBoost vs NeuralNet)
• Need to compare your work against a known strong baseline

Solution

Step 1: Run AutoGluon Baseline (5-15 minutes)

import time
from autogluon.tabular import TabularPredictor

label = 'target'  # Your target column
save_path = f'ag_baseline_{int(time.time())}'

predictor = TabularPredictor(
    label=label,
    path=save_path,
    eval_metric='accuracy',  # or 'rmse', 'roc_auc', 'log_loss'
    verbosity=1
).fit(
    train_data,
    presets='best_quality',  # 10+ algorithms, multi-level stacking
    time_limit=900  # 15 minutes
)

That's it. In 5-15 minutes you have:

• 10+ models trained (LightGBM, XGBoost, CatBoost, RF, ExtraTrees, KNN, NN)
• Multi-level stacking (Level 1, 2, 3)
• OOF predictions for validation
• Test predictions for submission

Step 2: Validate and Compare

# Get OOF score
oof = predictor.predict_oof()
test_pred = predictor.predict(test_data)

# Compare to your manual work
# - If AutoGluon matches you: Stop, don't waste time
# - If AutoGluon beats you: Learn from its ensemble
# - If AutoGluon loses: Use it as Silver signal, focus on what's different

5 Core Reasons AutoGluon Works

1. Multi-Algorithm Diversity

Manual ensembles fail because they use only 3 GBDTs (high correlation → high Cov). AutoGluon wins because it uses 10+ algorithms (low correlation → low Cov → low ensemble variance).

2. Multi-Level Stacking

Level 1: y1_i = model_i(X)   for i in 10 algorithms
Level 2: y2 = meta_learner(y1_1, y1_2, ..., y1_10)
Level 3: y3 = meta_meta_learner(y2, y1_combined)

Single-level averaging only learns linear combinations. Multi-level stacking learns conditional combinations:

• "If LightGBM predicts high AND KNN predicts low, use XGBoost"
• "If dataset is small, weight NeuralNet higher"

3. Automated Preprocessing

Each model needs different preprocessing:

• Tree models: Categorical encoding, missing value imputation
• NeuralNet: Standardization, OneHot
• KNN: Distance normalization
• Linear: Standardization

AutoGluon automatically does per-model preprocessing, while manual ensembles use uniform preprocessing (often suboptimal).

4. Bagging + Multi-Fold

For each model:
    for each fold (5-fold):
        train on 4 folds, predict on 1 fold
    average predictions

For small datasets especially, this dramatically reduces variance (the biggest bottleneck for small data).

5. Bayesian Optimization + Early Stopping

• Bayesian optimization predicts next hyperparams based on history
• Early stopping terminates bad models early
• Resource allocation trains good models more, bad models less

For small datasets: each train is cheap → can try more models → finds global optimum.

Validation Results (2026-06-14)

Win rate: 2/4 AutoGluon wins, 1/4 close, 1/4 manual wins (special method needed)

When to AVOID AutoGluon Tabular

How to Use AutoGluon in Custom Pipelines

Option 1: AutoGluon as Baseline (5-15 min)

• Run AutoGluon → get baseline OOF
• Compare to your manual work
• If AutoGluon matches you: Stop, don't waste time
• If AutoGluon beats you: Use it directly, focus elsewhere

Option 2: AutoGluon as Silver Signal (House Prices SST-style)

• Run AutoGluon → get OOF + test predictions
• Train your manual ensemble separately
• Blend: 0.30-0.50 × AutoGluon + 0.50-0.70 × Manual
• This adds diversity that AutoGluon's internal ensemble might miss

Option 3: AutoGluon + Special Methods

• Run AutoGluon first
• Add complementary approach (deep learning, external data, special features)
• Validate via CV that special method adds signal beyond AutoGluon

Anti-Patterns to Avoid

❌ Don't skip AutoGluon to "save time on small data" — Titanic 5 min AutoGluon = 0.8552 ❌ Don't trust AutoGluon for text/image — needs different approach ❌ Don't assume AutoGluon = best — TPS May 2022 had manual Silver 12% better ❌ Don't use AutoGluon as final without LB validation — CV-LB gap exists ❌ Don't spend days tuning GBDT if AutoGluon matches in 15 min

Empirical Pattern: When Manual Wins Over AutoGluon

• Special domain knowledge (chemistry, physics, biology)
• External data integration that AutoGluon doesn't have access to
• Custom feature engineering that captures domain-specific signals
• Advanced ensemble techniques beyond AutoGluon's multi-level stacking

Related Skills

• catboost-first-tabular — When AutoGluon doesn't work, try CatBoost (House Prices V17 lesson)
• cv-lb-validation — Always validate AutoGluon on LB (CV-LB gap exists)
• multi-model-diversity — AutoGluon implements this internally (10+ algorithms)
• asymmetric-blending — AutoGluon can be one part of a larger blend
• ml-sweet-spot — When to stop optimizing and accept AutoGluon baseline
• kaggle-competition-best-practices — Full competition workflow with AutoGluon

Validation Source

• House Prices: AutoGluon OOF 0.1180 vs V18 manual 0.1194 (2026-06-12)
• Leaf Classification: AutoGluon 0.9889 vs Gold 1.0 (2026-06-14)
• Titanic: AutoGluon 0.8552 vs classic 0.82 (2026-06-14)
• TPS May 2022: AutoGluon 0.8757 vs Silver 0.9975 (2026-06-14)