Scikit-learn

Scikit-learn is the foundational Python library for classical machine learning, providing robust implementations of regression, classification, clustering, dimensionality reduction, and pipelines.

When to Use

• Classical Machine Learning Algorithms: Linear models, decision trees, random forests, clustering, and dimensionality reduction.
• End-to-End ML Pipelines: Constructing robust pipelines combining imputers, encoders, scalers, and estimators.
• Cross-Validation & Hyperparameter Tuning: Stratified K-fold validation, GridSearchCV, and RandomizedSearchCV.
• Model Evaluation & Diagnostic Metrics: Confusion matrices, ROC-AUC curves, precision-recall tradeoffs, and classification reports.

Quick Start

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.datasets import load_breast_cancer

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Type-safe, leakage-free ML pipeline
pipe = Pipeline([
    ('scaler', StandardScaler()),
    ('classifier', LogisticRegression())
])

pipe.fit(X_train, y_train)
score = pipe.score(X_test, y_test)
print(f"Test Accuracy: {score:.4f}")

Core Concepts

Robust Pipeline with ColumnTransformer

Handling heterogeneous numeric and categorical columns without data leakage:

import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.model_selection import train_test_split

# Sample dataset
df = pd.DataFrame({
    'age': [25, 45, np.nan, 35, 52, 23],
    'income': [50000, 120000, 85000, 75000, np.nan, 42000],
    'department': ['sales', 'tech', 'marketing', 'tech', 'sales', 'tech'],
    'promoted': [0, 1, 1, 0, 1, 0]
})

X = df.drop(columns=['promoted'])
y = df['promoted']

numeric_features = ['age', 'income']
categorical_features = ['department']

numeric_transformer = Pipeline(steps=[
    ('imputer', SimpleImputer(strategy='median')),
    ('scaler', StandardScaler())
])

categorical_transformer = Pipeline(steps=[
    ('imputer', SimpleImputer(strategy='constant', fill_value='missing')),
    ('encoder', OneHotEncoder(handle_unknown='ignore', sparse_output=False))
])

preprocessor = ColumnTransformer(transformers=[
    ('num', numeric_transformer, numeric_features),
    ('cat', categorical_transformer, categorical_features)
])

# Full model pipeline
model_pipeline = Pipeline(steps=[
    ('preprocessor', preprocessor),
    ('classifier', HistGradientBoostingClassifier(random_state=42))
])

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42)
model_pipeline.fit(X_train, y_train)
print("Test Score:", model_pipeline.score(X_test, y_test))

Cross-Validation & Metric Evaluation

Evaluating classification performance comprehensively:

from sklearn.model_selection import cross_val_score, StratifiedKFold
from sklearn.metrics import classification_report

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
scores = cross_val_score(model_pipeline, X, y, cv=cv, scoring='accuracy')

print(f"5-Fold CV Accuracy: {scores.mean():.4f} (+/- {scores.std():.4f})")

Hyperparameter Search with HalvingGridSearchCV

Fast successive halving parameter search:

from sklearn.model_selection import HalvingGridSearchCV

param_grid = {
    'classifier__learning_rate': [0.01, 0.05, 0.1],
    'classifier__max_iter': [50, 100, 200]
}

search = HalvingGridSearchCV(model_pipeline, param_grid, cv=3, factor=2, random_state=42)
search.fit(X, y)
print("Best Parameters:", search.best_params_)

Common Patterns

ColumnTransformer for Heterogeneous Feature Types

Problem: Applying separate preprocessing (scaling to numericals, one-hot encoding to categoricals) without data leakage.

Solution: Use ColumnTransformer:

from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OneHotEncoder, StandardScaler

preprocessor = ColumnTransformer(
    transformers=[
        ('num', StandardScaler(), ['age', 'income', 'balance']),
        ('cat', OneHotEncoder(handle_unknown='ignore'), ['occupation', 'city'])
    ]
)

pipeline = Pipeline([
    ('prep', preprocessor),
    ('model', LogisticRegression())
])

Best Practices

Do:

• Always use Pipeline and ColumnTransformer to prevent data leakage between training and validation sets.
• Use HistGradientBoostingClassifier / Regressor for tabular data; it is much faster than GradientBoostingClassifier.
• Set sparse_output=False in OneHotEncoder when piping into estimators that expect dense NumPy arrays.
• Evaluate models using appropriate metrics (e.g. roc_auc, f1_weighted, brier_score_loss) for imbalanced datasets.

Don't:

• Fit transformers on the entire dataset before splitting into train/test; always fit only on X_train.
• Use standard GridSearchCV on massive parameter grids; use HalvingGridSearchCV or Optuna.
• Use StandardScaler on sparse data without with_mean=False.

Troubleshooting

References

• Scikit-learn Documentation