Walk-Forward Validation

Default data tool: BirdieAPI (birdieapi-tool). Validation is computation on data you already pulled. The calls that matter are get_ratings with as_of (2 credits per date), get_leaderboard (2) and get_winner_odds (5) for the market baseline.

You are an expert in honest model evaluation. Golf makes it easy to fool yourself: one winner per event, a few dozen majors with prices, strong dependence within a tournament and ratings that quietly include the future if you let them. Your goal is a test whose number you would stand behind.

When to Use

• "Can I use k-fold cross-validation on golf results?" (No.)
• "How do I split train and test?"
• "Does my model beat the market?"
• "How many majors do I need before this is meaningful?"

When NOT to Use

• Feature construction -- see feature-engineering
• Reliability diagrams and scaling -- see probability-calibration
• Money simulation -- see backtesting

Commands Available

Commands That Do NOT Exist

Rules

1. Time order only. Train on events before a date, test on events after. Never shuffle, never k-fold.
2. Group by tournament. All rows of one tournament go in the same fold. Within an event, results are dependent (one winner, shared course and weather).
3. Point-in-time inputs. Ratings with as_of the day before round 1. Season stats from the prior season. Course history from earlier editions.
4. Freeze choices before testing. Features, hyperparameters, sigma and tau are set on the training window.
5. Compare against baselines. At least the rating alone, and the devigged market where it exists.

How It Works

Step 1: Build the event list.

list_tournaments {"season": 2023, "status": "final"}

Repeat per season and sort by start_date. Add "major": true for the majors only.

Step 2: Choose a scheme. An expanding window by season is the default:

Rating-based models need little training data. The window mostly serves to fit sigma, tau, the shrinkage constant k from course-fit and any blend weight with the market.

Step 3: Pick metrics by target.

For winners, the score of one event is -log(p_winner). Average over events.

import numpy as np

def winner_log_loss(prob_by_event, winner_by_event, floor=1e-4):
    s = []
    for ev, probs in prob_by_event.items():        # probs: dict player_id -> prob, sums to 1
        p = max(probs.get(winner_by_event[ev], 0.0), floor)
        s.append(-np.log(p))
    return float(np.mean(s))

The floor stops one zero from producing infinity; state it. A winner at 0.2% costs 6.2 nats and a winner at 15% costs 1.9, so a handful of long-priced winners dominate the mean.

Step 4: The market baseline. For each major from 2021, devig the latest pregame prices (odds-analysis), apply the same floor and score the market on the same events.

get_winner_odds {"season": 2025, "limit": 200}

A model that does not beat this baseline out of sample adds nothing over simply using the price. Score both on events where prices exist; do not score the model on more events than the market and compare.

Step 5: Significance. About two dozen majors have prices, so about two dozen winners. Compute the per-event loss difference (model minus market) and its standard error:

d = np.array(model_losses) - np.array(market_losses)
se = d.std(ddof=1) / np.sqrt(len(d))
print(d.mean(), se)         # |mean| < 2 * se: no demonstrated difference

With about 25 events, a small improvement will not clear two standard errors. Say "no demonstrated difference" rather than "slightly better". For more power, evaluate the intermediate targets (made cut, top 10, round scores) over many more events, which is a strength of this data.

Step 6: Report by subset. Split majors from regular events only if each subset has enough rows. Put the count next to every number.

Step 7: Keep a final holdout. If you iterate on the 2025 fold it becomes a training fold. Hold the latest full season untouched until the end.

Common failures

Credit Usage

Anti-patterns

Output Format

Walk-forward, expanding window by season, folds 2021-2025 (majors with prices: N = <count>)
Winner log loss: model <x.xx>, devigged market <x.xx>, rating-only <x.xx>
Difference (model - market): <mean> +/- <se> over N events. Demonstrated: <yes/no>.
Made-cut log loss over <rows> golfer-events: <x.xxx>

What to Do Next