Regime Detection

name: regime-detection description: Market regime detection using HMM, Markov switching, and structural break methods. Use when identifying market environments.

When to Activate

• User wants to classify market environments (bull/bear/sideways, high/low volatility)
• Adapting strategy parameters to current market conditions
• Building regime-dependent allocation or risk management rules
• Detecting structural changes in market dynamics
• Evaluating whether a strategy works differently across regimes

First Questions

1. What defines a "regime" for your purposes (returns, volatility, correlation, macro)?
2. How many regimes do you hypothesize (2, 3, or more)?
3. What is the data frequency and how much history is available?
4. Do you need real-time regime classification or retrospective analysis?
5. Will regime detection drive allocation changes, risk limits, or strategy selection?

Core Concepts

Why Regimes Matter

Financial returns are not drawn from a single distribution. They exhibit:

• Volatility clustering: High-vol and low-vol periods persist
• Return asymmetry: Bear markets are faster and more violent than bull markets
• Correlation regime shifts: Correlations spike during crises (diversification fails when most needed)
• Fat tails: Partially explained by regime mixing — a mixture of normals produces fat tails

Regime-aware strategies can adapt position sizes, hedge ratios, and factor exposures to current conditions.

Regime Classification Frameworks

Returns-based:

• Bull: positive trend, low drawdown
• Bear: negative trend, rising drawdown
• Sideways/range-bound: no trend, choppy

Volatility-based:

• Low vol: VIX < 15 (or realized vol < 10%)
• Normal vol: VIX 15-25
• High vol: VIX > 25 (crisis regime)

Correlation-based:

• Normal: cross-asset correlations near historical norms
• Risk-off: correlations spike toward 1 (everything sells together)
• Divergence: correlations break down (dispersion regime)

Detailed Methodology

Hidden Markov Models (HMM)

HMM assumes observed returns are generated by an unobserved (hidden) state variable that follows a Markov chain.

Model specification:
- K states (typically 2-3)
- Each state k has: mean mu_k, variance sigma_k^2
- Transition matrix A: A_ij = P(state_t = j | state_{t-1} = i)
- Initial state distribution pi

Estimation (Baum-Welch / EM algorithm):
Step 1: Initialize parameters (K-means on returns is a good start)
Step 2: E-step: Forward-backward algorithm computes state probabilities
Step 3: M-step: Update mu_k, sigma_k, A given state probabilities
Step 4: Iterate until log-likelihood converges
Step 5: Viterbi algorithm recovers most likely state sequence

Typical 2-state result for equities:
  State 1 (Bull):  mu = +0.08%/day, sigma = 0.7%/day, duration ~200 days
  State 2 (Bear):  mu = -0.05%/day, sigma = 1.5%/day, duration ~50 days

Practical considerations:

• Number of states: Use BIC/AIC for selection, but 2-3 states is almost always sufficient
• Initialization sensitivity: Run from multiple starting points
• Look-ahead bias: Use online (filtering) probabilities for trading, not smoothed probabilities
• Regime persistence: Transition probabilities A_ii are typically > 0.95 (regimes persist)

Markov Switching Models (Hamilton 1989)

Extension of HMM that embeds regime switching within a regression framework:

y_t = mu_{s_t} + phi_{s_t} * y_{t-1} + sigma_{s_t} * epsilon_t

Where s_t in {1, 2, ..., K} follows a Markov chain

Advantages over simple HMM:
- Can include exogenous variables with regime-dependent coefficients
- Handles autoregressive dynamics within each regime
- Standard econometric framework with well-understood properties

Markov Switching GARCH: Allows both mean and variance to switch. Captures the empirical fact that volatility dynamics differ across regimes.

Structural Break Tests

Unlike HMM (which assumes recurring regimes), structural break tests identify permanent changes.

Chow Test:

• Known break date. Split sample, test equality of regression coefficients.
• F-statistic. Simple but requires specifying the break date a priori.

Bai-Perron (1998, 2003):

• Unknown number and location of breaks
• Sequential testing: test for 1 break, then 2, etc.
• Information criteria (BIC) for selecting number of breaks
• Provides confidence intervals for break dates
• Most widely used in practice

CUSUM / CUSUM-of-Squares:

• Monitors cumulative sum of recursive residuals
• Detects gradual change. Plot stays within bands under stability.
• CUSUM-of-Squares is more sensitive to variance changes

Practical Regime Detection Pipeline

Step 1: Choose observable(s) — returns, realized volatility, yield curve slope, credit spreads
Step 2: Estimate HMM with K=2 and K=3 states
Step 3: Compare BIC scores to select number of states
Step 4: Extract filtered (real-time) state probabilities
Step 5: Define regime threshold — e.g., P(bear) > 0.7 triggers defensive positioning
Step 6: Backtest regime-conditional strategy:
        - Full allocation in bull regime
        - Reduced allocation or hedged in bear regime
Step 7: Measure value-add vs static allocation
Step 8: Monitor for false signals (whipsaws during regime transitions)

Regime-Dependent Strategy Adaptation

Volatility Regime Detection

A simpler alternative to full HMM — use realized volatility with thresholds:

Compute: RV_t = sqrt(252 * mean(r_{t-20:t}^2))  [20-day realized vol]

Regime classification:
  Low vol:    RV_t < 30th percentile of historical RV
  Normal vol: 30th-70th percentile
  High vol:   RV_t > 70th percentile

Or use VIX term structure:
  Contango (VIX < VIX futures): complacency, low-vol regime
  Backwardation (VIX > VIX futures): fear, high-vol regime

Templates / Examples

HMM Regime Report Template

HMM Regime Detection Report — [Asset/Portfolio]
Estimation Period: [Start] to [End]
Number of States: 2 (selected by BIC)

State Parameters:
  State 1 (Bull):   mu = +12.5% ann. | sigma = 11.2% ann. | duration = 210 days
  State 2 (Bear):   mu = -18.3% ann. | sigma = 26.8% ann. | duration = 45 days

Transition Matrix:
            To State 1   To State 2
From State 1   0.9952      0.0048
From State 2   0.0222      0.9778

Current State: State 1 (Bull) with P = 0.94

Regime-Conditional Performance of Strategy X:
  Bull regime:  Sharpe = 1.45, Max DD = -8%
  Bear regime:  Sharpe = -0.30, Max DD = -28%
  Full sample:  Sharpe = 0.85, Max DD = -28%

With regime adaptation (50% allocation in bear):
  Full sample:  Sharpe = 1.10, Max DD = -16%
  Value-add:    +0.25 Sharpe, -12% max DD reduction

Regime Detection Evaluation Checklist

[ ] Number of states justified (BIC/AIC comparison for K=2,3,4)
[ ] Filtered (not smoothed) probabilities used for real-time signals
[ ] Regime durations are plausible (not too short for trading frequency)
[ ] No look-ahead bias — model estimated on expanding window
[ ] Whipsaw analysis: how many false regime transitions?
[ ] Transaction cost impact of regime switching measured
[ ] Robustness to parameter perturbation tested
[ ] Comparison vs simple rules (e.g., 200-day MA, VIX threshold)
[ ] Regime persistence is sufficient to be actionable

Quality Gate

Before using regime detection in production:

• Regime definitions have economic meaning (not just statistical artifacts)
• Filtered probabilities used for real-time decisions (no look-ahead from smoothing)
• Model retrained on expanding window with sufficient data per regime
• False signal rate is acceptable (regime transitions are not too frequent)
• Strategy improvement is meaningful after transaction costs from regime switching
• Simple baselines compared (moving average, volatility threshold)
• Regime classification validated against known market events (2008, 2020, etc.)
• Out-of-sample regime identification tested
• Latency of regime detection is acceptable (how many days to detect a new regime?)
• Fallback plan exists if the model produces ambiguous regime probabilities (P near 0.5)