regime-scan

A market-level daily read, the layer above the four name-level scans. It answers three questions that move at different speeds:

1. Trend: the direction of the primary trend (slow: weeks–months)
2. Breadth: whether the trend is healthy or narrowing (the early-warning layer; divergences show up here first)
3. Sentiment / Credit: whether fear/greed is stretched and positioning fragile (fast: days)

A turn tends to show up as internals deteriorating while the index still prints highs. The scan encodes that two ways: it raises divergence flags (breadth not confirming, equal-weight lagging, credit rolling over, defensive rotation, vol-curve inversion), and it logs each US market day once to state/history.csv, so the slope of breadth / VIX / credit across days (the real turn signal) stays visible beyond today's number.

This formalizes the "regime gate + cohort dashboard" idea from methodology.md (2026-06-03) and the breadth/narrowing observations in macro.md into one daily action.

What it pulls (all yfinance, ~516 tickers, one batched download)

• Indices: SPY, QQQ vs 50/200DMA + 200DMA slope; RSP/SPY (equal- vs cap-weight = the narrowing proxy)
• Breadth: % of the S&P 500 universe (state/breadth_universe.txt, ~500 names across all 11 sectors, regenerated quarterly by build_universe.py) above their 50/200DMA, plus 52-week new-highs − new-lows
• Vol / sentiment: ^VIX level + 5-day change; ^VIX/^VIX3M term structure (backwardation = acute stress)
• Credit / rotation: HYG/LQD (HY vs IG; credit tends to lead equities at turns); defensive (XLU/XLP/XLV) vs offensive (XLK/XLY/XLC) relative strength

Dependencies (auto-fetched by uv run --with): Python ≥ 3.10, yfinance>=1.3,<2, pandas>=2 (which pulls in numpy transitively; the script imports only pandas/yfinance). No persistent venv.

<SKILL_DIR> below is the directory containing this SKILL.md.

Run

# Standard daily run
uv run --with 'yfinance>=1.3,<2' --with 'pandas>=2' \
  python <SKILL_DIR>/scripts/scan.py

# Inspect the daily state log (no new scan); watch the slope of state/breadth/vix
... python <SKILL_DIR>/scripts/scan.py --show-history

# Machine-readable
... python <SKILL_DIR>/scripts/scan.py --format json

# Longer slope/relative-strength window (default 20 sessions ≈ 1 trading month)
... python <SKILL_DIR>/scripts/scan.py --lookback 30

# Signal-quality report + outcome grading (see "Signal quality & outcomes"). Re-run quarterly.
uv run --with 'yfinance>=1.3,<2' --with 'pandas>=2' --with 'numpy>=1.24,<3' \
  python <SKILL_DIR>/scripts/backtest_outcomes.py

Parameters

Refreshing the breadth universe

The breadth pool is the live S&P 500 constituents, pulled from Wikipedia (yfinance can't return index membership: ^GSPC exposes no constituents attribute, and an ETF's funds_data.top_holdings caps at the top 10). It's a quarterly snapshot: breadth's whole signal is the slope of "% above 50DMA" across days, and churning the universe between runs would inject compositional noise into that slope. Freeze it within a quarter; regenerate at quarter boundaries.

Fresh-clone bootstrap: state/breadth_universe.txt is gitignored as a regenerable cache, so a fresh checkout doesn't have it; run the command below once before the first scan. Until then scan.py warns and skips the breadth signals rather than crashing, which guts the early-warning layer.

# Regenerate state/breadth_universe.txt  (+ a dated state/breadth_universe.<YYYY>-Q<N>.txt snapshot)
uv run --with 'pandas>=2' --with lxml --with requests \
  python <SKILL_DIR>/scripts/build_universe.py

# Preview the per-sector breakdown, write nothing
... python <SKILL_DIR>/scripts/build_universe.py --dry-run

The generator dash-normalizes class shares (BRK.B→BRK-B), prints a per-sector count, and backs the prior list up to breadth_universe.bak.txt. As a guard it refuses to overwrite the live file if it parses fewer than 400 names (a truncated fetch or a Wikipedia re-layout), so a bad fetch can't shrink the breadth pool the daily scan depends on. The original hand-curated 97-name list lives at state/breadth_universe.legacy97.txt. scan.py reads every non-# line of breadth_universe.txt, so you can hand-edit it (it ignores # header lines).

How to read the output

Data line: first under the title, it names the session the read reflects. rebuilt from 30m intraday bars is the workaround for Yahoo's late daily bar succeeding (^VIX3M ran 0.4% off its settled close in the 2026-10-02 replay, everything else within 0.03%) and needs no mention. A ⚠️ Stale data warning means the rebuild failed too: the whole read, and the history row it files under the earlier session, is a day old. Lead with that in plain words.

State banner: one of three, in escalation order (mirrors the methodology ladder):

• 🟢 RISK-ON: trend gate on + layers confirm + ≤1 divergence. Trend healthy, hold per rules; new money can scale in on pullbacks.
• 🟡 CAUTION: trend still up but ≥2 divergence flags (or breadth weakening). Tighten trail stops, raise cash buffer; new money only on pullbacks, never chase 🔴.
• 🔴 RISK-OFF: either the trend gate is off (SPY below / rolling 200DMA), or price is still above 200DMA but ≥4 internals broke (the "price hasn't dropped yet but internals are already rotting" late-stage top). Cut gross exposure, let trail stops take over; don't bottom-fish an unconfirmed bounce.

Confirmed (2-day) line: the state under a 2-run-day confirmation rule (confirmed_state in JSON). The raw daily label chatters at the score threshold (6 of the first 10 logged transitions were single-day whipsaws), so conviction-funnel and premarket read this line for sizing; a first-day flip is "watch, don't act". The raw label above it remains the honest daily reading and is what history.csv logs; the confirmation costs one run-day of detection lag on genuine turns.

Score: sum of 10 per-signal votes (🟢 +1 / ⚪ 0 / 🔴 −1) across the four layers, with the 🟢/⚪/🔴 split in the banner. A blunt gauge; the divergence flags and the trajectory matter more than the absolute score.

Non-🟢 votes: only the ⚪/🔴 votes print, one line each with the raw reading — a 🟢 vote needs no explanation. The full 10-signal table is behind --verbose.

⚠️ Turn warnings (divergence flags): the turn detector. Each fires only while the uptrend is intact (a broken internal under an already-broken tape is the bear, not a divergence):

• Breadth divergence: SPY near its 52w high but < 50% of names above their 50DMA
• Narrowing rally: RSP/SPY falling (mega-cap-only rally)
• Credit weakening: HYG/LQD rolling over while stocks hold
• Defensive rotation: defensives outrunning offensives and the gap deepening vs 5 sessions ago (2026-07-31 retune: as a pure level alarm it was on 83% of days, chronic and habituating; the score vote stays level-based and only the flag became a change alarm, so flag base rates before/after that date aren't comparable)
• Vol-curve inversion: VIX > VIX3M
• VIX 5-day spike

None alone is a sell; 2–3 stacking = de-risk. This layer is the skill's founding requirement: catch sentiment turning before price confirms.

State strip + Trend lines: the trajectory layer, as glyphs instead of a table (2026-07-31 redesign). The strip is the last ~14 run-days of state (🟢🟢🟡🟢…, today last); below it, one sparkline per watched series (breadth, RSP/SPY, VIX, VIX/VIX3M term, credit, defensive−offensive) with the current value, a ↗/→/↘ arrow vs ~5 runs ago, and ⚠️ on any series whose divergence flag is firing. Read the slope: a one-day snapshot can't tell you if sentiment is turning; the multi-day drift can. The full numeric log stays in --show-history (which now ends with the same dashboard).

Presenting the result: the dashboard is already the readable form — relay it verbatim (keep the code block, or alignment breaks) rather than re-tabulating it, then add a short interpretation in the conversation's language: what the state is, which flag/series is the live concern, and what would change the call. Don't re-list every number the dashboard already encodes.

Write the interpretation for a reader with no finance background. Translate every term into everyday language the moment you use it — "breadth 60%" means "60% of the 500 big stocks are still in short-term uptrends, so fewer stocks are participating"; "defensive rotation" means "money moving into utilities/groceries/pharma, the stocks people hide in when nervous"; "VIX 17.9" means "the market's fear gauge; under 20 is calm". State plainly what, if anything, the reader should do — usually "nothing".

How it ties to the journal rules

• The trend gate (SPY > rising 200DMA) is the mechanical "market top" from methodology.md 2026-06-03: it flips you out rather than predicting the top.
• The divergence flags are the early warning: 🟡 means tighten trail stops / raise cash, a step short of dump everything. Consistent with "don't predict the top — build a position that survives without predicting it."
• Breadth < ~60% + gate-off is the kill switch (methodology.md 2026-06-03): cut gross exposure.
• Pairs with regime-scan → momentum-scan: read the market here first (are we in? add or trim?), then read names there.

Signal quality & outcomes (scripts/backtest_outcomes.py)

The script grades the signal itself in two separate halves: signal quality (readable now, even on weeks of data) and SPY forward returns (the grading framework is locked; its strata stay marked ⚠︎thin until each holds ~8 independent 20-session windows, so conclusions accrue with the log rather than getting invented from it). Re-run quarterly alongside the sister scans' backtest_outcomes.py.

First report (2026-07-31, 36 readings; findings about the signal, not about returns):

1. The daily label chatters at the state boundary. 11 spells over 36 run-days, 36% of them a single run-day; 10 raw transitions, of which a 2-day confirmation rule removes 6 as whipsaws at a 1-run-day detection lag (3-day: 8 removed, 2-day lag). The score oscillating around the RISK-ON/CAUTION threshold flips the label without the tape changing, and every flip reached downstream readers (funnel sizing, premarket framing) as a real regime change. Fixed 2026-07-31: the banner/JSON now carry a Confirmed (2-day) state (see "How to read the output"); funnel and premarket size off that line, and a first-day flip reads "watch, don't act". The raw daily label remains what history.csv logs.
2. The Defensive-rotation flag was chronic: on 83% of days (longest streak 14). An always-on warning can't be tested (no contrast group) and habituates the reader; premarket's 2026-06-09 miss (the flag was footnoted, then led the tape) is the same lesson from the other side. Fixed 2026-07-31: the flag now also requires the rotation to be deepening (> +0.5pp vs 5 sessions ago), a change alarm matching the skill's own "read the slope" doctrine; the score vote stays level-based. Flag base rates in the next report straddle this change; compare on-shares within one convention. Credit weakening (22%) and VIX spike (17%) were healthy and episodic already.
3. Forward returns verify the pipeline, nothing more yet. ~2 independent 20-session windows exist so far, all inside one regime; the by-state / by-score / by-flag / transition-day tables print with thin-markers, and you should read them as pipe-verification until a few genuine regime turns are on record.

Known limitations

• Breadth universe is the S&P 500 (~500 names), a quarterly snapshot, smaller than the full ~4,000-name market, and it applies today's membership to past prices (a mild survivorship bias, standard for a forward-looking gauge; a real backtest would need point-in-time membership). Broad enough to smooth the breadth %, and self-refreshing via build_universe.py; edit state/breadth_universe.txt to retune.
• Yahoo's daily bar can be late: since 2026-09-02 the just-closed session's bar often isn't out by the evening run. All 21 evening runs from 09-02 to 10-02 were therefore filed under the previous session (rows key on SPY's last bar), honestly labeled but a day late. The scan now rebuilds that session from 30-minute intraday bars, ^VIX and the other indices included, and says so on the Data line.
• No intraday / real-time: the scan uses daily closes. A run during market hours sees a partial today-bar in the MAs/breadth (the script warns); the post-close run overwrites that row, since history keys on the data's session date rather than the wall clock.
• Votes are mechanical: thresholds follow convention, without optimization. Treat the output as a structured dashboard to interpret rather than a trade signal. The signal compounds across days of history; a single run is a snapshot.
• Credit via HYG/LQD ETF ratio is a proxy for the OAS spread: good for direction, coarser than actual HY option-adjusted spreads.

Tests

cd <SKILL_DIR>/scripts && uv run --with 'yfinance>=1.3,<2' --with 'pandas>=2' \
  --with 'numpy>=1.24,<3' --with pytest pytest -q

Pure-logic tests (no network) cover the state machine, every divergence flag, the vote helper, the breadth/new-high-low math (test_classify.py), and the signal-report's spell/confirmation/flag-parsing/forward-window logic (test_backtest_outcomes.py).