CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Project Overview

Stock screening platform implementing CANSLIM (William O'Neil) and Minervini methodologies, with theme discovery, AI chatbot, and market analysis. Full-stack application with FastAPI backend, React frontend, PostgreSQL, Redis caching, and Celery for background tasks.

PR Workflow

Before pushing / opening a PR

• Run the FULL local test suite, including workflow/CI tests (e.g. tests that check .github/workflows). Never push workflow YAML changes without running those tests locally.
• Do not invent action version tags (e.g. setup-uv@v10). Check that the tag exists with gh api repos/<owner>/<repo>/git/refs/tags/<tag> first.
• Before pushing, self-review the diff for the bug classes reviewers keep finding here: swallowed exceptions or overly broad except, race conditions on first claim or lease, circuit breakers or state that can never reset, platform-specific exceptions (Windows ValueError), and missing imports.

PR Review Workflow

Addressing PR review comments

• For each bot or human thread: verify the finding, write a test that fails on the current code when applicable, fix it, then reply on the thread with the REAL commit SHA (never a placeholder).
• After fixing a round, look for sibling cases of the same bug elsewhere in the diff before pushing, so the next review round doesn't flag the gap.
• After pushing, request re-review (@codex review) and use ScheduleWakeup to poll CI and reviews. If gh returns a transient 401 or connection error, retry once before reporting failure.

Environment & Conventions

Environment & facts

• Shell is zsh: quote variables and use arrays, since unquoted $VAR does not word-split. Avoid source/nvm and rm -rf (blocked by safety hooks).
• Do one feature per git worktree. Never git checkout a commit hash (it detaches HEAD); use git switch with a branch.
• Schedules: the US cron runs in America/New_York, not UTC. CN/HK/JP market hours matter: don't trigger data runs while the market is open.
• Before stating facts in READMEs, issues or PR comments (versions, schedules, behavior), verify them against the code or config and cite the file.

Development Commands

Backend

cd backend
. venv/bin/activate
uvicorn app.main:app --reload --host 0.0.0.0 --port 8000

Frontend

cd frontend
npm run dev      # Vite dev server on :5173
npm run build    # Production build
npm run lint     # ESLint

Celery Workers (required for scans)

cd backend
./start_celery.sh    # Starts both queues

# Or manually:
./venv/bin/celery -A app.celery_app worker --pool=solo -Q celery -n general@%h
./venv/bin/celery -A app.celery_app worker --pool=solo -Q data_fetch -n datafetch@%h
./venv/bin/celery -A app.celery_app beat --loglevel=info  # Scheduler

Docker Deployment

Layered Docker Compose architecture with three scenarios:

# Local development
cp .env.docker.example .env   # Add API keys for chatbot
docker-compose up

# Homelab (behind reverse proxy like Traefik/nginx proxy manager)
cp .env.docker.example .env.docker
# Edit: CORS_ORIGINS=https://stocks.home.lan
docker-compose -f docker-compose.yml -f docker-compose.prod.yml up -d

# VPS with auto-HTTPS (Hostinger, DigitalOcean, etc.)
cp .env.docker.example .env.docker
# Edit: DOMAIN=stocks.yourdomain.com, CORS_ORIGINS=https://stocks.yourdomain.com
docker-compose -f docker-compose.yml -f docker-compose.prod.yml -f docker-compose.https.yml up -d

Docker files:

• docker-compose.yml - Base config (local dev)
• docker-compose.prod.yml - Production overlay (re limits, health checks, logging)
• docker-compose.https.yml - HTTPS overlay (Caddy with Let's Encrypt)
• .env.docker.example - Docker environment template
• Caddyfile - Caddy TLS configuration

Dependencies: Images install the hash-locked backend/requirements-runtime.lock. After changing backend/requirements-runtime.txt, run make lock-runtime, which ignores releases from the last 7 days. Add LOCK_ARGS=--upgrade to update existing pins.

Note: Backend runs as non-root user (uid 1000). After upgrade: sudo chown -R 1000:1000 ./data

Running Tests

Backend (pytest)

cd backend
. venv/bin/activate

# Run all tests
pytest

# Run unit tests only
pytest tests/unit/

# Run integration tests (requires running server at localhost:8000)
pytest tests/integration/ -m integration

# Run with verbose output
pytest -v

# Run specific test file
pytest tests/unit/test_canslim_scanner.py

Frontend (Vitest + React Testing Library)

cd frontend

# Run all tests once (CI mode)
npm run test:run

# Run tests in watch mode (development)
npm run test

# Run a specific test file
npx vitest run src/components/Scan/ResultsTable.test.jsx

# Lint test files
npm run lint

Note: Vitest 4.x requires Node 18+ (the Smithery CLI used for stockscreen-mcp needs 20+). On this machine the default node is already v24: nvm's default alias and /usr/local/bin/node are both Node 24, and nvm also has v22. No activation step is needed.

Test file locations:

• frontend/src/components/Scan/ — component tests (ResultsTable, FilterPanel, SetupEngineDrawer)
• frontend/src/components/Scan/filters/ — filter sub-component tests (CompactRangeInput, CompactCheckbox)
• frontend/src/utils/ — utility tests (formatUtils)
• frontend/src/test/fixtures/ — shared test fixtures
• frontend/src/test/renderWithProviders.jsx — MUI ThemeProvider test wrapper

Quality Gates (Setup Engine)

make help          # List all available targets
make gates         # Run all 5 SE quality gates
make gate-1        # Detector correctness
make gate-2        # Temporal integrity
make gate-3        # Integration coverage
make gate-4        # Performance baselines (advisory)
make gate-5        # Golden regression
make gate-check    # Verify all SE test files are in a gate
make all           # Full CI (backend gates + frontend)
make golden-update # Regenerate golden snapshots

Diagnostic Scripts

Utility scripts are in backend/scripts/:

cd backend
. venv/bin/activate

python scripts/inspect_redis.py            # Inspect Redis cache keys
python scripts/cache_diagnostic.py         # Trace cache flow (DB → Redis)
python scripts/check_cache_status.py       # Check price cache status
python scripts/clear_redis_price_cache.py  # Clear Redis cache after config change
python scripts/force_full_cache_refresh.py # Force full cache refresh
python scripts/cleanup_orphaned_scans.py   # Synchronously delete orphaned scans
python scripts/profile_scan_phases.py      # Time a bulk scan's prefetch/compute/persist phases (run in a worker container)

Manual orphaned scan cleanup runs directly:

python scripts/cleanup_orphaned_scans.py

The Celery task app.tasks.cache_tasks.cleanup_orphaned_scans remains the scheduled background path and requires a live worker to execute.

Architecture

Backend Structure

• FastAPI application in backend/app/main.py
• API routes in backend/app/api/v1/ - RESTful endpoints for stocks, scans, themes, chatbot, signals
• Services in backend/app/services/ - Business logic layer (70+ service files)
• Scanners in backend/app/scanners/ - Stock screening implementations
• Tasks in backend/app/tasks/ - Celery background tasks
• Models in backend/app/models/ - SQLAlchemy ORM models
• Schemas in backend/app/schemas/ - Pydantic request/response schemas

Key Architectural Patterns

Multi-Screener Orchestrator (scanners/scan_orchestrator.py):

• Coordinates Minervini, CANSLIM, IPO, Volume Breakthrough, and Custom screeners
• All screeners extend BaseStockScreener abstract class
• Data fetched once and shared across screeners
• Composite scoring via configurable aggregation (weighted_average, maximum, minimum)

Two-Queue Celery Architecture:

• celery queue: General compute tasks (4 workers local; Docker concurrency follows the deployed worker topology)
• data_fetch queue: API calls (1 worker, serialized to respect rate limits)
• All external API tasks route to data_fetch to prevent rate limit violations
• Exception — company exposure research: its tasks route only to the dedicated exposure_research queue and opt-in celery-exposure-research worker (docker-compose.exposure.yml, or EXPOSURE_WORKER_ENABLED=true in start_celery.sh), so long LLM/document stages never occupy price-fetch workers. Every research HTTP attempt still acquires the shared RateBudgetPolicy provider keys (e.g. sec_edgar); queue isolation grants no extra upstream rate. Research is disabled by default (EXPOSURE_RESEARCH_MODE=disabled). Beat entries company-exposure-* feed that queue with message expires, so they drop harmlessly when the worker is not deployed.

Per-Market Scan Queues:

• Manual user scans (run_bulk_scan) route to market-specific queues: user_scans_us, user_scans_hk, user_scans_jp, user_scans_tw (with user_scans_shared as fallback when no market is set)
• start_celery.sh spawns one worker per market in ENABLED_MARKETS (env, else backend/.env, else US), so with US,HK the US and HK scans run in parallel on separate workers
• @serialized_market_workload (see tasks/workload_coordination.py) holds a Redis lock keyed by market, so scans targeting the same market serialize while different markets remain independent
• Manual scans run in cache-only mode — they do not fall back to yfinance/Finviz. The API boundary rejects scans with 409 market_data_stale when cached prices haven't caught up to the last completed trading day for the market (see services/market_data_freshness.py::check_symbol_freshness). Inside the scan, RunBulkScanCommand(cache_only=True) reaches PriceCacheService.get_many(cache_only=True), which serves per-stock prices from whatever the database holds (even under 50 bars, or stale) and never calls a provider. The market benchmark lookup is not cache-only (it fetches from Yahoo if no cached benchmark reaches the last completed session), so the gate also refuses a market whose cached benchmark candidates are all behind that session (reason: benchmark_stale, #455)

Redis Caching Strategy (three-tier: Redis → PostgreSQL → API):

• DB 0: Celery broker
• DB 1: Celery results (24h TTL, auto-cleanup)
• DB 2: Application cache via shared connection pool (services/redis_pool.py)
  • Price cache: 7d TTL, stores 5 years of OHLCV (required for Volume Breakthrough Scanner)
  • Fundamentals cache: 7d TTL with DB fallback on Redis miss
  • Benchmark (SPY): 24h TTL with distributed locking to prevent thundering herd

LLM Integration (services/llm/):

• Supported provider path: Groq for chatbot/research, Minimax for primary theme extraction/merge, Z.AI fallback for extraction/merge
• The assistant (services/assistant_gateway_service.py) proxies chat to the Hermes sidecar. Hermes reads TAVILY_API_KEY and SERPER_API_KEY; the backend reads neither

Frontend Structure

• React 18 with Vite
• Material-UI for components
• React Query (TanStack) for data fetching
• MUI Table with TanStack Virtual row virtualization for results display
• Pages in frontend/src/pages/: ScanPage, ChatbotPage, ThemesPage, GroupRankingsPage, BreadthPage, SignalsPage

Frontend API Client Convention

CRITICAL: API paths must NOT include /api prefix

The axios client in frontend/src/api/client.js handles the /api prefix via baseURL:

• Local dev: baseURL = 'http://localhost:8000/api'
• Docker: baseURL = '/api' (set via VITE_API_URL build arg)

When adding new API endpoints, use paths starting with /v1/:

// ✅ CORRECT - path without /api prefix
const response = await apiClient.get('/v1/themes/rankings');

// ❌ WRONG - will cause double prefix in Docker (/api/api/v1/...)
const response = await apiClient.get('/api/v1/themes/rankings');

For API modules with a BASE_PATH constant:

// ✅ CORRECT
const BASE_PATH = '/v1/user-themes';

// ❌ WRONG
const BASE_PATH = '/api/v1/user-themes';

Why this matters: In Docker, nginx proxies /api/* to the backend. If paths include /api, you get /api/api/v1/... which returns 404.

Data Sources & Rate Limits

• yfinance: 1 req/sec (self-imposed)
• Finviz: Rate-limited via wrapper
• SEC EDGAR: 10 req/sec (150ms between requests)

Market Hours Awareness

Price data means different things during and after a trading session. Check the market's state before running anything that reads or fetches prices against a real database, and design for both states.

• Freshness is judged against the last completed session. MarketCalendarService.last_completed_trading_day(market) is that day; is_market_open(market) says whether a session is in progress. US regular hours are 09:30–16:00 ET (13:30–20:00 UTC in daylight time, 14:30–21:00 UTC otherwise). Every market has its own calendar.
• A fetch during a session stores a partial bar for today. The price cache marks it in Redis fetch metadata (needs_refresh_after_close) and refreshes it after the close. The marker lives in the cache Redis database, so a process pointed at a different CACHE_REDIS_DB writes the partial bar to PostgreSQL with no marker the live stack can see.
• Only the scan's cache_only path keeps per-stock prices off providers. batch_only_prices / batch_only_fundamentals stop the per-symbol fallback only, and a plain PriceCacheService.get_many still batch-fetches symbols with no usable database frame (stale rows, or fewer than 50 bars). The benchmark lookup can fetch in any mode; for manual scans the gate keeps it from needing to (#455). Code that must not reach a provider at all has to block providers itself, as backend/scripts/profile_scan_phases.py does for the benchmark and single-symbol fetches.
• The freshness cut-off moves at the close. A job that straddles it sees symbols turn stale part-way through. Start and finish long runs, benchmarks and profiles inside one session state, and record which state they ran in.
• Scheduled refresh jobs run after the close. They compete for CPU and rewrite the cache, so timings taken then are not comparable with timings taken mid-session.

When designing a feature that reads or writes prices, state what it does mid-session: whether it can see a partial bar, whether it can trigger a fetch, and what happens if the session closes while it runs.

Environment Variables

Local development: backend/.env (see backend/.env.example) Docker deployment: .env.docker in project root (see .env.docker.example)

Required for chatbot (at least one supported LLM provider):

• GROQ_API_KEY, GROQ_API_KEYS - Groq (fast inference, free tier)
• MINIMAX_API_KEY - Minimax (primary theme extraction)
• ZAI_API_KEY, ZAI_API_KEYS - Z.AI (theme extraction/merge fallback)

Web search (read by the Hermes assistant sidecar, not the backend):

• TAVILY_API_KEY, SERPER_API_KEY

Infrastructure:

• DATABASE_URL - PostgreSQL connection string (e.g., postgresql://user:pass@localhost/stockscanner)
• REDIS_HOST, CELERY_BROKER_URL - Redis/Celery configuration
• CORS_ORIGINS - Comma-separated allowed origins (for production)

LLM routing (optional):

• LLM_DEFAULT_PROVIDER - Optional override for sanctioned primary paths (Groq chatbot/research, Minimax extraction/merge)
• LLM_CHATBOT_MODEL, LLM_RESEARCH_MODEL - Model overrides (LiteLLM format)
• LLM_FALLBACK_ENABLED - Enable automatic provider fallback

Database

CRITICAL: Database Location

• Supported database: PostgreSQL referenced by DATABASE_URL
• docker-data/postgres/ holds the Docker Postgres data directory
• data/ remains for non-database state such as caches, backups, and Celery beat state
• If you see an empty database or missing data, verify DATABASE_URL points to the intended PostgreSQL instance

Persistent data paths:

• docker-data/postgres/ - PostgreSQL data directory (Docker)
• backend/celerybeat-schedule.db - Celery Beat scheduler state (legacy local path)

Key tables:

• stock_prices, stock_fundamentals, stock_universe - Core stock data
• scans, scan_results - Scan metadata and results with multi-screener scores
• ibd_groups, ibd_group_ranks - Industry group rankings
• theme_clusters, theme_constituents - Theme discovery
• signals - Technical signal detections
• chat_sessions, chat_messages - Chatbot conversation history

Migrations are versioned under backend/alembic/. PostgreSQL is the supported database for development and Docker deployment.

Screening Methodologies

Minervini Template: RS Rating > 70-80, Stage 2 uptrend, MA alignment (50 > 150 > 200), price 30%+ above 52-week low

CANSLIM: Current quarterly EPS > 25%, Annual EPS growth > 25% 3yr, new highs, volume patterns, RS > 70, institutional ownership 40-70%

macOS Development Note

For Celery on macOS, use --pool=solo (set via start_celery.sh) to avoid fork() crashes with curl_cffi. Also set:

export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES
export TOKENIZERS_PARALLELISM=false

Git Conventions

This project uses Conventional Commits for all commit messages. Format:

<type>[optional scope]: <description>

[optional body]

[optional footer(s)]

Types:

• feat: New feature
• fix: Bug fix
• docs: Documentation changes
• style: Code style changes (formatting, no logic change)
• refactor: Code refactoring (no feature or fix)
• perf: Performance improvements
• test: Adding or updating tests
• build: Build system or dependency changes
• ci: CI/CD configuration changes
• chore: Maintenance tasks (deps, configs)

Examples:

feat(scanner): add volume breakthrough screener
fix(chatbot): handle empty response from LLM provider
docs: update API endpoint documentation
refactor(api): consolidate stock data fetching logic
test(canslim): add unit tests for EPS calculation

Scopes (optional): api, scanner, chatbot, frontend, celery, db, cache, themes, signals