CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Project Summary

lixSearch is a multi-service intelligent search assistant (Python/Quart) that searches the web, fetches videos/images, and synthesizes answers with sources. It uses a pipeline architecture with distributed caching, vector embeddings via IPC, and Playwright-based search agents.

Build & Run

Docker (production)

./sops-reencrypt.sh --decrypt         # then fill in POLLINATIONS_API_KEY and optional HF_TOKEN
./deploy.sh build              # build image
./deploy.sh start 3            # start with 3 app containers
./deploy.sh scale 5            # scale to 5
./deploy.sh health             # check all services
./deploy.sh logs app           # tail app logs
./autoscale.sh                 # CPU-based autoscaler daemon (1-5 replicas)

Local development

source venv/bin/activate       # Python 3.11 venv at /mnt/volume_sfo2_01/lixSearch/venv/
redis-server --port 9530 &     # Redis on custom port
chroma run --host localhost --port 9001 &
APP_MODE=ipc python lixsearch/ipcService/main.py &      # embedding service (port 9510)
APP_MODE=worker WORKER_PORT=9002 python lixsearch/app/main.py &

Tests (integration, no unit test framework)

python tester/test_multi_turn_session.py
python tester/test_session_persistence.py
python tester/test_redis_semantic_cache.py

Health check

curl http://localhost:9002/api/health    # direct worker
curl http://localhost/api/health         # via nginx

Architecture

Network topology

Internal services communicate over a private Docker network (elixpo-network). The render service sits alone on a second network, render-network (172.31.250.0/24, bridge br-oreo-render), which only lixsearch-app also joins. None of the internal 9xxx ports are published to the host — they are only reachable within the Docker network (via expose). The only externally reachable entry point is nginx.

Internet → Cloudflare → search.elixpo.com
                              │
              ┌───────────────┴───────────────┐
              ▼                               ▼
     Next.js frontend (/)            nginx (:10001, API key)
     search.elixpo/                  ┌────────────────────────┐
     ├── /c/[session]               │  :80   internal/dev    │
     ├── /discover                  │  :10001 production     │
     ├── /library                   │  :443   reserved       │
     └── /api/* (proxy)             └──────────┬─────────────┘
              │                                │
              │  BACKEND_URL + X-Internal-Key   │ proxy_pass
              └───────────┬────────────────────┘
                          ▼
               lixsearch-app (:9002, N replicas, 10 Hypercorn workers each)
                 ├── Gateways (app/gateways/*.py)
                 ├── Pipeline (pipeline/lixsearch.py) — main async generator
                 │     ├── Tool execution (pipeline/optimized_tool_execution.py)
                 │     ├── RAG (ragService/)
                 │     └── LLM inference → Pollinations API (external)
                 ├── Session/cache (sessions/, ragService/cacheCoordinator.py)
                 ├── Render client (searching/scrape.py, X-Render-Token)
                 │     └─[render-network]→ render-service (:9520, singleton)
                 │           ├── Chromium, fresh context per render (screenshot/waitFor/fallback)
                 │           └── egress proxy (127.0.0.1) → public IPs only, ports 80/443
                 └── IPC client → ipc-service (:9510, singleton)
                                    ├── CoreEmbeddingService (sentence-transformers)
                                    ├── SearchAgentPool (Playwright text + image agents)
                                    └── Chroma vector DB (:9001)
Redis (:9530) shared across all containers:
  DB 0 — semantic query cache (5min TTL, per-session)
  DB 1 — URL embedding cache (24h TTL, global)
  DB 2 — session hot window (30min TTL, 20 msgs, LRU evicted to disk)

Ports & exposure

Authentication

• Port 80 (nginx internal server): No API key required. Used for local dev and inter-service health checks.
• Port 10001 (nginx authenticated server): API key required via X-API-Key header or ?key= query param. This is the production endpoint routed from search.elixpo.com through Cloudflare.
• Auth-exempt paths (both ports): /api/health, /docs, /api/docs, /openapi.json, /openapi.yaml
• App-level auth: INTERNAL_API_KEY env var in before_request middleware (app/main.py). Separate from the nginx API key — used for internal service-to-service calls.

Request flow

User → Cloudflare → nginx:10001 (API key check) → app:9002 → gateway (search.py / chat.py)
  → run_elixposearch_pipeline() async generator → query decomposition → tool routing
  → web_search/fetch/image_search/youtube → RAG retrieval → semantic cache check
  → LLM synthesis → SSE or JSON response

Conversation storage (two-tier hybrid)

• Hot: Redis DB 2 — last 20 messages per session
• Cold: Huffman-compressed .huff files in ./data/conversations/<session_id>.huff
• LRU eviction daemon migrates idle sessions (30min) from Redis to disk
• Disk archives have 30-day TTL, cleaned on startup

Key Files

Critical Patterns

• session_id is the single identifier for all cross-service communication. It must be passed through memoized_results["session_id"] for CacheCoordinator and friends. Always snake_case, always str.
• Config is centralized: all tunables live in pipeline/config.py with os.getenv() overrides. Don't hardcode values in service files.
• Logging convention: loguru with [Section] prefixes — [Pipeline], [RAG], [Cache], [IPC], [POOL], [APP].
• Pipeline yields strings: run_elixposearch_pipeline() is an async generator. Tools return strings that are yielded progressively to the client.
• IPC is a singleton: The ipc-service container runs one instance shared by all app replicas. The SearchAgentPool inside it has configurable SEARCH_AGENT_POOL_SIZE and SEARCH_AGENT_MAX_TABS (in config.py).
• Shared volumes: All app containers mount the same conversations-data, embeddings-data, cache-data volumes and connect to the same Redis instance. No per-container state.

Docker Services

Frontend (search.elixpo/)

Next.js app deployed on Cloudflare Pages at search.elixpo.com. Uses @cloudflare/next-on-pages for edge runtime.

Routes

Guest access

• Guests (no login) get 15 free requests per IP (tracked in RATE_LIMIT_KV, 24h TTL)
• Guest conversations navigate to /c/[sessionId] but are not saved to library
• clientId is generated per browser (stored in localStorage as elixpo_client_id)
• sessionId is generated per conversation (sess_ + 16 random chars)
• Rate limit returns 429 with X-RateLimit-Remaining header

Frontend → Backend flow

Browser → Cloudflare Pages (edge, Next.js API routes)
           → validates XID header (anti-abuse token)
           → checks guest rate limit (RATE_LIMIT_KV)
           → reads/writes D1 for conversations, discover articles
           → proxies search/session/surf to backend via BACKEND_URL
           → uses X-Internal-Key header for backend auth

Frontend data layer

• Cloudflare D1 (SQLite) — persistent storage for sessions, messages, bookmarks, discover articles
• Cloudflare KV (SESSIONS_KV) — read cache for session data (10min TTL, invalidated on write)
• Cloudflare KV (RATE_LIMIT_KV) — guest IP rate limiting (24h TTL)
• Schema: workers/migrations/0001_init/schema.sql (base), 0002_users/schema.sql (users + auth)
• Models: Session, Message, Bookmark, DiscoverArticle, User

Frontend API routes

Key frontend files

Deploy frontend

cd search.elixpo/
npm run pages:build            # build with @cloudflare/next-on-pages
npm run pages:deploy           # deploy to Cloudflare Pages
npm run db:migrate             # apply D1 migrations (production)
npm run db:migrate:local       # apply D1 migrations (local dev)

Cloudflare setup (one-time)

wrangler d1 create elixpo_search                    # create D1 database
wrangler kv namespace create elixpo_search_sessions     # create KV for session cache
wrangler kv namespace create elixpo_search_ratelimit    # create KV for rate limiting
# paste the IDs into wrangler.toml
wrangler secret put INTERNAL_API_KEY         # backend auth key
wrangler secret put XID                      # anti-abuse token
wrangler secret put API_KEY                  # nginx API key
wrangler secret put SSO_CLIENT_ID            # OAuth client ID from accounts.elixpo.com
wrangler secret put SSO_CLIENT_SECRET        # OAuth client secret from accounts.elixpo.com

SSO Authentication (Elixpo Accounts)

• Provider: accounts.elixpo.com — OAuth 2.0 Authorization Code flow
• Login: /api/auth/login redirects to SSO → callback exchanges code for tokens → sets httpOnly cookies
• Cookies: elixpo_access_token (15min), elixpo_refresh_token (7d), both httpOnly/Secure/SameSite=Lax
• User sync: On login, user profile is upserted from SSO into local D1 User table
• Rate limiting: Authenticated users skip guest rate limit; guests get 15 requests/IP/24h
• User profile: Local D1 stores bio, location, website, company, jobTitle, preferences (theme, language, searchRegion, safeSearch, deepSearchDefault), tier, usage stats
• Register an OAuth app at accounts.elixpo.com/dashboard/oauth-apps with redirect URI https://search.elixpo.com/api/auth/callback

Environment Variables

Backend (.env.local)

Required: POLLINATIONS_API_KEY, RENDER_SERVICE_TOKEN (shared secret between lixsearch-app and render-service; if unset the render service refuses all requests and reports unhealthy); optional: HF_TOKEN Optional overrides: REDIS_HOST, REDIS_PORT, QDRANT_MODE, QDRANT_URL, QDRANT_PATH, QDRANT_COLLECTION, IPC_HOST, IPC_PORT, WORKERS (Hypercorn workers per container, default 10), WORKER_PORT (default 9002), LOG_LEVEL

Frontend (wrangler.toml vars + secrets)

Vars: BACKEND_URL, GUEST_REQUEST_LIMIT, SSO_REDIRECT_URI Secrets (via wrangler secret put): INTERNAL_API_KEY, XID, API_KEY, SSO_CLIENT_ID, SSO_CLIENT_SECRET

Commit Rule

• All commits use a single author: Circuit-Overtime <ayushbhatt633@gmail.com>. Do not add Co-Authored-By or any other attribution trailers.

Render service isolation

lixsearch/renderService/ (APP_MODE=render) renders hostile pages for the scrape API. Layers:

1. Network: attached only to render-network; no route to redis/qdrant/ipc/nginx. No env_file, cap_drop: ALL, no-new-privileges.
2. Egress proxy (egress_proxy.py): Chromium runs with --proxy-server (loopback not bypassed, non-proxied UDP and QUIC disabled). The proxy resolves each target itself, requires every address to be public, and connects to that validated IP (no DNS rebinding). Ports limited by RENDER_PROXY_ALLOWED_PORTS.
3. Host firewall: sudo ./deploy.sh firewall (idempotent, --dry-run to preview) adds OREO-RENDER-FWD (DOCKER-USER) and OREO-RENDER-IN (INPUT) chains dropping render traffic to private/link-local/CGNAT/multicast ranges, the host, and non-DNS UDP. Not persistent across reboots. Config lives under RENDER_* in pipeline/config.py.