CLAUDE.md

IMPORTANT:

1. You must write automated tests for all code (pytest). Aim for both unit tests and integration tests against a real Postgres test database.
2. You must pass ALL tests before committing.
3. Maintain a minimum test coverage threshold of 85%. Per-module targets: pure logic (chunker, search, format) 95%, ingest pipeline 90%, CLI commands 85%.
4. NEVER commit or push without explicit user permission. No exceptions — even in bypass permissions mode.
5. USE Team mode (native agent teams — dispatch Agent teammates; the team auto-forms, no TeamCreate/TeamDelete needed) for any multi-task work to keep the main context window clear. Always use team-driven execution (not inline) when executing implementation plans. See "Team Mode Override" section below.
6. After edits, run the full test suite (pytest) and lint (ruff check). Fix bugs and update tests before claiming work complete.
7. NEVER jump straight to code. Produce a written plan FIRST for any multi-file task. Get explicit approval before writing code.
8. When referencing existing modules/functions, READ the actual source file first. Never guess field names, import paths, or function signatures.
9. All Python code MUST pass ruff check with zero warnings AND mypy src/ with zero errors. See "Linting" below.
10. Pre-commit check: Run ruff check && mypy src/ && pytest before committing.
11. Memory updates: After completing a task that adds/changes any of the following, update the corresponding memory file in the auto-memory directory: CLI commands → cli.md, database schema → schema.md, Python types/dataclasses → types.md, ingest extractors → extractors.md, search algorithm changes → search.md. 11a. Memory audits: When asked to "update memory", audit ALL memory files against the current codebase using parallel Explore agents (one per file). Update any drifted content and the MEMORY.md index. 11b. Plans, specs & docs: ALL plans, specs, design docs, and any .md files created as part of work MUST be stored under docs/ — specs in docs/specs/YYYY-MM-DD-<topic>-design.md, plans in docs/plans/YYYY-MM-DD-<topic>.md. Never leave planning/spec documents in the repo root.
12. MANDATORY regression tests for bug fixes: Every bug fix MUST include a test that reproduces the original bug and verifies the fix. No bug fix is complete without a regression test.
13. NO monkey-patching in tests. NO EXCEPTIONS.
    • BANNED: Reopening production modules/classes to inject constants, methods, or attributes.
    • BANNED: Direct attribute assignment on imported modules without restoration.
    • If a test needs monkey-patching to pass, the production code has a bug. Fix the production code, not the test.
    • monkeypatch (pytest), unittest.mock.patch, and mocker.patch (pytest-mock) are NOT monkey-patching — they are standard test doubles with automatic cleanup. These are fine.
14. Post-phase code review + completion audit LOOP (MANDATORY): After completing each phase/wave of a plan, run a review + audit loop that repeats until BOTH pass with zero issues:
    • Code review: Dispatch superpowers:code-reviewer on ALL changes in the phase against the plan spec.
    • Completion audit: Dispatch a separate completion auditor checking: (a) every task implemented — nothing skipped, (b) every fix matches spec exactly, (c) every required test exists and passes, (d) no regressions, (e) no dead code — no orphaned modules, unused functions, or untested branches.
    • THE LOOP: If EITHER review finds issues → fix ALL issues → re-run BOTH → repeat. Exit condition: code reviewer says "APPROVED" AND auditor says "AUDIT PASSED". No phase is complete until the loop exits clean.
15. No PII in checked-in code. NO EXCEPTIONS. The repo has been explicitly PII-scrubbed across tests, fixtures, docs, and comments — never reintroduce real personal data. Use synthetic / redacted values for names, email addresses, phone numbers, postal addresses, meeting attendees, calendar specifics, company names, customer references, and internal project codenames. When fixing or extending tests, copy the existing synthetic pattern; never paste production data (e.g. real meeting transcripts, real email bodies, real chat threads, employer / coworker names) even temporarily. If a real value seems required for a test to be meaningful, the test is wrong — parameterize the fixture or refactor the production code so it's testable with synthetic data. This rule covers commit messages and PR descriptions too.

Team Mode Override (MANDATORY — overrides superpowers skill routing)

ALL agent dispatching MUST use Team mode (native agent teams — dispatch Agent teammates; the team auto-forms and is torn down automatically, so there is no TeamCreate/TeamDelete and no team_name to pass) instead of standalone Agent subagents. This applies to every superpowers skill that spawns agents:

Non-agent superpowers skills are unchanged: brainstorming, verification-before-completion, using-git-worktrees, finishing-a-development-branch, test-driven-development, systematic-debugging, receiving-code-review, writing-skills.

Why: Team mode provides shared task lists, inter-teammate communication via SendMessage, and better coordination visibility than isolated subagents.

NEVER fall back to standalone Agent subagents for execution or parallel dispatch.

Plan header override: When writing plan documents, use > **For agentic workers:** REQUIRED SUB-SKILL: Use team-driven-development to implement this plan task-by-task.

Architecture Principles

DRY

• Search for existing shared code before creating new modules/utilities.
• Reusable patterns: src/brain/ for shared utilities. Extract anything used in 2+ places.
• Prefer composition over copy-paste.

SOLID Principles (MANDATORY)

• Single Responsibility: Each module has ONE reason to change. Extractors only extract. The chunker only chunks. The embedder only embeds. The search module only searches. The CLI only orchestrates.
• Open/Closed: Open for extension, closed for modification — adding a new ingest source (e.g., notion.py) means adding a new extractor module + dispatcher entry, not editing existing extractors.
• Liskov Substitution: All extractors return ExtractedDoc. All search backends conform to the same result shape. Callers don't care which subclass.
• Interface Segregation: Modules expose narrow, focused public APIs. Don't force a consumer to import a kitchen-sink module to get one helper.
• Dependency Inversion: High-level modules accept dependencies via constructor / function args (e.g., count_tokens callable on the chunker, client on the embedder). Tests pass fakes; production passes real ones.

Coding Conventions

• PEP 8, snake_case everywhere.
• Type hints on every function signature. mypy src/ must pass.
• f-strings for string formatting (no % or .format()).
• pathlib.Path for filesystem paths, never raw strings.
• Dataclasses for value objects (ExtractedDoc, Chunk, SearchResult).
• Custom exceptions inherit from a project-specific base (BrainError if needed); never raise Exception(...).
• Every module has a one-line docstring at the top describing its purpose.
• Use parameterized SQL (%s placeholders + tuple). NEVER concatenate user input into SQL strings.
• Every external HTTP/DB client MUST have explicit timeouts. The Voyage client gets an explicit timeout; it does not retry (voyageai.Client's max_retries defaults to 0 and embeddings.py leaves it there — corrected 2026-10-16, this line used to say retries were handled). Postgres connections set connect_timeout.
• No bare except: — always catch specific exceptions (psycopg.OperationalError, voyageai.error.RateLimitError, etc.).

File-size ceiling — 800 lines

The global rules set an 800-line ceiling per file (200–400 typical). It binds new modules and new growth. It is not a mandate to split the files that are already over it.

• A new module lands under 800. No exceptions.
• An existing file under 800 stays under it. Extract rather than grow past it (brain/backup/db_names.py, brain/format_search.py and brain/snippet_context.py all exist for that reason).
• A file already over may grow only for a written reason, recorded in both places: (1) an inline comment at the point of growth, for the reviewer reading the diff; (2) a one-line pointer in the module docstring, for the reader who opens the file. "It was already over" is not a reason. A change that leaves the file the same size or shrinks it owes no growth record.
• Every file over 800 has a row in the table in docs/ceiling-table.md. A file that crosses 800 gets its row in the same change. A file that drops back under loses its row and gets a "deliberately absent" note below the table.
• Rows carry no live line counts. The Lines cell holds the command (wc -l src/brain/<path>); the Why cell holds a trail of SHA-bound counts and the recorded reason. When you touch a file in the table, first promote its trailing descriptive hop to a real SHA and count, then add a descriptive hop for your own change. Read the trail rules in docs/ceiling-table.md before editing a row.
• scripts/check_ceiling_table.py parses that table and runs in CI's lint job and in bin/brain-ci. It fails when the row set and the tree disagree. It does not check what a row says; that is still a review.

Re-derive the over-800 set with find src/brain -name '*.py' | xargs wc -l | sort -rn. Never count it from memory.

The F6 draft/sensitivity residue audit

Every documents read that filters draft is a read of the browseable corpus, and almost every one is a place a confidential row could travel. Judge each statement, not each file: it must either carry a sensitivity predicate or carry a written reason it does not. An ungated statement with no recorded reason is the defect. Re-derive the set, never inherit it: grep -rn "draft = FALSE\|draft=FALSE\|draft IS FALSE\|NOT d.draft" src/brain/. The full audit and the known, judged residue are in docs/audits/f6-residue-audit.md.

Linting — Ruff + mypy

Run after every change: ruff check (lint) or ruff check --fix (auto-fix), then mypy src/. Config in pyproject.toml under [tool.ruff] and [tool.mypy].

Key rules: Line length 100, target Python 3.11, from __future__ import annotations not needed (3.11+ has native PEP 604 union syntax). Sort imports with ruff check --select I --fix.

Eval gate (CI)

• The eval pytest marker is off in the default pytest run (addopts = "... -m 'not eval'"). The harness needs a live Postgres and Ollama.
• .github/workflows/eval.yml checks harness health only. The retrieval-quality regression gate is a local pre-merge check. On 2026-10-12 the user decided there will be no hermetic CI gate. The gate step self-arms only when both tests/eval/baselines/ci.json and the gitignored tests/eval/golden_corpus.yaml exist.
• brain eval --baseline ci --diff --fail-below exits 3 when any mean metric (nDCG@5, MRR, Recall@20) regresses by more than 1e-4. It exits 1 on an error or when a query the baseline also has is degraded (a transiently failed embed), and 2 on bad flags. A spurious exit 3 almost always means the document set moved, not that search got worse: check brain status first.
• ci.json is keyed to one machine's document set. Whoever makes a change that invalidates it re-records it (brain eval --record-baseline ci) in the same PR and names the trigger in the commit message. Triggers: an ingest or deletion that changes the document set, a re-embed or embedder swap, a change to a config_signature key, a ranking-code change, an edit to golden_corpus.yaml, or recency-window drift (since_days queries only).
• A new committed baseline is allowlisted by name in tests/eval/baselines/.gitignore. Never blanket-allow.

Full detail (workflow steps, the three ways ci.json bites, recency pinning, the re-judging history, the degraded-run rules) is in docs/eval-gate.md and tests/eval/baselines/README.md.

Migration Safety

• Migrations are raw SQL files in src/brain/migrations/ (shipped inside the brain package so wheel/pip installs bundle them), applied in name order by brain init via db.migrations_dir().
• Never reference Python code in migrations — they are pure SQL, frozen in time.
• Every migration must be idempotent or applied to a fresh schema. During development, docker compose down && rm -rf data/postgres && docker compose up -d && brain init resets cleanly. (docker compose down -v alone won't wipe the data — Postgres is mounted from a host bind-mount at ./data/postgres, not a Docker-managed volume.)
• Schema changes = new numbered migration file. Never edit 001_init.sql once shipped.

Security Standards

• UUID primary keys everywhere.
• Parameterized SQL queries only.
• Never hardcode secrets — VOYAGE_API_KEY and DATABASE_URL come from .env (gitignored). Never commit .env.
• Never log full document content at INFO level — it can contain sensitive transcripts/emails. Log titles + IDs only.

Workflow

Plan → Approve → Implement → Verify

1. Plan — List affected files, modules, tests, risks.
2. Approve — Present plan to user. Do NOT write code until approved.
3. Implement — File by file. Use teammates for large tasks (per Team Mode Override above).
4. Verify — Run ruff check && mypy src/ && pytest. Then exercise the CLI end-to-end (brain doctor, brain status, a sample brain search).

Teammate Usage

Use teammates for: parallel test creation, codebase exploration, tasks consuming >30% of context window, implementation when plan is approved. See "Team Mode Override" — always Team mode, never standalone subagents.

Project Overview

Local personal knowledge base ("second brain") with hybrid search, designed to be queried by Claude from any conversation. Stores career documents, interview prep, Krisp call transcripts, Slack threads, and selected Gmail in Postgres + pgvector. Searches use Reciprocal Rank Fusion of FTS rank + vector cosine similarity.

Embeddings: Pluggable via the BRAIN_EMBEDDER env var. Default arctic = Snowflake Arctic Embed v2 (1024-dim, local Ollama, free, Apache 2.0). Alternatives: voyage (Voyage AI voyage-3.5 SaaS, paid, 1024-dim) and qwen3 (Qwen3-Embedding-8B, local Ollama, free, China-origin, 4096-dim — no HNSW index because pgvector caps at 2000 dims for vector).

Full design in docs/specs/2026-04-24-second-brain-design.md. Implementation plan in docs/plans/2026-04-24-second-brain.md. Pluggable-embedder retrofit plan in docs/plans/2026-04-28-local-embeddings-qwen3-8b.md.

Tech Stack

• Language: Python 3.11+
• CLI framework: Typer
• Database: PostgreSQL 16 + pgvector (Docker, port 55432)
• Embeddings: Pluggable backends behind a single Embedder Protocol. Default arctic (Snowflake Arctic Embed v2 over local Ollama, 1024-dim). Alternates: voyage (Voyage AI SaaS, 1024-dim) and qwen3 (Qwen3-Embedding-8B over local Ollama, 4096-dim). Selected at setup time via BRAIN_EMBEDDER.
• PDF/DOCX: pypdf, pdfplumber, python-docx
• Markdown: markdown-it-py
• Tokenization: tiktoken (offline cl100k_base, used by every backend for chunker budgeting)
• Output: Rich (colored tables, JSON)
• Graph retrieval (experimental): Apache AGE (openCypher graph in-Postgres) + networkx (Louvain community detection). Entity-centric GraphRAG alongside the vector/FTS search. Needs the custom AGE Postgres image; default-OFF via BRAIN_GRAPH_ENABLED.
• Tests: pytest, real Postgres test DB, fake embedder fixture
• Lint/Type: ruff, mypy

Build & Run Commands

# Setup (default arctic backend — local Ollama, free, no API key)
brew install ollama                      # macOS; on Linux follow ollama.com/install
brew services start ollama
ollama pull snowflake-arctic-embed2

cp .env.example .env                     # BRAIN_EMBEDDER=arctic by default
python3.11 -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"
docker compose up -d                     # Postgres + pgvector on port 55432
brain init                               # applies migrations + aligns chunks.embedding dim
brain reembed                            # backfill any NULL embeddings + finalize column
brain doctor                             # health check (env, Postgres, embedder, gws CLI)

# Daily use
brain ingest <file>                      # ingest a single file
brain ingest-dir <dir>                   # recursive ingest
brain search "..."                       # hybrid search
brain show <id-prefix>                   # full document
brain reembed                            # backfill missing embeddings; idempotent

# Switching backends (destructive — requires data wipe + re-ingest)
docker compose down && rm -rf data/postgres
# edit .env (BRAIN_EMBEDDER=qwen3 / voyage / arctic)
docker compose up -d && brain init && brain ingest-dir <…> && brain reembed

# GraphRAG (experimental — entity graph alongside vector/FTS search)
# Requires the custom Apache AGE Postgres image
#   (second-brain-age:pg16-v1.5.0-rc0-pgv0.8.6), NOT the stock pgvector prod
#   image. `pip install -e ".[dev]"` pulls the networkx dep. Set
#   BRAIN_GRAPH_ENABLED=true in .env to enable the ingest-time graph sync.
brain init                               # also bootstraps AGE + applies graph migrations when the image ships AGE
brain graphrag build --backfill          # backfill the people graph from existing docs
brain graphrag communities build         # detect + summarize communities (needed for --mode global)
brain graphrag search "..."              # graph retrieval (modes: auto|local|themes|global|fuse)
brain graphrag themes --person "Jane Doe"  # "themes in my conversations with X"
brain graphrag search "..." --mode fuse  # RRF of the graph leg + vector/FTS hybrid leg
brain graphrag communities list          # admin view of materialized communities
brain doctor                             # also reports AGE + graph health (soft check)
# Usage skill (Claude Code): skills/brain-graph/SKILL.md (brain-graph) — graph
#   retrieval for themes / patterns / connections across interactions (themes
#   with a person, what connects A and B, recurring themes), alongside plain
#   hybrid search.

# Testing
pytest                                   # full suite
pytest --cov=brain --cov-report=term     # with coverage
pytest tests/test_chunker.py -v          # single file

# Linting
ruff check                               # lint
ruff check --fix                         # auto-fix
mypy src/                                # type check

Architecture

Layout

src/brain/
  cli.py            — Typer app, all commands (init / doctor / status /
                      ingest* / search / show / list / tag / edit / rm / reembed)
  config.py         — env loading; selects BRAIN_EMBEDDER ∈ {arctic, voyage, qwen3}
  db.py             — psycopg connection + migration runner;
                      ensure_embedding_column() reconciles chunks.embedding
                      dim against the active embedder
  embeddings.py     — `_OllamaEmbedderBase` shared transport; concrete
                      `ArcticEmbedder`, `Qwen3Embedder`, `VoyageEmbedder`;
                      `make_embedder(cfg)` factory dispatched on cfg.embedder
  errors.py         — BrainError base + id-prefix exceptions, OllamaEmbedError
  queries.py        — read helpers (resolve_document_prefix, fetch_document,
                      list_documents, summary_counts) + reembed helpers
                      (iter_chunks_missing_embedding, finalize_embedding_index,
                      embedding_column_state)
  search.py         — hybrid FTS + vector via RRF
  format.py         — human + JSON output
  edit_session.py   — JSON-header + body editor flow used by `brain edit`
  mcp_server.py     — stdio MCP server (brain-mcp entry point)
  chat.py           — shared public chat_json() Ollama call (brief / ask / audio)
  ollama_budget.py  — leaf: BRAIN_OLLAMA_NUM_CTX default + the per-call token
                      budgets validated against it at config load; the
                      BRAIN_OLLAMA_EMBED_NUM_CTX default + floor (chunker
                      ceiling the chunker binds); the shared overflow regex
  activity.py       — shared time-windowed activity reader (brief + weekly review)
  resurface.py      — `brain resurface` spaced-repetition scoring core
  brief.py          — `brain brief` daily-digest assembly + next-step suggestions
  review/           — `brain review` package: scans (contradiction/staleness),
                      weekly synthesis, queue queries, emit/render
  timeline.py       — `brain timeline` temporal bucketing over graph entities
  connect.py        — `brain connect` auto-link scoring core
  cli_connect.py    — `brain connect` CLI sub-app (list/refresh/accept/reject/stats)
  ask.py            — `brain ask` agentic plan/reflect/synthesize loop
  audio.py          — `brain audio` two-host script generation + TTS Protocol
  gaps.py           — `brain gaps` search-failure clustering + detector
  eval/             — retrieval + answer eval harness; answer_eval.py backs
                      `brain eval --answer`
  ingest/
    __init__.py     — Embedder Protocol (declares `dim`);
                      ingest_document() / update_document() pipelines
    dispatch.py     — extractor registry: extension → ExtractorSpec (RM-07)
    chunker.py      — paragraph-aware chunking (uses embedder.count_tokens)
    text.py / markdown.py / pdf.py / docx.py — file extractors
    gmail.py        — shells out to gws CLI
    stdin.py        — generic stdin ingester (Krisp, Slack)

src/brain/migrations/ — numbered SQL files (packaged inside the brain package; applied by brain init)
  001_init.sql                    — base schema (chunks.embedding starts as vector(1024))
  002_qwen3_embedding.sql         — drops + re-adds chunks.embedding as vector(4096), nullable
                                    (ensure_embedding_column then resizes for the active backend)

scripts/embedding_smoke.py        — retrieval sanity check for a corpus +
                                    seeded queries (used during backend swaps)

tests/              — real-DB fixture, fake embedder, one test file per module
docs/specs/         — design specs
docs/plans/         — implementation plans

Key Patterns

• Idempotent ingest: documents.content_hash is UNIQUE on documents. Re-ingesting the same file is a no-op unless --force.
• Source dedup: sources(kind, external_id) is UNIQUE. Re-ingesting the same Krisp/Gmail message updates the existing row instead of duplicating.
• Hybrid search: Reciprocal Rank Fusion (k=60) combines FTS rank and vector cosine similarity. Ranks chunks; groups by document; returns top N docs with best matching chunk as snippet.
• Claude-orchestrated ingestion: Krisp and Slack don't have CLIs. Claude calls the MCP, then pipes content into brain ingest-stdin --source krisp/slack ....
• Pluggable embedders behind one Protocol. All three backends conform to brain.ingest.Embedder (dim: int, embed(texts, *, input_type), count_tokens). The factory make_embedder(cfg) is the only place that knows about concrete backends; ingest, search, and reembed depend only on the Protocol.
• Dim-aware finalize. queries.finalize_embedding_index applies NOT NULL on chunks.embedding for every backend, then conditionally creates an HNSW cosine index when embedder.dim ≤ 2000 (arctic, voyage). For Qwen3 (4096) the index is skipped — pgvector 0.8.x caps HNSW/IVFFlat at 2000 for vector; sequential cosine scan is acceptable at personal-corpus scale.
• Backend-aware brain init. init runs migrations, then db.ensure_embedding_column reconciles chunks.embedding's declared dim against embedder.dim — drop + re-add on a fresh DB, error with a destructive-reset hint if chunks already exist at a different dim.
• Switching backends is destructive. Embeddings cannot be re-projected across models; switching requires docker compose down && rm -rf data/postgres && brain init && brain ingest-dir … && brain reembed. Postgres data is a host bind-mount (./data/postgres), not a Docker volume — docker compose down -v alone is not sufficient.

Lessons Learned

These are universal mistakes — never repeat them.

1. Never guess fields/paths — Read the actual source file before referencing any field, function, or import path.
2. Never declare "done" without running it — Run the CLI command end-to-end before claiming a task complete. Tests passing ≠ feature working.
3. Never copy-paste code — Extract shared patterns into reusable functions. If you write something twice, refactor on the third write.
4. Test mocks must match signatures — After changing a function's signature, grep ALL call sites and update them. Run mypy src/ to catch drift.
5. Four Phase Tests — Every test follows setup → exercise → verify → teardown. Clear separation between phases. No mystery guests — make test data explicit, don't rely on invisible factory defaults.
6. No test failures are pre-existing — Every failure is a regression until proven otherwise. Investigate immediately, never dismiss as "pre-existing" or "flaky" without evidence.
7. The Scout Law — leave the code better than you found it — Before completing any task, ALL tests in the suite must pass — not just the ones related to your changes. If you encounter broken tests from other areas, fix them.