AI Vector Brain

Build a vector brain implementation. V1 default: Postgres + pgvector. Pick a corpus playbook.

For source code, first show a gain over ripgrep, tree-sitter, or code-graph retrieval (see Corpus Readiness Gate); pgvector fits docs and prose corpora.

Use this skill when the user asks to:

• build a vector brain, RAG brain, LLM brain, repo brain, docs brain, or compliance brain
• turn a repo, docs hub, policy corpus, guide set, or generated context artifacts into a repeatable retrieval layer
• choose a vector-brain backend and produce a concrete implementation path
• create SQL, manifests, ingestion scripts, eval seeds, or an agent retrieval tool contract

Boundary Contract

Quick Reference

The bundled SQL assets default to 1024-dimensional embeddings. If the chosen embedding model uses a different output dimension, update every vector(N) and bit(N) occurrence in assets/sql/001_schema.sql, 003_hybrid_search_function.sql, 005_eval_tables.sql, and 011_quantize_rescore.sql before loading data.

V1 Workflow

1. Inventory the corpus with scripts/inventory_corpus.py.
2. Classify corpus type: repo/codebase, docs hub, or compliance/policy.
3. Prepare normalized documents with scripts/prepare_documents.py.
4. Chunk with stable anchors using scripts/chunk_corpus_files.py for mixed repo/context corpora or scripts/chunk_markdown.py for Markdown-only corpora.
5. Choose retrieval unit: source chunk, parent-child chunk, or typed knowledge packet.
6. Deduplicate canonical facts before embedding when the corpus has repeated versions, copied procedures, or near-duplicate docs.
7. Create schema and indexes from assets/sql/.
8. Embed and load retrieval units with scripts/embed_and_load.py (provider-agnostic; swap the Embedder class for a new backend).
9. Optionally contextualize chunks at index time per contextual-retrieval.md when retrieval-failure rate is the bottleneck.
10. Run hybrid retrieval with scripts/retrieve.py (lexical + vector + RRF; sets the required hnsw.iterative_scan session knob).
11. Optionally rerank the top-N candidates with a cross-encoder per reranking-recipe.md before passing top-K to the generator.
12. Seed evals with scripts/build_eval_seed.py, then hand-label expected evidence.
13. Expose retrieval through the retrieve_context contract.
14. Operate freshness with ingest ledgers, corpus versions, tombstones, query logs (assets/sql/007_query_logs.sql), and metric gates.
15. Promote to production only after the selected backend runs its behavioral checks; for memory facts and RLS use server verification, and report missing prerequisites as not_run. Follow production-runbook.md (backups, RLS, SLOs, embedding migration drill).

Backend Stance

Postgres + pgvector is the V1 default because it is durable, scriptable, easy to inspect, and sufficient for most repo/docs/compliance brains. It is not the identity of the skill.

On-device iOS / Swift is out of scope here. When the retrieval target is an iPhone/iPad app running fully offline (sqlite-vec / NLEmbedding / Core Spotlight semantic index feeding an Apple Foundation Models or sentence-bank composer), this skill's pgvector + server scripts don't apply. Route to ../software-ios-ai-engine/SKILL.md for the on-device retrieval-stitch composer (Option C) and the shared { answer, grounding, followUps[], safetyBoundary } contract. Use this skill only for the upstream knowledge build (chunking, anchors, eval seeds) that ships into the bundle.

Dual-deployment pattern (server pgvector + on-device sqlite-vec mirror with shared content_hash so citations resolve identically across paths) plus full four-skill composition for natural conversational iOS surfaces is documented in composition-with-rag-context-vector.md — covers Path A (Foundation Models) and Path B (vector-DB-only) for three generic domain shapes (consumer reflection, regulated copilot, multi-turn emotional companion).

For non-iOS conversational surfaces (Android with Gemini Nano / AICore, web browser with Chrome window.ai or WebLLM, Telegram/Discord/WhatsApp/Slack bots via LangGraph + Mem0, voice surfaces), see conversational-surfaces-cross-platform.md — same composition skeleton, per-platform composer matrix (with-model and without-model paths), per-platform retrieval backend choice (ObjectBox Android, IndexedDB+WASM sqlite-vec web, server pgvector for bots).

Use backend-selection.md before choosing an alternative. The matrix covers four tiers:

• SQL-native: pgvector, pgvectorscale, ParadeDB
• Dedicated services: Qdrant, Weaviate, Milvus/Zilliz, Vespa, LanceDB, Chroma
• Serverless / object-storage-backed (cost-driven): AWS S3 Vectors, Turbopuffer, Pinecone Serverless, Cloudflare Vectorize, Upstash Vector
• Hyperscaler managed: AWS Bedrock Knowledge Bases, Azure AI Search, Vertex AI Vector Search, OpenAI File Search
• Search-engine k-NN (lift-and-shift): Elasticsearch / OpenSearch, Redis Stack, MongoDB Atlas Vector

For per-backend deep dives and look-alike comparisons (S3 Vectors vs Turbopuffer vs Pinecone Serverless, Bedrock KB vs Vertex vs Azure vs OpenAI File Search), see backend-selection-extended.md.

Corpus Playbooks

V1 ships three playbooks:

• Repo/codebase brain: exact path, symbol, module, ownership, structured profile, schema, and selected source retrieval. When scaffolding this brain, start from examples/repo-brain.manifest.json.
• Docs hub brain: architecture, guides, generated docs, and cross-page navigation. When scaffolding this brain, start from examples/docs-hub-brain.manifest.json.
• Compliance/policy brain: authority, effective-time, citation precision, and refusal-on-no-evidence. When scaffolding this brain, start from examples/policy-brain.manifest.json.

Support KB, note-vault, and generated graph + markdown playbooks are later increments unless the user explicitly asks for them.

Operational Defaults

• Keep source truth separate from chunks and embeddings.
• Store stable evidence IDs, source URIs, content hashes, freshness, ACL scope, and citation anchors.
• Treat chunks as a default transport unit, not always the best knowledge unit. For policies, docs, support, and repeated business knowledge, consider typed claim or question-answer packets with source anchors, version state, and access scope.
• Collapse near-duplicate units into canonical records before embedding when duplicate versions would crowd top-k results.
• Use lexical + vector hybrid retrieval from the first production version.
• Contextualize chunks at index time (not query time) when adopting contextual retrieval; embed the contextualized form, keep the original content for display and citation.
• Rerank in the app layer with a cross-encoder, never inside the database; oversample candidates before reranking and size N by ai-rag ranking-pipeline-guide.md.
• Add model_id to embeddings from day one; never overwrite embeddings in place during live migration.
• Keep the manifest embedding_model dimension aligned with the SQL assets and CLI --dim.
• Tombstone stale or deleted content; do not silently hard-delete normal corpus history.
• Build a corpus-specific eval set before tuning chunk size, model, backend, or index parameters.
• Treat retrieved chunks as untrusted external content until assembled into a grounded context bundle.

Corpus Readiness Gate

Do not embed until the corpus has stable source IDs, tenant or ACL scope, effective timestamps, deterministic extraction, duplicate handling, and a delete or tombstone path. Build a lexical or exact-search baseline and a gold set with evidence IDs first. The vector path may advance only when it improves the target retrieval slices and preserves authorization and deletion behavior. If the corpus contract is unstable, fix ingestion before tuning distance metrics or indexes.

Common Anti-Patterns

• vector database first, source-of-truth model later
• one chunking or retrieval-unit recipe for code, docs, and policies
• treating arbitrary token windows as facts when the corpus needs atomic claims, obligations, decisions, or Q&A records
• letting duplicate versions compete in embedding space instead of canonicalizing current/deprecated records
• embeddings and raw source text in one undifferentiated table
• pure vector search with no lexical leg for code, policy, or proper-noun-heavy docs
• treating HNSW as infinite scale; quantizing without a rescore step; rebuilding PPR/Louvain inside the brain instead of reusing dev-context-code-graph / dev-context-multi-repo runners
• contextualizing chunks at query time instead of index time (loses the precompute + prompt-caching cost win)
• embedding the contextualized form but discarding the original chunk so citations can no longer point at the source text
• reranking inside the database instead of app-layer; running rerank with N == K so there is nothing to rerank
• mixing rerank scores across reranker models in the same eval cohort
• no deletion, tombstone, or supersession path
• semantic cache without corpus-version invalidation
• tuning by vibes instead of labeled retrieval evals
• citing chunks without stable source anchors
• hiding operational SQL in theory skills where it drifts
• no re-indexing line in the FinOps model — re-embedding a corpus is recurring, not one-time, and its cost scales with tokens, not with vector count. Use N × avg_tokens_per_chunk × $/1M-tokens (see references/cost-calculation.md Formula 4) with the embedding model's current per-token rate. Do not budget from a flat $-per-N-vectors figure. Budget it at architecture-design time, not after the first surprise invoice.
• no embedding_model column on the vectors table — providers can silently upgrade models behind a stable API alias. Pin model name and version on every row (model_id, see Operational Defaults), and monitor cosine-similarity distribution between query embeddings and top-k results (rolling avg + stdev), alerting when the mean drifts more than 2σ from baseline. The pin-and-monitor mitigation holds whatever the size of a given provider-side change.
• assuming "same model" means "same neighbourhood" forever — even with a frozen embedding model, corpus drift (new docs, expired docs, evolving terminology) reshapes the semantic landscape of the index over months. Schedule a quarterly drift check against a held-out labeled eval; trigger re-embed when recall@k falls below threshold. When full re-embed is too expensive, consider Drift-Adapter (see references/embedding-drift-mitigation.md) — the source paper reports recovering 95-99% of fresh-index recall at over 100x lower compute than a full re-embedding or dual-index migration.
• picking a vector DB on raw k-NN throughput when the workload is filter-heavy — benchmark candidate backends with realistic filter cardinality and selectivity, not unfiltered top-k; metadata-filter performance varies widely by backend and payload-index design. Measure on your own workload rather than quoting a cross-vendor multiplier.
• ingesting tens of millions of vectors because storage is cheap — query-corpus concentration (a small fraction of the corpus serving most queries) is a common pattern in production retrieval systems, but the skew is workload-dependent; read it from your own query logs. Premature scale inflates RAM, re-embed cost, and tail latency for queries that would have been fine on a much smaller index. Measure query-corpus concentration before sizing.
• treating full-text search as identifier equality — the parser splits tokens and dictionaries normalize them; even simple lowercases. Keep exact identifiers in indexed key columns with explicit case semantics; use a tested simple+unaccent contribution for prose lookup. See references/postgres-fts-tuning.md.
• unaccent() wrapped IMMUTABLE to fit a generated column — index goes silently stale when the unaccent ruleset changes. Use a text-search configuration instead.
• assuming RRF makes lexical ranking irrelevant — RRF consumes rank positions, so normalization that changes the lexical order can change fusion. Evaluate lexical ordering separately; normalization cannot supply missing BM25 IDF.
• assuming "it indexed" means "it ranked" — positions past 16383 are silently clamped (lexeme still matches; ts_rank_cd proximity degrades), and positions beyond 256 per lexeme are discarded; tsvector caps at 1 MB. Guard chunk size.
• per-row language via a generated column — a generated column cannot pick its regconfig from a column; per-row language needs a trigger.
• trigram GIN on full body text — pg_trgm GIN bloats on long text; index only the identifier/symbol column.
• MTEB v1 and v2 scores are not directly comparable; confirm benchmark version when comparing embedding models.

Navigation

References

• references/framework.md
• references/backend-selection.md
• references/backend-selection-extended.md
• references/s3-vectors-backend.md — S3 Vectors cost model, GA limits, direct API vs Bedrock KB, anti-patterns
• references/cost-calculation.md
• references/postgres-pgvector-default.md
• references/corpus-playbooks.md
• references/eval-by-corpus-type.md
• references/agent-tool-contract.md
• references/contextual-retrieval.md
• references/reranking-recipe.md
• references/lexical-vs-vector-vs-hybrid.md — per-query leg decision matrix, smell test, worked examples, patterns/anti-patterns/traps
• references/bm25-when-ts_rank-isnt-enough.md — real BM25 (pg_search/OpenSearch/Vespa) vs ts_rank; per-backend verdict matrix
• references/learned-sparse-splade-leg.md — learned-sparse/SPLADE 4th leg (sparsevec/Qdrant/ELSER); per-backend verdict matrix
• references/embedded-local-brain.md — embedded/local brain (DuckDB-VSS, sqlite-vec); per-backend tier verdicts
• references/quantization-and-rescore.md — binary_quantize + two-pass rescore (default tier); per-backend verdict matrix
• references/embedding-runtime.md — self-hosted embedding runtime (TEI/Infinity/vLLM/Ollama/llama.cpp); per-runtime verdict matrix
• references/postgres-fts-tuning.md — tsvector weighting, exact-token, multilingual, debug, snippet, RUM toolkit
• references/graph-theory-at-scale.md
• references/deferred-extensions.md
• references/production-runbook.md
• references/embedding-drift-mitigation.md — detect model drift vs corpus drift; Drift-Adapter and dual-index alternatives to full re-embedding

SQL Assets

• assets/sql/001_schema.sql
• assets/sql/002_indexes_hnsw.sql
• assets/sql/003_hybrid_search_function.sql
• assets/sql/004_ingest_ledger.sql
• assets/sql/005_eval_tables.sql
• assets/sql/006_rls_multitenant.sql — optional, multi-tenant only
• assets/sql/007_query_logs.sql — observability
• assets/sql/008_fts_hardening.sql — reversible weighted unaccent fts_vector (supersedes 001's v1 column)
• assets/sql/009_bm25_pg_search.sql — reversible pg_search BM25 lexical-leg alternative to 008
• assets/sql/010_sparsevec.sql — reversible sparsevec column + HNSW inner-product index for the learned-sparse/SPLADE leg
• assets/sql/011_quantize_rescore.sql — binary_quantize expression index + two-pass rescore (reversible)
• assets/sql/012_bitemporal_facts.sql — bitemporal state facts (valid_from/valid_to + recorded_at/superseded_at), as-of and known-at queries, out-of-order backfill (reversible)

Scripts

• scripts/inventory_corpus.py
• scripts/prepare_documents.py
• scripts/chunk_corpus_files.py
• scripts/chunk_markdown.py
• scripts/embed_and_load.py
• scripts/retrieve.py
• scripts/build_eval_seed.py
• scripts/check_brain_manifest.py
• scripts/test_sql_asset_contracts.py — regression checks for SQL asset examples and reversible migration contracts
• ../ai-rag/scripts/exact_search_baseline.py — backend-neutral exact-search proof before index tuning
• ../ai-rag/scripts/hybrid_rrf_demo.py — portable hybrid/RRF smoke comparison for prediction files

Sources

• data/sources.json — curated primary docs for freshness validation

Learnings Loop

When prior decisions or pitfalls are relevant, consult learnings.consolidated.md if present; use learnings.md only for needed history or as the available fallback. Otherwise skip both.

After applying it, if you encountered a pattern worth remembering, a mistake worth preventing, or a domain fact that surprised you, append one dated bullet to learnings.md via agents-skills-feedback-loop/scripts/append_learning.py. Do not modify SKILL.md itself.