/semantic-search — Hybrid Vector + Lexical Retrieval

Semantic search is a workflow, not a library. The right embedding model depends on corpus size, latency budget, and quality bar. Hybrid ranking (dense + sparse) beats either alone. Every result carries a score and a reason.

When to invoke

• Athanasios needs to find prior art across .squad/decisions/ or past reports
• Superman asks "have we already decided this?"
• Coach is deduplicating near-identical patterns across game tapes
• Kevin is writing a section and needs related docs surfaced

Steps

1. Define corpus and query. Enumerate the document set (paths, IDs, count, total tokens). Write the exact query you will run against it. Small corpus (<10k docs) — in-memory is fine. Large — plan a vector store (Azure AI Search, pgvector, Qdrant). Verify current versions at author time.
2. Choose the embedding model. Verify latest at author time — as of this skill, OpenAI text-embedding-3-large (3072-dim, English + multilingual) or text-embedding-3-small (1536-dim, cheaper) are the defaults. On Azure OpenAI, deploy text-embedding-3-large in a region that supports it. For fully Microsoft-native, evaluate MAI embeddings when GA. Do not pull a model ID from memory — check the vendor docs.
3. Generate embeddings. Batch the corpus (100–500 docs per API call). Store the vectors keyed by doc ID. Cache — embeddings are re-generatable but not free. Log the model ID and dimension alongside the vectors so the index knows what it holds.
4. Hybrid rank. Run BM25 (sparse, lexical) over the corpus in parallel with vector cosine similarity (dense, semantic). Fuse scores with Reciprocal Rank Fusion (RRF, k=60 is a sensible default) or a weighted blend. Take top N (typically 5–20).
5. Return with scores and rationale. For each result: dense score, sparse score, fused rank, and a one-line rationale ("matches on query term X + close in embedding space to phrase Y"). Include the doc ID / path so the caller can open it.
6. Log cost. Note the number of embeddings generated and the per-call cost. Push to G's pricing.json if this runs regularly.

Deliverable format

• Ranked list of results — top N, each with {doc_id, dense_score, sparse_score, fused_rank, rationale, path}
• Embedding model ID and dimension noted
• Corpus size and total embedding cost for the run

Non-negotiables

• Never mix embeddings from different models in the same index — score comparisons are meaningless
• Always run hybrid (dense + sparse) unless the query is a pure semantic paraphrase — lexical catches exact-term hits dense misses
• Pin the embedding model version in the index metadata — re-embed on model upgrades
• Cost-log every batch — this is a per-token API surface

References

• OpenAI — Embeddings guide
• Azure OpenAI — embeddings models
• Azure AI Search — vector search
• Reciprocal Rank Fusion paper (Cormack et al., 2009)
• Superteam ADR — ontology/semantic skills