ai-rag-patterns

Overview

Production RAG in 2026 is no longer "embed, retrieve top-5, generate." Three measurable shifts changed the playbook: hybrid retrieval beating either BM25 or dense alone, reranking moving from optional to default, and Anthropic's Contextual Retrieval (September 2024) collapsing retrieval-failure rates dramatically. This skill encodes the patterns that actually move the metrics, with cited numbers and the failure modes they address.

Use this skill when designing a new RAG pipeline, diagnosing why an existing one underperforms, or evaluating retrieval quality before generation. Skip it for a small, stable corpus (under ~200K tokens) where putting everything in a cached prompt is the cheaper answer — see §6.

Mental model

A RAG pipeline is a chain of probabilistic filters: chunking decides what units the system can return, indexing decides what's searchable, retrieval ranks candidates, reranking re-orders them, and generation consumes the result. The output is bounded by the weakest link. The seven failure modes (Barnett et al., arXiv:2401.05856) map cleanly: missing content, top-ranked-missed, not-in-context, not-extracted, wrong format, incorrect specificity, incomplete. Diagnose at the layer that owns the failure; tuning generation when retrieval is the bottleneck is wasted effort.

Two compounding levers move 2026 production RAG most: contextual chunking (adding LLM-generated chunk context before indexing) and hybrid retrieval with reranking. The numbers in §1–§3 below are from Anthropic's Contextual Retrieval write-up and the Elastic / TOIS hybrid-retrieval studies.

Instructions

1. Chunking strategies

Anthropic Contextual Retrieval numbers (from a baseline top-20 retrieval-failure rate of 5.7%):

• Contextual Embeddings alone: −35% failure-rate reduction (5.7 → 3.7%)
• Contextual Embeddings + Contextual BM25: −49% (5.7 → 2.9%)
• Contextual Embeddings + Contextual BM25 + Reranking: −67% (5.7 → 1.9%)

The headline 67% requires reranking on top — it is not the chunking technique alone.

2. Retrieval — BM25, dense, hybrid

Hybrid numbers — Elastic Search Labs reports reciprocal rank fusion at +18% NDCG@10 over BM25 alone and +1.4% over ELSER alone on the BEIR benchmark (Elastic — hybrid retrieval). ACM TOIS (Bruch et al., 2023, doi:10.1145/3596512) shows convex combination at α≈0.5 reaches Recall@5 = 0.726 vs RRF k=60 at 0.695 — tuned weighting beats RRF given ~40 labeled query-result pairs to fit α.

Reciprocal Rank Fusion — Cormack, Clarke, Büttcher, SIGIR 2009 (canonical paper). The formula combines ranks across retrievers: RRF_score(d) = Σ 1 / (k + rank_i(d)), default k=60.

ColBERTv2 — Santhanam et al., NAACL 2022. jina-colbert-v2 reports +6.5% nDCG@10 over ColBERTv2; jina-reranker-v3 reports 61.85 nDCG@10 on BEIR, +4.79% over v2 (arXiv:2509.25085).

3. Query rewriting / expansion

Query rewriting is most impactful when user queries are short or ambiguous (chat agents, voice agents). For well-formed search queries, the lift is marginal.

4. Reranking

Reranking is the second filter: take 50–100 candidates from first-stage retrieval, score with a more expensive model, return top 3–10.

Anthropic's Contextual Retrieval write-up reports reranking adds ~18% relative failure-rate reduction on top of hybrid Contextual Retrieval. No single canonical "typical lift" exists — it depends on first-stage noise level. Vendor blogs cite "20–35% RAG accuracy lift" and "NDCG@5 +18 points" — these are vendor-reported, not peer-reviewed, so treat as ballpark.

5. Evaluation

Three frameworks dominate; use one as primary, cross-check with another.

RAGAS scores — original paper reports human-agreement 95% / 78% / 70% on faithfulness / answer relevance / context relevance. Production threshold: >0.8 on faithfulness and context precision is strong; depends on domain.

Recommended flow (practitioner consensus, not a single citation):

1. Offline golden-set: tune retriever with RAGAS context precision/recall.
2. Tune generator: faithfulness + answer relevancy.
3. Online tracing: Phoenix or LangSmith with LLM-as-judge sampling on production traffic.

Cross-check with the eval-design skill in the same domain for statistical-significance methodology and judge selection.

6. Long context vs RAG (the 2025–2026 question)

2026 frontier models take very long inputs (the Claude 5 family at 1M tokens; Gemini Pro and OpenAI price prompts above 200K and 272K tokens at a higher tier). The question: does long-context generation replace RAG?

• Advertised context is not effective context. RULER (Hsieh et al., arXiv:2404.06654) shows degradation well before the stated window; Chroma's "Context Rot" report (Jul 2025) shows reliability dropping with input length across 18 models even on simple tasks, worse with topically similar distractors.
• Anthropic's guidance: a knowledge base under ~200K tokens (~500 pages) can go straight into the prompt, with prompt caching making it fast and cheap (Contextual Retrieval).
• Databricks' long-context RAG study (blog, 2024 models) found more retrieved context helps only up to a model-specific point.

Practitioner consensus (no single citation): retrieve a wide shortlist, then let a long-context model reason over it. Pure long-context loses on cost, citation auditability, and freshness for large or changing corpora; pure short-RAG loses on multi-step reasoning across distant passages. Decide by corpus size, refresh rate, citation requirements, and budget.

7. Multi-modal RAG

ColPali beats text-extraction-then-embed pipelines on document-image benchmarks. Use it when the corpus is visually rich (scans, slides, layouts that don't OCR cleanly).

8. Agentic RAG

Singh et al., "Agentic Retrieval-Augmented Generation: A Survey" (arXiv:2501.09136, Jan 2025). Patterns: reflection (re-query if confidence low), planning (decompose then retrieve per sub-question), tool use (route between corpora and tools), multi-agent (specialist sub-agents per corpus).

Use agentic RAG when queries are heterogeneous (some need code search, some need docs, some need web), or when single-pass retrieval has known failure modes the agent can detect and recover from. Cost: 2–4× a single-pass RAG; justify with evidence the simpler design failed.

Agentic search is the 2026 variant for code and structured workspaces: give the agent grep/glob/read tools and let it retrieve just in time instead of pre-indexing everything (Anthropic, Effective context engineering for AI agents, Sep 2025). It trades index maintenance for more tool calls; measure both on your eval set.

9. The seven failure modes (Barnett et al.)

arXiv:2401.05856, Jan 2024 — canonical failure taxonomy:

1. Missing content — answer isn't in the corpus
2. Top-ranked missed — relevant chunk exists but wasn't retrieved
3. Not in context — retrieved but didn't make the generator's prompt window
4. Not extracted — in the prompt but generator missed it (often context pollution)
5. Wrong format — extracted correctly but presented incorrectly
6. Incorrect specificity — too general or too specific for the question
7. Incomplete — partial answer

Diagnose at the failing layer; map fix to the right intervention (corpus curation, retrieval tuning, reranking, chunk-size, generator prompting).

Checklist

• Chunking strategy chosen with rationale tied to corpus structure
• Retrieval is hybrid (BM25 + dense + RRF or convex combination)
• Reranking on top-50 to top-100 first-stage results
• Query rewriting where queries are short / ambiguous
• Eval suite using RAGAS or TruLens RAG Triad with a versioned golden set
• Production tracing via Phoenix / LangSmith / Langfuse
• Drift monitoring on retrieval (embedding centroid distance over time)
• Long-context vs RAG decision documented with rationale (corpus size, refresh rate, citation needs, measured context rot)
• Multi-modal corpus uses ColPali-family when documents are visually rich
• Agentic patterns only when single-pass has documented failure modes

Composition / References

Within-domain pairings:

• Pairs with the sibling AI skill on agent design (RAG often lives inside an agent loop with retrieval as a tool).
• Pairs with the sibling AI skill on evals (every RAG decision is gated by RAGAS / RAG Triad measurement).
• Pairs with the sibling AI skill on cost optimization (embedding + rerank + vector DB costs are the RAG-specific FinOps lever).

Primary sources:

• Anthropic — Contextual Retrieval
• Late Chunking, arXiv:2409.04701
• RAGAS, arXiv:2309.15217
• TruLens RAG Triad
• Seven Failure Points of RAG, arXiv:2401.05856
• Cormack et al. RRF, SIGIR 2009
• HyDE, arXiv:2212.10496
• Elastic Search Labs — hybrid retrieval
• Hybrid Fusion ACM TOIS
• ColPali, arXiv:2407.01449
• Agentic RAG Survey, arXiv:2501.09136
• RULER, arXiv:2404.06654
• Databricks long-context RAG performance
• Cohere Rerank docs
• DeepEval LLM evals