/craftsman:rag - Retrieval-Augmented Generation Design

Outcome Contract

• Outcome: a RAG pipeline design with explicit chunking, embedding, and retrieval decisions.
• Done when: chunk strategy, embedding model, and retrieval method are each chosen with a stated reason, and the evaluation method for retrieval quality is defined.
• Evidence: the design decisions listed with their trade-offs, and the evaluation plan.

Design production-ready RAG pipelines following best practices.

RAG Architecture Overview

┌─────────────────────────────────────────────────────────────────┐
│                        RAG PIPELINE                              │
├─────────────────────────────────────────────────────────────────┤
│                                                                  │
│  ┌──────────────┐    ┌──────────────┐    ┌──────────────┐       │
│  │   INGESTION  │───▶│   RETRIEVAL  │───▶│  GENERATION  │       │
│  └──────────────┘    └──────────────┘    └──────────────┘       │
│                                                                  │
│  Documents → Chunks   Query → Vectors    Context → LLM → Answer │
│  → Embeddings →       → Similar Chunks   → Response             │
│  Vector Store                                                    │
│                                                                  │
└─────────────────────────────────────────────────────────────────┘

Three Pipelines

1. Ingestion Pipeline

from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.embeddings import OpenAIEmbeddings
from langchain.vectorstores import PGVector

# Chunking Strategy
text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,        # Characters per chunk
    chunk_overlap=200,      # Overlap for context
    separators=["\n\n", "\n", ". ", " ", ""],
)

# Embedding Model
embeddings = OpenAIEmbeddings(
    model="text-embedding-3-small",  # Cost-effective
    # model="text-embedding-3-large", # Higher quality
)

# Vector Store
vectorstore = PGVector.from_documents(
    documents=chunks,
    embedding=embeddings,
    collection_name="knowledge_base",
    connection_string=DATABASE_URL,
)

2. Retrieval Pipeline

from langchain.retrievers import ContextualCompressionRetriever
from langchain.retrievers.document_compressors import LLMChainExtractor

# Base Retriever
retriever = vectorstore.as_retriever(
    search_type="similarity",  # or "mmr" for diversity
    search_kwargs={"k": 5},    # Top K results
)

# Optional: Contextual Compression
compressor = LLMChainExtractor.from_llm(llm)
compression_retriever = ContextualCompressionRetriever(
    base_compressor=compressor,
    base_retriever=retriever,
)

3. Generation Pipeline

from langchain.chains import RetrievalQA
from langchain.chat_models import ChatOpenAI

llm = ChatOpenAI(
    model="gpt-4-turbo-preview",
    temperature=0,  # Deterministic for factual answers
)

qa_chain = RetrievalQA.from_chain_type(
    llm=llm,
    chain_type="stuff",  # or "map_reduce" for long contexts
    retriever=retriever,
    return_source_documents=True,
)

Chunking Strategies

Vector Database Comparison

Design Checklist

## RAG Design: [Project Name]

### 1. Data Source Analysis
- [ ] Document types: [PDF, Markdown, HTML, etc.]
- [ ] Total volume: [X documents, Y MB]
- [ ] Update frequency: [Real-time, Daily, Weekly]
- [ ] Quality: [Clean, Needs preprocessing]

### 2. Chunking Strategy
- [ ] Strategy: [Recursive, Semantic, etc.]
- [ ] Chunk size: [X characters]
- [ ] Overlap: [Y characters]
- [ ] Metadata: [Title, Source, Date, etc.]

### 3. Embedding Choice
- [ ] Model: [text-embedding-3-small, etc.]
- [ ] Dimensions: [1536, 3072, etc.]
- [ ] Cost estimate: [$X per 1M tokens]

### 4. Vector Store
- [ ] Database: [pgvector, Pinecone, etc.]
- [ ] Index type: [HNSW, IVF, etc.]
- [ ] Hosting: [Self-hosted, Managed]

### 5. Retrieval Strategy
- [ ] Search type: [Similarity, MMR, Hybrid]
- [ ] Top K: [3-10]
- [ ] Reranking: [Yes/No]
- [ ] Filtering: [Metadata filters]

### 6. Generation
- [ ] Model: [GPT-4, Claude, etc.]
- [ ] Temperature: [0 for factual, 0.7 for creative]
- [ ] Context window: [Token limit]
- [ ] Fallback: [When no relevant docs found]

Common Pitfalls

Evaluation Metrics

# Retrieval Quality
- Recall@K: % of relevant docs in top K
- Precision@K: % of top K that are relevant
- MRR: Mean Reciprocal Rank

# Generation Quality
- Faithfulness: Does answer match sources?
- Relevance: Does answer address query?
- Groundedness: Is answer supported by context?

Process

1. Understand the use case (Q&A, search, chatbot)
2. Analyze data sources
3. Choose chunking strategy
4. Select embedding model
5. Design vector store schema
6. Implement retrieval pipeline
7. Configure generation
8. Set up evaluation
9. Deploy and monitor