agentleFS
nainishshafinainishshafi/developer-productivity-skills/.github/skills/rag-pipeline/SKILL.md

rag-pipeline

Skill1 starsChanged todayMITView on GitHub

End-to-end architectural and engineering guidance for designing, building, and evaluating production Retrieval-Augmented Generation (RAG) pipelines. Covers document ingestion & layout preservation, structure-aware & contextual chunking, Matryoshka embeddings, vector databases (PostgreSQL pgvector HNSW, Qdrant, Milvus, SQLite FTS5), hybrid search (BM25 + dense vectors + Reciprocal Rank Fusion), cross-encoder reranking, query transformations (HyDE, multi-query, step-back), grounded generation with inline citations, Agentic & Corrective RAG, RAG Triad & offline IR evaluation (Recall@K, MRR, NDCG@K), 5-layer RAG testing strategies, and document-level ACL security. Use when asked to "rag-pipeline", "rag-pipeline-creator", "build rag pipeline", "create rag pipeline", "design rag architecture", "chunking strategy", "hybrid search rag", "vector database indexing", "reranking pipeline", "agentic rag", "evaluate rag pipeline", "rag testing strategy", "test rag pipeline", or "rag grounding".

5 other projects use a version of this.

Adopted by

Most recently taken up by

The file

We split this skill into the rules it gives an agent

10 rules. About 0 are common, 5 are distinctive, and 5 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈5

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 5

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Structure-Aware & Contextual Chunking: Never slice documents at blind fixed character offsets when structural boundaries (Markdown …

    line 16
  2. Distinctive:

    Measure Retrieval Before Tuning Prompts: Never blame the LLM prompt for retrieval failures.

    line 19
  3. Distinctive:

    Before generating RAG infrastructure or application code, audit your design against these anti-bloat heuristics:

    line 60
  4. Distinctive:

    No Premature GraphRAG: Do not build an expensive LLM entity-extraction knowledge graph pipeline when Hybrid Search + Cross-Encoder …

    line 62
  5. Very specific:

    No Unnecessary Vector Cluster Sprawl: If the application already runs PostgreSQL and expects $< 5\text{M}$ chunks, use pgvector + tsvector …

    line 63
  6. Very specific:

    Avoid Deep Framework Wrapper Bloat: Do not wrap a straightforward 50-line embed-query-rerank-stream flow in 6 layers of BaseRetriever / …

    line 64
  7. Very specific:

    Direct Tool Audit: Run detect-overengineering on generated pipeline modules to verify low accidental complexity.

    line 66
  8. Very specific:

    Test chunking boundaries, simulate BM25 + dense vector RRF fusion, and compute offline IR metrics (Recall@K, Precision@K, HitRate@K, MRR, …

    line 72
  9. Distinctive:

    Run the sample 5-layer test harness using the repository virtual environment:

    line 103
  10. Very specific:

    [LLM-as-Judge](file:///.github/skills/llm-as-judge/SKILL.md): Use Pattern 3 (Claim-Level Hallucination & RAG Grounding Audit) to score …

    line 132

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

RAG Pipeline Skill

You are a Principal AI & Search Systems Architect specializing in production Retrieval-Augmented Generation (RAG) pipelines—spanning document parsing, chunking, embedding models, vector/lexical indexing, hybrid retrieval, cross-encoder reranking, grounded generation, and information retrieval (IR) evaluation.


Guiding Principles & Architectural Heuristics

  1. Hybrid Search by Default (BM25 + Dense + RRF): Dense embeddings capture semantic paraphrases but miss exact identifiers (SKUs, error codes, function names, acronyms). Combine sparse/lexical retrieval (BM25 / tsvector) with dense vector search via Reciprocal Rank Fusion (RRF).
  2. Structure-Aware & Contextual Chunking: Never slice documents at blind fixed character offsets when structural boundaries (Markdown headings, HTML sections, code AST nodes, table rows) exist. Prepend succinct document/section context before embedding and lexical indexing.
  3. Two-Stage Retrieval (Recall $\to$ Rerank): Retrieve a broad candidate pool ($N = 30\text{--}100$) with fast bi-encoders + BM25, then rescore (query, chunk) pairs jointly using a Cross-Encoder or late-interaction reranker to pass only the top $K = 4\text{--}10$ high-precision chunks to the LLM.
  4. Strict Pre-Retrieval Authorization: Enforce tenant isolation and document-level ACLs (WHERE tenant_id = ... AND acl_groups && ...) inside the vector/hybrid query filter—never retrieve unauthorized chunks and attempt to filter them post-LLM.
  5. Measure Retrieval Before Tuning Prompts: Never blame the LLM prompt for retrieval failures. Quantify offline retrieval quality (Recall@K, MRR, NDCG@K) first, then audit claim-level groundedness using llm-as-judge.

End-to-End 5-Stage RAG Architecture

┌───────────────────────────┐     ┌──────────────────────────────────┐     ┌──────────────────────────────────┐
│ 1. Ingestion & Parsing    │ ──► │ 2. Chunking & Context Enrichment │ ──► │ 3. Embedding & Hybrid Indexing   │
│  • Layout/table preserve  │     │  • Markdown/AST/Parent-Child     │     │  • Dense vector (MRL / halfvec)  │
│  • PII/secret redaction   │     │  • Contextual prefix prepend     │     │  • Sparse/BM25 (GIN tsvector)    │
│  • SHA-256 delta hashing  │     │  • Tenant & ACL metadata attach  │     │  • HNSW index + iterative scan   │
└───────────────────────────┘     └──────────────────────────────────┘     └─────────────────┬────────────────┘
                                                                                             │
┌───────────────────────────┐     ┌──────────────────────────────────┐                       │
│ 5. Grounded Generation    │ ◄── │ 4. Retrieval, RRF & Reranking    │ ◄─────────────────────┘
│  • XML source delimiters  │     │  • Query rewrite / step-back     │
│  • Inline [doc_i] cites   │     │  • Pre-filter tenant & ACLs      │
│  • Abstain below tau      │     │  • RRF(k=60) + Cross-Encoder     │
│  • RAG Triad & IR eval    │     │  • "Lost in the Middle" reorder  │
└───────────────────────────┘     └──────────────────────────────────┘

When to Use Which RAG Pattern & Store (Agent Decision Matrix)

Requirement / WorkloadOptimal SelectionAnti-Pattern / What to AvoidRationale
General enterprise docs, wikis, & support KBsHybrid Search (BM25 + Dense + RRF) + Cross-Encoder RerankerDense-only Top-5 cosine searchHybrid search catches exact error codes/IDs while reranking eliminates noisy false positives before the LLM context window.
Long technical manuals where chunks lose surrounding contextParent-Child (Small-to-Big) + Contextual RetrievalIsolated 500-token slices with no section headersEmbeds crisp 256-token child chunks for high vector precision while returning 1,024-token parent sections for full LLM reasoning.
Existing PostgreSQL stack (< 5M vectors)PostgreSQL + pgvector (halfvec HNSW) + tsvector GINSpinning up a separate distributed vector cluster immediatelySingle ACID database keeps documents, ACLs, relational joins, BM25, and vectors consistent with zero sync drift.
High-scale (> 10M vectors), multi-tenant quantization, sub-20ms SLADedicated Vector DB (Qdrant, Milvus) with Binary/Scalar QuantizationUnindexed IVFFlat or brute-force exact scansDedicated engines provide payload-aware HNSW graphs, memory-mapped segments, and quantized two-phase rescoring.
Local CLI / desktop / zero-infra edge searchSQLite (FTS5 BM25 + sqlite-vec)Running Docker vector clusters for single-user local appsZero-ops single-file persistence with fast lexical + vector fusion.
Multi-hop ambiguity or stale/low-confidence retrievalsCorrective RAG (CRAG) / Adaptive Query RouterBlindly stuffing low-scoring chunks into generationGrades retrieved relevance first, rewrites queries or routes to structured SQL (sql-connector) when needed.
Global corpus-wide synthesis ("Summarize themes across 5k docs")GraphRAG / Hierarchical Map-Reduce SummariesStandard Top-K chunk retrievalTop-K retrieval cannot aggregate global themes across thousands of documents; use GraphRAG only for global/multi-hop synthesis.

Over-Engineering Self-Check (Anti-Bloat Guardrail)

Before generating RAG infrastructure or application code, audit your design against these anti-bloat heuristics:

  1. No Premature GraphRAG: Do not build an expensive LLM entity-extraction knowledge graph pipeline when Hybrid Search + Cross-Encoder Reranking satisfies factual QA.
  2. No Unnecessary Vector Cluster Sprawl: If the application already runs PostgreSQL and expects $< 5\text{M}$ chunks, use pgvector + tsvector instead of introducing a separate vector store and dual-write synchronization pipeline.
  3. Avoid Deep Framework Wrapper Bloat: Do not wrap a straightforward 50-line embed-query-rerank-stream flow in 6 layers of BaseRetriever / Chain abstractions. Write direct, inspectable SDK and SQL calls.
  4. Skip HyDE on Latency-Sensitive Paths: Hypothetical Document Embeddings (HyDE) adds an extra LLM generation round-trip before search; reserve it for ambiguous exploratory queries, not real-time autocomplete or known-entity lookup.
  5. Direct Tool Audit: Run detect-overengineering on generated pipeline modules to verify low accidental complexity.

CLI Evaluation & Simulation Utility (scripts/rag-pipeline-eval.py)

Test chunking boundaries, simulate BM25 + dense vector RRF fusion, and compute offline IR metrics (Recall@K, Precision@K, HitRate@K, MRR, NDCG@K) using the repository virtual environment:

# Simulate structure-aware Markdown chunking with deterministic SHA-256 IDs
.\.venv\Scripts\python.exe .github/skills/rag-pipeline/scripts/rag-pipeline-eval.py chunk `
    --strategy markdown --input .github/skills/rag-pipeline/SKILL.md --chunk-size 600 --overlap 80

# Compare BM25 vs Dense vs Hybrid RRF (k=60) rankings on a query
.\.venv\Scripts\python.exe .github/skills/rag-pipeline/scripts/rag-pipeline-eval.py hybrid-search `
    --query "ERR_CONNECTION_REFUSED pgvector HNSW index recall" --top-k 5

# Compute offline IR metrics (Recall@K, Precision@K, HitRate@K, MRR, NDCG@K)
.\.venv\Scripts\python.exe .github/skills/rag-pipeline/scripts/rag-pipeline-eval.py evaluate --demo --k 5

(On POSIX, substitute .\.venv\Scripts\python.exe with .venv/bin/python).


5-Layer RAG Testing Strategy (scripts/rag-testing-strategy.py)

Validate ingestion, access controls, retrieval regressions, grounding, and prompt injection defenses across five automated layers before shipping pipeline changes:

LayerVerification TargetFailure Mode PreventedCI Quality Gate
1. Ingestion, Chunking & PIIDeterministic SHA-256 chunk IDs, max character/token bounds, Markdown breadcrumbs, pre-indexing secret scrubbingDuplicate index rows, severed section headers, secrets/PII baked into embeddings0 unredacted secrets ([REDACTED_*]), deterministic chunk IDs
2. Tenant & ACL Pre-Filteringtenant_id and acl_groups enforced inside retrieval before BM25/vector scoringCross-tenant leakage or RBAC bypass via exact keyword match0 cross-tenant & 0 unauthorized group leaks
3. Hybrid IR Quality GateGolden query suite scored with Recall@K, Precision@K, MRR, and NDCG@KSilent recall regressions after chunk size or embedding changesRecall@3 >= 0.90, MRR >= 0.80, NDCG@3 >= 0.85
4. Grounding & AbstentionRelevance threshold $\tau$ abstention (insufficient_evidence), valid [doc_id] citations, claim support ratioHallucinated answers on low-relevance hits or phantom citations0 phantom citations, abstention enforced below $\tau$
5. Prompt Injection DefenseInstruction-override/exfiltration regex detection and XML <sources> escapingRetrieved tickets/docs hijacking system instructionsAdversarial rules flagged & payload HTML-escaped

Run the sample 5-layer test harness using the repository virtual environment:

# Run all 5 testing layers and print a pass/fail summary table
.\.venv\Scripts\python.exe .github/skills/rag-pipeline/scripts/rag-testing-strategy.py --mode report

# Emit machine-readable JSON for CI quality gates
.\.venv\Scripts\python.exe .github/skills/rag-pipeline/scripts/rag-testing-strategy.py --mode report --format json

# Run the standard unittest.TestCase suite (RagPipelineTestingStrategyTests)
.\.venv\Scripts\python.exe .github/skills/rag-pipeline/scripts/rag-testing-strategy.py --mode unittest

Knowledge Modules

Load only the modules relevant to the task:

  • Document Ingestion, Parsing & Chunking Strategies — Layout/table preservation, recursive/Markdown/AST/Parent-Child/Contextual chunking, and SHA-256 idempotent delta ingestion.
  • Embeddings, Quantization & Vector Database Indexing — MRL dimension truncation, asymmetric prefixes, HNSW/IVFFlat tuning, quantization, and production PostgreSQL pgvector schemas.
  • Hybrid Search, Query Transformation & Reranking — BM25 + Dense Reciprocal Rank Fusion (RRF), PostgreSQL CTE queries, multi-query/step-back/HyDE, Cross-Encoder reranking, and context reordering.
  • Grounded Generation, Streaming APIs & Agentic RAG — Strict citation prompts, Pydantic v2 output contracts, abstention thresholds, Corrective/Self-RAG routers, FastAPI SSE streaming, and caching.
  • RAG Evaluation, IR Metrics & Security Boundaries — RAG Triad, Recall@K/MRR/NDCG@K formulas, document-level ACL pre-filtering, indirect prompt injection defense, PII scrubbing, and the 5-Layer Automated RAG Testing Strategy.

Integration with Repository Skills

  • LLM-as-Judge: Use Pattern 3 (Claim-Level Hallucination & RAG Grounding Audit) to score faithfulness (SUPPORTED / CONTRADICTED / UNSUPPORTED) against retrieved chunks.
  • SQL Executor & SQL Connector: Inspect pgvector HNSW/GIN execution plans and safely query structured relational stores for hybrid SQL+RAG routers.
  • Python API Writer: Build non-blocking FastAPI StreamingResponse SSE endpoints with lifespan connection pool management.
  • Caching & Queues Reviewer: Design Redis embedding caches, async ingestion worker queues (Arq/Celery), and rate limits.
  • Researcher PoC Tester: Benchmark competing chunk sizes, ef_search settings, or reranker latency percentiles (P50/P95/P99).
  • Save Memory & Load Memory: Persist embedding model choices, index dimensions, and retrieval evaluation baselines in .project-memory/memory.db (ledger & journal).

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.