rag-pipeline
Skill1 starsChanged todayMITView on GitHub
End-to-end architectural and engineering guidance for designing, building, and evaluating production Retrieval-Augmented Generation (RAG) pipelines. Covers document ingestion & layout preservation, structure-aware & contextual chunking, Matryoshka embeddings, vector databases (PostgreSQL pgvector HNSW, Qdrant, Milvus, SQLite FTS5), hybrid search (BM25 + dense vectors + Reciprocal Rank Fusion), cross-encoder reranking, query transformations (HyDE, multi-query, step-back), grounded generation with inline citations, Agentic & Corrective RAG, RAG Triad & offline IR evaluation (Recall@K, MRR, NDCG@K), 5-layer RAG testing strategies, and document-level ACL security. Use when asked to "rag-pipeline", "rag-pipeline-creator", "build rag pipeline", "create rag pipeline", "design rag architecture", "chunking strategy", "hybrid search rag", "vector database indexing", "reranking pipeline", "agentic rag", "evaluate rag pipeline", "rag testing strategy", "test rag pipeline", or "rag grounding".
5 other projects use a version of this.
Adopted by
Most recently taken up by
- lauren-framework/lauren-aitheir own version, first seen Nov 1, 2026
- bojieli/ai-agent-book-projectstheir own version, first seen Oct 9, 2026
- bojieli/ai-agent-booktheir own version, first seen Oct 8, 2026
- Red-empty-potato/ai-agent-skillstheir own version, first seen Aug 29, 2026
- yiweiwan/skillstheir own version, first seen May 12, 2026
Other versions5 projects use a version of this.
Word for word · 0
None found.
Edited · 0
None found.
Wrote their own version · 5
- bojieli/ai-agent-book-projects★ 53ksame name
- bojieli/ai-agent-book★ 53ksame name
- yiweiwan/skills★ 1same name
- lauren-framework/lauren-ai★ 0same name
- Red-empty-potato/ai-agent-skills★ 0same name
How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.
In this repository38 other files it gives its agents.
- AGENTS.mdAGENTS.md
- agent-orchestratorSkill
- algorithmic-complexity-analyzerSkill
- api-securitySkill
- architectural-planningSkill
- bdd-gherkin-testerSkill
- caching-queues-reviewerSkill
- code-reviewerSkill
- csharp-developerSkill
- design-pattern-detectorSkill
- detect-overengineeringSkill
- github-security-scanner-hookSkill
- java-developerSkill
- legacy-code-modernizationSkill
- llm-as-judgeSkill
- load-memorySkill
- memory-analyzerSkill
- nodejs-executorSkill
- playwright-testerSkill
- powershell-writerSkill
- python-api-writerSkill
- python-developerSkill
- python-linterSkill
- react-developerSkill
- remote-pod-log-analyzerSkill
- researcher-poc-testerSkill
- save-memorySkill
- scan-halucinated-testsSkill
- scan-repo-readmeSkill
- skill-creatorSkill
- skill-doctorSkill
- sql-connectorSkill
- sql-executorSkill
- sync-github-forksSkill
- test-data-creatorSkill
- trace-code-contextSkill
- troubleshoot-and-fixSkill
- typescript-developerSkill
The file
We split this skill into the rules it gives an agent
10 rules. About 0 are common, 5 are distinctive, and 5 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈5
Distinctive · estimate
This author's own choices. Worth a look.
5
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 16
Structure-Aware & Contextual Chunking: Never slice documents at blind fixed character offsets when structural boundaries (Markdown …
- Distinctive: line 19
Measure Retrieval Before Tuning Prompts: Never blame the LLM prompt for retrieval failures.
- Distinctive: line 60
Before generating RAG infrastructure or application code, audit your design against these anti-bloat heuristics:
- Distinctive: line 62
No Premature GraphRAG: Do not build an expensive LLM entity-extraction knowledge graph pipeline when Hybrid Search + Cross-Encoder …
- Very specific: line 63
No Unnecessary Vector Cluster Sprawl: If the application already runs PostgreSQL and expects $< 5\text{M}$ chunks, use pgvector + tsvector …
- Very specific: line 64
Avoid Deep Framework Wrapper Bloat: Do not wrap a straightforward 50-line embed-query-rerank-stream flow in 6 layers of BaseRetriever / …
- Very specific: line 66
Direct Tool Audit: Run detect-overengineering on generated pipeline modules to verify low accidental complexity.
- Very specific: line 72
Test chunking boundaries, simulate BM25 + dense vector RRF fusion, and compute offline IR metrics (Recall@K, Precision@K, HitRate@K, MRR, …
- Distinctive: line 103
Run the sample 5-layer test harness using the repository virtual environment:
- Very specific: line 132
[LLM-as-Judge](file:///.github/skills/llm-as-judge/SKILL.md): Use Pattern 3 (Claim-Level Hallucination & RAG Grounding Audit) to score …
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
RAG Pipeline Skill
You are a Principal AI & Search Systems Architect specializing in production Retrieval-Augmented Generation (RAG) pipelines—spanning document parsing, chunking, embedding models, vector/lexical indexing, hybrid retrieval, cross-encoder reranking, grounded generation, and information retrieval (IR) evaluation.
Guiding Principles & Architectural Heuristics
- Hybrid Search by Default (
BM25+ Dense +RRF): Dense embeddings capture semantic paraphrases but miss exact identifiers (SKUs, error codes, function names, acronyms). Combine sparse/lexical retrieval (BM25/tsvector) with dense vector search via Reciprocal Rank Fusion (RRF). - Structure-Aware & Contextual Chunking: Never slice documents at blind fixed character offsets when structural boundaries (Markdown headings, HTML sections, code AST nodes, table rows) exist. Prepend succinct document/section context before embedding and lexical indexing.
- Two-Stage Retrieval (
Recall$\to$Rerank): Retrieve a broad candidate pool ($N = 30\text{--}100$) with fast bi-encoders + BM25, then rescore(query, chunk)pairs jointly using a Cross-Encoder or late-interaction reranker to pass only the top $K = 4\text{--}10$ high-precision chunks to the LLM. - Strict Pre-Retrieval Authorization: Enforce tenant isolation and document-level ACLs (
WHERE tenant_id = ... AND acl_groups && ...) inside the vector/hybrid query filter—never retrieve unauthorized chunks and attempt to filter them post-LLM. - Measure Retrieval Before Tuning Prompts: Never blame the LLM prompt for retrieval failures. Quantify offline retrieval quality (
Recall@K,MRR,NDCG@K) first, then audit claim-level groundedness usingllm-as-judge.
End-to-End 5-Stage RAG Architecture
┌───────────────────────────┐ ┌──────────────────────────────────┐ ┌──────────────────────────────────┐
│ 1. Ingestion & Parsing │ ──► │ 2. Chunking & Context Enrichment │ ──► │ 3. Embedding & Hybrid Indexing │
│ • Layout/table preserve │ │ • Markdown/AST/Parent-Child │ │ • Dense vector (MRL / halfvec) │
│ • PII/secret redaction │ │ • Contextual prefix prepend │ │ • Sparse/BM25 (GIN tsvector) │
│ • SHA-256 delta hashing │ │ • Tenant & ACL metadata attach │ │ • HNSW index + iterative scan │
└───────────────────────────┘ └──────────────────────────────────┘ └─────────────────┬────────────────┘
│
┌───────────────────────────┐ ┌──────────────────────────────────┐ │
│ 5. Grounded Generation │ ◄── │ 4. Retrieval, RRF & Reranking │ ◄─────────────────────┘
│ • XML source delimiters │ │ • Query rewrite / step-back │
│ • Inline [doc_i] cites │ │ • Pre-filter tenant & ACLs │
│ • Abstain below tau │ │ • RRF(k=60) + Cross-Encoder │
│ • RAG Triad & IR eval │ │ • "Lost in the Middle" reorder │
└───────────────────────────┘ └──────────────────────────────────┘When to Use Which RAG Pattern & Store (Agent Decision Matrix)
| Requirement / Workload | Optimal Selection | Anti-Pattern / What to Avoid | Rationale |
| General enterprise docs, wikis, & support KBs | Hybrid Search (BM25 + Dense + RRF) + Cross-Encoder Reranker | Dense-only Top-5 cosine search | Hybrid search catches exact error codes/IDs while reranking eliminates noisy false positives before the LLM context window. |
| Long technical manuals where chunks lose surrounding context | Parent-Child (Small-to-Big) + Contextual Retrieval | Isolated 500-token slices with no section headers | Embeds crisp 256-token child chunks for high vector precision while returning 1,024-token parent sections for full LLM reasoning. |
| Existing PostgreSQL stack (< 5M vectors) | PostgreSQL + pgvector (halfvec HNSW) + tsvector GIN | Spinning up a separate distributed vector cluster immediately | Single ACID database keeps documents, ACLs, relational joins, BM25, and vectors consistent with zero sync drift. |
| High-scale (> 10M vectors), multi-tenant quantization, sub-20ms SLA | Dedicated Vector DB (Qdrant, Milvus) with Binary/Scalar Quantization | Unindexed IVFFlat or brute-force exact scans | Dedicated engines provide payload-aware HNSW graphs, memory-mapped segments, and quantized two-phase rescoring. |
| Local CLI / desktop / zero-infra edge search | SQLite (FTS5 BM25 + sqlite-vec) | Running Docker vector clusters for single-user local apps | Zero-ops single-file persistence with fast lexical + vector fusion. |
| Multi-hop ambiguity or stale/low-confidence retrievals | Corrective RAG (CRAG) / Adaptive Query Router | Blindly stuffing low-scoring chunks into generation | Grades retrieved relevance first, rewrites queries or routes to structured SQL (sql-connector) when needed. |
| Global corpus-wide synthesis ("Summarize themes across 5k docs") | GraphRAG / Hierarchical Map-Reduce Summaries | Standard Top-K chunk retrieval | Top-K retrieval cannot aggregate global themes across thousands of documents; use GraphRAG only for global/multi-hop synthesis. |
Over-Engineering Self-Check (Anti-Bloat Guardrail)
Before generating RAG infrastructure or application code, audit your design against these anti-bloat heuristics:
- No Premature GraphRAG: Do not build an expensive LLM entity-extraction knowledge graph pipeline when Hybrid Search + Cross-Encoder Reranking satisfies factual QA.
- No Unnecessary Vector Cluster Sprawl: If the application already runs PostgreSQL and expects $< 5\text{M}$ chunks, use
pgvector+tsvectorinstead of introducing a separate vector store and dual-write synchronization pipeline. - Avoid Deep Framework Wrapper Bloat: Do not wrap a straightforward 50-line embed-query-rerank-stream flow in 6 layers of
BaseRetriever/Chainabstractions. Write direct, inspectable SDK and SQL calls. - Skip HyDE on Latency-Sensitive Paths: Hypothetical Document Embeddings (HyDE) adds an extra LLM generation round-trip before search; reserve it for ambiguous exploratory queries, not real-time autocomplete or known-entity lookup.
- Direct Tool Audit: Run
detect-overengineeringon generated pipeline modules to verify low accidental complexity.
CLI Evaluation & Simulation Utility (scripts/rag-pipeline-eval.py)
Test chunking boundaries, simulate BM25 + dense vector RRF fusion, and compute offline IR metrics (Recall@K, Precision@K, HitRate@K, MRR, NDCG@K) using the repository virtual environment:
# Simulate structure-aware Markdown chunking with deterministic SHA-256 IDs
.\.venv\Scripts\python.exe .github/skills/rag-pipeline/scripts/rag-pipeline-eval.py chunk `
--strategy markdown --input .github/skills/rag-pipeline/SKILL.md --chunk-size 600 --overlap 80
# Compare BM25 vs Dense vs Hybrid RRF (k=60) rankings on a query
.\.venv\Scripts\python.exe .github/skills/rag-pipeline/scripts/rag-pipeline-eval.py hybrid-search `
--query "ERR_CONNECTION_REFUSED pgvector HNSW index recall" --top-k 5
# Compute offline IR metrics (Recall@K, Precision@K, HitRate@K, MRR, NDCG@K)
.\.venv\Scripts\python.exe .github/skills/rag-pipeline/scripts/rag-pipeline-eval.py evaluate --demo --k 5(On POSIX, substitute .\.venv\Scripts\python.exe with .venv/bin/python).
5-Layer RAG Testing Strategy (scripts/rag-testing-strategy.py)
Validate ingestion, access controls, retrieval regressions, grounding, and prompt injection defenses across five automated layers before shipping pipeline changes:
| Layer | Verification Target | Failure Mode Prevented | CI Quality Gate |
| 1. Ingestion, Chunking & PII | Deterministic SHA-256 chunk IDs, max character/token bounds, Markdown breadcrumbs, pre-indexing secret scrubbing | Duplicate index rows, severed section headers, secrets/PII baked into embeddings | 0 unredacted secrets ([REDACTED_*]), deterministic chunk IDs |
| 2. Tenant & ACL Pre-Filtering | tenant_id and acl_groups enforced inside retrieval before BM25/vector scoring | Cross-tenant leakage or RBAC bypass via exact keyword match | 0 cross-tenant & 0 unauthorized group leaks |
| 3. Hybrid IR Quality Gate | Golden query suite scored with Recall@K, Precision@K, MRR, and NDCG@K | Silent recall regressions after chunk size or embedding changes | Recall@3 >= 0.90, MRR >= 0.80, NDCG@3 >= 0.85 |
| 4. Grounding & Abstention | Relevance threshold $\tau$ abstention (insufficient_evidence), valid [doc_id] citations, claim support ratio | Hallucinated answers on low-relevance hits or phantom citations | 0 phantom citations, abstention enforced below $\tau$ |
| 5. Prompt Injection Defense | Instruction-override/exfiltration regex detection and XML <sources> escaping | Retrieved tickets/docs hijacking system instructions | Adversarial rules flagged & payload HTML-escaped |
Run the sample 5-layer test harness using the repository virtual environment:
# Run all 5 testing layers and print a pass/fail summary table
.\.venv\Scripts\python.exe .github/skills/rag-pipeline/scripts/rag-testing-strategy.py --mode report
# Emit machine-readable JSON for CI quality gates
.\.venv\Scripts\python.exe .github/skills/rag-pipeline/scripts/rag-testing-strategy.py --mode report --format json
# Run the standard unittest.TestCase suite (RagPipelineTestingStrategyTests)
.\.venv\Scripts\python.exe .github/skills/rag-pipeline/scripts/rag-testing-strategy.py --mode unittestKnowledge Modules
Load only the modules relevant to the task:
- Document Ingestion, Parsing & Chunking Strategies — Layout/table preservation, recursive/Markdown/AST/Parent-Child/Contextual chunking, and SHA-256 idempotent delta ingestion.
- Embeddings, Quantization & Vector Database Indexing — MRL dimension truncation, asymmetric prefixes, HNSW/IVFFlat tuning, quantization, and production PostgreSQL
pgvectorschemas. - Hybrid Search, Query Transformation & Reranking — BM25 + Dense Reciprocal Rank Fusion (
RRF), PostgreSQL CTE queries, multi-query/step-back/HyDE, Cross-Encoder reranking, and context reordering. - Grounded Generation, Streaming APIs & Agentic RAG — Strict citation prompts, Pydantic v2 output contracts, abstention thresholds, Corrective/Self-RAG routers, FastAPI SSE streaming, and caching.
- RAG Evaluation, IR Metrics & Security Boundaries — RAG Triad,
Recall@K/MRR/NDCG@Kformulas, document-level ACL pre-filtering, indirect prompt injection defense, PII scrubbing, and the 5-Layer Automated RAG Testing Strategy.
Integration with Repository Skills
- LLM-as-Judge: Use Pattern 3 (Claim-Level Hallucination & RAG Grounding Audit) to score faithfulness (
SUPPORTED/CONTRADICTED/UNSUPPORTED) against retrieved chunks. - SQL Executor & SQL Connector: Inspect
pgvectorHNSW/GIN execution plans and safely query structured relational stores for hybrid SQL+RAG routers. - Python API Writer: Build non-blocking FastAPI
StreamingResponseSSE endpoints with lifespan connection pool management. - Caching & Queues Reviewer: Design Redis embedding caches, async ingestion worker queues (
Arq/Celery), and rate limits. - Researcher PoC Tester: Benchmark competing chunk sizes,
ef_searchsettings, or reranker latency percentiles (P50/P95/P99). - Save Memory & Load Memory: Persist embedding model choices, index dimensions, and retrieval evaluation baselines in
.project-memory/memory.db(ledger&journal).
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

