agentleFS
Sign inStart free
remembervictorchatbot-vipremembervictorchatbot-vip/AI_Token_Saving/skills/rag-engineering/SKILL.md

rag-engineering

Skill0 starsChanged 3 months agoMITView on GitHub

Reusable RAG engineering playbook — how to build retrieval-augmented systems that actually work, distilled from infiniflow/ragflow, hkuds/lightrag, HKUDS/RAG-Anything, NirDiamant/RAG_Techniques, Graphify-Labs/graphify, raga-ai-hub/RagaAI-Catalyst. Use when designing, auditing, or fixing any RAG/retrieval pipeline: chunking, indexing, hybrid search, reranking, graph augmentation, evaluation, observability. Pairs with engineering-discipline (lifecycle) and token-savings (compression + the folded crl codebase-retrieval engine).

6 other projects use a version of this.

Adopted by

Most recently taken up by

Used with

Projects with it also use security-review.

Other files people use alongside this one.

Files found in the same public projects as a version of this one, more often than they turn up anywhere else. Each project counts once, large collections count for less, and this file's own repository is left out.
  • SentrySkill

    security-review

    Finds exploitable application security vulnerabilities in code changes. Use for Warden security scans, appsec review, OWASP-style checks, authentication or authorization bugs, injection, XSS, SSRF, path traversal, secrets, unsafe crypto, webhook verification, open redirects, or sensitive data exposure.

    Also in 3 projects that use this

The file

We split this skill into the rules it gives an agent

5 rules. 0 are common, 4 are distinctive, and 1 is very specific to this project.

  • 0

    Common

    Most of the 5 edited versions we read keep these.

  • 4

    Distinctive

    This author's own choices. Worth a look.

  • 1

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Always rerank top-k with a cross-encoder before feeding context.

    line 26
  2. Very specific:

    Safe-mode: run toks/safemode.py on retrieved chunks.

    line 30
  3. Distinctive:

    Trace: every answer back to its retrieved chunk (citation).

    line 36
  4. Distinctive:

    Run regression tests on a fixed query set when you change chunking/embedding.

    line 37
  5. Distinctive:

    No evaluation harness — tuning blind.

    line 44

“Common” is measured: kept by at least 6 in 10 of the 5 edited versions of this file we could read. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

RAG Engineering — Reusable Playbook

Synthesized from the reference repos. Principles first, repo specifics as evidence.

1. Chunking (ragflow's "deep doc understanding" lesson)

  • Chunk by semantic + structural boundaries, not fixed token counts. Tables, headings, and code fences must not be split mid-structure.
  • Preserve layout/order metadata (page, section, position) — free reranking signal.
  • For code: chunk by ast/function boundaries (the folded crl engine in token-savings/scripts/crl/ does closure retrieval) so retrieval returns whole units.

2. Indexing & retrieval (lightrag / RAG-Anything)

  • Hybrid search is the baseline: dense (embeddings) + sparse (BM25/keyword). Dense alone misses exact IDs/code; sparse alone misses paraphrase.
  • Graph augmentation (LightRAG): entity/relation graph for multi-hop queries.
  • Multimodal (RAG-Anything): keep modality-specific parsers; don't flatten images/PDFs into dumb text and lose tables/diagrams.

3. Reranking & compression (RAG_Techniques + token-savings)

  • Always rerank top-k with a cross-encoder before feeding context.
  • Apply context compression (token-savings toks/astrip.py, compress_json): drop boilerplate, keep evidence sentences. Never drop the cited source pointer.
  • Limit context to the fewest passages that answer — more isn't better, it dilutes.
  • Safe-mode: run toks/safemode.py on retrieved chunks; secrets/stack traces pass through verbatim so the answer never hallucinates from a truncated source.

4. Evaluation & observability (RagaAI-Catalyst / graphify)

  • Measure: retrieval recall@k, answer faithfulness (no hallucinated citations), end-to-end task success. A RAG you can't measure is a RAG you can't trust.
  • Trace every answer back to its retrieved chunk (citation). No citation = no ship.
  • Run regression tests on a fixed query set when you change chunking/embedding.

5. Anti-patterns

  • Naive fixed-size chunking that splits tables/code.
  • Dense-only retrieval dropping exact-match IDs.
  • Stuffing 20 passages into context "just in case."
  • Shipping without a citation traceability check.
  • No evaluation harness — tuning blind.
  • Compressing retrieved secrets/stack traces (safe-mode violation).

Integration

  • Lifecycle: engineering-discipline. Efficiency + dedup + safe-mode + the folded crl codebase-retrieval engine: token-savings (the single source of token-saving truth — this skill does NOT duplicate those rules, it references them).
  • Note: this playbook is a synthesis of established patterns + the named repos' published architectures; validate repo-specific claims against the latest docs before production use.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

Check your setup, free