break-it
Skill24 starsChanged 19 days agoMITView on GitHub
Find the bug — learn RAG by debugging intentionally broken pipelines
1 other project uses a version of this.
Adopted by
Most recently taken up by
- lifedever/skills-plugintheir own version, first seen Nov 16, 2026
Other versions1 project uses a version of this.
Word for word · 0
None found.
Edited · 0
None found.
Wrote their own version · 1
- lifedever/skills-plugin★ 14same name
How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.
In this repository23 other files it gives its agents.
- AGENTS.mdAGENTS.md
- CLAUDE.mdCLAUDE.md
- architectureSkill
- audit-contentSkill
- benchmarkSkill
- buildSkill
- challengeSkill
- code-reviewSkill
- compareSkill
- debug-ragSkill
- evaluateSkill
- explainSkill
- fixSkill
- glossarySkill
- journalSkill
- lessonSkill
- paper-reviewSkill
- quizSkill
- recapSkill
- roadmapSkill
- sandboxSkill
- startSkill
- triageSkill
The file
We split this skill into the rules it gives an agent
3 rules. About 0 are common, 2 are distinctive, and 1 is very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈2
Distinctive · estimate
This author's own choices. Worth a look.
1
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 14
If the learner specifies a difficulty (e.g., /break-it hard), use that.
- Distinctive: line 41
No grounding instruction — model ignores context and hallucinates
- Distinctive: line 46
Document not chunked before embedding (entire doc as one vector)
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Break It: Learn by Debugging
Introduce a realistic bug into a RAG pipeline and challenge the learner to find and fix it. Debugging teaches more than building because you have to understand why things work to figure out why they don't.
Language awareness: See
.claude/LANGUAGE_AWARENESS.md.
Step 1: Choose a Difficulty
If the learner specifies a difficulty (e.g., /break-it hard), use that. Otherwise, calibrate based on their progress:
- Easy — One obvious bug. The pipeline runs but gives clearly wrong results.
- Medium — One subtle bug. The pipeline runs and results look plausible but are wrong.
- Hard — Two bugs that interact. Fixing one makes the other more visible.
Step 2: Select a Bug Category
Pick from these common RAG failure modes. Rotate through categories so the learner sees variety:
Chunking Bugs
- Chunk size set way too small (10 tokens) — fragments lose all context
- Zero overlap — information at boundaries is lost
- Wrong separator for the document type (splitting code on paragraphs)
Embedding Bugs
- Mismatched embedding models between indexing and querying
- Missing query prefix for asymmetric models (e.g., no "query: " prefix for E5)
- Embeddings not normalized, breaking cosine similarity
Retrieval Bugs
- Top-k set to 1 — missing relevant context spread across chunks
- Wrong distance metric (L2 instead of cosine)
- Metadata filter too restrictive — filtering out relevant results
Generation Bugs
- Context placed after the question (lost-in-the-middle effect)
- No grounding instruction — model ignores context and hallucinates
- Context window exceeded — retrieved chunks silently truncated
Pipeline Bugs
- Stale index — new documents added but not re-embedded
- Document not chunked before embedding (entire doc as one vector)
- Query embedded with a different model than the corpus
Step 3: Present the Broken Pipeline
Show the learner a complete, runnable pipeline with the bug(s) already in place. Include:
- The code (with the bug hidden in plain sight)
- Sample data
- A query that exposes the bug
- The actual output (showing the wrong behavior)
- What the correct output should look like
Say: "Something's wrong with this pipeline. The query should return [expected], but it's returning [actual]. Can you find the bug?"
Step 4: Guide the Debugging
Let the learner investigate. If they're stuck after 2 attempts, offer graduated hints:
- Hint 1 (gentle): Point to the general area — "The issue is in the retrieval step" or "Look at how the chunks are created"
- Hint 2 (specific): Narrow it down — "Compare the embedding model used for indexing vs querying"
- Hint 3 (direct): Explain the bug and why it causes the behavior
Step 5: Explain and Connect
Once they find the bug (or after hints), explain:
- Why this bug causes the observed behavior
- How common this bug is in real RAG systems
- How to prevent it — what checks or tests would catch this early
- The concept behind the fix — link to the relevant
/lessonor/explaintopic
Step 6: Track and Suggest
Log the debugging session to progress/debug-log.md with the date, bug type, difficulty, and whether they found it independently.
Suggest next steps:
/break-it [harder difficulty]to level up/lessonon the concept behind the bug/buildto implement the correct pattern from scratch
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

