ingest-source
Skill1 starsChanged 6 months agoMITView on GitHub
Ingest one or more raw sources into the wiki. Two modes: INTERACTIVE (default) — one source at a time, with user-in-the-loop checkpoints. Triggers: "ingest", "ingest <file>", "process this", "read raw/unprocessed/<file>". BULK — recursively process the entire raw/unprocessed/ tree autonomously, no pauses, commit per file. Triggers: "bulk ingest", "process all", "process the queue", "process the unprocessed queue". In both modes the skill reads sources, distills into wiki pages using templates from wiki/_templates/, updates wiki/index.md and appends to wiki/log.md, moves sources into raw/processed/ preserving subfolder structure, and commits per source. Output bias: SYNTHESIS pages (wiki/<category>.md) cover all entities of a type. DEEP-DIVE pages (wiki/<category>/<name>.md) are created only when content warrants it.
4 other projects use a version of this.
Adopted by
Most recently taken up by
- mfoll/second-brain-templatetheir own version, first seen Nov 8, 2026
- leiverkus/research-superpowerstheir own version, first seen Nov 1, 2026
- Abilityai/corneliustheir own version, first seen Oct 8, 2026
- kylus/agentporttheir own version, first seen Aug 19, 2026
Other versions4 projects use a version of this.
Word for word · 0
None found.
Edited · 0
None found.
Wrote their own version · 4
- Abilityai/cornelius★ 109same name
- leiverkus/research-superpowers★ 9same name
- mfoll/second-brain-template★ 1same name
- kylus/agentport★ 0same name
How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.
In this repository3 other files it gives its agents.
- CLAUDE.mdCLAUDE.md
- lint-wikiSkill
- save-sessionSkill
The file
Show its rules and text
We split this skill into the rules it gives an agent
17 rules. 0 are common, 12 are distinctive, and 5 are very specific to this project.
0
Common
Most of the 4 edited versions we read keep these.
12
Distinctive
This author's own choices. Worth a look.
5
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 34
If the trigger is ambiguous, ask once which mode.
- Distinctive: line 48
If a relevant synthesis page already exists, update sections / table rows / lists.
- Very specific: line 52
Create a page at wiki/<category>/<name>.md ONLY when:
- Distinctive: line 74
Read the entire file. For long sources (>1000 lines), summarize first.
- Distinctive: line 98
Add or update rows / entries
- Distinctive: line 101
Don't rewrite the whole page
- Distinctive: line 109
If new synthesis or deep-dive pages were created, add a one-line entry under the right category section.
- Distinctive: line 115
If the right category section doesn't exist yet, create it.
- Very specific: line 119
Every ingest gets a log entry, even when only existing pages were updated.
- Very specific: line 137
Use git mv if under git. Create destination subfolders as needed (mkdir -p).
- Very specific: line 139
If the user has set up category folders under raw/processed/ (e.g.
- Distinctive: line 143
Use this exact format:
- Distinctive: line 154
Don't pause to ask the user about every conflict.
- Distinctive: line 203
Do not silently skip and continue.
- Distinctive: line 235
Bias to synthesis: Don't auto-create dozens of deep-dive pages.
- Very specific: line 236
Preserve subfolder structure: When moving sources to raw/processed/, mirror the input path.
- Distinctive: line 244
Create deep-dive pages by default — bias to synthesis updates
“Common” is measured: kept by at least 6 in 10 of the 4 edited versions of this file we could read. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Skill: Ingest source(s) into the wiki
This skill has two modes. Pick based on the user's trigger phrase.
Mode selection
| User says | Mode | Behavior |
"ingest <file>", "ingest this", "process raw/unprocessed/X" | Interactive | Process one source, pause for user input at key steps |
| "bulk ingest", "process all", "process the queue", "process unprocessed" | Bulk | Recursively process every file under raw/unprocessed/ autonomously |
If the trigger is ambiguous, ask once which mode. Default to interactive.
Output decision: synthesis vs deep-dive
Both modes use this rule. When ingesting any source, ask: "What output shape does this content warrant?"
Update a synthesis page (default)
Most ingests update a synthesis page at the wiki root: wiki/<category>.md. The agent picks the category from the source content — categories emerge from what your domain actually contains, not a fixed list. Examples:
- A legal source about a court ruling might update
wiki/cases.mdorwiki/rulings.md - A medical source about a clinical study might update
wiki/studies.md - An engineering source about a service might update
wiki/services.md
If a relevant synthesis page already exists, update sections / table rows / lists. Don't rewrite the whole page. If none exists and the source warrants one, create it.
Create a deep-dive page (only when warranted)
Create a page at wiki/<category>/<name>.md ONLY when:
- The user explicitly asks ("create a page for X")
- The source has design-level depth — full descriptions, edge cases, lifecycle decisions, debugging notes — that won't fit cleanly in a synthesis row
- The entity is being actively designed, modified, or repeatedly debugged
Default: synthesis update only, no deep-dive. Most entities get a one-line entry in the synthesis page and nothing else. A deep-dive can always be added later when the content warrants it.
Both
Some ingests warrant both — synthesis page update AND a deep-dive. Example: a source that catalogs 50 entities (synthesis) but also includes detailed coverage of one of them (deep-dive for that one).
INTERACTIVE MODE
1. Identify the source
The user named a specific file, or pointed at one. Confirm the path before proceeding.
2. Read the source in full
Read the entire file. For long sources (>1000 lines), summarize first.
3. Tell the user the takeaways and ask what to emphasize
In 3–6 bullets, surface what's actually in the source. Then ask:
- "Anything to emphasize or de-prioritize?"
- "Any context you have that isn't in this source?"
- "Should anything get a deep-dive page, or synthesis-only?"
Wait for the user's reply before writing.
4. Decide output shape
Based on the source content and user input:
- Which synthesis pages need updating? (likely 1–3)
- Which deep-dive pages need creating, if any?
- Which existing deep-dives need updating, if any?
List the plan before writing. Confirm with user.
5. Write/update pages
For synthesis page updates:
- Locate the right section (table, list, tree)
- Add or update rows / entries
- Bump
last_updatedin frontmatter - Append source path to frontmatter
sources: - Don't rewrite the whole page
For new deep-dive pages: copy the closest template from wiki/_templates/, rename, fill frontmatter, fill body, add cross-links to and from the synthesis page.
For decision, concept, plan, or reference pages: copy the matching template from wiki/_templates/. If a needed template doesn't exist, create it once and reuse for future ingests of the same type.
6. Update wiki/index.md
If new synthesis or deep-dive pages were created, add a one-line entry under the right category section. Format:
- [<page title>](<relative path>) — <one-line summary>If the right category section doesn't exist yet, create it. If only existing pages were updated, no index change is needed (the entry already exists).
7. Append to wiki/log.md
Every ingest gets a log entry, even when only existing pages were updated. Append to the end of wiki/log.md:
## [YYYY-MM-DD] ingest | <short title>
What happened in 1–3 lines.
Pages touched: <list>.
Output shape: synthesis | deep-dive | both.wiki/log.md is content-level audit — it captures what changed in the wiki, not what changed on disk. Git is the filesystem-level audit; log.md complements it.
8. Move the source
Move the file from raw/unprocessed/ into raw/processed/, preserving the subfolder path under unprocessed/.
Example: raw/unprocessed/foo/bar/doc.pdf → raw/processed/foo/bar/doc.pdf.
Use git mv if under git. Create destination subfolders as needed (mkdir -p).
If the user has set up category folders under raw/processed/ (e.g. raw/processed/docs/, raw/processed/exports/), route to the matching one. Otherwise mirror the input path directly. Don't invent a fixed taxonomy — let it emerge from the user's filing.
9. Flag contradictions, don't overwrite
Use this exact format:
⚠️ Conflict noted YYYY-MM-DD — <one-line title>
- Source A (<path>): <claim>
- Source B (<path>): <claim>
- Resolution: (empty until resolved)The Resolution: line is mandatory. Leave it empty when open. Fill it in when a later ingest resolves the conflict (and update the callout to past tense, e.g. "Conflict resolved 2026-06-15 — ...").
Don't pause to ask the user about every conflict. Note it and continue.
10. Commit
If under git:
git add -A
git commit -m "wiki: ingest <source filename>"One ingest = one commit.
BULK MODE
The same procedure as interactive, applied recursively to every file under raw/unprocessed/, with these differences:
Differences from interactive mode
| Step | Interactive | Bulk |
| 1 — identify source | Single file from user | Recursively walk raw/unprocessed/, alphabetical by full path, one at a time |
| 3 — takeaways + ask | Pause for user input | Skip — decide emphasis from the source alone |
| 4 — output shape decision | Confirm with user | Decide autonomously per the rule above (default synthesis, deep-dive only when warranted) |
| 6 — update index.md | Same — per ingest | Same — per ingest (still runs per file) |
| 7 — append log.md | Same — per ingest | Same — per ingest (content-level audit ≠ git audit) |
| 9 — conflicts | Same — note, continue | Same — note, continue |
| 10 — commit | One commit | One commit per source |
Bulk mode procedure
Survey the queue. Recursively find all files under
raw/unprocessed/:find raw/unprocessed/ -type f -not -name '.*' -not -name '_*' | sortSkip dotfiles (
.gitkeep,.DS_Store) and underscore-prefixed infrastructure files. Tell the user: "Found N files (across M subfolders). Starting bulk ingest. I will commit per source and produce a summary at the end. This may take a while."- Process each file in alphabetical order by full path. For each file, run steps 2, 4, 5, 6, 7, 8, 9, 10 from interactive mode, without pauses. Skip step 3 (takeaway discussion). Steps 6 (update
index.md) and 7 (appendlog.md) still run per-file — bulk mode commits per file, but the content-level audit trail is per ingest, not per run. - On error or unrecoverable confusion (e.g. source is malformed, can't be parsed, agent hits a contradiction so severe the rest of the run is unsound) — stop. Report:
- Which file caused the stop (full path)
- What the error was
- Which files were processed successfully (before the failure)
- Which files remain unprocessed
- Suggested next action (skip the bad file, re-run after fix, etc.) Do not silently skip and continue.
- At the end of a successful run, produce a summary report in chat:
Bulk ingest complete.
Processed: N files (across M subfolders)
- foo/bar/source-1.ext → updated <category>.md (K rows added), no deep-dive
- foo/bar/source-2.ext → updated <category>.md, created wiki/<category>/<name>.md
- ...
Synthesis pages updated: K (list)
Deep-dive pages created: J (list)
Index entries added: I (in wiki/index.md)
Log entries appended: N (in wiki/log.md, one per ingest)
Conflicts flagged: P (see callouts in <list of pages>)
Commits: N (one per source)
Recommended next steps:
- Review the P flagged conflicts
- Run lint-wiki to check for cross-reference gaps
- Inspect graph view in Obsidian to see new connectionsBulk mode rules
- One commit per source. Audit trail. Lets the user
git resetto recover from a bad run. - Stop on errors, don't skip. Better to halt and surface than silently produce garbage.
- Keep CLAUDE.md rules. Bulk doesn't bypass workflow rules — wiki-first, raw-immutable, conflict callouts all still apply.
- Process alphabetically by full path. Predictable order makes debugging easier when something fails.
- Bias to synthesis. Don't auto-create dozens of deep-dive pages. If unsure, write a synthesis row and skip the deep-dive. The user can always ask for a deep-dive later.
- Preserve subfolder structure. When moving sources to
raw/processed/, mirror the input path. - Skip dotfiles and underscore-prefixed files.
.gitkeep,.DS_Store,_meta.jsonetc. are infrastructure, not content.
What this skill does NOT do (both modes)
- Modify files in
raw/processed/after they've been moved (raw is read-only post-ingest) - Create deep-dive pages by default — bias to synthesis updates
- Auto-promote unclear content into wiki pages — when in doubt, flag and ask
- Process multiple sources in parallel — one at a time, fully complete before starting the next (applies to bulk too)
- Resume a failed bulk run silently — always require explicit user confirmation to continue after a stop
- Process dotfiles or underscore-prefixed infrastructure files
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

