agentleFS
artificialbeingzartificialbeingz/abz-karpathy-llm-wiki/.claude/skills/ingest-source/SKILL.md

ingest-source

Skill1 starsChanged 6 months agoMITView on GitHub

Ingest one or more raw sources into the wiki. Two modes: INTERACTIVE (default) — one source at a time, with user-in-the-loop checkpoints. Triggers: "ingest", "ingest <file>", "process this", "read raw/unprocessed/<file>". BULK — recursively process the entire raw/unprocessed/ tree autonomously, no pauses, commit per file. Triggers: "bulk ingest", "process all", "process the queue", "process the unprocessed queue". In both modes the skill reads sources, distills into wiki pages using templates from wiki/_templates/, updates wiki/index.md and appends to wiki/log.md, moves sources into raw/processed/ preserving subfolder structure, and commits per source. Output bias: SYNTHESIS pages (wiki/<category>.md) cover all entities of a type. DEEP-DIVE pages (wiki/<category>/<name>.md) are created only when content warrants it.

4 other projects use a version of this.

Adopted by

Most recently taken up by

  • Other versions4 projects use a version of this.

    Word for word · 0

    None found.

    Edited · 0

    None found.

    Wrote their own version · 4

    1. Abilityai/cornelius★ 109same name
    2. leiverkus/research-superpowers★ 9same name
    3. mfoll/second-brain-template★ 1same name
    4. kylus/agentport★ 0same name

    How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.

  • In this repository3 other files it gives its agents.

The file

Show its rules and text

We split this skill into the rules it gives an agent

17 rules. 0 are common, 12 are distinctive, and 5 are very specific to this project.

  • 0

    Common

    Most of the 4 edited versions we read keep these.

  • 12

    Distinctive

    This author's own choices. Worth a look.

  • 5

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    If the trigger is ambiguous, ask once which mode.

    line 34
  2. Distinctive:

    If a relevant synthesis page already exists, update sections / table rows / lists.

    line 48
  3. Very specific:

    Create a page at wiki/<category>/<name>.md ONLY when:

    line 52
  4. Distinctive:

    Read the entire file. For long sources (>1000 lines), summarize first.

    line 74
  5. Distinctive:

    Add or update rows / entries

    line 98
  6. Distinctive:

    Don't rewrite the whole page

    line 101
  7. Distinctive:

    If new synthesis or deep-dive pages were created, add a one-line entry under the right category section.

    line 109
  8. Distinctive:

    If the right category section doesn't exist yet, create it.

    line 115
  9. Very specific:

    Every ingest gets a log entry, even when only existing pages were updated.

    line 119
  10. Very specific:

    Use git mv if under git. Create destination subfolders as needed (mkdir -p).

    line 137
  11. Very specific:

    If the user has set up category folders under raw/processed/ (e.g.

    line 139
  12. Distinctive:

    Use this exact format:

    line 143
  13. Distinctive:

    Don't pause to ask the user about every conflict.

    line 154
  14. Distinctive:

    Do not silently skip and continue.

    line 203
  15. Distinctive:

    Bias to synthesis: Don't auto-create dozens of deep-dive pages.

    line 235
  16. Very specific:

    Preserve subfolder structure: When moving sources to raw/processed/, mirror the input path.

    line 236
  17. Distinctive:

    Create deep-dive pages by default — bias to synthesis updates

    line 244

“Common” is measured: kept by at least 6 in 10 of the 4 edited versions of this file we could read. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Skill: Ingest source(s) into the wiki

This skill has two modes. Pick based on the user's trigger phrase.

Mode selection

User saysModeBehavior
"ingest <file>", "ingest this", "process raw/unprocessed/X"InteractiveProcess one source, pause for user input at key steps
"bulk ingest", "process all", "process the queue", "process unprocessed"BulkRecursively process every file under raw/unprocessed/ autonomously

If the trigger is ambiguous, ask once which mode. Default to interactive.

Output decision: synthesis vs deep-dive

Both modes use this rule. When ingesting any source, ask: "What output shape does this content warrant?"

Update a synthesis page (default)

Most ingests update a synthesis page at the wiki root: wiki/<category>.md. The agent picks the category from the source content — categories emerge from what your domain actually contains, not a fixed list. Examples:

  • A legal source about a court ruling might update wiki/cases.md or wiki/rulings.md
  • A medical source about a clinical study might update wiki/studies.md
  • An engineering source about a service might update wiki/services.md

If a relevant synthesis page already exists, update sections / table rows / lists. Don't rewrite the whole page. If none exists and the source warrants one, create it.

Create a deep-dive page (only when warranted)

Create a page at wiki/<category>/<name>.md ONLY when:

  • The user explicitly asks ("create a page for X")
  • The source has design-level depth — full descriptions, edge cases, lifecycle decisions, debugging notes — that won't fit cleanly in a synthesis row
  • The entity is being actively designed, modified, or repeatedly debugged

Default: synthesis update only, no deep-dive. Most entities get a one-line entry in the synthesis page and nothing else. A deep-dive can always be added later when the content warrants it.

Both

Some ingests warrant both — synthesis page update AND a deep-dive. Example: a source that catalogs 50 entities (synthesis) but also includes detailed coverage of one of them (deep-dive for that one).


INTERACTIVE MODE

1. Identify the source

The user named a specific file, or pointed at one. Confirm the path before proceeding.

2. Read the source in full

Read the entire file. For long sources (>1000 lines), summarize first.

3. Tell the user the takeaways and ask what to emphasize

In 3–6 bullets, surface what's actually in the source. Then ask:

  • "Anything to emphasize or de-prioritize?"
  • "Any context you have that isn't in this source?"
  • "Should anything get a deep-dive page, or synthesis-only?"

Wait for the user's reply before writing.

4. Decide output shape

Based on the source content and user input:

  • Which synthesis pages need updating? (likely 1–3)
  • Which deep-dive pages need creating, if any?
  • Which existing deep-dives need updating, if any?

List the plan before writing. Confirm with user.

5. Write/update pages

For synthesis page updates:

  • Locate the right section (table, list, tree)
  • Add or update rows / entries
  • Bump last_updated in frontmatter
  • Append source path to frontmatter sources:
  • Don't rewrite the whole page

For new deep-dive pages: copy the closest template from wiki/_templates/, rename, fill frontmatter, fill body, add cross-links to and from the synthesis page.

For decision, concept, plan, or reference pages: copy the matching template from wiki/_templates/. If a needed template doesn't exist, create it once and reuse for future ingests of the same type.

6. Update wiki/index.md

If new synthesis or deep-dive pages were created, add a one-line entry under the right category section. Format:

- [<page title>](<relative path>) — <one-line summary>

If the right category section doesn't exist yet, create it. If only existing pages were updated, no index change is needed (the entry already exists).

7. Append to wiki/log.md

Every ingest gets a log entry, even when only existing pages were updated. Append to the end of wiki/log.md:

## [YYYY-MM-DD] ingest | <short title>

What happened in 1–3 lines.
Pages touched: <list>.
Output shape: synthesis | deep-dive | both.

wiki/log.md is content-level audit — it captures what changed in the wiki, not what changed on disk. Git is the filesystem-level audit; log.md complements it.

8. Move the source

Move the file from raw/unprocessed/ into raw/processed/, preserving the subfolder path under unprocessed/.

Example: raw/unprocessed/foo/bar/doc.pdf → raw/processed/foo/bar/doc.pdf.

Use git mv if under git. Create destination subfolders as needed (mkdir -p).

If the user has set up category folders under raw/processed/ (e.g. raw/processed/docs/, raw/processed/exports/), route to the matching one. Otherwise mirror the input path directly. Don't invent a fixed taxonomy — let it emerge from the user's filing.

9. Flag contradictions, don't overwrite

Use this exact format:

⚠️ Conflict noted YYYY-MM-DD — <one-line title>
- Source A (<path>): <claim>
- Source B (<path>): <claim>
- Resolution: (empty until resolved)

The Resolution: line is mandatory. Leave it empty when open. Fill it in when a later ingest resolves the conflict (and update the callout to past tense, e.g. "Conflict resolved 2026-06-15 — ...").

Don't pause to ask the user about every conflict. Note it and continue.

10. Commit

If under git:

git add -A
git commit -m "wiki: ingest <source filename>"

One ingest = one commit.


BULK MODE

The same procedure as interactive, applied recursively to every file under raw/unprocessed/, with these differences:

Differences from interactive mode

StepInteractiveBulk
1 — identify sourceSingle file from userRecursively walk raw/unprocessed/, alphabetical by full path, one at a time
3 — takeaways + askPause for user inputSkip — decide emphasis from the source alone
4 — output shape decisionConfirm with userDecide autonomously per the rule above (default synthesis, deep-dive only when warranted)
6 — update index.mdSame — per ingestSame — per ingest (still runs per file)
7 — append log.mdSame — per ingestSame — per ingest (content-level audit ≠ git audit)
9 — conflictsSame — note, continueSame — note, continue
10 — commitOne commitOne commit per source

Bulk mode procedure

  1. Survey the queue. Recursively find all files under raw/unprocessed/:

    find raw/unprocessed/ -type f -not -name '.*' -not -name '_*' | sort

    Skip dotfiles (.gitkeep, .DS_Store) and underscore-prefixed infrastructure files. Tell the user: "Found N files (across M subfolders). Starting bulk ingest. I will commit per source and produce a summary at the end. This may take a while."

  2. Process each file in alphabetical order by full path. For each file, run steps 2, 4, 5, 6, 7, 8, 9, 10 from interactive mode, without pauses. Skip step 3 (takeaway discussion). Steps 6 (update index.md) and 7 (append log.md) still run per-file — bulk mode commits per file, but the content-level audit trail is per ingest, not per run.
  3. On error or unrecoverable confusion (e.g. source is malformed, can't be parsed, agent hits a contradiction so severe the rest of the run is unsound) — stop. Report:
    • Which file caused the stop (full path)
    • What the error was
    • Which files were processed successfully (before the failure)
    • Which files remain unprocessed
    • Suggested next action (skip the bad file, re-run after fix, etc.) Do not silently skip and continue.
  4. At the end of a successful run, produce a summary report in chat:
Bulk ingest complete.

Processed: N files (across M subfolders)
- foo/bar/source-1.ext → updated <category>.md (K rows added), no deep-dive
- foo/bar/source-2.ext → updated <category>.md, created wiki/<category>/<name>.md
- ...

Synthesis pages updated: K (list)
Deep-dive pages created: J (list)
Index entries added: I (in wiki/index.md)
Log entries appended: N (in wiki/log.md, one per ingest)
Conflicts flagged: P (see callouts in <list of pages>)

Commits: N (one per source)

Recommended next steps:
- Review the P flagged conflicts
- Run lint-wiki to check for cross-reference gaps
- Inspect graph view in Obsidian to see new connections

Bulk mode rules

  • One commit per source. Audit trail. Lets the user git reset to recover from a bad run.
  • Stop on errors, don't skip. Better to halt and surface than silently produce garbage.
  • Keep CLAUDE.md rules. Bulk doesn't bypass workflow rules — wiki-first, raw-immutable, conflict callouts all still apply.
  • Process alphabetically by full path. Predictable order makes debugging easier when something fails.
  • Bias to synthesis. Don't auto-create dozens of deep-dive pages. If unsure, write a synthesis row and skip the deep-dive. The user can always ask for a deep-dive later.
  • Preserve subfolder structure. When moving sources to raw/processed/, mirror the input path.
  • Skip dotfiles and underscore-prefixed files. .gitkeep, .DS_Store, _meta.json etc. are infrastructure, not content.

What this skill does NOT do (both modes)

  • Modify files in raw/processed/ after they've been moved (raw is read-only post-ingest)
  • Create deep-dive pages by default — bias to synthesis updates
  • Auto-promote unclear content into wiki pages — when in doubt, flag and ask
  • Process multiple sources in parallel — one at a time, fully complete before starting the next (applies to bulk too)
  • Resume a failed bulk run silently — always require explicit user confirmation to continue after a stop
  • Process dotfiles or underscore-prefixed infrastructure files

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.