ingest-sources
Skill1 starsChanged 7 months agoMITView on GitHub
Review newly ingested raw sources and draft or update source summary notes in notes/Sources/.
In this repository9 other files it gives its agents.
- AGENTS.mdAGENTS.md
- .codex/AGENTS.mdAGENTS.md
- .claude/CLAUDE.mdCLAUDE.md
- CLAUDE.mdCLAUDE.md
- compile-wikiSkill
- lint-healSkill
- qa-agentSkill
- render-outputSkill
- research-workflowSkill
The file
We split this skill into the rules it gives an agent
8 rules. About 0 are common, 3 are distinctive, and 5 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈3
Distinctive · estimate
This author's own choices. Worth a look.
5
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 22
Update only the source notes that need to change.
- Very specific: line 50
Imported Report: If kind is imported-model-report, set authority: lead_only, verification_state: needs_primary_sources, evidence_strength: …
- Very specific: line 51
Imported Citation Stub: If kind is citation-stub, set canonical_url: "<url>", authority: lead_only, verification_state: needs_fetch, …
- Very specific: line 66
When data/raw/<source_id>/large_source_manifest.json exists AND the normalized source content exceeds 20 KB, the agent MUST use the …
- Very specific: line 85
If the note would exceed 60 KB, add source_summary_too_large: true and truncated_at_section: "<last section rendered>" to frontmatter, stop …
- Distinctive: line 88
Prefer updating an existing note over creating a duplicate.
- Distinctive: line 89
Keep the summary anchored in the raw capture and registry metadata.
- Very specific: line 90
If the source is too thin, use stub or citation-only rather than inventing detail.
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Ingest Sources
Goal
Turn already-fetched raw source folders into structured source summary notes that the vault compiler can use.
Context
Mechanical fetching and normalization is handled by kops/ingest_sources.py via kb.py ingest.
This skill covers the agent-driven step that follows: reading normalized content and writing or updating source summaries.
Inputs
data/registry.json- registry of all ingested sourcesdata/raw/<id>/normalized.md- normalized text for each sourcedata/raw/<id>/metadata.json- source metadata (URL, type, timestamps)- Existing
notes/Sources/<id>.mdif a prior summary exists
Output contract
- Update only the source notes that need to change.
- Preserve provenance and uncertainty explicitly.
- Flag weak or incomplete captures instead of smoothing them over.
Required note shape
Frontmatter (all fields mandatory per config/schema.yaml):
source_id(canonicalsrc-[0-9a-f]{10})titlesource_urlsource_kind(must map to a canonical schema value):arxiv-paper|paper-pdf|github-repo-snapshot|github-file|official-doc|spec|blog|news|local-file|imported-model-report|citation-stubevidence_strength(must map to a canonical schema value):primary-doc|primary-doc-partial|official-spec|strong|code|maintainer-commentary|changelog|pr-issue|secondary|model-generated|stub|citation-only|image-onlysource_status(defaults toactive)ingested_at(ISO-8601 date, e.g. 2026-06-14T10:53:44Z)tags(must includekb/source)
Kind-Specific Required Frontmatter:
- arxiv-paper:
authors,arxiv_id,published_date,abstract - paper-pdf:
page_count - github-repo-snapshot:
git_commit,branch,tracked_file_count,sampled_file_count - github-file:
github_url,git_commit - official-doc:
organization - spec:
organization,version,status
Strong PDF / Imported Report Constraints:
- PDF Coverage: Any PDF source (
paper-pdf,arxiv-paper, or URL ending in.pdf) with strong evidence strength (primary-doc,strong,official-spec) MUST includeextraction_coveragemetadata in its frontmatter (e.g.extraction_coverage: 1.0). - Imported Report: If kind is
imported-model-report, setauthority: lead_only,verification_state: needs_primary_sources,evidence_strength: secondary. - Imported Citation Stub: If kind is
citation-stub, setcanonical_url: "<url>",authority: lead_only,verification_state: needs_fetch,evidence_strength: stub.
Sections (in this order, all required):
## Summary## What this source is## Key claims## Important evidence / details## Candidate concepts## Open questions## Reliability notes(Required by linter)## Related Concepts(Required by linter)## Backlinks
Large source variant
When data/raw/<source_id>/large_source_manifest.json exists AND the normalized source content exceeds 20 KB, the agent MUST use the large-source section schema instead of the flat schema above.
Required sections in order:
## Document Summary— 1-3 paragraph synthesis of the full source (main thesis, audience, scope). Machine-owned.## Section Evidence Map— contains###subsections, one per high-signal manifest node. Only render: top-level nodes (level ≤ 1) plus child nodes that have claims, contradictions, or table/figure type. Machine-owned.- Each
###heading must use the manifest node'stitlefield verbatim (strip any "type: " prefix). - Each subsection may contain a
#### Key Claimsblock with inline citations anchored to the section heading. - Parent nodes must not restate child claim bullets verbatim — parent Key Claims are for cross-section synthesis only.
- Each
## Key Claims— top-level cross-document claims only; must not duplicate bullets from Section Evidence Map.## Candidate concepts## Open questions## Reliability notes(required by linter)## Related Concepts(required by linter)## Backlinks
Citation anchor format for section-level citations:
[[Sources/<subdir>/<source_id>#<heading-anchor>|<source_id>#<heading-anchor>]]
where <heading-anchor> is the Obsidian anchor derivation: lowercase, spaces → hyphens, punctuation removed.
If the note would exceed 60 KB, add source_summary_too_large: true and truncated_at_section: "<last section rendered>" to frontmatter, stop rendering, and append a TODO entry in notes/TODO.md.
Safe behavior
- Prefer updating an existing note over creating a duplicate.
- Keep the summary anchored in the raw capture and registry metadata.
- If the source is too thin, use
stuborcitation-onlyrather than inventing detail.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

