gather
Skill150 starsChanged 7 days agoMITView on GitHub
Web capture into raw/, two modes, one pipeline. SEED mode: given URL(s), fetch them and (optionally) the relevant links they cite. SEARCH mode: given a topic and no URL, web-search it, triage the hits, and present a dated shortlist for approval before anything is fetched — the vault's native deep-research entry (topic → sources → wiki; the cited report then comes from ingest + query/output). Use on /gather, "gather sources on <topic>", "search the web for X and save the sources", "deep-capture these links", "grab this article AND the repos/papers it cites". Preview-and-approve gates by default; conservative caps + a hard 100-page ceiling; expert flags (--search/--seed, --expand, --focus, --rounds, --max-pages, --include/--exclude, --same-domain, --yes, --ingest). Big budgets flow through a ranked policy gate (--shortlist, --queries tune the funnel); a per-run ledger plus a scripted sole write path harden caps, raw immutability and provenance. Opt-in; does nothing until invoked. It only CAPTURES into raw/ — /ingest compiles. Writes only new raw files, their asset images and its own log entry (never edits existing raw content).
5 other projects use a version of this.
Adopted by
Most recently taken up by
- sanmidable/skillstheir own version, first seen Nov 8, 2026
- jurisdictional/jurisdictional-skilltheir own version, first seen Nov 1, 2026
- raphaelsalaja/skillstheir own version, first seen Nov 1, 2026
- HurricaHjz/SecondSelfword for word, first seen Oct 20, 2026
- corygabrielsen/skillstheir own version, first seen May 19, 2026
Other versions5 projects use a version of this.
Word for word · 1
- HurricaHjz/SecondSelf★ 150100% same
Edited · 0
None found.
Wrote their own version · 4
- raphaelsalaja/skills★ 67same name
- sanmidable/skills★ 2same name
- corygabrielsen/skills★ 1same name
- jurisdictional/jurisdictional-skill★ 0same name
How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.
In this repository15 other files it gives its agents.
- CLAUDE.mdCLAUDE.md
- adoptSkill
- atticSkill
- deep-lintSkill
- delegateSkill
- export-templateSkill
- ingestSkill
- lintSkill
- outputSkill
- projectSkill
- qmd-searchSkill
- querySkill
- reflectSkill
- suggestSkill
- weekly-auditSkill
The file
We split this skill into the rules it gives an agent
11 rules. 0 are common, 6 are distinctive, and 5 are very specific to this project.
0
Common
Most of the 3 edited versions we read keep these.
6
Distinctive
This author's own choices. Worth a look.
5
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 29
One funnel: search wide → rank a pool → show a menu → the owner approves → capture.
- Distinctive: line 44
No URL, a topic/question present → search mode.
- Distinctive: line 196
Default X/Reddit posture (tier 3 inactive): login-walled — locate posts via WebSearch, read via the tier-1 Jina channel.
- Very specific: line 235
Ledger: Every run inits one; the write path requires it — a capture with a missing ledger is STOP-and-ask, the declared repair a fresh init …
- Very specific: line 236
STOP-and-ask, the declared repair a fresh init carrying the known state.
- Distinctive: line 263
Every run echoes its parsed run-spec at the first gate (mode · seeds/brief · focus · expand · rounds · caps.
- Very specific: line 298
Logging: After a completed capture run, append ONE gather entry to wiki/log.md via shell (cat >>): mode, seed URL(s) or the brief + the …
- Distinctive: line 333
Backend-pain warn: when a hard window forces ≥8 in-window verifications in one run, or the run's own row/drop reasons record region- or …
- Distinctive: line 358
Engine health: When a formerly-working engine fails on ordinary pages (rot, not a one-off), the run report says so.
- Very specific: line 407
Run gather_links.py so every gather applies the SAME rules.
- Very specific: line 439
STOP and ask; authority off → the --pages-captured flag, stated).
“Common” is measured: kept by at least 6 in 10 of the 3 edited versions of this file we could read. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
gather — web capture into raw/ (seed mode + search mode)
Goal
Turn either a set of links (seed mode) or a bare topic (search mode) into a high-quality set of
captured sources in raw/, without fan-out blow-ups, off-topic noise, or silent writes. /ingest
then compiles. Capture is document-granular — whole pages, verbatim; salience is ingest's job.
The whole logic in three lines (single-letter symbols live in the scripts):
- Two inputs: the owner's page budget, and the topic's uncovered sub-questions (round 1: the brief's facets; later rounds: the open gaps — same thing, next moment).
- One funnel: search wide → rank a pool → show a menu → the owner approves → capture.
- Three principles: numbers bound, never oblige (every cap is a cap, not a quota); nothing is written without the owner's yes, and noes outlive yeses (drops persist, rules don't); judgement declares itself, arithmetic is scripted (the agent states sub-questions/relevance/advice, the scripts derive and print every number).
How to read this skill. The CONTROL ENVELOPE is rigid — a smarter model obeys it identically. Procedure (how to phrase queries, rank, scope) is stated as goal + reason under JUDGEMENT & RATIONALE and applied with judgement — better models make better choices there. Judgement governs HOW a rationale entry is satisfied, never WHETHER: the imperative duties inside those entries (SAY SO · EVERY · STOP · never) bind exactly like the envelope. The CAPABILITY CATALOG lists every lever; a lever you don't know exists is one you can't reach for.
Modes & routing (deterministic — any agent, any model, same result)
- ≥1 URL among the arguments → seed mode. Free text alongside = scope + focus.
- No URL, a topic/question present → search mode. The text is the brief: goal, any include/exclude criteria, recency expectations.
- Explicit
--seed/--search "<brief>"overrides inference. Neither URL nor topic — or--seedwith no URL — → ask. An explicit--shortlistor--queriesstands as given (below 1 it is raised to 1), marked explicit at the gate; other out-of-range values are clamped to their bounds, the clamp reported at the gate. NL intent maps to flags ("two hops, only papers" →--expand 2 --include arxiv,doi,/paper; "only the parts about X" →--focus "X"; "show me 10" →--shortlist 10); unstated → the defaults in Flags. - Thin-brief clarify (search mode): a brief too thin to plan from — fewer than two derivable sub-questions, or scope/recency/depth genuinely ambiguous — gets up to THREE pointed questions ONCE, before any discovery spend; otherwise unstated choices become assumptions stated in the run-spec echo, and the gate stays the correction point. Never re-interrogate a brief that already answers them; the clarify OUTRANKS the survey probe (owner answers first, the probe serves residual unfamiliarity, never an unanswered clarify).
Flags
--search "<brief>" · --seed · --expand N — citation hops beyond the fetched base (default:
seed 1 · search 0; no fixed clamp — the page caps bound it; --max-depth accepted as a legacy
alias; link hops, unrelated to ingest's compile depth) · --focus "<topic>" — semantic
relevance filter (implied by the brief in search mode) · --rounds N — search-mode iterative
deepening (default 1, max 3) · --max-pages N (default 10; hard ceiling 100) · --shortlist N —
full-detail rows at the Discover gate (search mode; default min(12, remaining budget); an explicit N stands) ·
--queries N — search queries per round (search mode; default 2 per brief facet within 3–12;
an explicit N stands) · --same-domain · --include a,b / --exclude c,d (seed mode: URL substring
patterns for the classifier · search mode: domain names for the search engine, non-domain patterns
applied as substring post-filters) · --yes (skip the Step-4 previews and the seed --expand 0
batch preview — the Discover gate always waits) · --ingest (compile after capturing) ·
--no-ledger (bookkeeping continues in full — write path, gates, rounds, so anti-repeat keeps its
data; only cross-round AUTHORITY is off, numbers quoted from the transcript instead; authority is
otherwise auto-on when --rounds > 1 or budget ≥ 30).
Capability catalog (complete — every lever, none prescribed)
Scripts (.claude/skills/gather/; arithmetic and enforcement live here, never re-derived in
prose):
funnel_knobs.py --max-pages N --facets F [--shortlist N] [--queries N] [--pages-captured N | --ledger-id ID] [--advice N] [--date-window] [--json]— derives and prints every funnel number for a round: queries band, menu, pool, date-check policy, clamps, explicit values, budget warnings.gather_links.py <seed.md> --seed-url "<url>" --max-pages N [--same-domain] [--include a,b] [--exclude c,d] --json— the link classifier: returnsexpand/maybe/skipwith the page caps pre-applied. Heuristics live in the script (docs/papers/repos/READMEs expand; nav/ads/ login/social skip; a skip-path hit on the seed's OWN documentation-shaped host demotes tomaybe, so a docs login guide reaches the owner while a commercial login wall stays out).capture_write.py— three verbs.write --url U --engine E --ledger-id ID [--title T] [--probe-result P] < fetched.md: the SOLE write path (budget + ceiling enforced against the ledger, refuses existing paths, sanitises, stamps provenance frontmatter, appends to the ledger, runs the capture quality gate;--allow-degraded '<why>'for a declared acceptance). Every write also records the capture check's stamp,capture_check:in frontmatter and ledger: with--probe-result(capture_probe.py'sresult.jsonfor the page) the body is compared with the page the probe fetched, whichever engine produced it; without one the stamp readsnot run — no probe result given. The check warns and never blocks: nothing it finds, and no missing, unreadable, malformed or failed result, refuses a write, and a check that did not run is stampednot run, never as no loss.check: the quality gate alone on stdin — pre-write probe, no write, no ledger.dedup --urls u1,u2 [--titles 'T1' 'T2'] --control <known-present URL>: vault de-dup oversource_url/converted_fromacrossraw/+wiki/, plus the arXiv ids and DOIs in those URLs and each normalised title againstwiki/sources/+raw/frontmatter, with the key code in the ingest skill'sdedup_probe.py(a paper compiled from a local PDF has nosource_url) (candidates comma- or whitespace-separated, titles one argument each; the control URL must hit or the scan aborts — proof the probe ran, CLAUDE.md §11;--allow-no-controlonly for a provably fresh vault, declared). Beside those definitive keys runs one advisory key: the same host (www.dropped) and the same final path segment (case-folded; a trailing slash,.htmland.htmdropped; query and fragment ignored) as a raw capture'ssource_url,converted_fromorresolved_urlprints a second line,<url> ~> likely duplicate of <raw path> (same host and final slug …), while the->line stays the verdict. It catches a publisher page at a moved address, and it also joins versioned docs, generic index pages and query-selected items, so it is never definitive and never excludes a row. A title naming a review site that meets a page by a stripped key alone prints an advisory line in place of a match (step 1.3).capture_write.py, the browser-rung options:write … [--resolved-url R]stampsresolved_url:besidesource_url:when the body came from an open copy, anddedupgrepsresolved_url:too;check [--engine E] [--url U]adds the abstract-page rule when E nameschrome-burner, whichwriteapplies to such an engine as well and--allow-degradednever lifts.capture_probe.py— the structural routing for a web page, and the capture check.capture "<url>" --work <a new directory outside the vault>fetches the page once with curl (its exit status kept beside the HTTP status, content type and effective URL) and routes and converts that same copy: a page whose HTML carries a collapsed or tabbed block (adetailselement; ahidden,aria-hidden="true",display:noneorvisibility:hiddencontainer; acollapseoraccordion…class; atablistortabpanelrole; an mkdocs tabbed set; a target a collapsed toggle or a tab controls) goes throughpython3 -m markitdownwith each tab label set as a heading directly before its own panel (curl+markitdown (structural route)), any other page through defuddle on the fetched copy (curl+defuddle). It then compares the capture with the fetched original, never its own normalised copy: every collapsed text in its block's own order, every value counted (a cell holding only a check mark or a cross, and the sign of< 101), every tab label directly before its own panel, no relative link left (relative ones are resolved against the effective URL and any<base>first). The comparison warns and never blocks. Its stamp readssuspected loss — <findings>;no loss found by the unit check (<counts>); not compared: …(it reads only collapsed blocks and tab panels and sets aside literal punctuation Markdown also uses and empty-cell positions in tables, so it never says the capture is complete); ornot run — <why>(no result, an unreadable, malformed or failed one, a page the probe never fetched, tabs it cannot pair, a check that failed to run). It writescapture.mdandresult.jsonin that directory and nothing else. Exit 0: a capture was made, its stamp printed whatever the check found; exit 2 isPROBE FAILED, no capture made (an HTTP error however long its body, a failed curl under any status, a non-HTML response, ambiguous tab pairing, a failed converter or built-in control).capture_probe.py stamp <result.json> --body <file> --url <url> --engine <engine>prints the provenance lines and the stamp for ingest Step 0's save (stdout only, exit 0 whatever the check found;converted_by:is the engine named, never the probe's;--allow-degradedadds a declaration);route,normaliseandcheckrun the steps one at a time. In a lane, whose fence cannot see inside a script, it honours the lane's grants itself: it fetches only a host the network grant names, follows no redirect (the target is reported instead) and writes only within the lane's write grants or/tmp.oa_resolve.py <doi|arXiv id|url|title> [--json] [--hand-fetch]— the open-access resolver (stdout only): Unpaywall, OpenAlex, the arXiv API and Europe PMC in order; printsRESOLVED <url> via <source>,UNRESOLVEDorPROBE FAILED(exit 0/1/2);--hand-fetchprints the hand-fetch block for an unresolved row. SetOA_CONTACT_EMAILto your own API contact address before using the resolver. A missing or malformed value is a prerequisite failure and makes no request.run_ledger.py—init --id <run-id> --budget N·add-gate·add-drop·add-round·raise-budget --id ID --to N --reason '<why>'(the audited budget-raise path;init --forceis for genuine restarts only and refuses once the ledger records any state).repo_pack.py— repo study:clonea shallow sha-pinned working copy into~/.llm-wiki/repos/(outside the vault, disposable), select paths AFTER reading, thenpack-and-writeone immutable commit-pinned pack through capture_write (exit-verified, never a pipe; caps + fence escalation in the script; contract: the repo-pack design; since 2026-09-07 also the project store's snapshot path:--repotakes an absolute working-copy path, a dirty tree or a credential-bearing remote is refused,--allow-largeis accepted and changes nothing, since no pack is cut or refused for size; before the consent gate, run--no-write: it writes a temporary pack, prints its path and bytes, and checks for warnings over 120 KiB or ingest's helper ceiling (ingest cannot compile a pack above that ceiling whole until it is split). Remove the temporary pack after use;--no-writedoes not call capture_write or write to raw/. The resolved file list with byte totals is the consent gate for a pack, and its cost line uses actual pack bytes/4, not the per-page default.
Capture chain (type-routed, then fallback — same as ingest Step 0, carve-out included): web
pages through the structural routing, never a bare defuddle of the URL:
python3 .claude/skills/gather/capture_probe.py capture "<url>" --work <a new directory outside the
vault> fetches once, sends a page carrying a collapsed or tabbed block through markitdown
(curl+markitdown (structural route)) and any other through defuddle on the same fetched copy
(curl+defuddle), and compares the capture with the fetched page (see Scripts); its
capture.md is the body, written with --probe-result <work>/result.json and the engine the result
names · TED talk pages (ted.com/talks/…) as ingest Step 0's TED route: the structural route
keeps only the page's description, so capture the talk's YouTube upload by the YouTube route and
check it with python3 .claude/skills/ingest/ted_cue_check.py <the probe's page.html> <capture> ·
arXiv HTML / LaTeXML pages (arxiv.org/html/…, ar5iv) never through defuddle —
curl -sL the page to a temp file, then convert with python3 -m markitdown, exactly as ingest Step 0
routes them, because defuddle's Markdown converter crashes on LaTeXML and silently emits raw HTML ·
curl for raw/.md URLs · markitdown for binaries (PDF, Office, …) · repo study →
repo_pack.py (see Scripts) · Jina Reader as the
fallback when the routed engine fails (third-party routing — consented via the preflight, see the
envelope); for a page the probe made no capture of, the Jina body is written with that page's
--probe-result too, so it is compared with the page wherever the probe fetched one (Writes). The routing stays until
saved real pages, captured by both routes in an attended session, show defuddle keeping every
collapsed text and every tab label with its table; that evidence, not a defuddle release note, retires
it.
Discovery engines & probes (discovery informs the gate; only capture writes):
- WebSearch per derived query,
--include/--excludemapped to allowed/blocked domains. Results are US-region — say so when the topic suggests non-US sources (those may need direct URLs or the platform engines below). - Platform tier (only when
.agent-reach-onenables it; venv-only binaries — invoke via~/.agent-reach-venv/bin/<cmd>, incl.yt-dlp, never a guessed path; declare the tier withdoctor --json, venv on PATH):bili search "<q>" --type video -n 5(Bilibili) · the V2EX public API · RSS/Atom feeds via the tier-1 feedparser channel (~/.agent-reach-venv/bin/python3 -c "import feedparser; …"; blog/newsletter coverage). Hits are triaged identically, metadata-only. Default X/Reddit posture (tier 3 inactive): login-walled — locate posts via WebSearch, read via the tier-1 Jina channel. A 0-result or auth failure on ANY platform engine is an engine failure: report it, never a blank capture. Tier-3 governance: the agent-reach adoption design. - Bounded pre-consent probes, each declared at the gate it feeds: ≤6 throwaway WebFetch date
checks per round on unclear finalists (never a capture path; unknown → mark "date unverified") ·
≤2 locate probes per round (a
ghsearch, an arXiv listing lookup — when search returns only secondary coverage of a known primary artefact; discovery only) · ≤1 survey probe per run (one overview-shaped query whose result structure informs the sub-question list itself — perspective discovery for weak topic familiarity) · the seed host inventory probe (see rationale; onecurlofsitemap.xml, falling back to/llms.txt).
Control envelope (rigid — never relaxes, whatever the model)
Consent. Nothing is written before the owner's yes. Preview-and-confirm is the default; only
--yes skips the Step-4 previews and the seed --expand 0 batch preview — the search Discover
gate ALWAYS waits, even with --yes: discovered pages are unknowns and always get human curation.
An agent gap proposal (CLAUDE.md §6 — propose-only) becomes a run ONLY on the owner's explicit
yes; the consent covers the echoed run-spec alone; the agent never self-adds --yes. Under
delegation the same flow binds every agent, with three admission conditions — novelty verified
against the vault, essentiality for the live or a named future task, strong-agent judgement
(delegate skill §5, owner ruling 2026-08-29). The engine
preflight verifies the capture chain and declares any substitution UP FRONT ("defuddle absent →
Jina fallback") — third-party routing is consented before capture, never reported after. A web page
needs curl, defuddle and python3 -m markitdown: with one absent, the probe makes no capture of a page
it would convert (PROBE FAILED), so the preflight says so before the gate.
Approval semantics (pinned). A bare "approve" captures the MENU ONLY (the gate's prompt line
says so; if an explicit --shortlist made the menu exceed the remaining budget, it captures the
menu's top rows up to the budget, clamp reported). More takes a rule ("top N by relevance" — N
beyond the remaining budget is clamped and reported), a rule plus curation ("top 25, drop 9, add
31"), or row-by-row choices. An ambiguous rule is asked back, never guessed. Before capturing,
echo the RESOLVED set in one line (rows · count · cost) and record it in the ledger. A rule NEVER
carries across rounds — each round's gate is fresh ("same rule" must be said); an explicit DROP
persists — out of every later menu and rule resolution this run, re-listed only among the drops
("dropped at round 1") if re-surfaced, re-entering only when named explicitly.
Caps. --max-pages (default 10) and the non-overridable 100-page ceiling, cumulative
across rounds and hops — each repeat classifier invocation gets the REMAINING budget. Caps bound,
never oblige. No silent caps: every cap hit, clamp, drop, demotion, dedup and pool underfill
is reported at a gate with its reason. A budget raise is a mid-run constraint change — full
run-spec re-echo — and runs run_ledger.py raise-budget (ledger-audited; next round's funnel
reads the raised budget), NEVER init --force.
Ledger. Every run inits one; the write path requires it — a capture with a missing ledger is
STOP-and-ask, the declared repair a fresh init carrying the known state. Cross-round AUTHORITY
(feeding funnel_knobs.py --ledger-id, the mismatch rule, resume) engages when --rounds > 1 or
budget ≥ 30, unless --no-ledger. The ledger is evidence, never authority over the owner: any
ledger-versus-memory mismatch → STOP and ask; if the file goes missing mid-run, fall back to model
memory for DISCOVERY accounting only and DECLARE the fallback at the next gate.
Writes. capture_write.py is the sole write path — raw immutability is mechanical (it refuses
existing paths; never edit or delete existing raw bytes; only add new captures, CLAUDE.md §3.1's
"converted .md" provision). If the script is missing or broken, STOP and ask — never hand-write
a capture around it. Every capture carries full provenance frontmatter
(converted_from/converted_by/converted_on + source_url, keeping ingest's de-dup working).
Download body images to assets/ with relative paths. The capture quality gate (mechanical,
in the script) refuses unwritten: engine error stubs, Page-Not-Found shells, HTML-dominant bodies
(a crashed converter emitting markup), near-empty pages. Platform captures (tier-3 CLIs) pipe the
CLI's output VERBATIM inside a closed fenced block under a one-line header — the gate honours the
fence (fenced payloads sit outside the HTML heuristic; the sanitiser leaves them byte-exact) but
refuses a dominant fence that is a JSON error/empty payload or a served HTML page (a login wall),
and bare unfenced whole-body JSON. Treat any refusal as that engine failing and retry with the
next engine in the chain. A body you deliberately accept needs --allow-degraded '<why>' —
stamped into frontmatter + ledger and declared in the run report. A body the gate passes but you
know is damaged (lost spacing, flattened tables) takes the same flag, stamped declared — <why>.
The capture check warns and never blocks: a web page is written with the --probe-result of its
capture_probe.py capture run, a PROBE FAILED page's fallback body included, and every write records
the stamp capture_check: (suspected loss — <findings>, no loss found by the unit check (<counts>);
not compared: <what it does not compare>, or not run — <why>). The stamp never says the capture is
complete. The run report lists every capture stamped suspected loss, with its findings, and every web
page stamped not run, with its reason.
Reporting duties (gate content = informed consent — exact duties, never paraphrased away).
Every run echoes its parsed run-spec at the first gate (mode · seeds/brief · focus · expand ·
rounds · caps; search mode adds the brief's facet list, the declared provenance ranking frame,
every declared probe/engine/assumption, and the funnel_knobs.py block echoed VERBATIM, clamps
included — re-run with --advice N once triage has the count). A mid-run constraint change (caps,
date window, focus, mode — at an open gate or between rounds) re-echoes the FULL updated run-spec,
never just the changed value. The Discover gate prints the LITERAL queries run. Menu rows are NUMBERED (rules and curation
address rows by number) and carry: title · provenance class (·×k multiplicity where >1) · date (✓ = page-verified) · one-line
relevance · likely raw/ subfolder (informational — capture writes to the raw/ root; ingest
sorts). A coverage table (per sub-question: covered by which rows · still open) appears at
every gate and again at hand-off — approval is informed by what it buys AND what it leaves open.
The advice line states the judged worth-capturing count ("of the <pool> ranked, ~N look
genuinely worth capturing", echoed via --advice); advice widens what is SHOWN, never what a rule
may buy. The ranked remainder (one-liners: rank · title · provenance · date · relevance)
appears when the remaining budget OR the advice count exceeds the menu, rows beyond the remaining
budget carrying the block's raise marker (rules always cap at the remaining budget; capturing a
marked row needs an explicit budget raise); otherwise below-cutoff candidates are summarised in one
line ("+K ranked below — say 'show all'"), never silently dropped. Every dropped or deduped
candidate is listed with its reason. Every gate/preview shows the page count and a cost
estimate (assume ≈4k tokens per captured page + ≈2× that to compile, and state the assumption;
when the remainder is shown, price a full-budget rule too) and ends with ONE short
recommendation line — at a gate, the option the agent judges best and the trim it would make
("Recommend: approve all 20; drop rows 4/7 if trimming"); at hand-off, the next step. Advisory
only — it never self-executes.
Capture discipline. Capture, don't summarise: the chain captures verbatim; WebFetch is
triage-only (throwaway date checks) — NEVER a capture path. Never fill gaps with invented
content; mark anything uncertain unverified. A page whose whole chain fails (Jina included) is
reported — URL, engines tried — and its slot is never auto-backfilled from the remainder; offer a
follow-up rule or round instead (the date-window discard rule, mirrored). WebSearch unavailable →
say so and ask for seed URLs — never substitute model memory. Zero hits → print the queries run
(proof the probe ran) and offer reformulations.
Privacy & language. Skip anything behind a login or obviously private.
The Jina fallback routes URLs through a third party — don't use it for sensitive links. UK
English at COMPILE time, not capture time: non-English pages are captured VERBATIM in their
source language (raw/ is immutable evidence — CLAUDE.md §3.1); /ingest translates. Note the
source language in the gate row when known. Don't compile here — that's /ingest's job.
Logging. After a completed capture run, append ONE gather entry to wiki/log.md via shell
(cat >>): mode, seed URL(s) or the brief + the queries run (and rounds), any policy rule
approved at the gate, pages captured, where they landed in raw/, and that the batch awaits
/ingest. A preview-only or aborted run (nothing captured) is not logged. The run ledger
(/tmp/gather-run-<id>.json) may then be deleted — run-scoped scratch, never knowledge.
Judgement & rationale (procedure as goal + reason — judgement chooses the HOW, never the
WHETHER)
- Query craft. Goal: cover every sub-question, wide before narrow — unguided agents
over-narrow. Derive per sub-question one WIDE and one NARROW phrasing by default; across the
set cover the applicable expansion axes: terminology shift · entity-anchored (named
systems/people/orgs) · temporal (recency terms where freshness matters) · provenance-targeted
(phrasing that surfaces the source class sought). Add native-language queries when the topic's
primary literature is non-English (capture stays verbatim; ingest translates, §3.1). When the
scripted band overrides two-per-facet (floor 3 · cap 12), the band wins: keep every
sub-question represented first, then trim narrow variants. Round ≥2 queries narrow onto the
open gaps. The pool holds the top candidates by triage score up to the scripted pool value — a
cap, not a quota; underfill is reported with
--queries/--roundssuggested. - Triage & ranking. Metadata-only — no capture fetches at discovery; capture is what the owner consents to. Dedupe by URL, recording ×k multiplicity (a URL surfaced by k>1 distinct queries) as a declared centrality signal, not a verdict. Source trust is objective-relative, so declare the provenance ranking frame chosen for THIS brief ("operational how-to → official docs first"; "landscape scan → engineering blogs first") and rank under it — the frame is declared, never implicit. Classify provenance (official docs / paper / repo / engineering blog / news / aggregator); score relevance against the brief. Drop junk (listicles, undated marketing, off-topic) — every drop with its reason, per the envelope.
- Date windows & liveness. A hard date window is an explicit user-stated cutoff
("2026-04 onwards", "last three months") — a mere freshness preference ("recent work") is NOT
one. Under a hard window, EVERY menu row is date-verified pre-gate (the overrun past the ≤6
baseline declared at the gate) and rule-selected below-menu rows are date-checked at capture;
out-of-window pages are discarded unwritten, the spent fetch reported, the shortfall never
auto-backfilled — offer a follow-up rule or round. An UNDATED living surface (an active repo's
files, a vendor docs page) may pass the window on liveness evidence named in its row —
host/repo activity (e.g.
pushed_at) inside the window — while dated artefacts (posts, announcements) pass only on publication date. Backend-pain warn: when a hard window forces ≥8 in-window verifications in one run, or the run's own row/drop reasons record region- or language-skew for the topic, SAY SO at the gate and note that an opt-in agent-grade search backend (API-side date/domain filters: Exa/Tavily/Brave) is the designed, deliberately deferred remedy — surfacing the option is the duty; adopting it is the owner's call, never the run's. This warn is the deferred backend upgrade's live trigger (an IDEAS queue item) and must never rot out of the gate. - Seed inventory. Extraction strips site navigation, so a documentation host's structure
never reaches the classifier through body links alone. When the seed host serves
sitemap.xml, fetch it as a declared discovery-only probe (onecurl, never a capture path) and put BOTH counts in the run-spec echo ("<body count>linked from the seed body ·<sitemap count>on the host sitemap"); a missing, redirected, index-style or HTML-serving sitemap falls back to ONE further declared probe of/llms.txt(docs hosts increasingly serve it; precedent: developers.openai.com, 2026-08-25); only when both are absent or empty does the probe degrade to link-only discovery, the degradation STATED at the preview — it never aborts the run and never fabricates an inventory. The sitemap's focus-relevant pages reach the preview asmaybes, owner-curated; promotions count against the caps. - Rounds & gaps. From the round's captured material, list the brief's open gaps — derived from what was already read, no re-reads. No gaps → stop early and say so. The gap list IS the next round's facet list (printed at its gate) and gap-derived queries follow the same craft (2 per gap, within the band). Anti-repeat, ledgered: previously-issued queries are excluded from the new derivation ("excluded k repeat queries" reported at the gate), and a candidate already shown at an earlier gate this run returns annotated ("shown round n — not selected"), never presented as new — the dropped-row rule's softer sibling: drops stay OUT, passed-over rows return labelled. Every knob recomputes from the REMAINING budget, so later rounds shrink naturally; every round gets the same gate under the same semantics.
- Engine health. When a formerly-working engine fails on ordinary pages (rot, not a one-off), the run report says so; after the run, the engine's class-rule upgrade may apply automatically under the trusted-release criteria of the agent-reach adoption design, reported from → to — never mid-run, and never for a release failing the criteria (those are reported for the owner).
Pipeline (the funnel — order and gates; duties above)
0 — Scope (both modes)
Parse args + natural language into the run-spec (defaults from Flags). Run the engine
preflight and declare substitutions and the platform tier (per the envelope). Init the run
ledger: python3 .claude/skills/gather/run_ledger.py init --id <run-id> --budget <N>.
1 — Discover (search mode only)
1.1 State the brief's facets; run funnel_knobs.py (block echoed verbatim at the gate; re-run
with --advice after triage); one declared survey probe if familiarity is weak (the
thin-brief clarify outranks it); declare the provenance frame; derive queries per the craft
rationale; run WebSearch — plus any enabled platform engines, each named in the run-spec
echo — into the pool.
1.2 Triage on result metadata (multiplicity, provenance, relevance, dates — the hard-window
policy and the backend-pain warn live in the rationale); state the advice count.
Routing (parity gate G5, 2026-09-02): (applies under the multi regime, owner-set or head-resolved for the run under delegation auto; in single the head runs the step itself unless an instrument-rule reason holds — thin-lanes phase 4, 2026-09-04; delegation auto 2026-09-04) the metadata triage runs in a read-only verifier-shaped lane, at the class's per-call pick (delegate skill §2), blind to any expected shortlist, handed the pool WITH each hit's title, snippet, date and URL (a bare-URL pool loses exactly the signal-less rows, the gate's one caveat), the brief's facets, the declared provenance frame and the budget; it returns keep (ranked, one-line ground each), drop (ground each), anomalies (pool contamination, unverifiable rows, disclosed model priors) and its controls. The head reviews the lists, runs 1.3 and presents the Discover gate itself (1.4): the owner-facing step never routes.
1.3 Vault de-dup pre-check: capture_write.py dedup --urls … --titles '<title>' … --control <known-present URL>
(pass each hit's title, so a paper the vault compiled from a PDF still matches); matches shown as
"already in vault" with the key that matched, never proposed. A ~> likely duplicate of <raw
path> (same host and final slug …) line is not a match: the row stays on the menu, marked
"likely duplicate of <raw path>", for the owner to keep or drop; it is never excluded and never
counted as already in the vault. A title "…" ~> paper or its review: <path> … line is not a
match either: a title naming a review site (… | OpenReview) met the page by a stripped key
alone, and the site's name cannot say which it is. The row stays on the menu, marked "paper or
its review of <path>", for the owner to keep or drop (a review is never a duplicate, CLAUDE.md
section 3).
1.4 Present the Discover gate (contents per the envelope's reporting duties; never skipped,
even with --yes). Approval per the pinned semantics; record it in the ledger (add-gate;
every explicit drop → add-drop; the round's literal queries + shown rows → add-round,
"shown" = every row printed at this gate, menu and displayed remainder alike — feeding
Step 6's anti-repeat). With --expand 0 (the search default) this approval IS the capture
approval → approved rows go straight to Step 5. With --expand ≥ 1, approved rows enter
Steps 2–4 as seeds AND are captured at Step 5.
2 — Fetch the seed(s)
Capture each seed with the type-routed chain (a web page through
capture_probe.py capture, its capture.md the seed Markdown); save the seed
Markdown to a temp file OUTSIDE the vault (e.g. /tmp — not a vault write) for Step 3. Run the host inventory probe (per the
rationale). Base pages (seeds, or search-mode approved rows) are themselves captures: written at
Step 5, counted against the caps. Seed --expand 0 skips Steps 3–4 — the given URLs are the
whole capture set (batch-capture shorthand), previewed ONCE with the cost estimate BEFORE
anything is fetched (unless --yes), the de-dup's verdicts on the rows (a likely duplicate marked
and still approvable, as at 1.3); the inventory probe STILL RUNS under --expand 0 (a
discovery read, not a capture) and its counts join that one batch preview, its sitemap pages
promotable like any maybe.
3 — Plan (deterministic)
Run gather_links.py so every gather applies the SAME rules. The script's output is never
altered; nothing is silently dropped. With --focus, a labelled semantic pass runs AFTER the
classifier: each expand/maybe link annotated for topical fit, demotions proposed — both
verdicts shown side by side at the preview.
4 — Preview & confirm (expansion hops — skip only with --yes; the --expand 0 paths carry
their own approval)
Show the plan + cost estimate, headed by the run-spec, a row the de-dup marks as a likely duplicate
carrying that mark and its raw path. The owner approves / prunes / adjusts
caps / reclassifies the maybes and any focus demotions; promotions count against --max-pages
— the run never exceeds it. Under --yes: the conservative disposition — maybes skipped,
demotions applied.
5 — Capture (into raw/)
Fetch each APPROVED link (routed chain → Jina fallback; whole-chain failures and engine rot per
the envelope and rationale). Write every page through capture_write.py write — the sole write
path; a web page with --probe-result <work>/result.json and the engine its result names (a
fallback's body with the same --probe-result), its stamp reported as the Writes envelope says. Then score every capture where it is written, the same line as ingest Step 0 3c:
python3 -B .claude/skills/ingest/conversion_score.py "raw/<path>.md" — a capture carries
converted_by:, so it is a conversion whatever its extension and is scored like one. One verdict
line per capture with five measures (four text arms and recovery) and a scope; a web capture reads scope: other (a PDF, or an arXiv-HTML page taken through markitdown, reads scope: pdf), so a
suspect here is carried into the run report under Suspect conversions: with its scope and
measures (read the line: identifiers and product names fire it, real damage — lost tables, lost
inter-word spaces — does too) and raises no flagged: line by itself; the page that later compiles
the capture takes ingest 3c's decision. A capture of a TED talk page (ted.com/talks/…) also takes
python3 .claude/skills/ingest/ted_cue_check.py <the probe's page.html> <capture> before it is
written: exit 3 writes only with --allow-degraded 'TED check: <n> transcript lines missing' or
re-takes the capture by the TED route, and any result but a pass goes into the run report under
Suspect conversions: (exit 2 as undecidable, with its reason). The verdict is reported, not written back into the capture:
capture_write.py stays the sole write path. For --expand > 1, repeat Steps 3–5 on the newly
captured pages, re-previewing each hop (unless --yes).
6 — Rounds (search mode, --rounds > 1)
Per the rounds rationale: open gaps → the new facet list → re-enter Step 1, re-running
funnel_knobs.py with --ledger-id <run-id> (the ledger supplies pages-captured; mismatch →
STOP and ask; authority off → the --pages-captured flag, stated). Same gate, same semantics,
cumulative caps; no rule carries over; explicit drops persist.
7 — Hand off to ingest
The captures sit in the raw/ inbox. Offer /ingest (or chain with --ingest). Report what
was captured, skipped, the running page/cost total, and — search mode — the final coverage table
(sub-questions covered · still open); a facetless seed run reports plan-versus-captured instead.
For a synthesised report, follow with /query (filed into the wiki) or /output (a
deliverable) — discovery + capture here, reporting there. Then log per the envelope.
Relationship to the other skills
gather→ builds the Raw layer (search-driven or seed-driven multi-link capture intoraw/).ingest→ compilesraw/into linkedwiki/pages (run it after gather).query/output→ the reporting half of deep research: cited answers/deliverables from the compiled pages. gather + ingest + query/output together replace external deep-research tools.
Standing derivative:
output/user-notes/gather-ingest-quick-reference.md— a change to this skill's user-facing workflow (commands, flags, gates, report shape) updates that note in the same pass (§2 output contract).
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

