agentleFS
HurricaHjzHurricaHjz/second-yourself/.claude/skills/gather/SKILL.md

gather

Skill150 starsChanged 7 days agoMITView on GitHub

Web capture into raw/, two modes, one pipeline. SEED mode: given URL(s), fetch them and (optionally) the relevant links they cite. SEARCH mode: given a topic and no URL, web-search it, triage the hits, and present a dated shortlist for approval before anything is fetched — the vault's native deep-research entry (topic → sources → wiki; the cited report then comes from ingest + query/output). Use on /gather, "gather sources on <topic>", "search the web for X and save the sources", "deep-capture these links", "grab this article AND the repos/papers it cites". Preview-and-approve gates by default; conservative caps + a hard 100-page ceiling; expert flags (--search/--seed, --expand, --focus, --rounds, --max-pages, --include/--exclude, --same-domain, --yes, --ingest). Big budgets flow through a ranked policy gate (--shortlist, --queries tune the funnel); a per-run ledger plus a scripted sole write path harden caps, raw immutability and provenance. Opt-in; does nothing until invoked. It only CAPTURES into raw/ — /ingest compiles. Writes only new raw files, their asset images and its own log entry (never edits existing raw content).

5 other projects use a version of this.

Adopted by

Most recently taken up by

The file

We split this skill into the rules it gives an agent

11 rules. 0 are common, 6 are distinctive, and 5 are very specific to this project.

  • 0

    Common

    Most of the 3 edited versions we read keep these.

  • 6

    Distinctive

    This author's own choices. Worth a look.

  • 5

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    One funnel: search wide → rank a pool → show a menu → the owner approves → capture.

    line 29
  2. Distinctive:

    No URL, a topic/question present → search mode.

    line 44
  3. Distinctive:

    Default X/Reddit posture (tier 3 inactive): login-walled — locate posts via WebSearch, read via the tier-1 Jina channel.

    line 196
  4. Very specific:

    Ledger: Every run inits one; the write path requires it — a capture with a missing ledger is STOP-and-ask, the declared repair a fresh init …

    line 235
  5. Very specific:

    STOP-and-ask, the declared repair a fresh init carrying the known state.

    line 236
  6. Distinctive:

    Every run echoes its parsed run-spec at the first gate (mode · seeds/brief · focus · expand · rounds · caps.

    line 263
  7. Very specific:

    Logging: After a completed capture run, append ONE gather entry to wiki/log.md via shell (cat >>): mode, seed URL(s) or the brief + the …

    line 298
  8. Distinctive:

    Backend-pain warn: when a hard window forces ≥8 in-window verifications in one run, or the run's own row/drop reasons record region- or …

    line 333
  9. Distinctive:

    Engine health: When a formerly-working engine fails on ordinary pages (rot, not a one-off), the run report says so.

    line 358
  10. Very specific:

    Run gather_links.py so every gather applies the SAME rules.

    line 407
  11. Very specific:

    STOP and ask; authority off → the --pages-captured flag, stated).

    line 439

“Common” is measured: kept by at least 6 in 10 of the 3 edited versions of this file we could read. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

gather — web capture into raw/ (seed mode + search mode)

Goal

Turn either a set of links (seed mode) or a bare topic (search mode) into a high-quality set of captured sources in raw/, without fan-out blow-ups, off-topic noise, or silent writes. /ingest then compiles. Capture is document-granular — whole pages, verbatim; salience is ingest's job.

The whole logic in three lines (single-letter symbols live in the scripts):

  • Two inputs: the owner's page budget, and the topic's uncovered sub-questions (round 1: the brief's facets; later rounds: the open gaps — same thing, next moment).
  • One funnel: search wide → rank a pool → show a menu → the owner approves → capture.
  • Three principles: numbers bound, never oblige (every cap is a cap, not a quota); nothing is written without the owner's yes, and noes outlive yeses (drops persist, rules don't); judgement declares itself, arithmetic is scripted (the agent states sub-questions/relevance/advice, the scripts derive and print every number).

How to read this skill. The CONTROL ENVELOPE is rigid — a smarter model obeys it identically. Procedure (how to phrase queries, rank, scope) is stated as goal + reason under JUDGEMENT & RATIONALE and applied with judgement — better models make better choices there. Judgement governs HOW a rationale entry is satisfied, never WHETHER: the imperative duties inside those entries (SAY SO · EVERY · STOP · never) bind exactly like the envelope. The CAPABILITY CATALOG lists every lever; a lever you don't know exists is one you can't reach for.

Modes & routing (deterministic — any agent, any model, same result)

  • ≥1 URL among the arguments → seed mode. Free text alongside = scope + focus.
  • No URL, a topic/question present → search mode. The text is the brief: goal, any include/exclude criteria, recency expectations.
  • Explicit --seed / --search "<brief>" overrides inference. Neither URL nor topic — or --seed with no URL — → ask. An explicit --shortlist or --queries stands as given (below 1 it is raised to 1), marked explicit at the gate; other out-of-range values are clamped to their bounds, the clamp reported at the gate. NL intent maps to flags ("two hops, only papers" → --expand 2 --include arxiv,doi,/paper; "only the parts about X" → --focus "X"; "show me 10" → --shortlist 10); unstated → the defaults in Flags.
  • Thin-brief clarify (search mode): a brief too thin to plan from — fewer than two derivable sub-questions, or scope/recency/depth genuinely ambiguous — gets up to THREE pointed questions ONCE, before any discovery spend; otherwise unstated choices become assumptions stated in the run-spec echo, and the gate stays the correction point. Never re-interrogate a brief that already answers them; the clarify OUTRANKS the survey probe (owner answers first, the probe serves residual unfamiliarity, never an unanswered clarify).

Flags

--search "<brief>" · --seed · --expand N — citation hops beyond the fetched base (default: seed 1 · search 0; no fixed clamp — the page caps bound it; --max-depth accepted as a legacy alias; link hops, unrelated to ingest's compile depth) · --focus "<topic>" — semantic relevance filter (implied by the brief in search mode) · --rounds N — search-mode iterative deepening (default 1, max 3) · --max-pages N (default 10; hard ceiling 100) · --shortlist N — full-detail rows at the Discover gate (search mode; default min(12, remaining budget); an explicit N stands) · --queries N — search queries per round (search mode; default 2 per brief facet within 3–12; an explicit N stands) · --same-domain · --include a,b / --exclude c,d (seed mode: URL substring patterns for the classifier · search mode: domain names for the search engine, non-domain patterns applied as substring post-filters) · --yes (skip the Step-4 previews and the seed --expand 0 batch preview — the Discover gate always waits) · --ingest (compile after capturing) · --no-ledger (bookkeeping continues in full — write path, gates, rounds, so anti-repeat keeps its data; only cross-round AUTHORITY is off, numbers quoted from the transcript instead; authority is otherwise auto-on when --rounds > 1 or budget ≥ 30).

Capability catalog (complete — every lever, none prescribed)

Scripts (.claude/skills/gather/; arithmetic and enforcement live here, never re-derived in prose):

  • funnel_knobs.py --max-pages N --facets F [--shortlist N] [--queries N] [--pages-captured N | --ledger-id ID] [--advice N] [--date-window] [--json] — derives and prints every funnel number for a round: queries band, menu, pool, date-check policy, clamps, explicit values, budget warnings.
  • gather_links.py <seed.md> --seed-url "<url>" --max-pages N [--same-domain] [--include a,b] [--exclude c,d] --json — the link classifier: returns expand / maybe / skip with the page caps pre-applied. Heuristics live in the script (docs/papers/repos/READMEs expand; nav/ads/ login/social skip; a skip-path hit on the seed's OWN documentation-shaped host demotes to maybe, so a docs login guide reaches the owner while a commercial login wall stays out).
  • capture_write.py — three verbs. write --url U --engine E --ledger-id ID [--title T] [--probe-result P] < fetched.md: the SOLE write path (budget + ceiling enforced against the ledger, refuses existing paths, sanitises, stamps provenance frontmatter, appends to the ledger, runs the capture quality gate; --allow-degraded '<why>' for a declared acceptance). Every write also records the capture check's stamp, capture_check: in frontmatter and ledger: with --probe-result (capture_probe.py's result.json for the page) the body is compared with the page the probe fetched, whichever engine produced it; without one the stamp reads not run — no probe result given. The check warns and never blocks: nothing it finds, and no missing, unreadable, malformed or failed result, refuses a write, and a check that did not run is stamped not run, never as no loss. check: the quality gate alone on stdin — pre-write probe, no write, no ledger. dedup --urls u1,u2 [--titles 'T1' 'T2'] --control <known-present URL>: vault de-dup over source_url/converted_from across raw/ + wiki/, plus the arXiv ids and DOIs in those URLs and each normalised title against wiki/sources/ + raw/ frontmatter, with the key code in the ingest skill's dedup_probe.py (a paper compiled from a local PDF has no source_url) (candidates comma- or whitespace-separated, titles one argument each; the control URL must hit or the scan aborts — proof the probe ran, CLAUDE.md §11; --allow-no-control only for a provably fresh vault, declared). Beside those definitive keys runs one advisory key: the same host (www. dropped) and the same final path segment (case-folded; a trailing slash, .html and .htm dropped; query and fragment ignored) as a raw capture's source_url, converted_from or resolved_url prints a second line, <url> ~> likely duplicate of <raw path> (same host and final slug …), while the -> line stays the verdict. It catches a publisher page at a moved address, and it also joins versioned docs, generic index pages and query-selected items, so it is never definitive and never excludes a row. A title naming a review site that meets a page by a stripped key alone prints an advisory line in place of a match (step 1.3).
  • capture_write.py, the browser-rung options: write … [--resolved-url R] stamps resolved_url: beside source_url: when the body came from an open copy, and dedup greps resolved_url: too; check [--engine E] [--url U] adds the abstract-page rule when E names chrome-burner, which write applies to such an engine as well and --allow-degraded never lifts.
  • capture_probe.py — the structural routing for a web page, and the capture check. capture "<url>" --work <a new directory outside the vault> fetches the page once with curl (its exit status kept beside the HTTP status, content type and effective URL) and routes and converts that same copy: a page whose HTML carries a collapsed or tabbed block (a details element; a hidden, aria-hidden="true", display:none or visibility:hidden container; a collapse or accordion… class; a tablist or tabpanel role; an mkdocs tabbed set; a target a collapsed toggle or a tab controls) goes through python3 -m markitdown with each tab label set as a heading directly before its own panel (curl+markitdown (structural route)), any other page through defuddle on the fetched copy (curl+defuddle). It then compares the capture with the fetched original, never its own normalised copy: every collapsed text in its block's own order, every value counted (a cell holding only a check mark or a cross, and the sign of < 101), every tab label directly before its own panel, no relative link left (relative ones are resolved against the effective URL and any <base> first). The comparison warns and never blocks. Its stamp reads suspected loss — <findings>; no loss found by the unit check (<counts>); not compared: … (it reads only collapsed blocks and tab panels and sets aside literal punctuation Markdown also uses and empty-cell positions in tables, so it never says the capture is complete); or not run — <why> (no result, an unreadable, malformed or failed one, a page the probe never fetched, tabs it cannot pair, a check that failed to run). It writes capture.md and result.json in that directory and nothing else. Exit 0: a capture was made, its stamp printed whatever the check found; exit 2 is PROBE FAILED, no capture made (an HTTP error however long its body, a failed curl under any status, a non-HTML response, ambiguous tab pairing, a failed converter or built-in control). capture_probe.py stamp <result.json> --body <file> --url <url> --engine <engine> prints the provenance lines and the stamp for ingest Step 0's save (stdout only, exit 0 whatever the check found; converted_by: is the engine named, never the probe's; --allow-degraded adds a declaration); route, normalise and check run the steps one at a time. In a lane, whose fence cannot see inside a script, it honours the lane's grants itself: it fetches only a host the network grant names, follows no redirect (the target is reported instead) and writes only within the lane's write grants or /tmp.
  • oa_resolve.py <doi|arXiv id|url|title> [--json] [--hand-fetch] — the open-access resolver (stdout only): Unpaywall, OpenAlex, the arXiv API and Europe PMC in order; prints RESOLVED <url> via <source>, UNRESOLVED or PROBE FAILED (exit 0/1/2); --hand-fetch prints the hand-fetch block for an unresolved row. Set OA_CONTACT_EMAIL to your own API contact address before using the resolver. A missing or malformed value is a prerequisite failure and makes no request.
  • run_ledger.py — init --id <run-id> --budget N · add-gate · add-drop · add-round · raise-budget --id ID --to N --reason '<why>' (the audited budget-raise path; init --force is for genuine restarts only and refuses once the ledger records any state).
  • repo_pack.py — repo study: clone a shallow sha-pinned working copy into ~/.llm-wiki/repos/ (outside the vault, disposable), select paths AFTER reading, then pack-and-write one immutable commit-pinned pack through capture_write (exit-verified, never a pipe; caps + fence escalation in the script; contract: the repo-pack design; since 2026-09-07 also the project store's snapshot path: --repo takes an absolute working-copy path, a dirty tree or a credential-bearing remote is refused, --allow-large is accepted and changes nothing, since no pack is cut or refused for size; before the consent gate, run --no-write: it writes a temporary pack, prints its path and bytes, and checks for warnings over 120 KiB or ingest's helper ceiling (ingest cannot compile a pack above that ceiling whole until it is split). Remove the temporary pack after use; --no-write does not call capture_write or write to raw/. The resolved file list with byte totals is the consent gate for a pack, and its cost line uses actual pack bytes/4, not the per-page default.

Capture chain (type-routed, then fallback — same as ingest Step 0, carve-out included): web pages through the structural routing, never a bare defuddle of the URL: python3 .claude/skills/gather/capture_probe.py capture "<url>" --work <a new directory outside the vault> fetches once, sends a page carrying a collapsed or tabbed block through markitdown (curl+markitdown (structural route)) and any other through defuddle on the same fetched copy (curl+defuddle), and compares the capture with the fetched page (see Scripts); its capture.md is the body, written with --probe-result <work>/result.json and the engine the result names · TED talk pages (ted.com/talks/…) as ingest Step 0's TED route: the structural route keeps only the page's description, so capture the talk's YouTube upload by the YouTube route and check it with python3 .claude/skills/ingest/ted_cue_check.py <the probe's page.html> <capture> · arXiv HTML / LaTeXML pages (arxiv.org/html/…, ar5iv) never through defuddle — curl -sL the page to a temp file, then convert with python3 -m markitdown, exactly as ingest Step 0 routes them, because defuddle's Markdown converter crashes on LaTeXML and silently emits raw HTML · curl for raw/.md URLs · markitdown for binaries (PDF, Office, …) · repo study → repo_pack.py (see Scripts) · Jina Reader as the fallback when the routed engine fails (third-party routing — consented via the preflight, see the envelope); for a page the probe made no capture of, the Jina body is written with that page's --probe-result too, so it is compared with the page wherever the probe fetched one (Writes). The routing stays until saved real pages, captured by both routes in an attended session, show defuddle keeping every collapsed text and every tab label with its table; that evidence, not a defuddle release note, retires it.

Discovery engines & probes (discovery informs the gate; only capture writes):

  • WebSearch per derived query, --include/--exclude mapped to allowed/blocked domains. Results are US-region — say so when the topic suggests non-US sources (those may need direct URLs or the platform engines below).
  • Platform tier (only when .agent-reach-on enables it; venv-only binaries — invoke via ~/.agent-reach-venv/bin/<cmd>, incl. yt-dlp, never a guessed path; declare the tier with doctor --json, venv on PATH): bili search "<q>" --type video -n 5 (Bilibili) · the V2EX public API · RSS/Atom feeds via the tier-1 feedparser channel (~/.agent-reach-venv/bin/python3 -c "import feedparser; …"; blog/newsletter coverage). Hits are triaged identically, metadata-only. Default X/Reddit posture (tier 3 inactive): login-walled — locate posts via WebSearch, read via the tier-1 Jina channel. A 0-result or auth failure on ANY platform engine is an engine failure: report it, never a blank capture. Tier-3 governance: the agent-reach adoption design.
  • Bounded pre-consent probes, each declared at the gate it feeds: ≤6 throwaway WebFetch date checks per round on unclear finalists (never a capture path; unknown → mark "date unverified") · ≤2 locate probes per round (a gh search, an arXiv listing lookup — when search returns only secondary coverage of a known primary artefact; discovery only) · ≤1 survey probe per run (one overview-shaped query whose result structure informs the sub-question list itself — perspective discovery for weak topic familiarity) · the seed host inventory probe (see rationale; one curl of sitemap.xml, falling back to /llms.txt).

Control envelope (rigid — never relaxes, whatever the model)

Consent. Nothing is written before the owner's yes. Preview-and-confirm is the default; only --yes skips the Step-4 previews and the seed --expand 0 batch preview — the search Discover gate ALWAYS waits, even with --yes: discovered pages are unknowns and always get human curation. An agent gap proposal (CLAUDE.md §6 — propose-only) becomes a run ONLY on the owner's explicit yes; the consent covers the echoed run-spec alone; the agent never self-adds --yes. Under delegation the same flow binds every agent, with three admission conditions — novelty verified against the vault, essentiality for the live or a named future task, strong-agent judgement (delegate skill §5, owner ruling 2026-08-29). The engine preflight verifies the capture chain and declares any substitution UP FRONT ("defuddle absent → Jina fallback") — third-party routing is consented before capture, never reported after. A web page needs curl, defuddle and python3 -m markitdown: with one absent, the probe makes no capture of a page it would convert (PROBE FAILED), so the preflight says so before the gate. Approval semantics (pinned). A bare "approve" captures the MENU ONLY (the gate's prompt line says so; if an explicit --shortlist made the menu exceed the remaining budget, it captures the menu's top rows up to the budget, clamp reported). More takes a rule ("top N by relevance" — N beyond the remaining budget is clamped and reported), a rule plus curation ("top 25, drop 9, add 31"), or row-by-row choices. An ambiguous rule is asked back, never guessed. Before capturing, echo the RESOLVED set in one line (rows · count · cost) and record it in the ledger. A rule NEVER carries across rounds — each round's gate is fresh ("same rule" must be said); an explicit DROP persists — out of every later menu and rule resolution this run, re-listed only among the drops ("dropped at round 1") if re-surfaced, re-entering only when named explicitly. Caps. --max-pages (default 10) and the non-overridable 100-page ceiling, cumulative across rounds and hops — each repeat classifier invocation gets the REMAINING budget. Caps bound, never oblige. No silent caps: every cap hit, clamp, drop, demotion, dedup and pool underfill is reported at a gate with its reason. A budget raise is a mid-run constraint change — full run-spec re-echo — and runs run_ledger.py raise-budget (ledger-audited; next round's funnel reads the raised budget), NEVER init --force. Ledger. Every run inits one; the write path requires it — a capture with a missing ledger is STOP-and-ask, the declared repair a fresh init carrying the known state. Cross-round AUTHORITY (feeding funnel_knobs.py --ledger-id, the mismatch rule, resume) engages when --rounds > 1 or budget ≥ 30, unless --no-ledger. The ledger is evidence, never authority over the owner: any ledger-versus-memory mismatch → STOP and ask; if the file goes missing mid-run, fall back to model memory for DISCOVERY accounting only and DECLARE the fallback at the next gate. Writes. capture_write.py is the sole write path — raw immutability is mechanical (it refuses existing paths; never edit or delete existing raw bytes; only add new captures, CLAUDE.md §3.1's "converted .md" provision). If the script is missing or broken, STOP and ask — never hand-write a capture around it. Every capture carries full provenance frontmatter (converted_from/converted_by/converted_on + source_url, keeping ingest's de-dup working). Download body images to assets/ with relative paths. The capture quality gate (mechanical, in the script) refuses unwritten: engine error stubs, Page-Not-Found shells, HTML-dominant bodies (a crashed converter emitting markup), near-empty pages. Platform captures (tier-3 CLIs) pipe the CLI's output VERBATIM inside a closed fenced block under a one-line header — the gate honours the fence (fenced payloads sit outside the HTML heuristic; the sanitiser leaves them byte-exact) but refuses a dominant fence that is a JSON error/empty payload or a served HTML page (a login wall), and bare unfenced whole-body JSON. Treat any refusal as that engine failing and retry with the next engine in the chain. A body you deliberately accept needs --allow-degraded '<why>' — stamped into frontmatter + ledger and declared in the run report. A body the gate passes but you know is damaged (lost spacing, flattened tables) takes the same flag, stamped declared — <why>. The capture check warns and never blocks: a web page is written with the --probe-result of its capture_probe.py capture run, a PROBE FAILED page's fallback body included, and every write records the stamp capture_check: (suspected loss — <findings>, no loss found by the unit check (<counts>); not compared: <what it does not compare>, or not run — <why>). The stamp never says the capture is complete. The run report lists every capture stamped suspected loss, with its findings, and every web page stamped not run, with its reason. Reporting duties (gate content = informed consent — exact duties, never paraphrased away). Every run echoes its parsed run-spec at the first gate (mode · seeds/brief · focus · expand · rounds · caps; search mode adds the brief's facet list, the declared provenance ranking frame, every declared probe/engine/assumption, and the funnel_knobs.py block echoed VERBATIM, clamps included — re-run with --advice N once triage has the count). A mid-run constraint change (caps, date window, focus, mode — at an open gate or between rounds) re-echoes the FULL updated run-spec, never just the changed value. The Discover gate prints the LITERAL queries run. Menu rows are NUMBERED (rules and curation address rows by number) and carry: title · provenance class (·×k multiplicity where >1) · date (✓ = page-verified) · one-line relevance · likely raw/ subfolder (informational — capture writes to the raw/ root; ingest sorts). A coverage table (per sub-question: covered by which rows · still open) appears at every gate and again at hand-off — approval is informed by what it buys AND what it leaves open. The advice line states the judged worth-capturing count ("of the <pool> ranked, ~N look genuinely worth capturing", echoed via --advice); advice widens what is SHOWN, never what a rule may buy. The ranked remainder (one-liners: rank · title · provenance · date · relevance) appears when the remaining budget OR the advice count exceeds the menu, rows beyond the remaining budget carrying the block's raise marker (rules always cap at the remaining budget; capturing a marked row needs an explicit budget raise); otherwise below-cutoff candidates are summarised in one line ("+K ranked below — say 'show all'"), never silently dropped. Every dropped or deduped candidate is listed with its reason. Every gate/preview shows the page count and a cost estimate (assume ≈4k tokens per captured page + ≈2× that to compile, and state the assumption; when the remainder is shown, price a full-budget rule too) and ends with ONE short recommendation line — at a gate, the option the agent judges best and the trim it would make ("Recommend: approve all 20; drop rows 4/7 if trimming"); at hand-off, the next step. Advisory only — it never self-executes. Capture discipline. Capture, don't summarise: the chain captures verbatim; WebFetch is triage-only (throwaway date checks) — NEVER a capture path. Never fill gaps with invented content; mark anything uncertain unverified. A page whose whole chain fails (Jina included) is reported — URL, engines tried — and its slot is never auto-backfilled from the remainder; offer a follow-up rule or round instead (the date-window discard rule, mirrored). WebSearch unavailable → say so and ask for seed URLs — never substitute model memory. Zero hits → print the queries run (proof the probe ran) and offer reformulations. Privacy & language. Skip anything behind a login or obviously private. The Jina fallback routes URLs through a third party — don't use it for sensitive links. UK English at COMPILE time, not capture time: non-English pages are captured VERBATIM in their source language (raw/ is immutable evidence — CLAUDE.md §3.1); /ingest translates. Note the source language in the gate row when known. Don't compile here — that's /ingest's job. Logging. After a completed capture run, append ONE gather entry to wiki/log.md via shell (cat >>): mode, seed URL(s) or the brief + the queries run (and rounds), any policy rule approved at the gate, pages captured, where they landed in raw/, and that the batch awaits /ingest. A preview-only or aborted run (nothing captured) is not logged. The run ledger (/tmp/gather-run-<id>.json) may then be deleted — run-scoped scratch, never knowledge.

Judgement & rationale (procedure as goal + reason — judgement chooses the HOW, never the

WHETHER)

  • Query craft. Goal: cover every sub-question, wide before narrow — unguided agents over-narrow. Derive per sub-question one WIDE and one NARROW phrasing by default; across the set cover the applicable expansion axes: terminology shift · entity-anchored (named systems/people/orgs) · temporal (recency terms where freshness matters) · provenance-targeted (phrasing that surfaces the source class sought). Add native-language queries when the topic's primary literature is non-English (capture stays verbatim; ingest translates, §3.1). When the scripted band overrides two-per-facet (floor 3 · cap 12), the band wins: keep every sub-question represented first, then trim narrow variants. Round ≥2 queries narrow onto the open gaps. The pool holds the top candidates by triage score up to the scripted pool value — a cap, not a quota; underfill is reported with --queries/--rounds suggested.
  • Triage & ranking. Metadata-only — no capture fetches at discovery; capture is what the owner consents to. Dedupe by URL, recording ×k multiplicity (a URL surfaced by k>1 distinct queries) as a declared centrality signal, not a verdict. Source trust is objective-relative, so declare the provenance ranking frame chosen for THIS brief ("operational how-to → official docs first"; "landscape scan → engineering blogs first") and rank under it — the frame is declared, never implicit. Classify provenance (official docs / paper / repo / engineering blog / news / aggregator); score relevance against the brief. Drop junk (listicles, undated marketing, off-topic) — every drop with its reason, per the envelope.
  • Date windows & liveness. A hard date window is an explicit user-stated cutoff ("2026-04 onwards", "last three months") — a mere freshness preference ("recent work") is NOT one. Under a hard window, EVERY menu row is date-verified pre-gate (the overrun past the ≤6 baseline declared at the gate) and rule-selected below-menu rows are date-checked at capture; out-of-window pages are discarded unwritten, the spent fetch reported, the shortfall never auto-backfilled — offer a follow-up rule or round. An UNDATED living surface (an active repo's files, a vendor docs page) may pass the window on liveness evidence named in its row — host/repo activity (e.g. pushed_at) inside the window — while dated artefacts (posts, announcements) pass only on publication date. Backend-pain warn: when a hard window forces ≥8 in-window verifications in one run, or the run's own row/drop reasons record region- or language-skew for the topic, SAY SO at the gate and note that an opt-in agent-grade search backend (API-side date/domain filters: Exa/Tavily/Brave) is the designed, deliberately deferred remedy — surfacing the option is the duty; adopting it is the owner's call, never the run's. This warn is the deferred backend upgrade's live trigger (an IDEAS queue item) and must never rot out of the gate.
  • Seed inventory. Extraction strips site navigation, so a documentation host's structure never reaches the classifier through body links alone. When the seed host serves sitemap.xml, fetch it as a declared discovery-only probe (one curl, never a capture path) and put BOTH counts in the run-spec echo ("<body count> linked from the seed body · <sitemap count> on the host sitemap"); a missing, redirected, index-style or HTML-serving sitemap falls back to ONE further declared probe of /llms.txt (docs hosts increasingly serve it; precedent: developers.openai.com, 2026-08-25); only when both are absent or empty does the probe degrade to link-only discovery, the degradation STATED at the preview — it never aborts the run and never fabricates an inventory. The sitemap's focus-relevant pages reach the preview as maybes, owner-curated; promotions count against the caps.
  • Rounds & gaps. From the round's captured material, list the brief's open gaps — derived from what was already read, no re-reads. No gaps → stop early and say so. The gap list IS the next round's facet list (printed at its gate) and gap-derived queries follow the same craft (2 per gap, within the band). Anti-repeat, ledgered: previously-issued queries are excluded from the new derivation ("excluded k repeat queries" reported at the gate), and a candidate already shown at an earlier gate this run returns annotated ("shown round n — not selected"), never presented as new — the dropped-row rule's softer sibling: drops stay OUT, passed-over rows return labelled. Every knob recomputes from the REMAINING budget, so later rounds shrink naturally; every round gets the same gate under the same semantics.
  • Engine health. When a formerly-working engine fails on ordinary pages (rot, not a one-off), the run report says so; after the run, the engine's class-rule upgrade may apply automatically under the trusted-release criteria of the agent-reach adoption design, reported from → to — never mid-run, and never for a release failing the criteria (those are reported for the owner).

Pipeline (the funnel — order and gates; duties above)

0 — Scope (both modes)

Parse args + natural language into the run-spec (defaults from Flags). Run the engine preflight and declare substitutions and the platform tier (per the envelope). Init the run ledger: python3 .claude/skills/gather/run_ledger.py init --id <run-id> --budget <N>.

1 — Discover (search mode only)

1.1 State the brief's facets; run funnel_knobs.py (block echoed verbatim at the gate; re-run with --advice after triage); one declared survey probe if familiarity is weak (the thin-brief clarify outranks it); declare the provenance frame; derive queries per the craft rationale; run WebSearch — plus any enabled platform engines, each named in the run-spec echo — into the pool. 1.2 Triage on result metadata (multiplicity, provenance, relevance, dates — the hard-window policy and the backend-pain warn live in the rationale); state the advice count. Routing (parity gate G5, 2026-09-02): (applies under the multi regime, owner-set or head-resolved for the run under delegation auto; in single the head runs the step itself unless an instrument-rule reason holds — thin-lanes phase 4, 2026-09-04; delegation auto 2026-09-04) the metadata triage runs in a read-only verifier-shaped lane, at the class's per-call pick (delegate skill §2), blind to any expected shortlist, handed the pool WITH each hit's title, snippet, date and URL (a bare-URL pool loses exactly the signal-less rows, the gate's one caveat), the brief's facets, the declared provenance frame and the budget; it returns keep (ranked, one-line ground each), drop (ground each), anomalies (pool contamination, unverifiable rows, disclosed model priors) and its controls. The head reviews the lists, runs 1.3 and presents the Discover gate itself (1.4): the owner-facing step never routes. 1.3 Vault de-dup pre-check: capture_write.py dedup --urls … --titles '<title>' … --control <known-present URL> (pass each hit's title, so a paper the vault compiled from a PDF still matches); matches shown as "already in vault" with the key that matched, never proposed. A ~> likely duplicate of <raw path> (same host and final slug …) line is not a match: the row stays on the menu, marked "likely duplicate of <raw path>", for the owner to keep or drop; it is never excluded and never counted as already in the vault. A title "…" ~> paper or its review: <path> … line is not a match either: a title naming a review site (… | OpenReview) met the page by a stripped key alone, and the site's name cannot say which it is. The row stays on the menu, marked "paper or its review of <path>", for the owner to keep or drop (a review is never a duplicate, CLAUDE.md section 3). 1.4 Present the Discover gate (contents per the envelope's reporting duties; never skipped, even with --yes). Approval per the pinned semantics; record it in the ledger (add-gate; every explicit drop → add-drop; the round's literal queries + shown rows → add-round, "shown" = every row printed at this gate, menu and displayed remainder alike — feeding Step 6's anti-repeat). With --expand 0 (the search default) this approval IS the capture approval → approved rows go straight to Step 5. With --expand ≥ 1, approved rows enter Steps 2–4 as seeds AND are captured at Step 5.

2 — Fetch the seed(s)

Capture each seed with the type-routed chain (a web page through capture_probe.py capture, its capture.md the seed Markdown); save the seed Markdown to a temp file OUTSIDE the vault (e.g. /tmp — not a vault write) for Step 3. Run the host inventory probe (per the rationale). Base pages (seeds, or search-mode approved rows) are themselves captures: written at Step 5, counted against the caps. Seed --expand 0 skips Steps 3–4 — the given URLs are the whole capture set (batch-capture shorthand), previewed ONCE with the cost estimate BEFORE anything is fetched (unless --yes), the de-dup's verdicts on the rows (a likely duplicate marked and still approvable, as at 1.3); the inventory probe STILL RUNS under --expand 0 (a discovery read, not a capture) and its counts join that one batch preview, its sitemap pages promotable like any maybe.

3 — Plan (deterministic)

Run gather_links.py so every gather applies the SAME rules. The script's output is never altered; nothing is silently dropped. With --focus, a labelled semantic pass runs AFTER the classifier: each expand/maybe link annotated for topical fit, demotions proposed — both verdicts shown side by side at the preview.

4 — Preview & confirm (expansion hops — skip only with --yes; the --expand 0 paths carry

their own approval) Show the plan + cost estimate, headed by the run-spec, a row the de-dup marks as a likely duplicate carrying that mark and its raw path. The owner approves / prunes / adjusts caps / reclassifies the maybes and any focus demotions; promotions count against --max-pages — the run never exceeds it. Under --yes: the conservative disposition — maybes skipped, demotions applied.

5 — Capture (into raw/)

Fetch each APPROVED link (routed chain → Jina fallback; whole-chain failures and engine rot per the envelope and rationale). Write every page through capture_write.py write — the sole write path; a web page with --probe-result <work>/result.json and the engine its result names (a fallback's body with the same --probe-result), its stamp reported as the Writes envelope says. Then score every capture where it is written, the same line as ingest Step 0 3c: python3 -B .claude/skills/ingest/conversion_score.py "raw/<path>.md" — a capture carries converted_by:, so it is a conversion whatever its extension and is scored like one. One verdict line per capture with five measures (four text arms and recovery) and a scope; a web capture reads scope: other (a PDF, or an arXiv-HTML page taken through markitdown, reads scope: pdf), so a suspect here is carried into the run report under Suspect conversions: with its scope and measures (read the line: identifiers and product names fire it, real damage — lost tables, lost inter-word spaces — does too) and raises no flagged: line by itself; the page that later compiles the capture takes ingest 3c's decision. A capture of a TED talk page (ted.com/talks/…) also takes python3 .claude/skills/ingest/ted_cue_check.py <the probe's page.html> <capture> before it is written: exit 3 writes only with --allow-degraded 'TED check: <n> transcript lines missing' or re-takes the capture by the TED route, and any result but a pass goes into the run report under Suspect conversions: (exit 2 as undecidable, with its reason). The verdict is reported, not written back into the capture: capture_write.py stays the sole write path. For --expand > 1, repeat Steps 3–5 on the newly captured pages, re-previewing each hop (unless --yes).

6 — Rounds (search mode, --rounds > 1)

Per the rounds rationale: open gaps → the new facet list → re-enter Step 1, re-running funnel_knobs.py with --ledger-id <run-id> (the ledger supplies pages-captured; mismatch → STOP and ask; authority off → the --pages-captured flag, stated). Same gate, same semantics, cumulative caps; no rule carries over; explicit drops persist.

7 — Hand off to ingest

The captures sit in the raw/ inbox. Offer /ingest (or chain with --ingest). Report what was captured, skipped, the running page/cost total, and — search mode — the final coverage table (sub-questions covered · still open); a facetless seed run reports plan-versus-captured instead. For a synthesised report, follow with /query (filed into the wiki) or /output (a deliverable) — discovery + capture here, reporting there. Then log per the envelope.

Relationship to the other skills

  • gather → builds the Raw layer (search-driven or seed-driven multi-link capture into raw/).
  • ingest → compiles raw/ into linked wiki/ pages (run it after gather).
  • query / output → the reporting half of deep research: cited answers/deliverables from the compiled pages. gather + ingest + query/output together replace external deep-research tools.

Standing derivative: output/user-notes/gather-ingest-quick-reference.md — a change to this skill's user-facing workflow (commands, flags, gates, report shape) updates that note in the same pass (§2 output contract).

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.