agentleFS
buildinternetbuildinternet/releases/.claude/skills/backfilling-sources/SKILL.md

backfilling-sources

Skill14 starsChanged 13 days agoApache-2.0View on GitHub

Backfill a changelog source's full history locally in Claude Code via the backfill-source / backfill-sweep dynamic Workflows — preflight-gated, window-capped, budget-gated extraction written through the idempotent /batch upsert, with no remote extraction inference bill. Use when a source has a lot of history to pull in and running the remote update workflow per window would be too expensive. Local Claude Code only.

The file

We split this skill into the rules it gives an agent

2 rules. About 0 are common, 1 is distinctive, and 1 is very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈1

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 1

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    Does NOT spend: no server-side extraction (the update workflow's incremental-Haiku / tool-loop passes), no metered Anthropic API bill.

    line 26
  2. Distinctive:

    Always dry-run first (the default) — it maps + estimates and writes nothing.

    line 27

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Backfilling Sources

Backfill a source's changelog history without the remote extraction inference bill (the update workflow's server-side Haiku / tool-loop passes), using the backfill-source dynamic Workflow (and backfill-sweep for several at once). The Workflow wraps the local-ingest primitives — preflight, fetch + extract, /batch upsert, parity rules — in a deterministic harness that owns the window cap, the budget gate, the dedup, and the safety gate, so the disciplines that are fragile when left to prose are enforced in code.

Local Claude Code only. The Workflow fans out agent() sub-agents and relies on a persistent local filesystem (~/.releases/work/) and the CLI's RELEASES_API_* env. It is not deployed to the managed-agent fleet.

When to use

  • A source has substantial history to backfill and running the remote update workflow per window is too expensive.
  • A remote fetch burned an extraction loop and wrote 0 releases — extracting locally sidesteps the loop.

When NOT to use

  • A clean feed / GitHub source — just add it (managing-sources) and let cron fetch it.
  • Inside a managed-agent session — this is local-only.
  • A publisher opt-out — the preflight gate refuses it (see below).

Cost contract

  • Spends: your Claude Code session tokens for the agent() sub-agents, hard-capped by the turn's budget.total (set with a +Nk directive). Extraction runs at Sonnet; the mechanical phases (preflight, run-setup, write, validate, report) run at Haiku.
  • Does NOT spend: no server-side extraction (the update workflow's incremental-Haiku / tool-loop passes), no metered Anthropic API bill. POST /v1/workflows/update is never called. /batch runs no AI on insert.
  • Always dry-run first (the default) — it maps + estimates and writes nothing.
  • When a dry-run is worth it: on index → detail sources, where it enumerates the per-release pages it would pull and shows how many are new vs already-ingested. For a known single-page source it adds little — it can only confirm shape/reachability, not preview record counts (those need extraction), and the workflow already routes single-page through a lighter recon path (no run-setup or known-URL agent on the way to the estimate). Going straight to dryRun: false with a turn budget is reasonable there.

Preflight gate (non-negotiable)

The Workflow runs local-ingest's preflight.ts first and fails closed: refuse (an ai-input=no opt-out — the gate is on ai-input only; ai-train=no alone proceeds) or a persistent unknown stops the run before any fetch or write. conductor.build (Content-Signal: ai-train=no, ai-input=no) is the regression target — it must be refused, via its ai-input=no. Override only with explicit, documented publisher permission.

Launch recipe

Launch by scriptPath, not by name — project .claude/workflows/ scripts are not registered in the Workflow name registry (only plugin-shipped workflows like deep-research resolve by name), so Workflow({ name: "backfill-source" }) errors not found (#1407). The path is repo-relative to the session cwd.

Dry-run one source, review the plan + counts, then commit:

Workflow({
  scriptPath: ".claude/workflows/backfill-source.ts",
  args: { source: "acme/changelog" },
}); // dry-run (default)
Workflow({
  scriptPath: ".claude/workflows/backfill-source.ts",
  args: { source: "acme/changelog", dryRun: false, maxReleases: 50 },
});

Set a turn budget to cap spend (e.g. prefix the request with +300k). Several sources at once:

Workflow({
  scriptPath: ".claude/workflows/backfill-sweep.ts",
  args: { sources: ["acme/changelog", "globex/releases"], dryRun: false },
});

args: source (slug | src_ id | URL), maxReleases (default 50), dryRun (default true), model (sonnet default; haiku for bulk/simple). Sweep takes sources: string[] and resolves each child backfill-source by the same sibling path (override with backfillScriptPath for a non-standard checkout layout).

Re-seed vs. top-up (scrape sources)

A dry-run showing skippedKnown=0 while the source already has rows is a signal, not a success: scrape-ingested releases are stored with url = null, so the workflow's known-URL dedup finds nothing to skip. A dryRun:false run would then write the overlap as new url'd rows alongside the old url-null rows — duplicates, not a gap-fill.

Correct path for the first backfill of an already-ingested scrape source:

  1. Hard-delete the existing rows to free the UNIQUE(source_id, url) dedup slot:
    releases admin release delete --source <slug> --hard
  2. Then run the backfill with dryRun: false.

Truly incremental re-runs (skippedKnown > 0) only happen after the source already carries per-release URLs from a prior backfill.

Caveat: releases tail --json does not expose url or content — only id/title/summary/publishedAt/source/contentChars/contentTokens. To confirm whether existing rows are url-null, use releases admin release get <id> --json.

Extraction altitude (granularity)

The same page can be split at very different altitudes — per feature, per period (month/quarter heading), per version, or one coarse rollup — and left unguided the extractor chooses per-run, so an unchanged page can yield different record counts on different runs. Pin it by setting the source's metadata.granularity once at onboard to one of feature | period | version | rollup. The workflow reads it (via resolve-source) into the extract prompt so the altitude is deterministic across runs, and the Report phase logs an altitude check when the record count runs high relative to the date span (≫4 records per covered month) and the altitude wasn't pinned to feature — surfacing over-splitting instead of letting it pass silently.

After a run

The Workflow records the run under ~/.releases/work/ in an isolated run dir it mints itself (summary per source, cost line grounded in budget.spent(), cross-run sweep report). It deliberately does not use releases admin work start / the shared .current-run pointer — that pointer is global and leaks across concurrent sessions (#1396), so an automated sweep that held it would absorb an unrelated session's mutations. A sweep threads its run dir to each nested source; per-source summaries are namespaced summary-<slug>.md to avoid clobbering. Because no pointer is set, these runs don't appear in releases admin work status. Review the cross-run report (reports/<date>-backfill-sweep.md) for data-quality findings (empty content, thin pages, deferred-for-budget pages). See maintenance-workspace.md → Concurrency.

Before declaring a source "clean" or "recovered" (required)

A dry run's notStored: 0 — or a top-up's skippedKnown matching every extracted URL — is a claim about what extraction found in the window it looked at, not proof the live page has nothing newer. During the 2026-07-23 → 07-28 outage recovery, dry runs across ten sources all reported new=0 and were read as "no content was lost." A commit run on one of those sources then inserted 5 rows, including a changelog entry dated Jul 22 that was visibly live on the source page the whole time (#2168 item 5b).

Before reporting any source "recovered," "up to date," or "nothing missing," fetch the live page yourself — through the same preflight gate above — and compare its newest 3-5 entries against what's actually stored (releases tail <slug> --json or admin release get). This applies whether the conclusion comes from one source or a whole sweep; a clean tool summary is a starting point for the check, not a substitute for it. If the two disagree, the tool number is what's wrong, not the page.

Media and content backfills (server-side, no local extraction)

The local Workflows above rebuild release rows. Four admin routes heal what's already stored — all idempotent, all dryRun-defaulting where noted, none on any cron:

  • POST /v1/workflows/backfill-media { sourceId?, all?, limit?, dryRun? } — R2-mirrors media stored before ingest-time mirroring was active (third-party URLs, no r2Key). This is the ONLY path that re-mirrors populated media — the ingest upsert never overwrites non-empty media. Bounded per call; remaining in the report tells you to loop. dryRun default.
  • POST /v1/workflows/backfill-video { releaseId?|sourceId?|all?, limit?, dryRun? } — re-runs inline-video detection (Wistia/Loom/Vimeo/YouTube) over stored bodies and APPENDS the type:"video" poster card to media[] (dedup by linkUrl). For rows ingested before #1549.
  • POST /v1/workflows/reextract-source { sourceId, snapshotId?, maxWindows?, dryRun? } — re-runs extraction from a stored raw snapshot (released-raw, 90-day lifecycle → 410 snapshot_expired) with no live scrape. Use when parse logic improved and you want history reprocessed without re-fetching.
  • POST /v1/workflows/refetch-release { releaseId, url?, dryRun?, force? } — heals ONE release in place from its live page (same rel_ id, title/content/date replaced, AI fields nulled for regen; media replaced only on extractor hit). A stored #fragment URL requires an explicit same-host url; placeholder content is refused, and a >50% body shrink needs force: true. The fix for thin index-scrape rows left by a crawl-mode flip.

Rule of thumb: wrong/missing media → the two media routes; stale extraction with captured raw → reextract-source; one bad row → refetch-release; missing history → the local backfill Workflows above.

Related

  • local-ingest — the primitives this wraps (preflight, /batch, parity); use it for one-off interactive ingests.
  • parsing-changelogs — the extraction conventions (type, dates, rollups) the extract prompt follows.
  • managing-sources — create the org/source first; naming, primary, playbooks.
  • maintenance-workspace.md — the ~/.releases/work/ run-recording convention.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.