forge-ingest

When this fires

Phase position: INGEST — first phase fired when the input is a multi-source-config.yaml (detected by forge-analyze). Replaces the single-input prep phase for multi-source runs.

After INGEST, the standard pipeline continues from AUDIT:

INGEST → AUDIT → SYNTH → SHAPE → PLAN_FIT → ... (unchanged)

Config format

version: 1
run_name: dental-pubmed-jan2026     # optional
domain: dental.ai.research           # optional override
budget_usd: 75                       # optional override

sources:
  - name: local-papers
    kind: local_dir
    path: ./Publications/
    enable_plugins: [archive, ocr]   # opt-in heavy tiers per-source

  - name: papers-archive
    kind: archive
    path: ./papers-2024.tar.gz

  - name: pubmed-snapshot
    kind: http
    url: https://example.org/pubmed-dental.jsonl
    sha256: abc...                   # optional verification

  - name: pubmed-live
    kind: database
    config: ./db-sources.yaml        # delegates to forge-ingest-db

  - name: prior-run
    kind: jsonl
    path: ./prior-runs/2025-12.jsonl

  - name: hf-corpus
    kind: hf_dataset
    repo: nlpaueb/dental-qa
    split: train
    text_field: question

options:
  dedup_across_sources: true          # default true
  min_len: 200
  max_len: 100000

Source kind: handlers

All cloud sources support optional include: / exclude: glob filters applied after the listing call.

Output

slm-forge/.runs/<run-id>/prepped.jsonl — same canonical schema as single-source forge-prep. Each chunk's metadata.source_file carries the source name as prefix (<source-name>:<original-path>) so provenance survives the merge.

slm-forge/.runs/<run-id>/ingest-stats.json — per-source breakdown + cross-source dedup count.

Cross-source dedup

When options.dedup_across_sources: true (default), a single SHA-256 set is maintained across all sources. So if the same paragraph appears in:

• a PDF in local_dir
• the same paragraph cached in a DB row from database

…only the first occurrence wins. Source order matters — the source listed FIRST in sources: is the canonical winner; later sources lose their duplicates. Stats record which source each dup was attributed to.

Within-source dedup ALWAYS runs (catches symlink loops + duplicated files inside a single source) regardless of the dedup_across_sources setting. The hash key is text for pretrain chunks and role:content\nrole:content... for chat chunks (role differences DO matter — same prompt with different system priming does not collapse).

Dry-run

bash slm-forge/skills/forge-ingest/run.sh <run-id> --dry-run

Parses the config, validates handler dispatch for every source, sizes locally-resolvable inputs, and prints a structured plan as JSON. NO downloads, NO writes, NO prepped.jsonl produced. Use this to preview "would this S3 prefix be 10 GB or 100 GB?" or "is the HF token I set actually valid for this gated dataset?" before committing to a long run.

Returns rc=0 on a clean plan, non-zero if any handler dispatch failed.

Disk-space pre-check

Before fan-out begins, the skill sums the bytes of all local_dir and jsonl sources and refuses (rc=4) if free disk at $RUN_DIR is less than 1.5x that estimate. Cloud / HF / DB sources are not pre-sized (can't be cheaply); operators should --dry-run first if unsure.

Re-run safety

If prepped.jsonl already exists in the run dir and is non-empty, the skill refuses to clobber it (rc=3). Override with FORGE_INGEST_FORCE=1 or use a fresh run-id.

Failure policy

Per source: failures (HTTP 404, DB unreachable, archive corrupt) are logged but do not abort the overall run. Other sources continue. The final ingest-stats.json carries failures: [{source, error, ...}].

If ALL sources fail OR the merged JSONL is empty, the skill exits 1 and the dispatcher tears down.