agentleFS
Sign inStart free
DshamirDshamir/slm-forge/skills/forge-source/SKILL.md

forge-source

Skill1 starsChanged 6 months agoMITView on GitHub

Ingests the raw corpus from manifest.spec.corpus_ref into the forge's S3 raw/ prefix. v1 implementation supports local: paths only; s3:/hf:/http: schemes are stubbed (exit 78) — use multi-source config.yaml + forge-ingest for those instead. Writes a per-shard manifest. Hard-caps at 500 GB per D-006.

We split this skill into the rules it gives an agent

3 rules. About 0 are common, 0 are distinctive, and 3 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈0

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 3

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    Read manifest.spec.corpus_ref and detect scheme (local: / s3: / hf: / http:)

    line 16
  2. Very specific:

    Write data/raw/_raw-corpus-manifest.json listing every shard + its bytes

    line 18
  3. Very specific:

    If manifest.artifacts.raw_corpus_s3 is set, exits 0.

    line 34

≈ “Common” is an estimate: at least 1 in 5 of the 308,232 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

forge-source

When this fires

Phase position: SOURCE — first data phase in the legacy v1 flow, between forge-intake and forge-curate. v2 runs use forge-prep or forge-ingest instead.

What it does

  1. Read manifest.spec.corpus_ref and detect scheme (local: / s3: / hf: / http:)
  2. For local:, walk the path, copy to s3://FORGE_BUCKET/forge/<id>/data/raw/
  3. Write data/raw/_raw-corpus-manifest.json listing every shard + its bytes
  4. Hard-fail if total bytes > 500 GB (D-006 S3 layout cap)
  5. Set manifest.artifacts.raw_corpus_s3 and advance to CURATE

Inputs

  • $1 = forge-id
  • manifest.spec.corpus_ref (currently local:/abs/path only)
  • FORGE_BUCKET env or default

Outputs

  • s3://FORGE_BUCKET/forge/<id>/data/raw/<shards>
  • s3://.../data/raw/_raw-corpus-manifest.json (sizes + sha256 of each shard)
  • manifest.artifacts.raw_corpus_s3
  • manifest.state.current_phase = CURATE

Idempotency

If manifest.artifacts.raw_corpus_s3 is set, exits 0.

Failure modes

ExitReason
0corpus uploaded or already-set
1local path missing OR corpus > 500 GB cap
64no forge-id provided
78scheme is s3: / hf: / http: (not yet implemented in v1; use multi-source config.yaml)

External resources

  • AWS S3 (writes raw shards)
  • Local filesystem (reads local: source)

Cost class

free — local I/O + S3 PUTs (negligible at the M2 size class).

Depends on

forge-intake (populated corpus_ref in spec)

See also

For non-local sources (HF dataset, S3 prefix, GCS, Azure, git, archive, HTTP), prefer the v2.2+ multi-source path: write a config.yaml and invoke via /slm-forge ./config.yaml <budget>. That route uses forge-ingest which has all 10 source kinds wired.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

Reviews can't be read right now.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

Check your setup, free