agentleFS
Sign inStart free
DshamirDshamir/slm-forge/skills/forge-curate/SKILL.md

forge-curate

Skill1 starsChanged 6 months agoMITView on GitHub

Filters the raw corpus by length and dedups, normalizes to the unified {id, text, metadata} schema, writes per-shard curated JSONL to S3. Runs LOCALLY (not on EC2) — pure CPU work that doesn't justify GPU spend. Replaced by forge-audit on v2 runs (audit also covers slop / off-domain / safety).

We split this skill into the rules it gives an agent

3 rules. About 0 are common, 1 is distinctive, and 2 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈1

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 2

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    Write curated shards to s3://.../data/curated/shard-NNNN.jsonl

    line 20
  2. Distinctive:

    Write a curation stats file (input/output counts, rejection histogram)

    line 21
  3. Very specific:

    If manifest.artifacts.curated_corpus_s3 is set, exits 0.

    line 36

≈ “Common” is an estimate: at least 1 in 5 of the 308,232 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

forge-curate

When this fires

Phase position: CURATE — after forge-source, before forge-shape. v1-only phase; v2 runs use forge-audit which adds slop detection, off-domain filtering, and safety boilerplate scrubbing on top of the v1 length+dedup baseline.

What it does

  1. Stream-read every shard from s3://.../data/raw/
  2. Apply length filters (drops too-short and too-long) and SHA-based dedup
  3. Normalize each surviving doc to {id, text, metadata}
  4. Write curated shards to s3://.../data/curated/shard-NNNN.jsonl
  5. Write a curation stats file (input/output counts, rejection histogram)
  6. Set manifest.artifacts.curated_corpus_s3 and advance to SHAPE

Inputs

  • $1 = forge-id
  • s3://FORGE_BUCKET/forge/<id>/data/raw/* (shards from forge-source)
  • manifest.spec.domain (only used for the metadata tag, not filtering in M2)

Outputs

  • s3://.../data/curated/shard-NNNN.jsonl (each ≤ 100k docs)
  • s3://.../metadata/curation-stats.json
  • manifest.artifacts.curated_corpus_s3
  • manifest.state.current_phase = SHAPE

Idempotency

If manifest.artifacts.curated_corpus_s3 is set, exits 0.

Failure modes

ExitReason
0curated or already-set
1zero docs survive filtering (corpus is junk)
64no forge-id provided

A non-fatal warning is emitted when output / input < 10% (severe loss).

External resources

  • AWS S3 (read raw, write curated, write stats)

Cost class

free — runs on the operator's laptop; no GPU spend.

Depends on

forge-source (raw corpus already in S3)

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

Reviews can't be read right now.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

Check your setup, free