agentleFS
Sign inStart free
DshamirDshamir/slm-forge/skills/forge-prep/SKILL.md

forge-prep

Skill1 starsChanged 6 months agoMITView on GitHub

Walks a directory of mixed documents and emits a unified canonical JSONL. Plugin-driven (v2.1) — handles ~30 file formats including documents, code, notebooks, tabular data, archives, images (OCR), audio/video (transcription), 3D meshes, and orphaned MySQL data dirs (v2.2 auto-revive). Runs as the first DATA-PHASE in the pipeline (after PREFLIGHT/ANALYZE/PLAN, before AUDIT). Skipped when input is already JSONL.

We split this skill into the rules it gives an agent

1 rule. About 0 are common, 0 are distinctive, and 1 is very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈0

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 1

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    Every plugin emits chunks in this shape (validated by prep_plugins/schema.py before write):

    line 48

≈ “Common” is an estimate: at least 1 in 5 of the 308,232 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

forge-prep

When this fires

Phase position: PREP — first data-phase. Skipped when input is already JSONL or a multi-source config.yaml (the latter routes to forge-ingest instead).

What it does

Dispatches each file in target_dir to a plugin in scripts/prep_plugins/ based on the file extension. Plugins are auto-discovered from the registry in prep_plugins/__init__.py and emit chunks in the canonical schema (see prep_plugins/base.py).

Plugin family (v2.1 + v2.2)

PluginSource formatExtensionsDefaultDisable env
pdfpdf.pdfon–
docx_plugindocx.docxon–
pptx_pluginpptx.pptxon–
text_simpletxt/md/html/....txt .md .markdown .mdx .rtf .html .htm .xml .tex .rst .org .logon–
epubepub.epubon–
notebookipynb.ipynbon–
codecode-*30+ language extensionson–
tabularcsv/xlsx/parquet.csv .tsv .xlsx .ods .parquet .pqon–
archivearchive.zip .tar .gz .tgz .bz2 .tbz2 .7z .rar (recursive, depth ≤ 3)on–
mesh_metadatamesh.stl .vtp .obj .ply (sparse, metadata-only)onFORGE_DISABLE_MESH_METADATA=1
binary_metadatabinary.bin .so .dll .exe .o .a .class .jar .db ... (sparse)onFORGE_DISABLE_BINARY_METADATA=1
ocrimage (OCR).png .jpg .jpeg .tif .tiff .bmp .gif .webp .heic .heifonFORGE_DISABLE_OCR=1
audioaudio (transcription).mp3 .wav .m4a .flac .ogg .opus .aaconFORGE_DISABLE_TRANSCRIBE=1
videovideo (transcription).mp4 .mkv .mov .avi .webm .wmv .flvonFORGE_DISABLE_TRANSCRIBE=1
mysql_revive (v2.2)mysql_revive.frm .myi .myd .ibd .ibdata (auto-revive via docker run mysql:8)on (needs docker)FORGE_DISABLE_MYSQL_REVIVE=1

Heavy plugins (ocr, audio, video) install their system deps lazily when their extensions appear in the corpus. The orchestrator does a dry-walk first to compute extraction_profile.plugins_needed so forge-prep/run.sh only materializes the install tier you actually need.

Canonical chunk schema

Every plugin emits chunks in this shape (validated by prep_plugins/schema.py before write):

{
  "id": "<source-stem>-<chunk-suffix>",
  "text": "...",
  "format": "pretrain",
  "metadata": {
    "source_file": "...",
    "source_format": "pdf|docx|...|mysql_revive|...",
    "section": "<top-level-dir>",
    "doc_title": "...",
    "chunk_type": "page|slide|paragraph|full|cell|row|ocr|transcript|metadata_only",
    "chunk_idx": 0,
    "char_count": 1479
  }
}

Sparse "metadata-only" chunks (mesh, binary, fallback paths) set chunk_type: "metadata_only" so forge-audit can filter them out if the operator wants to keep the training corpus dense.

Inputs

  • target_dir (from analysis.json)
  • run_id

Outputs

  • slm-forge/.runs/<run_id>/prepped.jsonl
  • slm-forge/.runs/<run_id>/prep-stats.json (per-plugin chunk counts)

Idempotent — if prepped.jsonl exists, skip and emit existing.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

Reviews can't be read right now.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

Check your setup, free