chunk-text-extraction
Skill1 starsChanged 55 days agoMITView on GitHub
Deterministically slice a mainframe or IBM i (AS400) source file (COBOL, CBLLE, SQLCBLLE, CL, CLP, CLLE, JCL, PROC, copybook, BMS) into one raw-text file per chunk. USE FOR: extract CL or COBOL chunks as txt and prepare source text for section-doc-writer. DO NOT USE FOR: chunking, facts extraction, facts slicing, copybook expansion, or markdown generation.
In this repository17 other files it gives its agents.
- cobol-chunkingSkill
- cobol-facts-extractionSkill
- diagram-rendererSkill
- facts-slicingSkill
- functional-analysis-finalizerSkill
- functional-analysis-group-finalizerSkill
- functional-analysis-group-writerSkill
- functional-analysis-writerSkill
- group-doc-finalizerSkill
- qa-reviewerSkill
- requirements-finalizerSkill
- requirements-group-finalizerSkill
- requirements-group-writerSkill
- requirements-writerSkill
- section-doc-finalizerSkill
- section-doc-writerSkill
- technical-analysis-writerSkill
The file
We split this skill into the rules it gives an agent
5 rules. About 0 are common, 1 is distinctive, and 4 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈1
Distinctive · estimate
This author's own choices. Worth a look.
4
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 20
Do not use this skill for chunking (cobol-chunking), facts extraction (cobol-facts-extraction), per-chunk facts slicing (facts-slicing), …
- Very specific: line 30
No header, no chunk-id banner, no line-number gutter.
- Distinctive: line 80
When the user asks to extract chunks for more than one file, do not loop the python script inline.
- Very specific: line 121
Stale manifest: If the source was edited after cobol-chunking ran, the line numbers in the manifest no longer point at the right ranges.
- Very specific: line 122
Encoding mismatch: If the manifest claims utf-8 but the source is actually EBCDIC or CP1252, the extractor will still write bytes — they …
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Chunk Text Extraction
Deterministic phase-B sibling. Reads a source file + its chunk-manifest.json and emits one raw-text file per chunk under docs/_shared/<basename-with-ext>/chunks/<chunk_id_sanitized>.txt. No LLM calls — pure byte/line slicing, sha-gated against stale manifests.
The .txt files are the only raw-source view a downstream per-section subagent (section-doc-writer, qa-reviewer) needs. They sit beside facts-slices/ so a phase-C agent for chunk X reads exactly two files: its slice JSON and its .txt.
When to Use
- The user asks to "extract chunks as txt", "save each chunk to a file", "dump chunk source", or "give me the source for chunk_id X".
- Phase A has just (re-)written a
chunk-manifest.jsonand the matchingchunks/directory is missing or stale (different sourcesha256). - A single chunk needs to be re-extracted on demand (
--chunk-id <id>). - Preparing per-section inputs before launching phase-C subagents.
Do not use this skill for chunking (cobol-chunking), facts extraction (cobol-facts-extraction), per-chunk facts slicing (facts-slicing), markdown writing (phase C), or copybook expansion.
Output Contract
One UTF-8 text file per chunk at docs/_shared/<basename-with-ext>/chunks/<chunk_id_sanitized>.txt.
<chunk_id_sanitized>=chunk_id.replace("/", "__")— the default transformation. Same rule asfacts-slicing.- Duplicate-chunk-id disambiguation. Phase A should guarantee unique chunk ids; some real-world manifests violate that (e.g. two
data-division/FILLERrecords at different line ranges). On collision, the second+ occurrence is suffixed with__L<start_line>(and_<n>if even that collides) and aDUPLICATE_CHUNK_IDentry is recorded inerrors[]. The authoritative chunk-id → filename map is alwaysindex.json::chunks[].file— downstream agents MUST read that, not re-derive the filename from the chunk id. - Body = the raw source bytes for the inclusive 1-based line range
[chunk.start_line .. chunk.end_line]from the manifest, decoded with the manifest'sencoding_detected(defaultutf-8). - Line endings are normalized to
\nto keep re-runs byte-identical across OSes; the original column content (including significant trailing spaces inside columns 8–72) is preserved verbatim. - No header, no chunk-id banner, no line-number gutter. The companion slice JSON already carries
chunk.start_line/chunk.end_line. - Empty chunks (
start_line == end_line + 1due to upstream parsing edge cases) produce a zero-byte file and a singleerrors[]entry on stdout, but still write the file.
Alongside the per-chunk files, a single index is emitted at docs/_shared/<basename-with-ext>/chunks/index.json:
{
"index_version": "1.0",
"source_path": "repos/sample1/COBOL/DEMO100.CBL",
"source_sha256": "<64-hex>", // copied from manifest
"manifest_sha256": "<64-hex of the chunk-manifest.json bytes>",
"encoding": "utf-8",
"chunks": [
{
"id": "DEMO100/procedure-division/MAIN-PROCESSING/A000-MAIN",
"file": "DEMO100__procedure-division__MAIN-PROCESSING__A000-MAIN.txt",
"start_line": 22,
"end_line": 26,
"line_count": 5,
"byte_count": 187,
"sha256": "<64-hex of the emitted .txt bytes>"
}
/* one per chunk, sorted by start_line */
],
"errors": []
}chunks[].sha256 is the integrity gate downstream consumers use to detect a stale .txt without re-reading every file.
Procedure
- Resolve inputs. Locate the source file (same rules as upstream skills: relative to repo root first, then to
source_rootfromconfig/pipeline.yaml). Default manifest is atdocs/_shared/<basename-with-ext>/chunk-manifest.json. Reject if either is missing. - Stale-input check. Hash the source bytes on disk and compare with
chunk-manifest.json::sha256. On mismatch: exit code4, error on stderr, no.txtwritten. Re-runcobol-chunkingfirst. - Run the extractor. Invoke scripts/extract-chunks.py:
Flags:python .github/skills/chunk-text-extraction/scripts/extract-chunks.py ` --source repos/sample1/COBOL/DEMO100.CBL--chunk-id <id>— extract ONLY that chunk (must exist in the manifest); default is all chunks.--output-dir <path>— overridedocs/_shared/<basename-with-ext>/chunks.--source-root <root>— overridepipeline.yaml::source_root.--manifest <path>— override the default chunk-manifest location.--prune— delete any pre-existing.txtin the output directory whose chunk id is NOT in the current manifest (default: keep stale files and warn).--no-index— skip writingindex.json(useful when extracting a single chunk on demand).
- Report. The script prints
OK: wrote N chunks to <dir>plus total.txtwritten / skipped / pruned, the source/manifest sha256 prefixes, and any entries fromerrors[]. Do not echo the chunk bodies.
Batch / Parallel Extraction
When the user asks to extract chunks for more than one file, do not loop the python script inline. Use scripts/extract_chunks_batch.ps1:
pwsh -File .github/skills/chunk-text-extraction/scripts/extract_chunks_batch.ps1 `
-Paths repos/sample1/COBOL/*.CBLParameters (mirrors cobol-facts-extraction / facts-slicing):
-Files,-Paths,-Manifest <json|csv>— input selection.-SourceRoot,-OutputDir— per-run overrides.-Prune— pass--pruneto every child invocation.-Throttle <int>— defaults to[Environment]::ProcessorCount.-ResultsXml <path>— defaults tochunk_text_results.xml.-ExcludeExtensions <string[]>— case-insensitive extensions skipped when expanding-Paths. Default:.cob, .inp, .itt(copybooks and JCL inline-data are consumed by reference upstream, not chunked standalone).
The runner is deterministic (sorted inputs/results, no timestamps), captures per-file stdout/stderr/exit, persists results via Export-Clixml, and exits 1 if any child failed.
Ad-hoc helpers go in temp/. Same convention as the other phase-B skills.
Report only the summary line (DONE total=… ok=… failed=…) plus any failures, matching the single-file reporting contract in step 4.
Determinism Rules
- No LLM calls.
- Re-running on an unchanged source + manifest pair MUST yield byte-identical
.txtfiles AND a byte-identicalindex.json(sortedchunks[], stable key order, no timestamps). - Line endings always normalized to
\nin the output — the input may be CRLF, LF, or mixed; the emitted bytes are LF-only. - Filename sanitization is
chunk_id.replace("/", "__")by default. The only non-deterministic-looking case is duplicate chunk ids (a phase-A bug), where the second+ occurrence gets a__L<start_line>suffix — still deterministic given the same manifest. Chunk ids that exceed the OS path limit fail loudly withcode=CHUNK_ID_TOO_LONG. - Stale
source sha256(manifest vs. on-disk source) is a hard error, not a warning. Extracting against a manifest whose line numbers no longer match the source would silently mis-bind every chunk. --pruneis opt-in. Default behavior is additive (write/overwrite per-chunk files; never delete) so a partial run does not destroy a known-good directory.
Resources
- scripts/extract-chunks.py — the extractor (entry point).
- scripts/extract_chunks_batch.ps1 — parallel batch runner.
Pipeline Wiring
config/pipeline-dag.yaml phase B lists skills/chunk-text-extraction/scripts/extract-chunks.py as a sibling of cobol-facts-extraction's extract-facts.py. Both run per source after phase A, both consume the same chunk-manifest.json, and neither depends on the other — they can be parallelized freely. Outputs land at docs/_shared/<file>/chunks/<chunk_id>.txt (with / → __ filename sanitization).
Common Pitfalls
- Stale manifest. If the source was edited after
cobol-chunkingran, the line numbers in the manifest no longer point at the right ranges. Step 2 catches this. Always re-run phase A first. - Encoding mismatch. If the manifest claims
utf-8but the source is actually EBCDIC or CP1252, the extractor will still write bytes — they just may not be human-readable. Trust the manifest'sencoding_detectedand fix it in phase A if wrong. - CRLF on Windows checkouts. The extractor reads via universal-newline mode and writes LF, so the output is stable regardless of git's
core.autocrlfsetting. Do not introduce a--preserve-eolflag unless a downstream agent explicitly needs it. - Filename collisions. Phase A is supposed to guarantee unique chunk ids, but some legacy manifests have duplicates. This skill disambiguates by suffixing
__L<start_line>and emits aDUPLICATE_CHUNK_IDerror entry. Always look up filenames viaindex.json::chunks[].file, never by re-sanitizing the chunk id, or you will mis-bind to the wrong .txt for duplicated ids. --chunk-idwithout--no-index. Extracting one chunk without--no-indexwill overwriteindex.jsonwith a single-entry file. Pass--no-indexfor surgical single-chunk re-extraction.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

