compressed-ndjson-boundaries
Skill0 starsChanged 39 days agoMITView on GitHub
Use when compressed NDJSON ingestion loses trailing records, fails at chunk boundaries, grows memory on long lines, or needs safe parallel decompression with simdjson and zstd.
In this repository66 other files it gives its agents.
- AGENTS.mdAGENTS.md
- postgres-locking-joinsSkill
- ai-bug-patternsSkill
- http-resume-integritySkill
- linux-io-latency-triageSkill
- macos-disk-space-recoverySkill
- minimal-sufficient-evidenceSkill
- modularize-large-filesSkill
- patch-reproducer-validitySkill
- prompt-cache-boundariesSkill
- proton-drive-selective-backupSkill
- transactional-outbox-deliverySkill
- conventional-commitsSkill
- git-advanced-mergingSkill
- git-bisect-forensicsSkill
- git-data-recoverySkill
- git-hooksSkill
- git-rerereSkill
- git-reset-mechanicsSkill
- git-revisions-syntaxSkill
- git-rewriting-historySkill
- git-submodulesSkill
- git-worktreesSkill
- github-ssh-keysSkill
- npm-publish-ci-pitfallsSkill
- sapling-stacked-pr-workflowSkill
- go-concurrency-patternsSkill
- go-data-idiomsSkill
- go-error-handlingSkill
- go-interfaces-embeddingSkill
- go-pprof-profilingSkill
- marketing-funnel-diagnosisSkill
- marketing-geo-experimentsSkill
- marketing-incrementalitySkill
- marketing-positioning-testsSkill
- python-descriptors-mroSkill
- python-logging-cookbookSkill
- python-scaling-regression-testsSkill
- python-unicode-gotchasSkill
- rust-compile-time-invariantsSkill
- rust-deterministic-concurrency-testsSkill
- rust-error-identity-boundariesSkill
- rust-regex-search-contractsSkill
- rust-runtime-bug-classesSkill
- rust-tokio-actor-deliverySkill
- rust-tokio-cancellation-safetySkill
- sales-discovery-qualificationSkill
- sales-mutual-action-plansSkill
- sales-pipeline-forecast-calibrationSkill
- sales-value-negotiationSkill
- ai-triage-evaluationSkill
- container-image-patchingSkill
- dependency-artifact-integritySkill
- dnssec-sentinel-diagnosticsSkill
- iam-policy-change-replaySkill
- rbvm-applicabilitySkill
- rbvm-asset-evidenceSkill
- rbvm-guarded-remediationSkill
- rbvm-priority-decisionsSkill
- rbvm-risk-exceptionsSkill
- The first 60 shown.
The file
Files
Compressed NDJSON boundaries
Compression frames, read chunks, and JSON records are different boundaries. Fixed chunks do not bound memory: an unfinished record, decoder window, parser, or work queue can still grow.
Boundary contract
| Layer | Required invariant |
| Decompressor | Check every zstd result; physical EOF is successful only after frame completion (ZSTD_decompressStream returns zero). Consume concatenated frames and reject trailing garbage. |
| Framer | Cut after the last LF; retain only the unfinished suffix. Impose a record-byte limit before growing/copying. Accept CRLF; explicitly choose blank-line and final-LF policies. |
| Parser | Keep input alive and padded by SIMDJSON_PADDING; batch size must accommodate the largest document. Check setup, iteration, field-access errors, then truncated_bytes() after successful exhaustion. |
| Publication | Stage results until compressed integrity and record validation succeed; otherwise earlier callbacks can leak partial success before a bad footer. |
Recipe
- Set independent caps: compressed input buffer, output chunk, record bytes, zstd window (
ZSTD_DCtx_setParameter(ctx, ZSTD_d_windowLogMax, 20)limits the window to 1 MiB), total decompressed bytes, workers, and queued chunks. Include parser allocations and application-retained results in the memory budget. - Parse complete lines; preserve split UTF-8 bytes until a complete record exists. Reject an oversized suffix instead of repeatedly doubling storage. At EOF, distinguish a permitted complete unterminated JSONL record from a truncated JSON value.
- With simdjson, destroy/exhaust the stream before moving/refilling its buffer. Copy retained strings; document references expire on iterator advancement. On-Demand skips validation of untouched values: fully traverse and validate every scalar and key when whole-document validity matters.
- Parallelize independent frames only when the writer guarantees record-aligned boundaries and bounded frame sizes. Check advertised content size before allocation; unknown size requires bounded streaming or rejection. Give each worker its own parser. A generic
.zstfile does not guarantee record alignment.
From repository root:
python3 skills/engineering/compressed-ndjson-boundaries/scripts/reproduce.pySources and validation limits: Lemire (2026-10-01), simdjson 5.0.1, zstd 1.5.7. Tests exercise installed zstd plus Python's JSON oracle; simdjson and parallel throughput remain untested.
Pitfalls
truncated_bytes()==0does not prove skipped values are valid.- Newline alignment does not authenticate input; checksums detect corruption, not attackers.
- Bounded queues cannot compensate for uncapped per-record/frame allocations.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.