streaming-csv-json-normalizer

Streams a delimited text file into row objects, and streams row objects back out to CSV. Every value is a string unless you explicitly ask for a cast, and every row-level problem is reported as data rather than as a crash.

node scripts/index.js inspect  data.csv
node scripts/index.js parse    data.csv --output data.json
node scripts/index.js normalize data.csv --trim --lowercase email --sort name
node scripts/index.js stringify data.json out.csv --columns name,email

When to use

• A file or an export ends in .csv, .tsv, .txt, or is comma/semicolon/tab/pipe separated and you need JSON.
• You need to know a file's shape before committing to it (rows, columns, duplicate or blank headers, ragged rows, empty cells).
• You need row-level errors with source line numbers instead of a parser stack trace.
• You need to clean a messy export: surrounding whitespace, mixed case keys, empty-string cells, over-long values, duplicate rows, unsorted rows.
• You need 007, 1e3 or true to become 7, 1000 or true, and you want the report to prove which columns were cast.
• You need to go back to CSV after an agent edited the JSON, without losing fields that only appear on later rows.
• The CSV is arriving on stdin (curl ... | node scripts/index.js parse -).

When not to use

• The file is XLS/XLSX, ODS, or a PDF table: convert it to CSV first.
• The file is not UTF-8 (latin-1, UTF-16, Shift-JIS): transcode it first; only UTF-8, with or without a BOM, is supported.
• The CSV has more than one table, or the header spans several rows: flatten it first.
• You need spreadsheet semantics: formulas, 001 displayed as a number, locale decimal commas, or type inference from the cell format. This tool has none of that.
• You need fuzzy or approximate header matching, dedup by business key, or schema inference. Those are your code's job, not this tool's.
• You need to load a 200k-row CSV into a database: use parse --output and then a real importer.

Choose a workflow

1. Start with inspect when the delimiter, header quality, row shape, or file size is unknown. Check truncated, ragged_rows, duplicate/generated columns, and the error count before using the data.
2. Use parse to preserve source values as strings. Name columns explicitly with --coerce only when numeric or boolean conversion is intended, and review the conversion summary for skipped or emptied values.
3. Use normalize for deliberate transformations such as trimming, selected lowercase conversion, empty-value handling, deduplication, or sorting. Keep the original file and inspect the reported changes before replacing any downstream source.
4. Use stringify when the JSON records are ready to return to a delimited format. Specify columns when order is part of the contract and check missing_cells when records have different keys.

Never treat a truncated result as a full dataset. For imports, financial data, identifiers, or other high-impact values, preserve the raw source and validate the transformed output against the receiving system's expectations.

Detailed data-handling procedure

1. Protect and identify the source

Confirm that the file is the intended export and choose an output path that does not overwrite it. CSV often contains personal or business-sensitive records; inspect only the fields and sample needed for the task. The tool reads local files or stdin and does not fetch remote resources itself. If a pipeline downloads data, verify the source and keep credentials out of shell history.

2. Profile before transforming

Run inspect first when the delimiter, header, or row shape is uncertain. Review columns, duplicate_columns, generated_columns, ragged_rows, empty_values, errors, and truncated. A detected delimiter is a guess based on the first line only; supply --delimiter if that line is atypical or the report shows every row is ragged. Confirm the first non-empty line really is the header because there is no headerless mode.

If headers are blank or duplicated, inspect the source and decide whether the generated/renamed column names still identify the right values. Do not assume the original header wording can be reconstructed from generated names.

3. Parse while preserving meaning

Use parse when converting rows to JSON without cleanup. Values remain strings unless --coerce is requested. That protects identifiers such as 007, long account numbers, ZIP codes, and values with leading zeros. Coercion chooses a kind by majority in the selected column; mismatches remain strings and are reported. Review coercion counts before passing mixed-type output downstream.

Ragged rows are represented with padded or dropped fields and listed in errors. Long-row extra fields are not retained in the returned row object. For important data, correct the export or handle malformed rows explicitly rather than treating the normalized rows as lossless.

4. Normalize with an explicit change list

Enable only transformations the user asked for. Trimming changes values; lowercasing should be limited to columns where case is semantically insensitive; mapping empty cells to null changes their type; truncating long values loses information; sorting and deduplication change row order or row count. --dedupe compares whole rows only, so it is not a business-key merge. Use changes to report exactly what the tool changed, and retain the source for audit or rollback.

5. Export and verify

For stringify, pass --columns when the receiving system requires stable column order. Check missing_cells, output bytes, delimiter, and EOL. Objects and arrays are serialized as JSON text and null becomes an empty cell, so a CSV round trip is not a general reversible encoding. Verify a sample of important rows and confirm the receiving application accepts the output.

If people will open the CSV in spreadsheet software, consider formula-injection risk from untrusted cells beginning with formula-significant characters. This tool does not add spreadsheet-safe escaping; use a policy appropriate to the consumer before distributing such a file.

Scenario playbooks

Quick export review

Run inspect; summarize row count, detected delimiter, headers, malformed rows, and whether the result is truncated. Do not print sensitive sample rows unless needed.

Agent-friendly JSON conversion

Run parse with an output file and pretty formatting only if a human needs to read the file. Keep values as strings unless type conversion was requested. Include errors and truncation in the handoff so downstream reasoning knows which rows may be incomplete.

Cleaning a contact or event export

Profile first, then use targeted --trim, selected --lowercase, and --empty-as options. Avoid case-folding identifiers or names without explicit approval. If sorting matters, name the column and verify null behavior and truncation before claiming global order.

Returning edited records to CSV

Use stringify after reviewing the JSON structure. Specify an explicit column list when downstream schema/order matters, and check the reported missing-cell count. Keep JSON alongside CSV if nested objects or arrays must remain machine-readable.

Completeness checklist

Before reporting a conversion as complete, confirm the run was not truncated, the intended delimiter and header were used, ragged rows were reviewed, any casts and transformations are described, and the output path was produced. If any of those conditions cannot be established, label the result partial or lossy and explain why.

Requirements

• Node.js >= 20 (uses node:stream, Buffer.subarray, Readable.iterator).
• Dependencies are already installed in the repository: csv-parse (streaming reader) and csv-stringify (streaming writer). Nothing is downloaded at run time and this skill never touches the network.
• No input file is modified. --output and stringify <output.csv> create or overwrite exactly one file, whose path comes from your argument.

Output contract

Success (exit code 0, one line of JSON on stdout, nothing on stderr):

{"data": <object described per command below>}

Failure (exit code 1, one line of JSON on stderr, nothing on stdout):

{"error": "<human readable message>"}

Nothing else is ever written to stdout or stderr: no banners, no progress logs, no stack traces. --help prints {"data":{"usage":"..."}} and exits 0. --version prints {"data":{"name":"@unreadlogs/skill-streaming-csv-json-normalizer","version":"0.1.0"}} and exits 0.

inspect

{"data":{"input":"data.csv","bytes":4063232,"bytes_human":"3.9 MB","delimiter":",",
"columns":["id","name","email"],"duplicate_columns":[],"generated_columns":[],
"truncated":false,"truncated_reason":null,"error_count":0,"rows":199998,
"empty_values":12,"ragged_rows":2,"errors":[]}}

• delimiter is the sniffed character; a tab is reported as the string "\\t".
• columns is the de-duplicated header. duplicate_columns lists names that appeared more than once, generated_columns lists the column_<n> placeholders created for blank header cells. ragged_rows counts rows whose field count differs from the header, empty_values counts cells that are the empty string.
• rows is a count here (an array in parse/normalize).
• bytes is the number of bytes actually consumed, not ls -l.

parse

{"data":{"input":"data.csv","bytes":68,"bytes_human":"68 B","delimiter":",",
"columns":["name","email","age"],"duplicate_columns":[],"generated_columns":[],
"ragged_rows":1,"empty_values":0,"truncated":false,"truncated_reason":null,
"error_count":1,"output":"data.json",
"rows":[{"name":"Ada","email":"ada@example.com","age":"36"}],
"errors":[{"line":4,"code":"CSV_RAGGED_ROW","message":"expected 3 fields, got 2",
"expected":3,"got":2,"row":{"name":"Grace","email":"","age":""}}],
"coerced_columns":["age"],"coercion":[{"column":"age","to":"integer","converted":1,"skipped":0,"emptied":0}]}}

• rows is the array of row objects. errors is the array of row problems; each entry has a 1-based line counted from the physical file, including the header, so line: 4 is the fourth line of the file.
• output is the path written by --output, or null.
• coerced_columns and coercion are only populated by --coerce; otherwise both are empty arrays.

normalize

Same envelope as parse, plus rows_in and changes:

"changes":{"trimmed_values":2,"lowercased_values":2,"empty_values_as":"null",
"truncated_values":0,"duplicates_removed":1,"sorted_by":"name"}

empty_values_as is null unless --empty-as was passed; sorted_by is null unless --sort was passed.

stringify

{"data":{"input":"data.json","output":"out.csv","rows":2,"columns":["name","email","age"],
"delimiter":",","eol":"lf","missing_cells":0,"bytes":65,"bytes_human":"65 B"}}

Commands

Flag details:

• --delimiter <char> — exactly one character. Without it, , ; tab | are counted outside quotes in the first line and the winner wins (ties go to ,).
• --limit <n> — maximum number of data rows collected, 1..100000. Reaching the limit always sets truncated: true.
• --max-errors <n> — maximum number of row errors kept, 0..100000. Reaching it stops the scan with truncated_reason: "max_errors".
• --keep-empty — treat a blank line as a one-field row instead of skipping it. Such a row is ragged unless the file has exactly one column.
• --output <file> — parse and normalize write the row array (a bare JSON array, plus a trailing newline) to that path. inspect rejects --output; stringify takes its output path as a positional.
• --coerce <cols> — comma-separated column names, or nothing to disable. The cast kind (integer, float, boolean) is decided per column by majority over the whole column. Empty cells in a coerced column become null. Values that do not match the majority kind stay strings and are counted in skipped. An unknown column name is an error.
• --trim — trim surrounding whitespace from every value.
• --lowercase <cols> — comma-separated column names to lowercase.
• --empty-as null|empty — after trimming, an empty value becomes null or stays "".
• --max-length <n> — truncate any value longer than n characters.
• --dedupe — drop rows whose key/value list is identical to a row already kept.
• --sort <col> — sort ascending; numeric when both values are numbers, otherwise localeCompare with numeric collation. null sorts last.
• --columns a,b — stringify only. Explicit column order. A column no record has is an error; a record missing some columns contributes missing_cells and empty cells.
• --eol lf|crlf — stringify only, \n or \r\n.
• --pretty — 2-space indented JSON in the --output file. stdout is always one compact line.

Limits

Every partial result carries truncated: true and a truncated_reason of max_rows, max_errors, max_bytes or parse_error. A dataset is never silently cut short.

Examples

Profile a file before you trust it:

node scripts/index.js inspect exports/customers.csv

rows tells you how much you are dealing with, duplicate_columns and generated_columns tell you the header needed repair, ragged_rows tells you the file is inconsistent, truncated tells you whether you saw all of it.

Turn it into JSON and see the ragged rows with their real line numbers:

node scripts/index.js parse exports/customers.csv --output customers.json --pretty

{"data":{"input":"exports/customers.csv","bytes":68,"bytes_human":"68 B","delimiter":",",
"columns":["name","email"],"duplicate_columns":[],"generated_columns":[],
"ragged_rows":1,"empty_values":0,"truncated":false,"truncated_reason":null,
"error_count":1,"output":"customers.json",
"rows":[{"name":"Ada","email":"ada@example.com"},{"name":"Grace","email":"grace@example.com"}],
"errors":[{"line":4,"code":"CSV_RAGGED_ROW","message":"expected 2 fields, got 1",
"expected":2,"got":1,"row":{"name":",,","email":""}}],
"coerced_columns":[],"coercion":[]}}

Exit code 0 even though a row was malformed: the file is written and the problems are in data.errors with the line number you can open in an editor.

Clean it up without changing anything you did not ask about:

node scripts/index.js normalize exports/customers.csv --trim --lowercase email \
  --empty-as null --dedupe --sort name --output customers.clean.json

data.changes counts exactly what moved (trimmed_values, lowercased_values, duplicates_removed, sorted_by); if it is all zero, the file was already clean.

Cast a column on purpose:

node scripts/index.js parse orders.csv --coerce total --pretty

coercion: [{"column":"total","to":"float","converted":812,"skipped":3,"emptied":0}] tells you the column became numbers and that 3 cells did not match the majority kind and stayed strings. "007" is only ever a number if you name that column.

Pipe an export straight in and go straight back out:

curl -s 'https://example.internal/export.csv' | node scripts/index.js parse - --output out.json
node scripts/index.js stringify out.json roundtrip.csv --eol crlf

missing_cells: 0 proves the round trip lost nothing; a non-zero value means some record did not have a key that later records introduced.

Notes and limitations

• Values are always strings. "007" stays "007", "1e3" stays "1e3", "TRUE" stays "TRUE". --coerce is the only thing that casts, and it reports what it cast.
• The first non-empty line is always the header. There is no --no-header.
• Blank header cells become column_1, column_2, ... by 1-based field position. Repeated names become name, name_2, name_3, ... Both are reported in inspect; the original header text is never recovered, because the file does not contain it.
• Header names are trimmed of surrounding whitespace before comparison, so "email " and "email" collide and become email and email_2.
• Ragged rows are padded with "" or truncated to the header width, and every one is listed in errors. Extra fields on a long row are dropped: their content is in the row preview of the error entry, but not in rows.
• Delimiter sniffing looks at the first line only and counts only unquoted characters. A file whose first line has no delimiter at all falls back to ,.
• rtrim is on: trailing whitespace outside quotes is dropped from every field. Leading whitespace is preserved unless --trim is used.
• Quoting follows RFC 4180. An unclosed quote, or a stray quote in the middle of an unquoted field, is a hard parse error: rows parsed so far are still returned, with truncated_reason: "parse_error" and the parser's own message in errors.
• bytes counts bytes consumed from the input, so on a truncated run it is smaller than the file size on disk.
• stringify writes null and undefined as an empty cell and objects/arrays as JSON text ({"x":[1,2]}), which is not reversible to the original shape.
• stringify unions the keys of all records in first-seen order, so later records never silently lose a field, but a 10 000-key union is still a 10 000-column CSV.
• Nothing is sorted, deduplicated or deduplicated-by-key implicitly. --dedupe is exact whole-row equality only.

Troubleshooting

• input not found: x.csv / input is not a regular file — the path is wrong or is a directory/FIFO. - is the only way to read stdin.
• input is 52.0 MB which exceeds the 50.0 MB limit — the file is over the hard byte guard. Split it or raise your own limits by chunking.
• columns contains column_4 and you expected date — that header cell was blank, so the name could not be recovered.
• Every row is CSV_RAGGED_ROW — the file uses a delimiter that is not , ; tab |. Pass --delimiter '|' (or whatever it is).
• unknown command: ... or unexpected argument: ... — you are past the flags; run node scripts/index.js <command> --help.
• --coerce names a column that is not in the file: total — check inspect's columns array; the header cell was blank or renamed to column_n.
• truncated: true with truncated_reason: "max_rows" — the file is bigger than --limit; raise it or narrow the query upstream.