token-squeeze

This skill wraps a deterministic CLI. It does NOT ask you to compress text by hand — invoke the tool so results are reproducible and guard-checked.

WHEN IT PAYS (state this if the user aims it at live chat). Compressing text already in the context window saves nothing — those tokens were billed on arrival. token-squeeze only nets tokens when its output is reused: a saved prompt, a system prompt, a skill body, a doc pasted across many sessions. For a one-off message, skip it.

RUN IT. From the skill directory: npm install once (pulls gpt-tokenizer), then:

• node cli.js <file> — compressed text to stdout.
• node cli.js <file> --stats — adds a token-before/after + guard line on stderr.
• node cli.js <file> --json — structured {tokensBefore, tokensAfter, savedPct, subs, subsDetail, guard}; subsDetail is the per-substitution ledger ("what changed"), show it when the user asks what was done.
• node cli.js - --stats — read from stdin.
• add --clean only for text carrying junk whitespace (pasted logs, tables); it's off by default because it saves ~0% on normal prose. Exit code is non-zero if the guard trips, so it's safe to script.

WHAT IT DOES. Layer B (default): token-aware dictionary substitution — "in order to"->"to", "due to the fact that"->"because", "utilize"->"use", plus deletion of pure filler. Each dictionary entry is self-filtered at load against the tokenizer and dropped unless it strictly reduces tokens, so a bad entry can't make output worse. Layer A (--clean, opt-in): lossless whitespace reclamation.

GUARANTEES AND LIMITS (report honestly, don't oversell).

• Measured on the bundled corpus: ~25% overall, 37-42% on verbose prose, 13-17% on technical text, 0% on already-tight prompts (a tight prompt returns byte-identical). Run node test.js to reproduce.
• Protected spans are never altered: fenced/inline code, URLs, Windows paths, double-quoted strings, dotted identifiers (auth.js, example.com, decimals). A five-way guard verifies every run and fails loudly: numbers; negation/constraint words in ANY case (no/not/never/must/cannot + contractions); dotted identifiers; protected spans; idempotence (running the output through again must be byte-identical).
• Homonym-risky words are deliberately NOT in the dictionary (e.g. "leverage" — financial noun; "interface with" — UI noun phrase); a deterministic tool can't disambiguate, so it doesn't try. Same for phrases whose compression would delete a negation ("there is no doubt that") — the guard outranks the saving.
• Token counts use the o200k (tiktoken) tokenizer as a PROXY for Claude's tokenizer, which Anthropic does not publish. Real Claude savings will be close but not identical. For exact numbers, count via the Anthropic count-tokens API (needs an API key).
• Meaning-leaning, not string-lossless: substitutions preserve propositional content but change wording; emphasis from deleted filler is lost by design. Do not use where exact wording is contractual (legal text, quoted material) — those cases are protected only if quoted.

After running, show the user the savedPct and the guard result; if the guard fails, surface it rather than returning the output silently.

REPORTING FORMAT. Show before/after with the savedPct, then two short lists: Preserved (protected spans, negations, numbers — what the guard confirmed survived) and Dropped (pure-filler phrases removed entirely, not substituted — from subsDetail, the "-> (deleted)" entries). Never silently swap the user's text; the diff and the guard result are always shown.

FOR LONG REUSABLE PROMPTS (~200+ words). Beyond running the dictionary substitution, restructure the result as: INTENT (one sentence — what the prompt is for), CONSTRAINTS (bullets — the non-negotiable rules), CONTEXT (only what the model can't infer on its own). This is a manual restructuring step on top of the deterministic pass, not something the CLI does — do it only when asked to prep a prompt for reuse (a saved prompt, a system prompt, a skill body), not for a one-off message.

node cli.js --canary — MUST print CANARY PASS 15/15 before you trust a result.