claude-watermark-remover

# CLAUDE.md

Guidance for Claude Code (or any agent) working in this repository.

## What this project is

Stdlib-first Python tool that removes AI provenance marks from text and
files. Two structurally separate halves — read the README's "What this
actually does" section first; it draws the exact lines this codebase
enforces in code, not just prose:

1. **Layer A** (`src/mark_hygiene/layer_a.py`) + **file cleaners**
   (`src/mark_hygiene/formats/*.py`) — deterministic, verifiable, byte-diff
   tested. A `confirmed` finding is provably gone from the output.
2. **Module 2** (`src/mark_hygiene/rewrite/`) — best-effort, model-dependent
   statistical-watermark disruption. Never certifiable. Kept in its own
   package on purpose; do not blur this boundary by, e.g., importing
   `rewrite/` into a file cleaner or vice versa.

## Non-negotiable conventions (read before editing)

- **Stdlib only** in `src/mark_hygiene/`. No new runtime dependency without
  discussing it with the user first. Optional external tools (`exiftool`,
  `qpdf`) are fine when shelled out to with an availability check and an
  honest degraded-mode fallback (see `formats/pdf.py`).
- **Fixtures first, byte-diff assertions.** Every format cleaner was built
  by constructing a real, structurally valid input containing the target
  marker (see `tests/*_fixture.py`), then asserting the marker's bytes are
  actually absent from cleaned output — never just "the function returned
  without raising." Follow this order for new format work.
- **Three-tier confidence classification**, not blanket removal:
  `confirmed` (near-zero false-positive risk, removed by default),
  `probable` (legitimate non-AI uses exist too, removed only with
  `--aggressive`), `informational` (never auto-removed). See
  `findings.py`. This exists specifically to keep false-positive rate
  measurable instead of asserted — don't collapse it back to a flat
  strip-everything rule.
- **C2PA detection is structural, not string matching.** `formats/jumbf.py`
  parses real JUMBF boxes and validates the 16-byte manifest UUID. A
  same-named container (PNG `caBX`, JPEG/WebP `APP11`/`C2PA`) with the
  wrong UUID must land at `probable`, not `confirmed` — see the README's
  "C2PA: manifest-structure validation" section for why a plain substring
  search was replaced (it both misses compressed PDF manifests entirely
  and false-positives on coincidental text).
- **Never reparse-and-reserialize a markup format you can't guarantee
  byte-identical output from.** SVG and HTML cleaning use regex byte-span
  removal specifically because round-tripping through a DOM/XML parser
  doesn't guarantee untouched content stays byte-for-byte untouched. If
  you're about to add a full parser for a new format, read the docstrings
  in `formats/svg.py` and `formats/docx.py` first.
- **Every documented gap states which kind of gap it is.** Two are
  currently real and must not be conflated: the PDF hard-bound C2PA
  attachment case (detection reliable, removal genuinely out of scope —
  needs a full PDF object-model library) vs. DOCX's `customXml/*` (not
  detected at all — no reliable signal, plus a real corruption risk).
  Read `formats/pdf.py` and `formats/docx.py` docstrings before touching
  either. If you add a new "detected but can't remove" case, state which
  kind it is explicitly, in code comments and in the README.
- **Module 2's report never claims removal.** `rewrite/rewriter.py`'s
  `RewriteReport` always carries a fixed `DISCLAIMER`. If you touch that
  package, keep language to "best-effort disruption," measured quantities
  only (tokens selected/changed, divergence, drift-guard pass/fail) — never
  "removed," "clean," or "detector will fail."
- **API keys: environment only, never argv.** `MARK_HYGIENE_REWRITE_API_KEY`
  is read from `os.environ` inside the CLI wiring; there is deliberately no
  `--api-key` flag. Keep it that way.
- **Atomic, symlink-safe writes everywhere.** Use `io_safety.safe_write_bytes`/
  `safe_write_text` for any new write path, not a raw `open(..., "w")`.

## Layout

```
src/mark_hygiene/
  layer_a.py          # Module 1: text Unicode scrubber
  io_safety.py         # binary-input guard, atomic/symlink-safe writes
  findings.py           # shared Finding dataclass + confidence constants
  file_clean.py         # magic-byte/structural-signature router -> formats/*
  formats/
    jumbf.py             # shared C2PA/JUMBF box parser
    ai_vendor.py          # shared AI-vendor-keyword check (markdown/docx/html)
    pdf.py, png.py, jpeg.py, webp.py, docx.py, svg.py, html.py, markdown.py
  rewrite/               # Module 2 -- separate package, see above
    stopwords.py            # closed-class stopwords + discourse-connector allowlist
    selection.py              # entropy-proxy token selection
    scoring.py                  # bigram-Jaccard divergence + drift guards
    backends.py                   # print-prompt / ollama / openai-compatible
    rewriter.py                     # orchestration + RewriteReport
  cli.py                # argparse: inspect/clean/inspect-file/clean-file/rewrite
tests/
  test_*.py              # one test file per module, mirroring src/ layout
  *_fixture.py            # fixture builders, imported by the matching test file
```

## Running things

```bash
python3 -m venv .venv && .venv/bin/pip install -e ".[dev]"
brew install exiftool qpdf   # needed for PDF cleaner + its end-to-end tests
make test                     # full suite
make smoke                    # quick CLI smoke check
```

CI (`.github/workflows/ci.yml`) runs the full suite on Python 3.10–3.13
with `exiftool`/`qpdf` installed, so PDF end-to-end tests aren't skipped
there — don't assume they're always skipped in CI when writing new ones.

## When asked to add a new format

1. Confirm whether it's deterministic/verifiable (→ `formats/`) or
   model-dependent (→ nowhere yet; this project has no third category —
   raise it with the user).
2. Write the fixture builder first (`tests/<format>_fixture.py`), with a
   real, structurally valid marked sample and a clean control sample.
3. Write the cleaner module: `inspect(data) -> list[Finding]` and
   `clean(data, *, aggressive=False) -> tuple[bytes, removed, reported]`,
   matching the signature convention every other `formats/*.py` module
   uses.
4. Route it in `formats/__init__.py::detect_format` and
   `file_clean.py::_BYTES_CLEANERS`.
5. Write the byte-diff test asserting marker absence, plus the
   confirmed/probable precision tests if the format has a JUMBF-style
   structural channel.
6. Update the README's coverage matrix and supported-formats table.