AGENTS.md — read this before changing anything in this repository

This file is the standing context for every agent (and every human) working on this codebase. Read it before proposing, writing, reviewing, or merging any change. It is not background reading — it is the rule that decides arguments.

⛔ THE SPEC-DRIVEN DEVELOPMENT MANDATE (Smart Ralph Protocol)

This repository operates strictly under Spec-Driven Development (SDD). Inspired by Smart Ralph, every change in this repository is spec-governed:

1. Spec First, Code Second: Every agentic change to the code must first declare the spec for the change and implement exactly what was declared. You must never modify, add, or delete .java files, configuration files, or build scripts without first declaring or updating the specification in the specs/ directory.
2. Strict Implementation Fidelity: Implement exactly what was declared in the specification. Do not add unrequested features, speculative abstractions, extra helper functions, or unsolicited refactorings.
3. Dual Verification Requirement: Every change must be verified against:
   • Empirical Tests: Unit tests, integration tests, and value-level comparisons.
   • Formal Verification Alignment: The formal Lean 4 replication model in formal_specs/lean/ proving that the change preserves log sequence monotonicity, version consistency, and convergence.
   • Spec Validation Tool: Run python3 scripts/validate_specs.py before submitting any PR.

See specs/CONSTITUTION.md for core invariants (I1–I13) and specs/SMART_RALPH_PROTOCOL.md for the complete agentic development protocol.

⛔ THE PRIME DIRECTIVE: MySQL is the source of truth

This project is a replication engine from a transactional source database (MySQL, PostgreSQL) into ClickHouse.

That single sentence settles most design questions in this repo, so state it plainly:

1. The source data is the truth, and the source is always correct. The source database defines what the data is. There is no scenario in which this connector decides the source is wrong. (Throughout this file "MySQL" stands for whichever source you are replicating; the rule is identical for PostgreSQL.)
2. The ClickHouse side must CONFORM to MySQL. ClickHouse is the replica. Its schema, its column set, its values, and its types exist to reproduce what MySQL holds.
3. Every mismatch is fixed by making ClickHouse match MySQL — always. A divergence is never resolved by adjusting, reinterpreting, or excusing the MySQL side, and never by declaring the ClickHouse state acceptable. The direction of the fix is fixed: ClickHouse moves to MySQL.
4. The connector is the ENFORCER of that rule, not a reporter of it. It does not merely detect divergence and describe it — it makes ClickHouse correct according to MySQL. When the ClickHouse side is in a shape that prevents the source value from being stored, the connector changes the ClickHouse side so it can be stored. Reporting is the fallback for when enforcement genuinely fails, never the default.

Enforcement, concretely

Enforcement is an action, so it has a shape. Whenever you find the replica unable to hold what the source sent:

• Correct the ClickHouse side automatically where a safe, bounded operation exists. A MATERIALIZED column shadowing a source value is fixed with ALTER TABLE ... MODIFY COLUMN <col> <type>, restating the type with no default expression — metadata-only, no part rewrite, on the same DDL path the connector already uses for schema evolution.
• Enforcement has two halves, and the second is easy to forget. Correcting the definition fixes the write path forward. Rows already written while the definition was wrong still hold the wrong values, and reconciling those is a backfill of the affected range. A change that fixes only the forward path is half-done — say so explicitly in the log and in the PR, so the remaining work is visible rather than assumed done.
• Log the enforcement, loudly. An automatic schema correction is a real event: record what was wrong, what was changed, and what still needs backfilling.
• Only when enforcement cannot be performed — the DDL is rejected, the privilege is missing, the type cannot be read — do you fall back to a warning naming the manual remediation.

Do not write code whose response to a divergence is only a log line. If you find yourself describing a fix for a human to apply, ask first whether the connector can apply it.

What this means when you are deciding something

Any time you are weighing a design choice, a bug fix, a schema rule, a conflict-resolution policy, or a "which side is right?" question, resolve it with the prime directive:

• If a value differs between MySQL and ClickHouse → the MySQL value is correct. Write the MySQL value. Do not preserve, prefer, or defend the ClickHouse value.
• If a column exists in MySQL but not usefully in ClickHouse → the ClickHouse side is wrong or incomplete. Fix the ClickHouse side.
• If a type or precision cannot round-trip → that is a defect in the ClickHouse mapping, not an acceptable loss. Report it loudly; never silently narrow, truncate, or default the value.
• If you are tempted to write "the connector correctly excludes X" — stop. Verify against MySQL first. "Correct" is defined by the source, not by what the current code happens to do.

Column-kind rules (ClickHouse side)

These follow directly from the prime directive and are the specific rules that have caused real production incidents:

The trap: a column that is MATERIALIZED in ClickHouse and present in MySQL looks, to code that only inspects the ClickHouse catalog, like a column ClickHouse owns. It is not. If MySQL sends it, it belongs to MySQL. Any code path that decides column membership must therefore ask what MySQL sent, not only what the ClickHouse metadata says.

Failures this directive is meant to prevent

Every one of these has actually happened in this codebase:

• A NULL-valued source column being dropped from the INSERT, so ClickHouse substituted a column DEFAULT (0 / '' / 1970-01-01) instead of NULL. Row counts matched; only a value-level checksum caught it.
• A newly added source column being replicated as schema but never backfilled, leaving a large historical range blank on the ClickHouse side while MySQL held the values.
• Treating a MATERIALIZED column as "correctly excluded" ClickHouse-owned state, rather than asking whether MySQL defines it.
• A per-record metadata re-read loop caused by mistaking a permanently ClickHouse-computed column for a stale cache.

The common shape: the connector inferred correctness from the ClickHouse side alone. Do not do that. Check the source.

Silence is the enemy

Because MySQL is the truth, any divergence is a defect — so it must be loud:

• Never swallow, mute, or downgrade a replication error to make a batch succeed. A failed batch that retries is recoverable; a batch that writes wrong data and reports success is not.
• Never let row counts stand in for correctness. Every incident above had matching row counts. Value-level comparison (db_compare/ checksum tooling) is the only proof of agreement.
• Never declare "in sync", "converged", or "false positive" on a checksum failure without comparing actual column VALUES against the MySQL host the checksum job locked, and naming why the checksum differed in the first place. "It just converged" is not an explanation.

Working rules for changes in this repo

1. Declare the spec first. Create or update the relevant document under specs/ before modifying code.
2. Verify against the live source, not the code. Reading a DDL file or a config does not tell you what MySQL holds. Query it.
3. Trace the full path before changing a flag. config → generator → template → consumer → runtime effect. A flag whose mechanism does not achieve the intent is not a fix.
4. Do not break existing behaviour to fix new behaviour. Search for consumers of anything you change; fix them in the same change or abandon it.
5. Add a regression test that fails without your fix. Mutation-check it: break the fix, confirm the test goes red. A test that passes either way protects nothing.
6. Schema-cache and metadata code is high-risk. It decides INSERT column membership, so a mistake there is silent data divergence, not a crash. Changes here need value-level verification, not just green unit tests.
7. Run python3 scripts/validate_specs.py to confirm specification integrity.

Reviewing changes: skills and gates

Every pull request, your own or someone else's, is reviewed with the same method:

1. Load the review skills in .claude/skills/ (index):
   • Start with pr-self-review. Its five gates are contract, impacted surface, failure and divergence, evidence, and then lower-priority quality.
   • Then load code-quality-<language> for every changed language (jvm, python, go, bash, sql) and code-quality-architecture for the design pass.
   • Use code-reuse-review and design-note-debt for duplication and deliberate shortcuts.
   • Use destructive-operation-safety for any code that drops, truncates or deletes.
   • Use repro-harness-negative-proof before calling anything "not reproducible" or "unaffected".
   • Use sink-connector-mysql-source-of-truth for any "which side is right" question.
   • Before writing code, code-reuse-first and mandatory-code-change-validation apply.
2. Run the deterministic gates: python3 scripts/review_gates.py --base origin/<base> (Spec 11.06, doc/review_gates.md). CI runs them on every pull request. A destructive statement needs a DESTRUCTIVE: comment and a Destructive-Op-Check: sites=<N>; result=pass commit trailer. Code may never stop merges, and no GPL/LGPL/AGPL dependency may reach a runtime scope.

The gates prove what a machine can prove. The skills cover the judgement. A green gate run is not an approval.

Repository shape

• specs/ — Component and system specifications (001–009, Constitution, Smart Ralph protocol).
• formal_specs/lean/ — Formal verification state machine models and machine-checked proofs in Lean 4.
• scripts/ — Spec validation and developer tooling.
• sink-connector/ — the Kafka Connect sink: batching, schema cache, query construction, ClickHouse writes.
• sink-connector-lightweight/ — the standalone (embedded Debezium) runtime.
• sink-connector-client/ — client/CLI.
• sink-connector/python/db_compare/ — source↔ClickHouse count and checksum comparison tooling. This is the ground truth for "are the two sides actually equal?".
• sink-connector/python/ch_sink_tools/db_load/mysql_resync.py (ch-mysql-resync, spec 11.04) — the repair when the checksum fails because the source was changed outside the binlog (sql_log_bin=0 patch, schema reload): dump → count-reconciled scratch load → atomic REPLACE PARTITION → connector offset rewind. Never TRUNCATE/DELETE a live table by hand.
• doc/ — feature matrix and reference docs.

Build with JDK 17 and Maven; mvn -o test works offline once dependencies are cached.

If this file conflicts with something else

The prime directive wins. If a comment, a doc, an old commit message, or your own reasoning suggests that the ClickHouse side should be preserved over the MySQL side, that source is wrong and should be corrected in the same change.