Expand and contract

During any rolling deploy, blue-green cutover, or canary, old and new code run simultaneously against the same database, often for minutes and after a rollback for hours.

Every intermediate state must be compatible with both the old and the new version.

A migration correct only after every process restarts causes an outage. This makes ALTER TABLE ... RENAME COLUMN unsafe regardless of how fast it runs: the moment it commits, running old code breaks.

Three phases

1. Expand: add the new thing. Additive only. Old code untouched and unaware.
2. Migrate: write to both, backfill, move readers to the new thing, stop writing the old.
3. Contract: delete the old thing once nothing reads or writes it.

Each phase is at least one deploy, independently deployable and revertible. The gap between phases is where problems surface, so do not compress it.

Schema recipes

Rename a column: six steps

Stopping between any two steps leaves a correct system. Wait between steps 4 and 6 long enough to be confident there will be no rollback.

Add a NOT NULL column

ALTER TABLE users ADD COLUMN plan text;                    -- 1. nullable, instant on PG 11+

-- 2. code writes it on every new row

UPDATE users SET plan = 'free' WHERE plan IS NULL AND id BETWEEN $1 AND $2;   -- 3. batched

ALTER TABLE users ADD CONSTRAINT users_plan_nn CHECK (plan IS NOT NULL) NOT VALID;  -- 4
ALTER TABLE users VALIDATE CONSTRAINT users_plan_nn;       -- takes only SHARE UPDATE EXCLUSIVE
ALTER TABLE users ALTER COLUMN plan SET NOT NULL;          -- PG 12+ uses the constraint, no scan

Other operations

The lock queue trap

In Postgres a DDL statement waiting for a lock blocks every query arriving behind it, including plain SELECTs. One slow ALTER behind one long transaction stalls the whole table. Always cap the wait:

SET lock_timeout = '3s';
ALTER TABLE ...;   -- fails fast instead of building a queue

Backfills

• Batch a few thousand rows per statement with a pause between batches.
• Never hold one long transaction: it blocks vacuum and grows replication lag. See mvcc-and-bloat.
• Make the loop resumable and idempotent by tracking the last id processed. See keyset-pagination.
• Throttle on replica lag rather than wall-clock time.

API and event changes

Never repurpose a name. A field whose meaning changed is undetectable to consumers, while a field that disappeared is a clear error. Prefer the loud failure.

For queues, add one constraint: a queue holds messages written by the previous version for its full retention window. Consumers must handle both shapes until that window passes after the last old producer stops.

Instrument before deleting

Prove nothing uses the old thing:

• API field: count reads per field, tagged by client, over a full business cycle including monthly and quarterly jobs
• Column: check query logs and grep every repository, not only this one
• Endpoint: metrics by caller identity, not total request count

"Nobody uses it" without a metric means "nobody I asked". Batch jobs appear at month end, right after the column is dropped. See hyrums-law.

Tools

Consumer habits that lower the cost

• Never SELECT * in application code, since a new column changes the result shape.
• Never rely on column order or JSON key order.
• Be a tolerant reader: ignore unknown fields. Strict validation belongs on fields actually used, not on the whole payload.
• Keep migrations in the repository with the code that needs them.