duckgres

# Claude Code Context for Duckgres

This file provides context for Claude Code sessions working on this codebase.

## Project Overview

Duckgres is a PostgreSQL wire protocol server backed by DuckDB. It allows any PostgreSQL client (psql, pgAdmin, lib/pq, psycopg2, JDBC, etc.) to connect and execute queries against DuckDB databases.

## Architecture

Duckgres has three deployment topologies, built from three primary run modes (`standalone`, `control-plane`, `duckdb-service`; a fourth utility mode, `reshard-runner`, is the entrypoint of the dedicated per-operation reshard pods the control plane spawns — see the Resharding section):

**1. Standalone** — single process. One binary running in `standalone` mode handles the PG wire protocol, auth, TLS, transpilation, and DuckDB execution itself. Each user gets their own DuckDB database in-process.
```
PG Client → TLS → Server (standalone) → DuckDB
```

**2. Control plane + local process workers** — single host, multiple processes. A `control-plane` parent process owns client connections (TLS, auth, PG wire, transpilation) and spawns child `duckdb-service` worker processes, communicating via Arrow Flight SQL over Unix sockets. Used for stronger isolation between sessions on a single host. Selected with `--worker-backend process` (the default).
```
PG Client → TLS/Auth/PG Protocol → Control Plane (process)
                                 → Flight SQL (UDS) → local Worker process (DuckDB)
```

**3. Control plane + remote workers on Kubernetes** — multitenant cluster deployment. The `control-plane` runs as its own pod and routes per-org traffic to dedicated `duckdb-service` worker pods over TCP+TLS. Worker pods are scheduled by the control plane via the K8s API; org config and worker state are persisted in a Postgres-backed config store. Selected with `--worker-backend remote`; requires a binary built with `-tags kubernetes`.
```
PG Client → TLS/Auth/PG Protocol → Control Plane pod
                                 → Flight SQL (TCP+TLS) → per-org Worker pod (DuckDB)
```

### Native metadata Postgres proxy

The Kubernetes control plane can expose explicitly opted-in CNPG-backed
warehouse metadata databases on a separate SNI suffix
(`DUCKGRES_METADATA_HOSTNAME_SUFFIXES`; managed-warehouse deployments use
`.md.dev.postwh.com`, `.md.us.postwh.com`, or `.md.eu.postwh.com`). This is an
early connection branch, not a DuckDB
executor: the existing org `root` password authenticates at Duckgres, the
startup database MUST be exactly `metadata`, and the control plane resolves
the real endpoint/database/tenant role/password internally through
`SharedWorkerActivator.MetadataPostgresURL`. After authenticating upstream
with that internal credential, protocol traffic is relayed byte-for-byte. The
endpoint and password remain internal; the upstream role and database can be
visible to the fully privileged client through normal PostgreSQL introspection
such as `current_user` and `current_database()`.

Access is fail-closed on the warehouse row's `metadata_proxy_enabled` flag,
`state=ready`, and `metadata_store_kind=cnpg-shard`. Never infer publication
from shard placement and never pass a client-supplied upstream database,
username, host, or password. `DUCKGRES_METADATA_PROXY_MAX_CONNECTIONS_PER_ORG`
(default 20 per control-plane replica) bounds public sessions so they cannot
exhaust the internal PgBouncer pool. These sessions participate in the
existing per-user kill / disable fan-out, and an established session is closed
if the warehouse gate stops being eligible. An admin warehouse PUT that
explicitly includes `metadata_proxy_enabled` reloads the local config snapshot
and notifies peer replicas; established sessions observe the new gate on their
next five-second recheck after snapshot propagation.

The initial scope is dedicated, single-customer CNPG shards only. Do not enable
an org on a shared shard until upstream `CONNECT` ACLs and role hardening are in
place. The exact virtual database check prevents selecting a different startup
database, but after connection the customer `root` credential has the full
access of the internally resolved metadata role.

Observability stays explicit at the branch boundary:
`duckgres_connections_open` includes these client sockets because it is the
process-wide accepted-connection gauge, while
`duckgres_metadata_proxy_connections_open`,
`duckgres_metadata_proxy_connection_attempts_total`,
`duckgres_metadata_proxy_connection_duration_seconds`,
`duckgres_metadata_proxy_upstream_connect_duration_seconds`, and
`duckgres_metadata_proxy_bytes_total` isolate proxy load and failures. The
relay is intentionally opaque, so DuckDB query metrics, query logs, and query
traces do not include SQL executed through it. Use the CNPG/PgBouncer metrics
and the fixed upstream `application_name=duckgres-metadata-proxy` for
database-side attribution; never make one org's metadata target part of the
control-plane health check. Target resolution plus upstream
connect/auth/synchronization has a fixed 10-second bootstrap deadline; the
deadline is canceled after hijack and never applies to established relay
traffic. `duckgres_auth_failures_total` includes wrong-password proxy attempts;
the dedicated attempt counter with `outcome="auth_failed"` is the proxy split.
Pre-TLS `duckgres_rate_limit_rejects_total` events cannot be assigned to either
endpoint because SNI has not yet been observed.

Metadata-proxy `CancelRequest` handling is session-terminating. Synthetic
backend keys map to the exact established frontend/upstream connection pair on
the owning control-plane replica, which closes both rather than redialing a
PgBouncer Service where instance-local cancel keys could reach the wrong pod.
Raw cancel connections remain control-plane-local behind the NLB, matching the
existing cancellation locality: a synthetic-key miss on another replica is
absorbed and counted by
`duckgres_metadata_proxy_cancel_requests_total{outcome="not_local"}`.

In topologies 2 and 3, the control plane exposes only PostgreSQL wire protocol to clients. Arrow Flight SQL is internal transport between the control plane and workers.

### Key Components

- **main.go / config_resolution.go**: CLI flags; effective config resolution (CLI > env > YAML > defaults), including env-only K8s knobs.
- **server/** — PG wire protocol server and DuckDB execution
  - Wire protocol & connections: `server.go`, `conn.go`, `conn_errors.go`, `conn_query_exec.go`, `conn_results.go`, `conn_copy.go`, `conn_extended_query.go`, `conn_pg_stat_activity.go`, `conn_cursor.go`, `protocol.go`, `exports.go`
  - Execution: `executor.go`, `flight_executor.go`, `chsql.go`, `transient.go`
  - Catalog & types: `catalog.go`, `types.go`, `session_database_metadata.go`
  - Auth, TLS, rate limiting: `auth_policy.go`, `ratelimit.go`, `certs.go`, `acme.go`
  - DuckLake: `ducklake_migration.go`, `checkpoint.go`
  - Observability: `querylog.go`, `tracing.go`
  - ProcessIsolation child workers: `parent.go`, `worker.go`, `worker_activation.go`, `worker_control.go`
- **controlplane/** — Multi-process / multi-tenant control plane
  - Core: `control.go`, `session_mgr.go`, `worker_mgr.go`, `worker_pool.go` (process/k8s abstraction), `validation.go`, `sdnotify.go`
  - Runtime loops: `janitor.go`, `leader_loop.go`, `memory_rebalancer.go`, `runtime_tracker.go`
  - K8s / multitenant under build tag `kubernetes` (including: `multitenant.go`, `k8s_pool.go`, `k8s_pool_acquire.go`, `k8s_pool_spawn.go`, `k8s_pool_lifecycle.go`, `k8s_pool_reconcile.go`, `k8s_pool_helpers.go`, `k8s_factory.go`, `org_router.go`, `org_reserved_pool.go`, `sts_broker.go`, `shared_worker_activator.go`, `worker_rpc_security.go`, `janitor_leader_k8s.go`)
  - Subpackages: `admin/` (HTTP admin API + dashboard, `kubernetes` tag; includes the models explorer UI `static/models.html` + `models_api.go`, and `devserver/` for local UI dev against a port-forwarded CP — see `admin/README.md`), `provisioner/` (k8s controller, `kubernetes` tag), `provisioning/` (HTTP API), `configstore/` (Postgres-backed config)
- **duckdbservice/** — DuckDB Arrow Flight SQL service
  - Core: `service.go`, `flight_handler.go`, `arrow_helpers.go`, `auth.go`, `config.go`
  - Lifecycle, caching, profiling, metrics: `activation.go`, `transient.go`, `cache_proxy.go`, `profiling.go`, `progress.go`, `metrics.go`
- **transpiler/** — AST-based PostgreSQL → DuckDB SQL transpiler
  - Top-level: `transpiler.go`, `config.go`, `boolpredicates.go`, `show_create.go`
  - `transform/`: individual transforms; see registered pipeline in `transpiler.go` `New()`

## Run Modes

- **standalone** (default): Single process, handles everything including TLS, auth, PG protocol, and DuckDB execution.
- **control-plane**: Multi-process. Owns pgwire client connections end-to-end (TLS, auth, protocol handling, SQL transpilation) and routes queries to a worker pool.
  - **Process backend** (default, `--worker-backend process`): local Flight SQL workers over Unix sockets.
  - **Remote backend** (`--worker-backend remote`): per-org Kubernetes worker pods over TCP+TLS. Multitenant; requires `-tags kubernetes` and a Postgres-backed config store. Adds config store, org router, runtime tracker, janitor/leader election, and a provisioning/admin HTTP API.
- **duckdb-service**: Thin DuckDB execution engine exposed via Arrow Flight SQL. Spawned automatically by the control plane as worker processes, or run standalone for testing.
- **reshard-runner** (`-tags kubernetes` only): entrypoint of the dedicated per-operation reshard pod (`duckgres-reshard-op-<id>`), spawned by the control plane when an operator starts a reshard. Claims ONE op (`DUCKGRES_RESHARD_OP_ID`), executes the reshard step machine to a terminal state, exits (0 unless an infrastructure error). See "Resharding" below.

Key CLI flags for control-plane mode:
- `--mode control-plane|duckdb-service|standalone|reshard-runner`
- `--worker-backend process|remote`
- `--process-min-workers N` / `--process-max-workers N`
- `--process-retire-on-session-end`
- `--worker-queue-timeout DURATION` / `--worker-idle-timeout DURATION`
- `--statement-timeout DURATION` — cancels any single statement that runs longer than this; `0`/empty (the default) leaves statements **unbounded**, which is today's behavior. Blast-radius containment only: nothing else in the stack bounds a wedged statement (DuckDB has no statement timeout, client `statement_timeout` is an ignored SET, the cnpg shards run `idle_in_transaction_session_timeout=0`, pgwire clients wait forever), so one engine deadlock cost a tenant 5h33m of their hourly refresh pipeline on 2026-09-16. The deadline rides on the STATEMENT context, never the connection context — a timeout ends one statement, not the session — and surfaces as `57014` with PostgreSQL's exact wording `canceling statement due to statement timeout` (drivers string-match it). The statement context comes from `queryContextInner`, the single registration point every execution path passes through on its way to an engine: simple, batched, extended Execute, the writable-CTE rewrites on both protocols, the extended Describe schema probes (they really execute at `LIMIT 0`), COPY in both directions, and cursors. Classification is error-driven and feature-gated (`statementTimedOut(err)`): an expired deadline only classifies as the timeout when the knob is on, so deployments that never set it keep byte-for-byte behavior for internal deadlines. Two lifetime notes: a **cursor is bounded over its whole lifetime, not per `FETCH`** (one context, taken when the cursor opens at first `FETCH`), and a **suspended portal is bounded over its whole lifetime** too (its open rowset dies with the context, so the teardown moves to `closeExec`) — both stricter than PostgreSQL, deliberate (an unbounded cursor or suspended portal pins a worker).
  - **Every execution path must acquire the statement context through `queryContext`/`queryContextInner`.** That is the ONLY place `RegisterQuery` is called, so a path that skips it is reachable by neither the timeout nor a pgwire `CancelRequest`. Extended-protocol `Execute` used to skip it and called the context-LESS `executor.Query`/`Exec` (which run on `context.Background()` — see `PinnedExecutor`), leaving prepared statements — what pgx, psycopg3 and JDBC send — unbounded and uncancellable. Regression: `TestStatementTimeoutReachesExtendedProtocolExecute`. If you add an execution path, route it through `queryContext` and the `*Context` executor methods.
  - **Timeout classification is driven by the ERROR and gated on the timeout being configured** (`statementTimedOut(err)`), never by stored per-connection state. An earlier design recorded the live statement context on the connection; it went sticky — once any statement timed out, later unrelated failures were reported as timeouts and their logging skipped. The gate also keeps a deployment with the feature OFF byte-for-byte identical, so internal deadlines (attach, exec, worker gRPC) do not start surfacing as `57014`. With the feature ON an internal deadline racing a statement can still be labelled a statement timeout — an accepted narrowing.
  - **A cursor is bounded over its whole lifetime, not per `FETCH`**: `queryContextForCursor` takes ONE context at `DECLARE` and holds it across every `FETCH`, so a cursor open longer than the timeout dies — stricter than PostgreSQL. Deliberate (an unbounded cursor also pins a worker), but size the timeout with long-lived cursors in mind.
  - Caveat: an engine that ignores cancellation (duckdb/duckdb#24961) will not stop work; the client is freed with a clear error but the worker can stay wedged — retiring a timed-out worker is a follow-up that must respect the destroy-before-reuse ordering in `SessionManager.DestroySession`.
- `--idle-timeout DURATION` — connection idle timeout: a client connection with no traffic for this long is closed and its worker released to hot-idle (in control-plane mode an idle connection otherwise pins a worker forever). The close is preceded by a `FATAL` 57P05 ErrorResponse naming the effective timeout (matching PostgreSQL's `idle_session_timeout`), so clients see a reap, not a network fault. **Control-plane default is `5m`** (`server.DefaultControlPlaneIdleTimeout`; standalone defaults to `24h`); a negative value disables it. `server.New` applies the standalone default, so the control plane sets it explicitly before `InitMinimalServer` (which skips that defaulting).
- `--memory-budget SIZE` (default 75% RAM) / `--memory-rebalance`
- `--socket-dir /path` (process backend)
- `--handover-drain-timeout DURATION` (default `24h` process; **remote default is `0` = unbounded** — the CP waits for active sessions for as long as it takes and the pod's k8s `terminationGracePeriodSeconds` is the only hard wall. cloudflare/tableflip FD passing applies to process/standalone single-host upgrades, not k8s pod replacement.)
- `--ducklake-delta-catalog-enabled` / `--ducklake-delta-catalog-path`
- Remote backend (requires `--config-store`; `-tags kubernetes` for K8s pool):
  - Config store: `--config-store`, `--config-poll-interval`, `--internal-secret`
  - K8s pool: `--k8s-worker-image`, `--k8s-worker-namespace`, `--k8s-control-plane-id`, `--k8s-worker-port`, `--k8s-worker-secret`, `--k8s-worker-configmap`, `--k8s-worker-image-pull-policy`, `--k8s-worker-service-account` (no global worker cap — per-org `Org.MaxWorkers`, 0=unbounded, is the only cap)
  - AWS / STS: `--aws-region`
  - Compute-usage billing needs no config: metering is always on for the remote backend and billing PULLS usage over the internal-secret-authed HTTP API (`GET /api/v1/billing/usage` + `POST /api/v1/billing/ack`). See `docs/design/billing-pull-api.md` and "Compute-Usage Billing" below.
  - Pod scheduling knobs (CPU/memory requests, node selector, tolerations) are env-only — see `config_resolution.go`.

Key CLI flags for duckdb-service mode:
- `--duckdb-listen` (e.g., `unix:///...` or `:8816`)
- `--duckdb-listen-fd` (internal; set by control plane)
- `--duckdb-token` (bearer auth)
- `--duckdb-max-sessions` (0=unlimited)

## Configuration

Configuration is resolved in `config_resolution.go` with the following precedence (highest to lowest):
1. CLI flags (`--port`, `--config`, etc.)
2. Environment variables (`DUCKGRES_PORT`, etc.)
3. YAML config file
4. Built-in defaults

Note: `--mode` is CLI-only (not loadable from YAML/env). A handful of K8s pod-scheduling knobs are env-only (no CLI flag).

## Keep docs in sync with behavior

When you change a behavior, default, flag, or invariant that is documented
anywhere in the repo, **update that documentation in the same PR.** Stale docs
are worse than no docs — they actively mislead the next reader (human or agent).
This applies to, at least: this `CLAUDE.md`, `README.md`, `docs/`, CLI flag help
text (`main.go` / `cliflags.go`), and any design/plan docs that pin the changed
behavior. Concretely: if you change a default value, a flag's meaning, a drain or
shutdown semantic, an activation/routing/teardown order, or any of the
LOAD-BEARING CONTRACT sections below, grep for the old value/term across `*.md`
and help strings and fix every mention. A behavior change that leaves a doc
asserting the old behavior is incomplete, the same way a behavior change without
a test is incomplete.

## Development

The project uses [just](https://github.com/casey/just) as a command runner. Run `just` to see all available recipes for building, testing, running, metrics, and scripts.

## Testing

**Every feature, behavior change, bugfix, AND refactor that affects runtime or
cluster behavior MUST ship with a solid end-to-end test case in
`tests/mw-dev/e2e/` (`harness.sh`).** This is not just for new features — any
change to how the system behaves at runtime (new capability, changed semantics,
a fixed bug, a new config knob, an activation/routing/teardown tweak) extends or
adds a harness assertion in the same PR. Refactors count too: when you move or
rewrite a code path the harness covers, confirm the relevant assertion still
exercises it (and update it if the path moved) — a refactor that quietly drops
e2e coverage is a regression in the test suite even if behavior is unchanged. Unit/package tests are necessary but not sufficient: a
change is only "done" once it is exercised against the real mw-dev cluster —
real worker pods, real Crossplane ducklings, real cnpg/RDS metadata, real
S3/STS. "Solid" means a deterministic pass/fail
assertion of the actual user-visible behavior (not just "it didn't error"), with
transient/cold-pool conditions handled, on both metadata backends (cnpg + ext)
where it touches metadata. A bugfix gets a regression assertion that would have
caught the bug. If a change genuinely cannot be asserted in-Job (e.g. it needs
cnpg-shards exec, or warm-pool-only state), say so explicitly in the
harness/README with the reason — don't silently skip. The harness is the gate
that catches what unit tests fake.

Three test lanes worth knowing about, in increasing order of blast radius:

- **Unit / package tests** (`go test ./...`): in-process, no external deps. Where most coverage lives. Includes `tests/manifests/` (static-manifest artifact asserts for `k8s/rbac.yaml` + `k8s/networkpolicy.yaml`).
- **`tests/integration/`** (`just test-integration`): spins up the standalone server binary against a real MinIO + Postgres metadata store via docker compose. Covers wire protocol, DuckLake on real S3-compatible storage, transpilation against a live server.
- **`tests/mw-dev/e2e/`** (per-PR GitHub workflow `e2e-mw-dev.yml`): the full multi-tenant activation pipeline against the **real posthog-mw-dev EKS cluster** — real Cilium, real Crossplane ducklings, real cnpg-shard + external-RDS metadata, real AWS S3. A shell harness (`harness.sh`) runs as an in-cluster Job per PR; `run.sh` orchestrates deploy/test/teardown/e2e-cleanup. **Replaces the retired kind suite** (`tests/k8s/`) — that suite's `k8s-integration-tests` CI job and its Go tests are gone; the supporting `k8s/` scripts/manifests + Dockerfiles are kept for now. See `tests/mw-dev/README.md`.

### When code changes obligate test changes

`tests/mw-dev/e2e/` is the only place we exercise the full activation pipeline (control plane → STS broker → worker pod → DuckDB → ATTACH against real cloud storage). If your change touches any of the following, treat updating the harness as part of the change, not a follow-up:

- `controlplane/shared_worker_activator.go`, `controlplane/sts_broker.go`, anything in the activation payload shape (`TenantActivationPayload`, `server.DuckLakeConfig`)
- `server/server.go::AttachDeltaCatalog`, `server.attachDuckLake*`, `server.refresh*Secret`
- `controlplane/configstore/models.go` — new columns flow through the provisioning API the harness calls; exercise them via a provision body field
- `duckdbservice/activation.go`, `worker_activation.go` — worker-side activation order
- Any code path that wires AWS credentials through to DuckDB SECRETs

The contract: if the harness no longer exercises a path you changed, **update `harness.sh`**; if your change removes a path it asserts against, **delete the assertion**. The DuckLake round-trip / durability / concurrent-writers checks in `harness.sh` are the load-bearing ones for catalog wiring — keep them honest.

## Dependencies

- `github.com/duckdb/duckdb-go/v2` - DuckDB Go driver
- `github.com/pganalyze/pg_query_go/v6` - PostgreSQL SQL parser (CGO, uses libpg_query)
- `gopkg.in/yaml.v3` - YAML config parsing

## Known Limitations

- No replication
- Some pg_catalog tables are stubs (return empty)
- Unmapped DuckDB types (MAP, STRUCT, UNION, ENUM, BIT) fall back to OidText
- DML RETURNING is not supported via extended query protocol (see below)

## DML RETURNING Detection

DML with RETURNING is rejected at extended-query Describe time with SQLSTATE `0A000` — the Describe path probes schema by executing the query, which would cause an unintended mutation. Detection lives in `isDMLReturning` and friends in `server/conn.go` (heuristic SQL-aware lexer, with any-depth scanning for WITH-prefixed writable CTEs). Invariants for anyone editing this code:

- **False negatives are dangerous** — they cause silent mutations during Describe. False positives are safe (just an error to the client). Err toward false positives.
- All detection is heuristic string scanning. If precision becomes critical, switch to `pg_query_go` AST parsing.
- LIMIT 0 does NOT prevent CTE side effects — Postgres CTEs are optimization fences, so writable CTEs execute even with LIMIT 0.
- DuckDB does not currently support MERGE. If it adds MERGE RETURNING, add `MERGE` to the prefix check in `isDMLReturning`.

## Worker Session Model (k8s / remote backend) — LOAD-BEARING CONTRACT

In the **control-plane remote/k8s backend** a worker pod serves **exactly one
client query session at a time**. This is deliberate: `workerDuckDBLimits`
(`controlplane/control.go`) gives the single session the *whole pod's* RAM
less a headroom reserve, plus 2.5 DuckDB threads per requested CPU, rounded
up. It does NOT divide by session count. Two sessions on one pod would each
believe they own the whole budget → 125-150% of the pod's RAM committed →
nondeterministic OOM / a heavy query killed by a co-resident one. Do not break
the following:

Headroom is `max(25% of pod, min(6GiB, 40% of pod))`
(`workerMemoryHeadroomBytes`), not a flat percentage — 40% below 15Gi, exactly
6GiB in `[15Gi, 24Gi)`, 25% at and above 24Gi. The 24Gi region is the old
flat-75% rule exactly, so the 120Gi pool default and the 360Gi client-profile
ceiling keep the sizing they have always had.

The reserve covers what `memory_limit` does not govern — DuckDB C++ catalog
objects, postgres_scanner/libpq result buffers, Arrow Flight batches, the Go
runtime, page cache — and it is CUMULATIVE, not a per-query peak: DuckDB does
not return buffer-pool pages to the OS when a session ends, so a hot-idle
worker's RSS ratchets across the sequential sessions it serves and settles near
`memory_limit` + overhead. In mw-prod-us a 16Gi worker ratcheted to 14.8GiB —
2.8GiB past its 12GiB limit, 1.2GiB of pod left — and was OOMKilled, while
120Gi workers ran 15.6GiB past a 90GiB limit inside 14.4GiB of remaining pod
and were not. The overhead is not constant across pod sizes, but it grows
sub-linearly, so a flat fraction leaves proportionally less usable margin the
smaller the pod. Size headroom by the margin it must absorb, not by percentage
alone.

DuckDB budgets below the crossover are formatted in MB, not GB: truncating a
fractional-GiB budget (8Gi - 40% = 4.8GiB) to whole GB would hand back the
very margin the cap exists to reserve.

- **One session per worker is enforced, not emergent.** The CP spawns remote
  worker pods with `DUCKGRES_DUCKDB_MAX_SESSIONS=1` (`k8s_pool.go::spawnWorker`).
  A 2nd concurrent `CreateSession` on a worker is rejected, not silently
  overcommitted. Internal control/maintenance work uses the worker's side
  connections (`controlDB`/`warmupDB`), which are NOT counted sessions — so
  cap=1 does not starve them. Do not raise this to >1 for k8s workers, and do not
  route internal work through `CreateSession`.
- **`OrgReservedPool` (remote/multitenant) must never co-assign.** It reuses only
  idle (`activeSessions==0`, Hot, org-owned) workers via
  `findIdleAssignedWorkerLocked`, or claims/spawns a fresh one. There is NO
  least-loaded "share onto a busy worker" path (that exists only in the
  single-tenant flat `K8sWorkerPool.AcquireWorker`, which is not used in remote
  mode). Do NOT add one, and do not resurrect a `leastLoaded*` helper here.
- **At org max workers + all busy → fail fast with the clear org-cap message**
  (`WorkerClaimMissReasonOrgCap`, see `capacity_policy.go`). Never busy-wait at cap.
- **Under cap → spawn a worker on demand** (`spawnReservedWorkerForSlot`). There
  is no warm pool to wait on; the cap is re-checked authoritatively cross-CP in
  `CreateSpawningWorkerSlot`. The spawn+activate runs DETACHED from the request
  ctx (`context.WithoutCancel` + `workerSpawnActivateTimeout`): the requester
  waits for the result or its own ctx, but a requester that gives up must NOT
  kill the in-flight pod (doomed-spawn thrash). An abandoned spawn that succeeds
  is parked hot-idle (`ReleaseWorker`/`TransitionToHotIdleIfNoSessions`, record
  persisted) for the org's next connection; one that fails is retired. Nothing
  may leak in Reserved/Activating.
- **FIFO anti-snatch:** the slow acquisition path's DECISION section (idle-reuse
  re-check → hot-idle claim → spawning-slot creation; `acquireDecision` in
  `org_reserved_pool.go`) is serialized per org by `orgAcquireGate`
  (`org_acquire_gate.go`) so a worker the CP scaled up for an earlier waiter
  cannot be snatched by a later connection. The multi-minute spawn+activate runs
  OUTSIDE the gate — each waiter is 1:1 bound to the claim/slot it owns and the
  session is pre-claimed before the worker becomes Hot, so a cold burst ramps N
  spawns in parallel without breaking anti-snatch. Keep the gate cancel-safe (a
  queued waiter whose ctx is cancelled must be skipped, not deadlock the gate).
- **Destroy-before-reuse ordering:** `SessionManager.DestroySession`
  (`session_mgr.go`) MUST await the worker-side `DestroySession` RPC *before*
  `ReleaseWorker`, so a reused (hot-idle) worker's prior session is gone before
  the next one is assigned (otherwise cap=1 spuriously rejects the reuse).
- **A hot-idle claim refuses a pod that is already being disrupted.**
  Karpenter honours `karpenter.sh/do-not-disrupt` only while a session is
  assigned (#757, `k8s_pool_disruption.go`), so an idle worker can be
  SIGTERM'd between the durable `ClaimHotIdleWorker` CAS and adoption; a
  session created on it fails on its first RPC. `adoptClaimedWorker` therefore
  checks the pod it just read (`claimedPodDoomReason`,
  `k8s_pool_doomed_claim.go`): a `deletionTimestamp`, a non-Running phase, or
  a node carrying Karpenter's `karpenter.sh/disrupted` taint → the claim is
  retired (`retired`, reason `pod_doomed`, origin `reserve_doomed_pod` — not
  `lost`/crash), counted on
  `duckgres_control_plane_hot_idle_claim_skipped_total{reason}`, and the
  acquisition re-runs its decision (next candidate or a fresh spawn) instead
  of failing the client. A node lookup failure is "unknown", never "doomed":
  refusing every claim during an API-server blip would turn a transient into
  a spawn storm. The local idle-reuse path is unaffected — it already reuses
  only `Hot` workers, and the lifecycle state machine forbids
  `Draining → Reserved`.
- **Cap-drift is recovered, not fatal:** if a worker still rejects a CP-scheduled
  session at its cap (CP↔worker accounting drift — should never happen),
  `SessionManager.CreateSessionWithProtocol` does NOT fail the client: it logs
  loudly (ERROR), bumps `duckgres_control_plane_worker_session_cap_drift_total`,
  retires (recycles) the inconsistent worker, and re-acquires a fresh one
  (bounded by `maxWorkerSessionCapDriftRetries`). Detection is
  `isWorkerSessionCapError` (matches the worker's "max sessions reached"
  message). A nonzero drift metric means the scheduling invariant is broken —
  fix the root cause, don't just lean on the retry.

- **An invalidated DuckDB instance retires the worker; it is NEVER reused.**
  A DuckDB Internal- or Fatal-class exception does not just fail one statement,
  it poisons the whole instance: every later statement on ANY connection to it —
  including a brand new session — fails with "database has been invalidated
  because of a previous fatal error" until the process restarts. DuckLake's
  commit path is a known source (an InternalException inside the commit retry
  loop is rethrown by `ErrorData::Throw` with its original type). Detection is
  `isInstanceFatalError` (`duckdbservice/instance_fatal.go`). The **error TYPE
  is authoritative**: a typed `*duckdb.Error` of
  `ErrorTypeInternal`/`ErrorTypeFatal`. **Never add substring matches for
  `"INTERNAL Error"` / `"FATAL Error"`** — DuckDB echoes the offending SQL back
  in its error text (`LINE 1: <query>`), so those matched the USER'S OWN QUERY
  and handed every tenant a one-statement worker kill via
  `SELECT 'INTERNAL Error' + 1` (regression:
  `TestInstanceFatalIgnoresEchoedQueryText`). The one string fallback is the
  `database has been invalidated…` marker, for an error that arrives already
  flattened. **OOM and DuckLake transaction conflicts are distinct error types
  and must never classify here**; they retry in place. The flag is sticky
  (invalidation is permanent) and is set from the statement paths, the
  `CreateSession` path, and an async `SELECT 1` liveness probe kicked by each
  health check — the probe is what catches a fatal thrown on a session that has
  since been destroyed. **The stored reason MUST be redacted before it is kept**
  (`usersecrets.RedactErrorForLog`, or `noteInstanceErrorOpaque` where no
  statement is available to classify, as on the secret-replaying `CreateSession`
  path): it is logged on the worker, shipped to the CP as
  `instance_invalid_reason`, and logged again on retire, so an un-redacted
  engine error leaks a failed `CREATE SECRET`'s credential into three sinks.
  **The probe must stay OFF the health check's critical path**: the CP's
  health-check budget is 3s and already shared with progress polling, so a
  blocking probe would get healthy workers killed for unresponsiveness. It is
  single-flight and a wedged CGO call can outlive its context, so a probe stuck
  past the threshold raises `duckgres_worker_instance_probe_stuck` rather than
  silently disabling detection. The worker reports
  `instance_invalidated` + `healthy:false`; the CP retires it on the FIRST
  report, bypassing `maxConsecutiveHealthFailures` (the process answers RPCs
  fine, so the failure counter would never fire) and rejects it for reuse in
  `validateReservedWorkerHealth` — the hot-idle reuse path is what previously
  turned one bad statement into "the warehouse is down until someone restarts
  it". `duckgres_control_plane_worker_instance_invalidated_total` and
  `duckgres_worker_instance_invalidated_total` /
  `duckgres_worker_instance_invalidated_state` are the signals; nonzero means
  a tenant hit an engine bug and lost a worker — contained, but chase the root
  cause. This is blast-radius containment ONLY: it does not fix the engine bug,
  which is fixed by shipping a DuckLake extension build that guards the read
  (see `DUCKLAKE_EXTENSION_TAG` in `Dockerfile`/`Dockerfile.worker`).

Touching any of: `controlplane/org_reserved_pool.go`, `org_acquire_gate.go`,
`k8s_pool.go::spawnWorker`/`AcquireWorker`, `control.go::workerDuckDBLimits`,
`duckdbservice/instance_fatal.go`, or
`duckdbservice` session counting → update the unit tests
(`org_reserved_pool_test.go`, `org_acquire_gate_test.go`,
`k8s_pool_doomed_claim_test.go`,
`duckdbservice/service_test.go`, `duckdbservice/instance_fatal_test.go`,
`controlplane/instance_invalidated_test.go`) AND the
`one_session_per_worker` + `cold_burst_parallel_spawns` assertions in
`tests/mw-dev/e2e/harness.sh`.

## Exploratory Worker Tier (small-first routing) — LOAD-BEARING CONTRACT

Design: `docs/superpowers/specs/2026-08-04-exploratory-worker-tier-design.md`.
On the **remote/k8s backend only**, a connection that does not ask for a worker
shape starts on a small "exploratory" pod and grows into a normal one only when
it proves it needs to. Two mechanisms: **lazy acquisition** (no worker at
connect) and **escalation** (small → standard, one-way). Env-only knobs:
`DUCKGRES_EXPLORATORY_TIER_ENABLED` / `_WORKER_CPU` / `_WORKER_MEMORY` /
`_WORKER_TTL` (default 48h), resolved by `exploratoryWorkerProfile`
(`controlplane/worker_profile.go`). Server side lives in `server/conn_tier.go`
+ `tier_classify.go`; control-plane side in `controlplane/session_activation.go`
+ the activator/switcher closures in `control.go::handleConnection`.

- **Eligibility is decided once, at connect** (`useExploratoryTier`): remote
  backend AND a usable exploratory profile AND not a passthrough user AND no
  client `duckgres.worker_*` startup option (`clientSuppliedWorkerGUCs`). A
  half-configured tier (missing/invalid size) resolves to nil and degrades to
  today's eager behavior — never to a BestEffort pod. **Passthrough users are
  excluded** (they bypass the compat layer the classification is built on), and
  a **GUC-sized connection bypasses the tier entirely** — it acquires the
  requested shape eagerly at connect, as before.
- **Nothing is acquired until a statement needs an engine.** The connection
  reaches the message loop with `c.executor == nil` and a `SessionActivator`;
  `activateForStatement` acquires on first need. Statements the control plane
  answers itself — `SET`/`SHOW duckgres.query_source`, ignored SETs, no-ops,
  `pg_stat_activity`, the empty query — MUST NOT acquire. That is the point of
  the whole feature; adding an acquire to an engine-free path silently deletes
  the benefit. The `duckgres.s3_cache` and `duckgres.worker_ttl` GUCs are the
  exception, on all three protocol paths, because unlike the other duckgres
  GUCs they are WORKER state (the s3_cache secret swap; the worker_ttl
  pool-side hot-idle TTL): **`SET` always acquires** — the apply needs a
  worker to land on — and **`SHOW duckgres.s3_cache` acquires only when
  `c.hasPendingS3Cache`**, i.e. when a connect-time option has not been applied
  yet and answering first would report a transport the session is about to
  leave (see the s3_cache section below). `SHOW duckgres.worker_ttl` never
  acquires: until a worker exists, the connect-time baseline is the truthful
  answer (there is no pending worker-side TTL state).
- **A pinning FIRST statement acquires the standard profile directly** (one
  acquire, `pinned=true` → `MarkConnectionPinned`), never small-then-escalate.
- **The pin set is the state boundary, and every member is load-bearing:** DML,
  DDL, `COPY` (BOTH directions — COPY TO can reference session state and is
  routed above the transpile-time hook), `SET`, `BEGIN` (so an open transaction
  can never exist on a worker that is about to be replaced), `DECLARE` (simple,
  batched AND extended — `FETCH`/`CLOSE` are unhooked and rely entirely on the
  DECLARE pin, because a cursor's worker-side RowSet must not open on a session
  about to be destroyed), secret DDL **before** the user-secrets interception
  (that interception owns its own execution and sits ABOVE the general hook, so
  a plain/TEMPORARY `CREATE SECRET` would otherwise land on the small worker and
  be silently dropped), and the extended-protocol **Describe probe** of a
  pinning statement (Describe really executes it). A parse failure pins by
  default — false pins are free, a missed pin loses state.
- **Escalation is one-way and sticky.** `escalateWorker` destroys the small
  session BEFORE acquiring the standard one, so once it is entered there is no
  session to fall back to. Both reason and outcome are closed sets:
  `duckgres_exploratory_escalations_total{reason="state"|"oom"|"heuristic",
  outcome="ok"|"canceled"|"capacity"|"draining"|"disabled"|"error"}` — every
  ATTEMPT is counted, so a cluster that cannot escalate anything is
  distinguishable from one nobody escalates on (v1 ships no heuristic tier; the
  constant marks the hook point). The failure classes come from the CLASSIFIED
  SQLSTATE via `server.AcquisitionFailureOutcome`, the same helper
  `duckgres_session_activation_total{org,outcome}` uses — one helper, so the two
  acquisition metrics can never drift into different or unbounded label sets.
- **A failed acquisition — activation OR escalation — is CONNECTION-FATAL**
  (the one exception is the post-escalation s3_cache re-apply, below).
  `failWorkerAcquisition` sends a FATAL ErrorResponse, suppresses
  ReadyForQuery, and unwinds the message loop; there is no session left to
  resynchronize to. SQLSTATE comes from `escalationErrorSQLState`: a
  `*server.SessionAcquireError` (classified by the CP with the SAME logic the
  eager connect path uses, `sessionCreationErrorResponse`) is authoritative —
  28000 disabled / 53300 capacity / 57P03 draining / 57014 cancel / 3D000
  catalog / XX000 s3_cache apply / 53400 other; the substring fallback covers
  only the paths that still return a plain error. Keep the client message the
  classified one, never the wrapped internal chain. Extended-protocol handlers
  are void, so the error is ALSO parked on `c.fatalErr`, which
  `runExtendedQueryMessage` hands to the message loop. **`fatalErr` is one-shot
  by construction** — it is set on the way out and never cleared, because the
  connection is terminating; do not add a path that sets it and then continues.
- **OOM re-execute is transparent only under all three conditions:** the
  statement is a READ, ZERO DataRows have been sent, and `txStatus` is idle. A
  wire stream cannot be restarted, so a partial result must surface the error
  instead. RowDescription is NOT resent on the retry (same query, same engine
  version → identical schema), and the retry runs on the escalated worker only.
  Detection is `isWorkerOutOfMemoryError` (`server/conn_errors.go`) — DuckDB's
  engine OOM only; a pod-level OOMKill is `ErrWorkerDead` and is deliberately
  never re-executed.
- **`duckgres.s3_cache` interplay (both directions).** A connect-time
  `-c duckgres.s3_cache=...` cannot be applied at connect on this path (no
  worker), so the CP parks it (`SetPendingS3CacheOption`) and
  `ensureSessionActive` applies it AFTER installing the executor — applying it
  inside the activator would find a nil executor, no-op the worker swap, and
  still flip the session flag, the exact divergence `applyS3CacheSetting`
  exists to prevent. A failed apply fails the activation (XX000, fatal). On
  escalation the bypass is RE-APPLIED to the new worker
  (`reapplyS3CacheAfterWorkerSwitch`); that failure is the ONE
  non-connection-fatal escalation outcome, because the swap already succeeded:
  the statement fails with a normal `ERROR` (XX000, naming the re-apply), the
  session flag is reset to the worker's REAL transport (proxied — a fresh
  session always starts on the cache proxy), the connection stays alive (a
  ReadyForQuery on the simple protocol; Sync's on the extended one), and the
  pin is deliberately NOT rolled back. `escalateWorker` tags it
  `errS3CacheReapplyFailed` and `failEscalation` routes it — every call site
  goes through that dispatcher so the two severities can never be confused at
  one of them. **`SHOW` must never lie**: it activates iff an option is still
  pending.
- **Billing is largest-size-wins over the whole connection** (v1). The size is
  stamped at activation and re-stamped at escalation with the target profile;
  escalation only ever goes small→standard, so that stamp IS the maximum. The
  pre-activation idle prefix bills at the first acquired size.
- **Accepted gaps (do not "fix" silently — they are decisions):** per-org
  connection admission and the vCPU lease now happen at the FIRST STATEMENT,
  not at connect (the connect-time reshard/migration/draining gates are
  unchanged); a one-shot per-user `kill` landing inside the switcher's
  destroy→create window is missed, and so is a `kill` against a connection that
  authenticated but has never activated (no session for
  `DestroySessionsForUser` to iterate and no registered conn-closer until the
  first statement) — `disable` covers BOTH, because the activation/escalation
  re-check refuses the session outright (28000); extending `kill` to
  authed-but-unactivated connections is a named follow-up, not implemented; and
  a client that sends TCP FIN mid-activation does not abort the in-flight spawn
  (the message loop is blocked in the acquire) — the completed worker parks
  hot-idle for the org's next connection.
- Touching classification, the pin hooks, activation, escalation, the OOM
  retry, or the profile resolution → update `server/tier_classify_test.go`,
  `server/conn_tier_test.go`, `server/conn_tier_exec_test.go`,
  `server/conn_lazy_activation_test.go`, `server/s3_cache_test.go`,
  `controlplane/session_activation_test.go`,
  `controlplane/worker_profile_test.go`, `controlplane/compute_size_test.go`,
  AND the `exploratory_tier` / `exploratory_lazy_activation` /
  `exploratory_state_pin` / `exploratory_oom_escalation` /
  `org_default_profile` / `sized_worker` (GUC bypass) / `assert_worker_pod`
  assertions in `tests/mw-dev/e2e/harness.sh`. The harness header above
  `exploratory_tier` records which existing assertions the tier's connect
  semantics changed and why — keep that audit current.

## Hot-Idle Pool Reporting + Per-Org Caps (remote backend)

Two operator surfaces for the hot-idle pool, both over the durable runtime
store (`worker_records`, the only source that sees parked workers — they hold
no session):

- **Reporting**: `configstore.ListHotIdleByOrg` aggregates `hot_idle` rows
  per org (count, summed vCPU/memory, oldest park) and
  `GET /api/v1/workers/hot-idle` (`controlplane/admin/live.go`) joins each
  org's configured caps. Backs the Workers page "Hot idle by org" card
  (sortable, default memory-pinned desc). **Worker shape resolution is a
  three-step chain, everywhere**: the worker's explicit profile wins, else
  the org's default worker profile, else the CP-global default worker shape
  (`cfg.K8s.WorkerCPURequest`/`MemoryRequest`, else the 8/16Gi constants) —
  an unsized worker on a default-less org must never report zero cpu/memory
  (it pins real pod requests). The same chain backs the fleet rollup
  (`ListWorkerLifecycleStats`) and the cap sweep's shape math
  (`orgHotIdleLimitsFromSnapshot`); unparseable quantities still contribute 0.
- **Caps**: `max_hot_idle_workers` (count), `max_hot_idle_cpu` and
  `max_hot_idle_memory` (k8s quantity strings, e.g. `"16"` / `"64Gi"`) on
  `duckgres_orgs` (migration 000037; 0/"" = unlimited). Editable via the
  admin org PUT + the org detail form. Invariants:
  - **Enforcement is a convergent janitor sweep** (`reapHotIdleCaps`), NOT a
    park-time check: on each tick it retires the OLDEST parked workers
    (oldest-first listing) until the org is within ALL configured limits.
    This uniformly covers every park path AND cap decreases — lowering a cap
    drains the excess on the next ticks (config-poll reload, then the 5s
    janitor tick). Retires go through the fenced `RetireFromSnapshot` CAS
    with origin `janitor_hot_idle_cap` (a distinct metric origin — a nonzero
    rate means an operator's cost ceiling is biting, not stale capacity).
    On a retire error the sweep STOPS that org for the tick (never marches
    on and over-reaps on a transient failure).
  - **Cap wins over the floor** (`DefaultWorkerMinHotIdle`): the floor only
    guards the TTL reaper; the cap is the explicit operator intent. An org
    with floor > cap is a contradictory config that resolves to the cap.
  - **Validation is fail-closed on the write path**: a cap quantity that
    doesn't parse (or is zero/negative) is 400'd by the admin org PUT,
    because the sweep reads those as UNLIMITED — accepting them would
    silently mean "no cap".
  - The sweep runs even when the TTL reaper is disabled (`hotIdleTTL == 0`)
    — a cap is a hard ceiling, not a freshness rule.
- Touching any of this → update `tests/configstore/hot_idle_reporting_postgres_test.go`
  (+ the migration asserts in `migrations_postgres_test.go`),
  `controlplane/janitor_test.go` (the cap sweep cases +
  `TestOrgHotIdleLimitsFromSnapshot` glue test),
  `controlplane/admin/api_test.go` (`TestUpdateOrgHotIdleCaps*`) +
  `live_test.go` (`TestHotIdleRoute`), `ui/src/pages/Workers.test.tsx` +
  `OrgDetail.test.tsx`, AND the `/workers/hot-idle` envelope +
  `hot_idle_reporting_and_cap` assertions in `tests/mw-dev/e2e/harness.sh`.

## Worker Drain Protocol (graceful shutdown, #690)

Remote worker pods drain on SIGTERM (pod deletion): they reject new work, keep
in-flight work alive, then exit; the CP marks them `Draining` (not crashed) and
retires them cleanly. Drain readiness is tracked by a refcount (`activeWork` in
`duckdbservice/service.go`) of "drain tokens" — one taken per unit of in-flight
work (query, txn, metadata stream, COPY, activation), released when it finishes.
Invariants: take exactly one token when work starts and release exactly one when
it ends on **every** path (a leak hangs drain to the shutdown timeout, an early
release lets shutdown kill live work); `reapIdle` releases tokens stranded by a
`GetFlightInfo` whose `DoGet` never arrived. `terminationGracePeriodSeconds=3600`
(`k8s_pool.go`) must stay above `workerShutdownDrainTime` (55m).

**An adopted worker gets NO informer event on its owning CP.** Each CP's pod
informer (`startInformer`, `k8s_pool_reconcile.go`) is label-scoped to pods
THAT CP spawned (`duckgres/control-plane=<POD_NAME>`), so a worker claimed
from another CP's hot-idle pool (`k8s_pool_acquire.go`, fresh `done`
channel) never has `w.done` closed when its pod exits. The `HealthCheckLoop`
draining branch therefore MUST NOT rely on the informer as its only exit:
after `drainingPodVerifyThreshold` consecutive failed probes it `Get`s the
pod itself and, on NotFound / terminal phase, takes the same drained-lease
path the informer would (`handleTerminatedWorkerLease`, origin
`health_check_drained_pod_gone`). Without this, an adopted worker that was
SIGTERM'd (node disruption) and exited is probed every 2s tick forever —
one zombie is ~300 failed health checks per 10m, and four of them leaked for
three days in a production cluster and tripped the health-check-ratio alert.
The leader janitor's `reapOrphanedDrainingWorkers` (origin
`janitor_draining_orphan`, grace `defaultDrainingOrphanGrace`) is the
cluster-wide backstop: a durable `draining` row past its grace whose pod is
gone is retired via the fenced `RetireFromSnapshot` CAS, which also unsticks
the owner's local object (its next probe sees a non-draining row and drops
it). Both are pod-verified — a pod that still exists is left to drain, however
many probes fail. A local object whose durable row is draining under ANOTHER
owner is not waited on: the durable check fails and the ordinary 3-failure
mark-lost path drops it as a stale lease. Expect a steady trickle of
`duckgres_worker_lifecycle_transitions_total{operation="drain",origin="worker_drain",outcome="fence_miss_lease"}`:
a SIGTERM'd worker reports `draining` to EVERY CP still holding a local object
for it (previous owners keep theirs after a sibling claims the worker from
hot-idle), and only the current owner's lease can land the CAS — the misses
are the fence working, not the owner's drain failing. Touching the informer
selector, the draining branch, or the
sweep → update `controlplane/k8s_pool_draining_orphan_test.go` +
`janitor_draining_orphan_test.go` (unit-only; see `tests/mw-dev/README.md`
for why the e2e cannot stage a cross-CP adoption + pod exit).

## PostHog Logs (OTLP) — LOAD-BEARING CONTRACT

Process-level slog → PostHog Logs via OTLP (`internal/cliboot.InitLogging`).
This is **not** a replacement for `ducklake.system.query_log`. Product-analytics
events (`POSTHOG_ANALYTICS_API_KEY`) stay on the capture API.

- **`service.name` is the process role**, not `duckgres-<identifier>`:
  `duckgres-control-plane` / `duckgres-worker` / `duckgres-reshard` /
  `duckgres` (standalone). `DUCKGRES_IDENTIFIER` is resource attr
  `duckgres.deployment`. Logs and traces share `otelResource(bi)`.
- **Default PostHog level is WARN** (`DUCKGRES_POSTHOG_LOG_LEVEL`); stderr
  stays at `DUCKGRES_LOG_LEVEL`. User-class `Query execution failed.` stays
  Info and does not export at the default.
- **Query text default is `redacted`** (`RedactForLog`+4096). Secret DDL is a
  placeholder; ordinary SELECT text still leaves. Never redact on `LINE 1:`.
- **Public connection key is `pid`**, not `connection_id`. Org/worker are
  process-scoped on workers (`stampWorkerLogIdentity`); `user`/`pid` live on
  the session logger and must never be `SetDefault`'d.
- **Worker `POSTHOG_API_KEY` is a named `env:` `secretKeyRef`** copied from
  the CP pod spec (`controlplane/pod_env.go`, `k8s_pool_spawn.go`). Never
  `envFrom`. Never `os.Getenv` → `value:`. If Get/`POD_NAME`/named env misses:
  one WARN and omit — **never fail spawn**. Do not forward
  `ADDITIONAL_POSTHOG_API_KEYS` or `POSTHOG_ANALYTICS_API_KEY`.
- **`FlushLogging()`** only on the listed drain `os.Exit` sites (worker
  success/timeout; CP SIGTERM after drain; `drainAfterUpgrade`). Do not wrap
  every startup `os.Exit(1)`.
- **Exporter health** is scraped on the CP:
  `duckgres_otlp_log_export_failures_total{source,reason}` (no `{org}`).
  Workers report `otlp_export_enabled` / `otlp_export_failures` on the
  health-check JSON; the CP `Add`s last-seen **deltas** only. Workers do not
  call `InitMetrics`.
- Touching spawn env, health JSON, or the handler stack → update
  `controlplane/k8s_pool_spawn_test.go`, `controlplane/otlp_export_test.go`,
  `internal/cliboot/*_test.go`, AND `assert_worker_pod` in
  `tests/mw-dev/e2e/harness.sh` (plaintext-key assert; secretKeyRef copy when
  the CP named env has one).

## Per-Session S3 Cache Bypass (`duckgres.s3_cache`, remote backend)

`SET duckgres.s3_cache = on|off|passthrough` (default `on`; also a `-c`
startup option) controls the node-local S3 cache-proxy DaemonSet. `off`
bypasses its cache transport for cold-read benchmarking; `passthrough` keeps
requests flowing through cache-proxy while skipping its cache, so cache-off
workloads retain per-request proxy instrumentation.
Mechanism: the CP intercepts the duckgres-namespaced GUC (never forwarded to
DuckDB) and, on every state flip, calls the worker's `SetSessionS3Cache`
action (`flightclient.FlightExecutor.SetS3CacheMode` →
`SessionPool.SetS3CacheMode`), which rebuilds the `ducklake_s3` secret:
`off` = the org's native HTTPS transport, so httpfs CONNECT-tunnels through
the proxy as opaque TLS (no cache reads/fills — the deliberate, reversible
form of the mw-prod-us 2026-07-17 incident); `passthrough` = the same
`overrideS3EndpointForCacheProxy` transport as `on`, with a worker-local
marker that makes cache-proxy use `forwardUncached` and strips that marker
before it reaches S3; `on` = the normal caching transport. Global `http_proxy`
is never touched (post-attach propagation to DuckLake subcatalogs is
unreliable; secrets are consulted per request). Instance-global secret + one
session per worker = session-scoped in effect. Invariants:

- **State follows the worker, never leads it.** The session flag
  (`clientConn.s3CacheMode`) flips only after the worker swap succeeds; a
  failed swap fails the SET (`XX000`) / the batch / the connect (startup
  option), so `SHOW` can never report a transport the worker isn't using.
- **A bypass must never leak into the org's next session.** `CreateSession`
  restores the proxy transport before the session starts and a restore
  failure fails the create (CP retries); `DestroySession` restores
  best-effort. Both no-op unless a bypass is actually in effect.
- **Credential rotation respects the flag.** The hot-idle/mid-session refresh
  (`reuseExistingActivation`) rebuilds the secret with or without the proxy
  transport according to `s3CacheMode`; all secret rebuilds serialize on
  `secretSwapMu` so the last write always matches the flag. (Without this, an
  hourly STS rotation silently re-enables the cache mid-benchmark — the
  inverse of the 2026-07-17 incident.)
- **On an exploratory-tier connection the startup option is applied at
  ACTIVATION, not at connect** (there is no worker at connect). The CP parks it
  (`SetPendingS3CacheOption`) and `ensureSessionActive` applies it right after
  installing the executor — never inside the activator, where the executor is
  still nil and the swap would silently no-op while the session flag flipped. A
  failed apply is a fatal `XX000` activation failure, and a tier escalation
  re-applies the bypass on the new worker. `SHOW` activates iff an option is
  still pending, so it can neither lie nor spend a pod needlessly. See the
  Exploratory Worker Tier section.
- **Closed enum, validated on every set path** (`transform.NormalizeS3Cache`):
  PostgreSQL boolean spellings, normalized to on/off, plus `passthrough`; anything else is `22023`
  (simple/batched SET, extended Parse, and the startup option — which the CP
  validates BEFORE acquiring a worker). The rejection never echoes the value.
- **Scope: remote/k8s shared-warm workers with a cache proxy.** Elsewhere
  (standalone, process backend, no `DUCKGRES_CACHE_ENABLED`) the worker swap
  is a no-op and the GUC is session-state only — there is no cache to bypass.
  The query-log sink's activation stays proxied unconditionally.
- Touching the interception, swap, restore, or refresh interplay → update
  `transpiler/s3_cache_test.go`, `server/s3_cache_test.go`,
  `duckdbservice/s3_cache_test.go`, AND the `s3_cache_guc` assertion in
  `tests/mw-dev/e2e/harness.sh` (mw-dev has no cache proxy, so the e2e covers
  the client-visible plumbing incl. the worker action round-trip; the swap
  itself is unit-only — see the harness header note).

## Mid-Session Worker TTL (`duckgres.worker_ttl`, remote backend)

`SET duckgres.worker_ttl = '20m'` / `SHOW` / `RESET` let a client that cannot
set startup options change its bound worker's pool-side hot-idle TTL
mid-session (transpiler interception in `transform/setshow.go`, connection
apply in `server/conn_worker_ttl.go`, control-plane hook
`controlplane/worker_ttl.go::workerTTLControlFor` → pool
`SetWorkerTTLForPID`). Full design: `docs/design/worker-ttl-pool.md`.
Invariants:

- **Same trust boundary as the `duckgres.worker_*` startup options**: gated
  on `DUCKGRES_K8S_ALLOW_CLIENT_WORKER_PROFILE` (SET rejected 22023 when off)
  and clamped to `DUCKGRES_K8S_WORKER_MAX_TTL`; a clamped value is stored
  clamped.
- **State follows the worker, never leads it** (same rule as s3_cache): the
  session override flips only after the apply hook succeeds, and SHOW falls
  back to the bound worker's CURRENT pool TTL (`Current`), so SHOW never
  reports a TTL the worker won't park with.
- **Whole-minute granularity, enforced at validation.** The parked TTL is
  persisted as `ttl_minutes` (integer; 0 = "reaper applies the deployment
  default"), so `NormalizeWorkerTTL` REJECTS zero and sub-minute values with
  22023 — accepting them would park the worker for the deployment default
  while SHOW reported the shorter value. (Sub-minute STARTUP options still
  truncate at park — pre-existing.)
- **Both reapers read the persisted override** (`ttl_minutes` stamped at the
  hot→hot_idle park): the leader janitor's expiry query and the per-CP
  fallback.
- **Exploratory tier**: SET on a lazily-activated connection acquires a
  worker first (the TTL is pool-side per-worker state — there is nothing to
  apply to otherwise); SHOW never acquires (the connect-time baseline is the
  truthful answer pre-worker). Escalation re-applies the override on the new
  worker; on failure the override is RESET (statement error, not
  connection-fatal) — and BOTH session-worker-state re-applies (s3_cache +
  worker_ttl) always run (an s3 failure must not skip the TTL re-apply).
- **Describe must cover BOTH 'S' and 'P'** for the intercepted SET/SHOW — a
  portal-Describe miss returns NoData and probes DuckDB (acquiring a worker
  on lazy connections) for a GUC it does not know.
- **Standalone/process backends** accept SET/SHOW as session state only (no
  per-worker hot-idle TTL exists there).
- Touching the interception, apply hook, park persistence, or the reapers →
  update `transpiler/worker_ttl_test.go`, `server/worker_ttl_test.go`,
  `controlplane/worker_ttl_test.go` + `k8s_pool_worker_ttl_test.go`, AND the
  `worker_ttl_guc` assertion in `tests/mw-dev/e2e/harness.sh`.

## User Persistent Secrets (multitenant remote backend)

`CREATE PERSISTENT SECRET` from a client survives across sessions and worker
pods: the CP intercepts it (`server/conn_user_secrets.go`, classification in
`server/usersecrets/`), executes it on the live session first (DuckDB
validates), then stores the statement AES-GCM-encrypted in the config store
(`duckgres_org_user_secrets`, keyed org/user/name) and replays it in the
`CreateSession` payload on the user's future sessions
(`duckdbservice/user_secrets.go`). Enabled by the env-only
`DUCKGRES_USER_SECRET_KEY` (base64 32-byte AES key); unset → clear 0A000
error. Plain/TEMPORARY `CREATE SECRET` stays session-scoped passthrough.
Invariants for anyone touching this path:

- **Cross-user isolation is the wipe at session create, not the destroy-time
  cleanup.** DuckDB secrets are instance-global, and a hot-idle worker is
  reused across users of an org: `wipeUserSecrets` drops ALL user-created
  secrets — persistent ones AND non-persistent (plain/TEMPORARY `CREATE
  SECRET`) ones, which pass through to the worker and would otherwise leak to
  the next user. It preserves only the system-managed allowlist
  (`usersecrets.IsReservedName`: `ducklake_s3`
  + the `__default_*`/`duckgres_*` prefixes, which activation re-creates). It
  MUST run before replay on every CreateSession in shared-warm mode, and a
  wipe failure MUST fail the session.
- **Client-ATTACHed catalogs get the same treatment, in the same place.**
  `detachUserCatalogs` (`duckdbservice/user_catalogs.go`) runs right after the
  secrets wipe on every shared-warm CreateSession and a failure MUST fail the
  session (best-effort again at DestroySession). Attached databases are
  instance-global exactly like secrets, and an attached catalog freezes its
  connection string — credentials included — at ATTACH time, so it outlives
  the wipe of the secret it was built from: without the detach, user B can
  query user A's attached Postgres without ever holding A's credential. It is
  also a correctness bug, not just isolation: clients attach with
  `ATTACH IF NOT EXISTS ... AS db`, a no-op when the previous session left a
  `db` behind, so the new session silently reads the PREVIOUS session's target
  (2026-09-18: a tenant's sqlmesh run inherited an analyst's `db` endpoint and
  its parallel postgres scans died with `SET TRANSACTION SNAPSHOT ... snapshot
  does not exist`). Preserved: DuckDB-internal catalogs, the instance's primary
  database (resolved by lowest oid — it is a file stem, not `memory`, when
  DataDir is set), and the `isSystemCatalog` allowlist (`ducklake`, `delta`,
  `__ducklake_metadata_*`). **A new worker-managed ATTACH must be added to that
  allowlist** or the next session create will detach it.
- **Execute-then-persist ordering.** Persist only statements DuckDB accepted;
  a store failure after a successful exec is an ERROR telling the user the
  secret will NOT survive the session. Replay failures at session create are
  warnings, never connection refusals.
- **No silent non-persistence.** Any path where persistent-secret DDL would
  execute but not persist must REJECT instead, including multi-statement batches
  and parameterized statements in the control-plane interception path. Otherwise
  the secret works for one session and is silently deleted by the next session's wipe.
- **DROP's store-fallback is gated on DuckDB's not-found error only**
  (`isSecretNotFoundError`). Any other exec failure (cancel, RPC error,
  ambiguity, aborted txn) must surface and leave the store untouched — a
  false "DROP succeeded" is fatal for a credential revocation.
- **Never log/store secret statement text.** `usersecrets.RedactForLog` guards
  client-query and worker-statement logs, `logQueryError`, the query log, spans,
  and pg_stat_activity
  (`currentQuery`); keep new logging of query text behind it. Engine **error
  messages echo the offending SQL** (DuckDB emits `LINE 1: ... SECRET '...'`),
  so a failed CREATE SECRET leaks the credential via the `error` attribute /
  query-log `Exception` even when the query attribute is redacted —
  `usersecrets.RedactErrorForLog(query, errMsg)` guards those error sinks
  (`logQueryError`, `logQuery`); keep new error logging behind it too, and pass
  the original (un-redacted) query so it can classify.
- Touching the interception, wipe/replay, catalog detach, or payload shape →
  update `server/conn_user_secrets_test.go`,
  `duckdbservice/user_secrets_test.go`, `duckdbservice/user_catalogs_test.go`,
  and the `persistent_user_secret`(+`_isolation`, which also carries the
  attached-catalog leak check) assertions in `tests/mw-dev/e2e/harness.sh`.

## Admin Console (VPC-private web UI, `kubernetes` tag)

`controlplane/admin/` serves a React admin console + REST API on `:8080` — the
operate-everything surface (metrics, live queries/sessions/connections, recent
errors, worker fleet, live cluster node/pod topology, full config store, user
impersonation, audit log; sliceable by org + user). Design + decisions:
`docs/design/admin-ui.md`; package details:
`controlplane/admin/README.md`. Exposed VPC-privately via an internal-scheme ALB
+ Cognito (Google SSO) behind Tailscale (charts: `ingress-admin.yaml`). Invariants:

- **Frontend is an embedded React/Vite SPA** (`ui/`, built to `ui/dist/`,
  `//go:embed all:ui/dist` in `embed_ui.go`, SPA-fallback served by Gin; the SPA
  owns `/`). `ui/dist` is a **gitignored build artifact** — only `ui/dist/.gitkeep`
  is tracked, so the embed has a target and `go build` compiles without node
  (the server then serves a "UI not built" notice). `just ui-build` builds it
  locally; both `Dockerfile` and `Dockerfile.controlplane` run `npm run build`
  **before** `go build`. Do not delete `.gitkeep` and do not commit `ui/dist`.
- **Two-tier authz** (`authz.go`): `AuthMiddleware` resolves every `/api/v1`
  request to admin (valid `TokenSet` internal secret — service/break-glass) or to
  an SSO identity from the ALB `X-Amzn-Oidc-Data` JWT. The SSO email
  (`@posthog.com` + `email_verified != false`, else 401) is mapped to a role
  **per-request** by a `RoleResolver` backed by the `duckgres_operators` config-schema
  table (goose migration `000006_create_operators.sql`) — `admin` row → admin, else
  viewer. Admins manage operators in the admin-only **Operators** page
  (`ui/src/pages/Operators.tsx` → `/api/v1/operators`, `operators_api.go`) — a
  deliberately distinct surface from the **Org Users** page (`ui/src/pages/Users.tsx`,
  `/users`, per-org *database* logins). The two are labelled + cross-linked in the
  UI so "users" is never ambiguous: Operators = who can sign in to this console;
  Org Users = customer DB accounts. The operators GET is `RequireAdmin`, so
  `useOperators` only fires for admins (viewers see an "admin only" notice). The
  first SSO login auto-provisions a create-only **viewer** row, and the first
  admin is minted by logging in over the break-glass internal token and patching
  that row to `admin`. The **last-admin guard** (`operators_api.go`) refuses to
  demote/delete the final admin (409) so the console can't be locked out.
  `RoleGate` requires admin for all mutating verbs + the audit GET.
  `AuditMiddleware` records every mutation. Keep new mutating routes under this
  gate; never add a write path that bypasses RoleGate/audit.
- **Impersonation is a real session** (`impersonate.go` + `admin_providers.go`):
  it reuses `SessionManager.CreateSessionWithProtocol` (workers trust the CP — no
  password) and **always** `DestroySession` in a defer. Admin-only, every
  statement audited with the admin actor + `usersecrets.RedactForLog` SQL; writes
  require `allow_write=true` (conservative classifier — WITH/CTEs count as
  writes). It consumes a worker under one-session-per-worker and counts against
  the org's connection limits — do not silently exempt it.
- **Metrics proxy is allow-listed** (`metrics_proxy.go`): the client passes a
  panel KEY, PromQL is built server-side from `rangePanels` (never an open PromQL
  relay) and forwarded to `DUCKGRES_PROMETHEUS_URL`. Org-labelled panels keep
  slicing enforced.
- **Product monitoring API is tenant-safe** (`monitoring.go`):
  `GET /api/v1/orgs/:id/monitoring/{snapshot,series}` is internal-secret-only
  for the PostHog backend. It fixes tenant identity from the path, uses only
  org-scoped runtime-store reads and Prometheus selectors, returns CP coverage
  for partial live-state fan-out, and strips usernames, PIDs, pod/image names,
  CP ownership, SQL, client/trace data, and secrets. The series metric and
  window are closed enums; never turn it into arbitrary PromQL or reuse the
  operator payloads wholesale.
- **Env-only knobs**: `DUCKGRES_PROMETHEUS_URL` (read in
  `multitenant.go`; set by the chart). The audit table `duckgres_admin_audit` is
  AutoMigrated at startup (operational state, not goose-migrated tenant config).
  The `duckgres_operators` table is authoritative access-control data, so it lives
  in the config schema via goose migration `000006_create_operators.sql`, not
  AutoMigrate.
- `ManagedSession.Username` is populated at session create so the console can
  slice live sessions/queries by user; keep it set on every create path.
- **Errors page is a redacted, in-memory live-triage buffer** — NOT durable
  history. Every failed query is captured into a bounded per-server ring
  (`server/recent_errors.go`, `DefaultRecentErrorCap=500`) at the single
  `logQueryError` tap (`server/conn.go`), surfaced at `GET /api/v1/errors` and
  merged across CP replicas by `PeerFetcher.FetchPeers` (each error belongs to
  exactly one CP — disjoint union, no worker-id dedup; sorted newest-first, then
  capped). The ring stores ONLY the redacted forms: `Query` via
  `RedactForLog`, `Message` via `RedactErrorForLog` — a failed CREATE SECRET
  must never leak its credential into the ring. Keep the capture behind those
  redactors; long-term error history lives in the external query-log pipeline
  (Kafka sink), not here.
- **Per-user kill switch** (`live.go` routes + `admin_providers.go` +
  `session_mgr.go::DestroySessionsForUser` + `configstore` `disabled` column):
  - `POST …/users/:username/kill` is a **one-shot** terminate — it tears down all
    of a user's sessions + in-flight queries but does NOT block reconnects. It
    reaches only connections that HAVE a session: on the exploratory worker tier
    a connection that authenticated but has not yet run its first statement has
    no session (and no registered conn-closer) for `DestroySessionsForUser` to
    iterate, so `kill` misses it. Only `disable` covers that connection — its
    first statement's activation re-check refuses the session with 28000.
    Extending `kill` to authed-but-unactivated connections is a follow-up.
  - `POST …/users/:username/disable` is the **persistent block**: it sets the
    `duckgres_org_users.disabled` column (goose migration
    `000011_add_org_user_disabled.sql`), kills the user's live sessions, AND
    refuses the user's NEW pgwire connections at auth time (`control.go`,
    distinct `28000` "account is disabled" error, emitted only after the
    password checks out so it never leaks account existence). `enable` reverses
    it. The disabled state is read from the in-memory
    snapshot, so disable/enable call `ConfigStore.ReloadSnapshot()` to make the
    flip effective immediately instead of one config-poll later.
  - These are **cluster-wide**: a user's sessions live on whichever CP replica
    owns each connection, so the handlers fan out the kill/disable/enable to peers
    via `PeerFetcher.PostPeers` (POST sibling of the read fan-out, same
    `?scope=local` recursion guard) and sum the per-CP `killed` counts. The
    snapshot reload is fanned out too so every replica enforces the block at once.
  - Kill must be **scoped to the target user** — never tear down another user's
    sessions on the shared org stack (the regression the e2e asserts with a
    concurrent root query that must survive).
- **Live Nodes view** (`ui/src/pages/Nodes.tsx` + `pages/nodes/peepernetes.{ts,css}`,
  a port of the standalone peepernetes visualizer): a full-bleed, animated
  cluster node/pod TV — nodes grouped by karpenter nodepool (or by namespace /
  deployment), CPU/MEM request bars, pod chips colored per deployment,
  placeholder/system-pod classification, Karpenter empty-node reclaim countdown,
  draining-duration on nodes (deletion-timestamp or client-tracked first-seen)
  and terminating-duration on pods, each pod's running image, unscheduled tray,
  and a synthesized event ticker. Its header carries only the filters; the
  cluster counters live in the shared admin **Topbar** and show on EVERY page —
  the Topbar polls `GET /cluster/summary` (`useClusterSummary`), a server-side
  aggregate (`cluster.go`) of nodes (duckgres nodepools) · CP replicas · running
  **workers** (label `app=duckgres-worker`, NOT every app pod — so it matches the
  worker chips + the CP's own worker accounting) with their vCPU/GiB request
  totals as a sub-line · **placeholders** with their vCPU/GiB and the cpu%/mem%
  those headroom pods are OF the worker totals · pending. Computing it server-side
  (not from the Nodes view's pushed counts) is what lets the totals appear on
  every page, not just while the view is mounted. The view has no separate live
  indicator — the Topbar's "Connected" dot (admin-API reachability) is the single
  green pulsing live signal. It's imperative DOM (mounted
  by the React page into a `.peeper` root, scoped CSS + `pn-`-prefixed keyframes)
  and does NOT use native K8s watch — the browser can't reach the API, so it
  POLLS four **read-only** projected endpoints (`server/`-free; `cluster.go`):
  `GET /cluster/{nodes,pods,events}` project the in-cluster objects down to the
  minimal K8s-shaped subset the view reads (annotations trimmed to
  `kubernetes.io/config.mirror`; no raw objects), and `GET /cluster/nodepools`
  proxies the karpenter NodePool CRD (v1→v1beta1, degrading to an empty list when
  karpenter is absent). Backed by the shared K8s pool's clientset
  (`Extras.ClusterClient`, nil on non-k8s backends → routes unregistered). All
  four are GETs so RoleGate admits viewers; there is no mutation path. **RBAC:**
  these reads are cluster-scoped / cross-namespace, which the CP's in-namespace
  Role doesn't cover — the grant lives on its own `duckgres-control-plane-cluster-topology`
  ClusterRole in the `charts` repo (`charts/duckgres/templates/rbac.yaml`), bound
  to the CP SA. It's a *separate* role (not folded into `duckgres-duckling-reader`)
  so binding duckling-reader elsewhere doesn't drag these broader reads along and
  trip RBAC escalation-prevention. When the ClusterRole is absent the handlers
  **degrade a Forbidden to an empty `{items:[]}` (200)** and log a warning, so the
  view shows nothing rather than 500ing — the e2e CP hits exactly this path (its
  SA can't be granted cluster-scoped RBAC from CI), so `admin_console_api` only
  asserts the `{items:[...]}` envelope; projection shape is covered by
  `cluster_test.go`. Touching
  the projection/endpoints or the view → update `controlplane/admin/cluster_test.go`
  and the `/cluster/{nodes,pods,events,nodepools}` checks in `admin_console_api`
  (`tests/mw-dev/e2e/harness.sh`).
- **Trino cell views** (`trino.go` + `trino_client.go`, UI
  `pages/Trino{Cluster,Queries}.tsx` + `pages/OrgTrinoCard.tsx`, derivations
  in `lib/trino.ts`): live queries with an admin-only, audited kill; the
  cell's coordinator/node health; per-org Trino provisioning state (the
  `duckgres_managed_warehouse_trino` state/status_message/ready_at/failed_at
  columns, which were surfaced nowhere before, so a failed Trino provision
  was silent). Read as the OBSERVER principal, never the provisioner's
  admin — see the Trino Cells section and `admin/README.md`. Payloads are
  redacted (SQL) and projected (never `TrinoEnabledOrg.RootPasswordHash`). A
  ready, reachable org includes non-secret `status.connection` coordinates;
  passwords remain write-only provision/reset results.
  Every read is cached + timeout-bounded and degrades to `available:false`
  plus a reason rather than erroring the page: the console must render
  during exactly the incident it exists for. A shared-pool cell has no fixed
  coordinator, so its observer (`trino_pool_observer.go`) lists the pool's
  live instances on every call and fans out to each instance's Service over
  forwarded HTTPS; usage metering polls the same observer. Never build a
  pool observer from `CoordinatorURL` - it is empty (#1216). Unset
  legacy URL and registry configuration leaves the routes unregistered.
- Touching any of the above → update `controlplane/admin/*_test.go` (esp
  `authz_test.go`, `kill_switch_test.go`, `operators_api_test.go`,
  `trino_test.go`, `trino_client_test.go`),
  `controlplane/session_mgr_test.go`
  (`TestDestroySessionsForUser`), `controlplane/configstore/store_test.go`
  (`TestDisabledUserEnforcement`), the `Operators`/`Org Users` UI pages +
  `ui/src/pages/Operators.test.tsx`, AND the `admin_*` / `admin_operators` /
  `impersonation_*` / `user_kill_switch` / `user_disable_block` assertions in
  `tests/mw-dev/e2e/harness.sh`.

## Project-Scoped Logins (`project_reader` / `project_user`) — LOAD-BEARING CONTRACT

A team can hold two generated logins, both bound to one `duckgres_org_teams`
row via `duckgres_org_users.access_mode` + `team_id` (migrations `000026`,
`000031`): `project_reader` (`posthog_team_<id>`, read-only — the PostHog SQL
editor's login) and `project_user` (`posthog_team_<id>_rw`, read/write). Minted
by `PUT /api/v1/orgs/:id/teams/:team_id/{project-reader,project-user}` (admin
API only). Both are enforced by the **query gateway**, not PostgreSQL `GRANT`.
Invariants:

- **The two modes derive the SAME namespaces.** `ConfigStore.OrgUserQueryAccess`
  computes `AllowedSchemas` (`<schema>`, `<schema>_data_imports`,
  `shadow_<team>_models`) + `AllowedRelations` (the legacy `posthog.<events|
  persons>` overrides — a non-NULL override grants it even when it spells the
  derived default) identically for both; **`ReadOnly` is the only difference**.
  Write authorization must NEVER widen the reachable relation set — a project
  user is a project reader that may also mutate what it can already see. The
  e2e asserts cross-project denial in read AND write shapes for this reason.
- **Fail closed on an unresolvable team.** A scoped user whose team row is
  missing or disabled gets an empty, `ReadOnly` policy — a project user is
  DOWNGRADED, never left writing into a scope nothing can confirm.
- **`QueryAccessPolicy.Authorize` (`server/query_access.go`) is the single tap**,
  called on simple query and extended Parse. It is
  default-deny: the parser is the authorization boundary, so anything
  `pg_query` cannot describe is rejected (this is why DuckDB-only spellings
  fail). `CREATE SCHEMA` / `ALTER … SET SCHEMA` stay denied: the schema set IS
  the project boundary.
- **A WRITE TARGET is checked by `authorizeWriteTarget`, never by the walk's
  `authorizeRangeVar`.** The walk's check is the READ-position one: it accepts a
  bare name that is a visible CTE (sound, because a defined CTE provably shadows
  a base relation of the same name) or an unqualified pg_catalog compat name.
  **Neither is sound for a target** — a write target does not bind to the CTE, so
  it resolves against the session `search_path` (`sessionmeta` leaves it at
  `main,memory.main`) and reaches a real relation outside the grant;
  `WITH shared AS (…) INSERT INTO shared VALUES (1)` was a working escape into
  `ducklake.main.shared` before this split. `authorizeWriteTarget` therefore
  requires schema qualification unconditionally. Do NOT infer target-ness from
  proto position: `walk()` descends into a Node's own oneof, so a bare
  `*RangeVar` message reaches `walkMessage` for reads too. **Adding a case to
  `authorizeWriteStatement` REQUIRES enumerating that statement's target
  field(s) there by name** (`DropStmt` scopes its dotted-name lists itself); the
  `walkMessage` `RangeVar` branch is only the lenient defense-in-depth net for
  positions that enumeration does not cover (e.g. `Constraint.pktable`).
  `TestQueryAccessPolicyRejectsUnqualifiedWriteTargets` is the tripwire.
- **Neither mode may carry passthrough** (it bypasses the compat layer that
  enforces the scope) — DB check constraint + admin API guard.
- **A policy change tears down live sessions.** `changedProjectScopedUsers`
  (`org_router.go`) fires on credential, team, disable AND **mode** flips; the
  demotion direction is load-bearing (a demoted user must not keep a
  write-authorized session). Deleting a team deletes BOTH of its logins.
- **One of each per team**, via two SEPARATE partial unique indexes on
  `(org_id, team_id)` — deliberately not one index across both modes, so a
  reader and a writer coexist.
- **Service credentials are NOT project-scoped and never touch
  `duckgres_org_users`.** `POST /api/v1/orgs/:id/service-credentials` is how a
  PostHog backend job (dagster) fetches a short-lived credential: a
  per-credential grant row with its own `svc_` identity, root-shaped
  (unrestricted) at the org level. See "Service Credentials" below for the
  full contract.
- Touching any of this → update `server/query_access_test.go`,
  `controlplane/configstore/query_access_test.go`,
  `controlplane/admin/api_test.go`, `controlplane/org_router_test.go`,
  `tests/configstore/org_teams_postgres_test.go` +
  `migrations_postgres_test.go`, `docs/postgres-compatibility.md`, AND the
  `project_reader_isolation` / `project_user_isolation` assertions in
  `tests/mw-dev/e2e/harness.sh`.

## Service Credentials (`POST /orgs/:id/service-credentials`) — LOAD-BEARING CONTRACT

How PostHog backend jobs (dagster today) authenticate to duckgres WITHOUT a
long-lived password living in Django. Replaces the org-root credential read
from a `DuckgresServer` row with a per-credential grant minted on demand by
the CP — AWS AccessKey/Secret style: each minted credential is its own
`duckgres_service_grants` row with its OWN identity (`credential_id`), its
own TTL, and its own rotation clock. Storage NEVER touches
`duckgres_org_users` — there is no shared login row for an operator rotation
or password update to clobber mid-run, and service credentials are not
project-scoped (root-shaped: unrestricted at the org level).

**Caller contract:**
`POST /api/v1/orgs/:id/service-credentials` with
`{principal, ttl_seconds?}`. `principal` is audit attribution
(`"dagster:events-backfill"`) only: every request creates an independent grant
with a new `credential_id` and secret, even when principals are identical.
`ttl_seconds` is clamped to [1 min, 1 h]
(default 15 min, the RDS-IAM precedent). Response is
`{credential_id, credential_secret, expires_at, connect}`; all fields are
always present. `POST /api/v1/orgs/:id/service-credentials/refresh` with
`{credential_id, ttl_seconds?}` rotates the named grant's secret by default.
Explicit `rotate_secret: false` renews only a live, non-revoked grant without changing its secret, preserving Trino gateway query ownership.
Mint and default refresh return `secret_rotated: true`; non-rotating renewal returns `secret_rotated: false` and omits `credential_secret`.
Clients must require the explicit false marker before retaining their in-memory secret, because an old server may ignore the renewal option and rotate.
When Trino service authentication is configured for the assigned cell and that cell is ready, responses also include `trino_connect` from its authoritative client endpoint.
`DUCKGRES_TRINO_SERVICE_AUTH_SECRET_FILE` is an opt-in JSON map of configured immutable stored cell IDs to dedicated token rotation sets (see the Trino service credential runbook).
The validation endpoint derives the calling coordinator cell from its bearer token and checks the org's current persisted cell assignment on every request; empty assignments and cross-cell grants are denied.
The caller is the internal-secret-authed PostHog backend — the routes sit
next to the other provisioning routes for exactly that trust class, NOT on
the admin/console side.

The `connect` block tells the caller WHERE to use the credential from the same
authoritative CP response that issued it, so nothing downstream re-derives its
own idea of the warehouse endpoint out of band (a Django `DuckgresServer` row
is exactly the drift this kills). It is
`{host, port: 5432, database: "ducklake", sslmode: "require"}`.
**`connect.host` is ALWAYS the org's canonical ingress name
`<org-id>` + the CP's configured managed-ingress suffix** (e.g.
`<org-id>.dw.us.postwh.com`) — the very value the pgwire TLS `server_name`
pins (the wildcard cert is `*<suffix>` and the SNI router resolves the
single-label prefix as the org; see "Native metadata Postgres proxy" /
`controlplane/sni_kubernetes.go`). It is one logical name handed back verbatim
for every caller: NEVER a pod IP, NEVER a ClusterIP, NEVER resolved per caller
network. How that name resolves for a given caller — public ingress vs an AWS
PrivateLink endpoint for dagster workers — is the caller network's business,
not the CP's. The CP wires the suffix from its first configured
`ManagedHostnameSuffixes` entry (`DUCKGRES_MANAGED_HOSTNAME_SUFFIXES`) at the
`RegisterAPI` site (`controlplane/multitenant.go`), falling back to
`provisioning.DefaultManagedIngressSuffix` when unwired.

**The load-bearing invariants:**
- **The credential IS its own grant row.** `credential_id` is
  `svc_<24 random hex>`, generated server-side, never caller-supplied; the
  client presents it as the pgwire username and the plaintext secret as the
  password. The bcrypt hash lives on the `duckgres_service_grants` row —
  `duckgres_org_users` holds NO service-credential material, so operator
  writes to the org's users table can never invalidate (or be depended on by)
  a minted credential.
- **Auth resolves `svc_`-prefixed usernames against the grants snapshot map
  ONLY** (`Snapshot.OrgServiceGrant`, loaded alongside `OrgUserPassword`).
  Expiry and revocation are enforced at that lookup: unknown, revoked,
  expired, and hash-blanked credentials all fail identically (with equalized
  bcrypt time — which state a credential_id is in is not probeable). A live
  grant is `revoked_at IS NULL AND expires_at > now()`.
- **Mint always creates.** `principal` is non-unique audit metadata, never an
  identity or reuse key. Every mint inserts a new row, returns its new ID and
  plaintext once, and leaves every same-principal credential untouched. This
  lets concurrent jobs keep a stable principal without sharing secrets.
- **Management is by `credential_id`.** A caller keeps the ID returned by its
  mint and supplies it to refresh/revoke. Losing the plaintext means minting a
  new credential; plaintext cannot be recovered from the stored bcrypt hash.
  New Duckgres servers ignore the removed `force_rotate` JSON field so an old
  caller can still reach the always-create endpoint, but that compatibility
  is one-way: deploy the always-create Duckgres version to the whole fleet
  BEFORE removing PostHog's `force_rotate`/reuse fallback. A new caller talking
  to an old server could otherwise receive a reused ID without plaintext.
- **Refresh always rotates and never creates.** Unknown `credential_id` →
  404; a REVOKED grant → 410 (revocation is terminal: refresh never
  resurrects). An EXPIRED (but unrevoked)
  grant MAY be refreshed — expiry only refuses NEW handshakes, so refresh is
  how a caller that missed the window recovers without minting a second
  identity for the same principal.
- **Established sessions are NEVER torn down** — not on expiry, not on
  rotation, not on refresh (the mint plane is separate from connection
  scheduling). Freshness is enforced only at the pgwire handshake — the
  RDS-IAM / Cloud-SQL-IAM contract. A long job's existing connection rides
  to completion; each NEW connection mints or refreshes afresh.
- **Revoke keeps the row** (`DELETE /api/v1/orgs/:id/service-grants/:credential_id`):
  `revoked_at` is stamped and the hash is BLANKED server-side, so a leaked
  credential can never authenticate again and the provenance (principal,
  mint/rotation times) survives for investigation. `GET
  /api/v1/orgs/:id/service-grants` is the flat, all-statuses list — no
  plaintext, no hashes (`PasswordHash` is `json:"-"`). Durable history is
  intentional because expired, unrevoked IDs remain refreshable and audit
  provenance must survive; do not add expiry deletion. If grant volume makes
  the operator list expensive, add pagination/archival as a separate API
  change rather than weakening lifecycle semantics.
- **Org deletion deletes all of that org's grant rows.** This is the explicit
  lifecycle boundary to the normal durable-history rule: leaving a grant
  behind would let its old secret authenticate if the org name were reused.
  Org creation takes the same advisory lock and clears pre-existing orphan
  grants only when the locked lookup confirms no current org row; duplicate
  creation and re-provisioning an existing org must preserve live grants.
- **Concurrency**: each mint owns a new row. Mint/refresh/revoke still take
  `LockOrgConnectionAdmissionTx` so org lifecycle changes cannot leave orphan
  grants and management of a named ID remains serialized.
- **Admission recognizes authenticated `svc_` identities as root-shaped.**
  They consume the org vCPU budget but do not require an org-user row or carry
  a normal per-user cap. Admission and post-acquisition worker switching do
  not reapply expiry/revocation: those are handshake-only checks, so an
  established session rides to completion.
- **After the write, THIS replica's snapshot is reloaded immediately**
  (`ReloadSnapshot`), then a best-effort `/api/v1/internal/reload-snapshot`
  fan-out to peer replicas (`PeerFanout`, wired from the same
  `clusterPeerFetcher` the admin kill-switch uses). Without both, a freshly
  minted credential routinely fails its first auth on whichever replica the
  pgwire connection lands on.
- Touching any of this → update `controlplane/provisioning/service_credential.go` +
  `_test.go`, `controlplane/configstore/service_credential.go` +
  `_postgres_test.go`, `controlplane/admin/api.go` (list/revoke), and the
  caller-side minter in the PostHog repo
  (`products/managed_warehouse/backend/service_credentials.py`).

## Compute-Usage Billing (managed-warehouse, remote backend only)

duckgres meters per-org compute usage of worker pods into 60s buckets in the
config store; the billing service **pulls** the accumulated usage over an HTTP
API and acks a watermark, at which point duckgres deletes the acked buckets.
Full design + decisions: `docs/design/billing-pull-api.md` (supersedes the
push/capture reporting hop of `billing-compute-seconds-plan.md`; the metering
side of that doc still applies). Scope is **only** the remote/k8s backend
(per-org worker pod with a known `WorkerProfile` size). Pipeline:

```
compute: conn end → in-proc counter keyed (org, informational team, query_source, worker size)
              │  flusher (~15s) UPSERT-increment → config-store buffer (cross-CP sum)
              ▼  duckgres_org_compute_usage (+ duckgres_compute_billing_cursor)
storage: leader sampler (~30m) → org's DuckLake metadata Postgres
              SUM(data+delete file sizes) × interval → duckgres_org_storage_usage
billing: GET /api/v1/billing/usage (usage + storage arrays, per key per UTC day, watermarks)
       → POST /api/v1/billing/ack {watermark_high} → cursor advance + delete ≤ it (BOTH tables)
safety:  leader-only GC hard-deletes buckets older than 30 days (WARN, alertable)
```

Two raw metrics per connection over its full lifetime, using the **provisioned**
worker size: `cpu_seconds = vCPU × ceil(conn_secs)`, `memory_seconds = GiB ×
ceil(conn_secs)`. Counted internally in integer **millicore-seconds** /
**MiB-seconds** (`compute_meter.go`) to avoid truncating a fractional-core /
sub-GiB worker; worker size is stored in the bucket key as exact NUMERIC
decimals (vCPU / GiB). `team_id` is **informational only** (an integer —
PostHog's `Team.id`; a JSON NUMBER on every API surface): duckgres does NOT
own team-level billing attribution — the external billing service maps
org → team(s) itself. The stamp is resolved from the config snapshot at
record time: compute buckets get the CONNECTING USER's team
(`duckgres_org_users.team_id`, e.g. a project-reader login) when it has one,
else the org's OLDEST team (min `created_at`, ties broken by the smaller
`team_id` — in practice the provision-time first team; `ConfigStore.OrgUsageTeamID`);
storage buckets always get the oldest team (`OrgOldestTeamID`). 0 appears
only defensively (unknown org / stale snapshot — a committed org always has
at least one team). Team changes/deletions NEVER re-attribute existing
buckets; `query_source` is the
`duckgres.query_source` session GUC (`standard` unless set; a mid-connection
change bills the whole connection under the final value). The GUC is a **closed
enum validated at SET time** (`transform.NormalizeQuerySource`): only
`standard` | `endpoints` (case-insensitive, normalized to lowercase; empty =
reset to default) — anything else is rejected with `22023` on every set path
(simple/batched SET, extended Parse, and the `-c` startup option, which rejects
the connection like invalid `duckgres.worker_*` options), and
`server.ConnectionBilling` clamps a non-canonical value to `standard` as
defense in depth so client junk can never become a billing bucket key.
Invariants for anyone
touching this path:

- **Metering is strictly best-effort and off the hot path.** A metering error
  (counter, flush) must NEVER block or fail a query or connection teardown. The
  connection-end record is added to an in-process counter (map+mutex,
  microseconds, no I/O); the flush is async. `cp.computeMeter` is nil outside
  the remote backend — every call site is nil-safe. There is no enable knob:
  the remote backend always meters.
- **Worker size is plumbed onto the connection** (`server.SetConnectionWorkerSize`
  → `clientConn.workerMillicores/workerMiB`, set in `control.go::handleConnection`
  from `workerBillingSize(workerProfile)`, remote-only). `workerMillicores==0`
  (non-remote / unknown) → metering skipped. The metric is computed once at the
  SAME teardown point as `CloseConnectionMetrics` (the `#841` lifetime defer),
  via `server.ConnectionBilling` (which also carries the query source).
- **Bucket = connection-end time floored to 60s.** Flush carries the sub-unit
  remainder forward so rounding never loses counts across flushes. Buffer flush
  is UPSERT-increment so all CP pods sum into one row per key.
- **Serve only closed buckets.** `watermark_high` = the newest bucket with
  `bucket_start ≤ now − 60s − 30s grace` (grace > flush interval, so every
  CP's contribution has landed before a minute is served). The GET aggregates
  the window `(cursor, watermark_high]` into one row per
  `(org, team, query_source, cpu, mem_gib)` per **UTC day** — response size is
  bounded by active keys × days, so billing downtime can't make it explode.
- **Ack is the only deletion path (plus the 30d GC).** `POST /billing/ack`
  advances the single global cursor monotonically and deletes buckets
  `≤ watermark_high` in one TXN (`AckComputeUsage`). Idempotent — re-acks and
  stale acks are no-ops. An ack beyond the latest closed bucket is rejected
  (400) so it can never delete buckets that were never served. Auth is the
  admin internal secret (`RequireAdmin` on both routes, registered inside the
  audited `/api/v1` group in `multitenant.go`).
- **Safety GC is leader-only** (`runComputeUsageGC`, attached under the janitor
  lease): hard-deletes buckets older than 30 days regardless of ack and logs a
  WARN with the dropped count — nonzero means billing stopped pulling (alert).
- **Graceful shutdown does a final flush** after connections drain to their
  natural end (`shutdown`/`drainAndShutdown`), so a departing CP pod lands its
  last interval before exit.
- **Org team CRUD (`duckgres_org_teams`)**: the PostHog backend manages an
  org's team rows via `GET/POST /api/v1/orgs/:id/teams` +
  `DELETE /api/v1/orgs/:id/teams/:team_id` (internal secret,
  `controlplane/provisioning`). The POST is the **grandfather upsert**: it MAY
  overwrite an existing row's `schema_name` and the legacy
  `events_table_name`/`persons_table_name`/`schema_data_imports_name`
  overrides (NULL = derive from `schema_name`: `<schema>.events`,
  `<schema>.persons`, `<schema>_data_imports`), because the PostHog backfill
  replaces migration 000024's `team_<id>` placeholder through it. Two teams in
  one org can never share a schema (unique `(org_id, schema_name)`, migration
  000025 → 409). Provisioning a warehouse for a NEW org REQUIRES `team_id`
  (`ErrProvisionTeamRequired` → 400; `default_team_id` is accepted as a
  transitional alias) and creates the org's first plain team row — a
  warehouse cannot exist without a team. DELETE removes CONFIG only (never
  warehouse data) and never touches usage buckets; the org's LAST team is
  undeletable (409 — an org must always have at least one team; delete the
  org instead). The admin console mirrors this on a user-facing surface
  (`GET /teams`, `POST /teams`, `PUT /orgs/:id/teams/:team_id`) where
  `schema_name` is immutable. Shared rules live in
  `configstore.UpsertOrgTeamTx` / `DeleteOrgTeamTx`; tests:
  `tests/configstore/org_teams_postgres_test.go`, the provisioning/admin API
  tests, and `org_teams_crud` in the e2e harness.
- **Storage metric** (`managed_warehouse_storage_gib_seconds`,
  `storage_meter.go`): a LEADER-ONLY sampler (double writers would
  double-bill — the UPSERT is additive) visits each Ready warehouse's DuckLake
  metadata Postgres every 30m (env-only `DUCKGRES_STORAGE_SAMPLE_INTERVAL`;
  e2e uses 60s) and credits exactly `tracked_bytes × interval` byte-seconds —
  no elapsed-time tracking, a missed sample under-bills one interval. The SUM
  is over `ducklake_data_file` + `ducklake_delete_file` with NO snapshot
  filter (never `ducklake_table_info()`/`ducklake_table_stats` — current-
  snapshot-only / approximate). byte-seconds are NUMERIC (BIGINT overflows);
  served as exact-decimal GiB-seconds (÷2³⁰ terminates;
  `byteSecondsToGiBSeconds` big-int math). Connection resolution reuses the
  cross-org activator (`MetadataPostgresURL`: duckling pgbouncer → sslmode
  disable, direct RDS → require). Drift gauges:
  `duckgres_org_storage_pending_delete_files` (alert on sustained nonzero) +
  `duckgres_org_storage_tracked_bytes`.
- **The admin console usage views read the SAME buffer** —
  `GET /api/v1/usage/monthly` (the **Usage** page) and
  `GET /api/v1/orgs/:id/usage/daily` (the org detail page's **Usage** charts)
  in `controlplane/admin/usage_api.go`, backed by
  `configstore.Aggregate{Compute,Storage}Usage{Monthly,Daily}`, sum retained
  buckets per UTC month / per UTC day per (org, team), merging the compute and
  storage families and joining the team schema name for display. Both
  self-gate with `RequireAdmin` (per-team cost data across all orgs is as
  sensitive as the raw billing families — viewers get 403, and the UI hides
  the nav item / fires no query for them). The daily endpoint's org scope is
  the `:id` path segment flowing into the queries' WHERE clause — one org's
  usage must never leak into another org's page (the e2e asserts
  `.org_id == $o` on the response). These are operations views, NOT invoices:
  acked buckets are already deleted and >30d buckets are
  GC'd, so responses carry the ack cursor as `watermark_low` and the UI
  shows the retention caveat instead of implying all-time totals. They add NO
  second accounting pipeline — keep them pure reads over the buffer.
  The Usage page also carries a **client-side pricing-sensitivity calculator**
  (`ui/src/pages/UsagePricing.tsx` + `lib/pricing.ts`): named unit-price
  scenarios ($/CPU-min, $/GiB·min, $/GiB·h) priced against each org's month
  totals. It is pure browser math over the monthly rows — no endpoint, no
  persistence beyond the operator's own localStorage — so it inherits the
  page's admin-only gate and needs no server-side access control of its own
  (a PM gets it by holding the console admin role; a lighter pricing-viewer
  role is a named follow-up, not implemented).
- Touching the meter/flush/API/GC, the worker-size or query-source plumbing,
  the storage sampler, or the bucket keys → update
  `controlplane/compute_meter_test.go`, `compute_billing_api_test.go`,
  `compute_size_test.go`, `storage_meter_test.go`,
  `configstore/storage_usage_test.go`, the migration assertion in
  `tests/configstore/migrations_postgres_test.go`, and the
  `compute_usage_pull_api` assertion (compute + storage, incl. the
  `usage-monthly` checks) in
  `tests/mw-dev/e2e/harness.sh`. Touching the monthly/daily aggregation or
  the usage views → update `controlplane/admin/usage_api_test.go`,
  `tests/configstore/usage_monthly_postgres_test.go` +
  `usage_daily_postgres_test.go`,
  `ui/src/pages/Usage.test.tsx` + `OrgUsage.test.tsx`, and the
  `usage-monthly` / `usage-daily` harness checks.

## Discovery Endpoints (external-writer tenant listing)

`GET /api/v1/warehouses` + `GET /api/v1/warehouse-team-ids` on the internal
provisioning API (`controlplane/provisioning/discovery.go`) are the read-only
"which tenants exist and where do I write" surface for EXTERNAL writers
(viaduck destination discovery; millpond's include-values poller). Semantics
are load-bearing for those consumers:

- **Warehouse set = ready + resharding** (`discoveryStates`). Resharding is
  listed with `writable=false` — vanishing would read as tenant REMOVAL
  downstream, not a pause. The state enum is open: a new state MUST be
  classified into `discoveryStates`/`discoveryExcludedStates`
  (`TestDiscoveryStateClassification` is the tripwire; an unclassified state
  reads as fleet-wide removal to every consumer).
- **Teams come from `duckgres_org_teams`**, with RESOLVED table locations:
  `events_table`/`persons_table` = `<schema_name>.<override-or-derived>`,
  `data_imports_schema` = override-or-`<schema>_data_imports`. The
  derivation lives ONCE, in `resolveTeamTables`; the legacy overrides are
  BARE identifiers (never schema-qualified), enforced at every write surface
  by `configstore.ValidateOrgTeamTableName`.
- **`enabled` is passed through as information only** — it is the per-team
  query-serving switch (migration 000024, not yet enforced), NOT an
  ingestion signal. Disabled teams stay in BOTH endpoints; deriving
  "stop ingesting" from it would turn a serving hold into permanent event
  loss. The only ingestion-stop signal is row absence.
- **Error contract:** transient store failures fail the WHOLE request (a
  polling consumer keeps last-known-good — the safe direction); only a
  warehouse with zero team rows degrades, per-warehouse, to an empty teams
  array (`duckgres_discovery_broken_team_rows_total{reason}` counts it — a
  sustained nonzero means a live tenant is silently unroutable).
- **`config_generation` is an opaque change token**: max `updated_at` over
  ALL warehouse+org+org-team rows regardless of state, read BEFORE the data
  queries. Its delete-visibility depends on `DeleteOrgTeamTx` touching the
  parent org row (a bare team DELETE leaves no timestamp behind) —
  `TestLatestConfigChangeCoversTeamsPostgres` pins that pair; keep them in
  sync. Compare for equality only.
- **No plaintext credentials in the payload** — metadata-store passwords are
  k8s SecretRefs. cnpg connection details are MIRRORED from the Duckling CR
  status into the row by the provisioner's ready-reconcile
  (`reconcileMetadataStoreRow`; drift-only writes so steady-state ticks
  never bump `updated_at`/the change token; the state-CAS write is the
  reshard fence — the runner owns those columns mid-flip). External rows
  are provision-time inputs; only their credential Secret ref is mirrored.
- **Auth is a SEPARATE, scoped surface** (`RegisterDiscoveryAPI` + its own
  group in `multitenant.go` behind `admin.AnyTokenAuthMiddleware`): the
  read-only discovery secret (`--read-only-secret` /
  `DUCKGRES_READ_ONLY_SECRET`, sent in `X-Duckgres-Internal-Secret`, same
  fallback-rotation semantics as the internal secret) works ONLY on these
  two GETs; the admin internal secret also works here (operator/debug +
  rotation window). Never register discovery routes inside the admin
  `api` group and never accept `readOnlyTokens` anywhere else — external
  writer pods carry this credential, and its blast radius must stay "read
  the tenant list and its connection topology (RDS endpoints, bucket
  names, k8s Secret names — never values)". Tripwires:
  `TestAnyTokenAuthMiddlewareScoping` (token matrix incl. cross-surface
  rejection) and `TestReadOnlyGroupTopology` (the group's exact route
  set, against the real `registerReadOnlyGroup` wiring). A discovery
  value equal to the internal secret (or any fallback) FAILS STARTUP —
  `validateDistinctReadOnlySecret` — because a shared value silently
  un-scopes the credential.
- Touching the payload shape, states, team derivation, or the generation →
  update `controlplane/provisioning/discovery_test.go`,
  `tests/configstore/org_teams_postgres_test.go`, AND the
  `discovery_endpoints` assertion in `tests/mw-dev/e2e/harness.sh`.

## Resharding (metadata-store migrations) — LOAD-BEARING CONTRACT

Operator-driven moves of an org's DuckLake catalog between metadata stores
(cnpg↔cnpg, ext→cnpg, cnpg→ext escape hatch), admin-console-driven with a
verbose op log. Full design: `docs/design/resharding.md`. Pieces:
`configstore/reshard.go` (+ migrations `000018`/`000021`/`000022`),
`provisioner/reshard_runner.go` + `catalog_copy.go` + `catalog_backup.go`
(the step machine, executed in a DEDICATED per-op pod — see below),
`controlplane/reshard_runner_mode.go` (`--mode reshard-runner`, the pod's
entrypoint), `controlplane/reshard_pod.go` (spawner) +
`reshard_reconciler.go` (leader-only pod janitor), `admin/reshard.go`, UI
`ReshardForm.tsx`/`ReshardOperation.tsx`. Invariants:

- **Reshards execute in a dedicated per-operation pod, NEVER in a CP
  process** (`duckgres-reshard-op-<id>`, labels `app=duckgres-reshard` +
  `duckgres-op-id`, restartPolicy Never, TGPS 600, requests=limits from the
  env-only `DUCKGRES_RESHARD_POD_CPU`/`DUCKGRES_RESHARD_POD_MEMORY`, default
  2/8Gi): a catalog pg_dump/copy must not compete with live traffic for CP
  resources (a ~20k-table dump OOM-killed a 512Mi CP pod). The pod runs the
  CP's OWN image + ServiceAccount (no new RBAC) and inherits an ALLOWLIST of
  the CP's env spec verbatim (`reshardPodEnvAllowlist` — secretKeyRefs stay
  refs; nothing secret lands in a pod spec). The start handler creates the op
  PENDING and spawns the pod; the pod claims the row via the standard CAS,
  runs the step machine to a terminal state, and exits 0 — the op row is the
  OUTCOME's source of truth (failed/rolled-back still exits 0; only infra
  errors — store unreachable, claim lost/fenced — exit nonzero). A leader-only
  reconciler (attached under the janitor lease) respawns the pod of a
  pending-past-grace or stale-heartbeat op (bounded, 3 attempts, then
  force-fail with an operator-facing error) and reaps pods of terminal ops.
  The e2e reshard pod netpol lives in `k8s/networkpolicy.yaml`
  (`duckgres-reshard-runner-boundaries`, egress-only: DNS/5432/443/8080); the
  production chart needs the equivalent (charts repo).

- **The sound connection barrier is the lease-GRANT check**, not the
  connect-time 57P03 gates: the grant transaction refuses `resharding` orgs
  under the same per-org advisory lock `SetWarehouseResharding` takes for the
  `ready→resharding` CAS. Never rely on the snapshot-polled connect gate
  alone — a lease can be granted up to a queue-timeout after it ran.
- **Drain, never kill**: live queries always finish. Drain = leases==0 AND
  queue==0 (one tx) AND zero live org workers (each runs a catalog-writing
  `DuckLakeCheckpointer`). Lingering hot-idle workers are retired via the
  standard CAS retire path only — never raw pod deletes.
- **Flip semantics differ by direction**: a `cnpgShard` change re-points
  role/DB in place (source ORPHANED — explicit cleanup after verify); a TYPE
  flip to external un-renders the cnpg MRs, and whether Crossplane then DELETES
  or ORPHANS the role/DB depends on `spec.metadataStore.retainCnpgOnFlip`
  (charts). **External stores are never modified/deleted.**
- **cnpg→ext escape hatch = orphan-adopt then verified-delete** (charts
  `retainCnpgOnFlip`): copy → verify source → set `retainCnpgOnFlip=true` AND
  poll the cnpg Role/Database MRs until they carry the no-Delete policy (two-step
  flip, closes the un-render-before-policy race; this MR read needs an explicit
  get grant on roles+databases in `postgresql.sql.m.crossplane.io` — NOT covered
  by duckling-reader; a Forbidden fails the wait immediately with the missing
  grant named, and the periodic wait log + timeout error carry the observed
  `managementPolicies`) → flip type to external (now
  ORPHANS, not deletes, the cnpg role/DB) → verify the external catalog row
  counts match the copy EXACTLY → only THEN `DROP DATABASE` the retained source +
  clear the flag. ANY failure before that drop → flip back to cnpg-shard +
  clear flag in one patch (`SetMetadataStoreCnpgAdopt`), provider-sql re-ADOPTS
  the still-present role/DB by external-name — NO copy-back, NO empty-recreate
  (replaces the old recreate+copy-back recovery that caused a data-loss
  incident). **XRD-compat**: if the read-back shows the cluster's XRD lacks
  `retainCnpgOnFlip` (patch pruned), REFUSE the reshard ("deploy charts first")
  — safer than the destructive delete-on-flip.
- **DEPROVISION-UNAFFECTED (non-negotiable)**: `retainCnpgOnFlip` defaults false
  and is true only transiently mid-reshard; a normal never-resharded cnpg tenant
  always has it false → its cnpg Role/DB render with full lifecycle `["*"]`
  (Delete) → deprovision (Duckling delete → finalizer) drops them exactly as
  before. The orphan is bound to the reshard type-flip, NEVER to Duckling
  deletion. `lifecycle_teardown_cnpg` in the e2e is the regression net.
- **Pre-flip catalog backup (safety net, `backup_catalog` step,
  `catalog_backup.go`)**: after drain + pause-compaction + `recordSource` and
  BEFORE any flip (for EVERY direction), the runner `pg_dump`s the SOURCE catalog
  to the org's OWN S3 data bucket under
  `s3://<bucket>/_reshard_catalog_backups/op-<id>-<ts>.dump` (custom format →
  `pg_restore`; password via `PGPASSWORD`, never argv; bucket/region/IAM-role
  from the duckling CR status; upload creds via STS AssumeRole of the org's own
  role, injected as an `AssumeRoleFunc` from the reshard-runner mode
  (`controlplane/reshard_runner_mode.go`) to avoid the import cycle). The dump
  is STREAMED: pg_dump stdout flows straight into an S3 multipart upload
  (16MiB parts × concurrency 2 bounds memory regardless of catalog size — no
  temp file, no whole-dump buffering; a nonzero pg_dump exit or empty dump
  deletes the partial object). **Gate by direction**: the destructive cnpg→external direction
  (its verified-delete drops the source) HARD-FAILS the op before the flip if the
  backup fails; non-destructive directions (source survives) log a warning and
  continue. The URI is recorded on `backup_s3_uri` (migration `000021`) + the op
  log, with the exact `pg_restore` recovery command. Retention is an S3 lifecycle
  rule on the reserved key prefix (30d suggested) — no in-app GC; backups are
  kept on success. The objects carry NO object tag: PutObject with x-amz-tagging
  needs `s3:PutObjectTagging`, which the org duckling roles do not grant (a
  tagged upload 403s on the real cluster — mw-dev e2e regression). pg_dump/pg_restore ship in the CP image
  (Wolfi's `postgresql-18-client` apk, in BOTH `Dockerfile` and
  `Dockerfile.controlplane`). Full design + restore procedure:
  `docs/design/resharding.md`.
- **Source identity: the duckling STATUS is authoritative, validated at
  submit.** The start handler derives source_kind/from_shard/the external
  source block from the duckling STATUS (where the catalog actually lives) and
  400s an op whose identity is incomplete or contradicts the config-store row
  (kind drift, cnpg source with unresolvable shard, external source with an
  incomplete row block). The runner re-checks at recordSource (pre-flip).
  Never re-introduce the old "empty row kind defaults to cnpg-shard" fallback
  — it pointed a flip-before-copy reshard at a phantom cnpg source while the
  org lived on an external RDS (prod incident: org re-pointed onto an empty
  catalog, rollback patch rejected).
- **Metadata credentials come from a referenced Secret, never CR status.**
  `status.metadataStore.credentialSecretRef` names one key in a Secret in the
  `ducklings` namespace; `DucklingClient.Get` resolves it before returning the
  status used by activation, probes, metering, and resharding. References to
  any other namespace are rejected. The name, namespace, and key are all
  mandatory; an incomplete reference or an unreadable/missing Secret is a hard
  error. Duckgres never reads credential material from Duckling CR status.
- **Rollback patches the source shard VALUE back — never removes the key**
  (precedence would fall through to the freshly-stamped bogus status pin);
  ext→cnpg rollback must null `cnpgShard` (XRD CEL forbids it on external).
  **Never emit a patch the XRD would reject**: an empty/invalid recorded
  from_shard (`isValidCnpgShardName`, mirrors the XRD pattern) or an
  incomplete external source block skips the flip-back/adopt patch, logs
  ERROR operator instructions, and **intentionally leaves the warehouse
  blocked in `resharding`** — unblocking onto the wrong store is worse than a
  blocked org (see docs/design/resharding.md's never-stranded carve-out).
  Takeover/endpoint reconstructions derive sslmode from the metadata-store
  KIND via `sslModeFor` (cnpg → disable, external → require) — never
  hardcode an sslmode.
- **The compaction pause must never let XRD defaulting disable DuckLake**:
  `SetCompactionEnabled` pins the org's effective `spec.ducklake.enabled`
  (legacy type coupling, as the activator derives it) into the patch when the
  CR has no `spec.ducklake` object — otherwise materializing the object gets
  `enabled: false` stamped by the XRD default and every later activation
  fails ("tenant activation requires a ducklake metadata_store"). The pinned
  value is never removed afterwards (deliberate — see
  `TestSetCompactionEnabledPinsDuckLakeEnablement`).
- **The ext target password is ephemeral**: request → creating replica's
  in-memory stash → one-shot pull by the runner pod → runner memory; never in
  the op row, log, audit, k8s Secret, or any pod spec. The runner pod fetches
  it at startup from `GET /api/v1/reshards/:id/password` on the CREATING
  replica's pod IP (the URL — never the password — is persisted on
  `password_url`, migration `000022`, so a reconciler respawn re-wires the
  same handoff). The endpoint is INTERNAL-SECRET identity ONLY (an SSO admin
  is 403'd — operators must never read tenant credentials); the stash is
  pruned once the op turns terminal. A pull that 404s (creating replica gone)
  is not an infra error: the runner proceeds stashless and the step machine
  fails the op with the clear "password is not available … cancel and re-run"
  message, then rolls back. (This replaces the old claim-on-create /
  `CreateReshardOperationClaimed`+`AdoptClaimedOperation` path — with the op
  no longer executing in a CP process, pinning it to the creating replica is
  neither possible nor needed.)
- **The ext SM secret must be ESO-readable and a raw string**: the ESO IAM
  policy only allows `posthog-*`/`duckling-*` names, so the start handler
  enforces a POSITIVE allowlist (`esoReadableSecretPrefixes`) — a name outside
  it is 400'd, with RDS-managed `rds!…`/`rds/…/master` names
  (`rdsManagedSecretNamePattern`) detected first for a more specific message.
  The composition's ExternalSecret copies the whole value verbatim (no JSON
  property). An unreadable name that slips through just hangs the cutover wait
  until the per-op timeout, then recovers (flip-back + copy-back). The form
  teaches the same rules (`ui/src/lib/reshard.ts::classifySecretName`).
- **cnpg→ext fails fast at submit, before the destructive flip**: the flip
  DELETEs the cnpg source, so both submit-time gates (the ESO-name allowlist
  above + a bounded `SELECT 1` pre-flight connection check to the external
  target, `admin.ExternalTargetProber` over `provisioner.PGCatalogCopier.Probe`,
  `sslmode=require`, wired in `multitenant.go`) 400 a doomed op before anything
  is created. The prober nil-degrades (tests / non-k8s → check skipped; the
  runner's copy still catches a bad credential); when present and the connect
  fails, the op is refused. The 400 never echoes the password.
- **Runner fencing**: claim bumps `runner_epoch`; every runner write is
  CAS-fenced on (runner, epoch); stale-heartbeat (>5m) ops are takeover-able;
  the copy holds a target-DB advisory lock.
- **Takeover rollback reconstructs progress from the persisted row.** The
  in-process rollback flags (`blocked`, `compactionPaused`, `flipped`,
  `retainRequested`) start false on a fresh `opRun`. A runner that CLAIMS an op a
  prior epoch advanced (crash-takeover or replica switch) MUST call
  `reconstructProgress()` before `run()` — otherwise a cancel/failure that
  short-circuits before the steps re-execute (e.g. `run()`'s first
  `cancelRequested()` check) rolls back with all-false flags and silently skips
  unblocking the warehouse (org stuck in `resharding`) and restoring compaction.
  Reconstruction is conservative: `blocked` ← `blocked_at` set OR step past
  `blocking`; `compactionPaused` ← step past `pausing_compaction` (proves the
  prior setting was recorded, else restore could wrongly re-enable compaction);
  `flipped`/`retainRequested` ← step reached `cutover`/`orphaning_source`
  (over-marking is safe — the flip-back/adopt patches are idempotent no-ops when
  the store never moved). This was a real mw-dev incident: the blocking runner
  OOM-crashed mid-backup, a sibling replica took over, saw the cancel flag, and
  marked the op `cancelled` while leaving the warehouse blocked.
- Touching any of this → update `tests/configstore/reshard_postgres_test.go`,
  `provisioner/reshard_runner_test.go` (incl. the `TestReshardBackup*` cases),
  `provisioner/k8s_client_test.go` (the compaction/ducklake-pin cases),
  `admin/reshard_test.go`, `controlplane/reshard_pod_test.go` +
  `reshard_reconciler_test.go` + `reshard_runner_mode_test.go`, the
  migration asserts in `tests/configstore/migrations_postgres_test.go`, the
  netpol assert in `tests/manifests/manifests_test.go`, AND
  the `reshard_*` + `lifecycle_teardown_cnpg` assertions in
  `tests/mw-dev/e2e/harness.sh` (validation, cancel-during-drain,
  bogus-shard-rollback, ext→cnpg positive path incl. the pre-flip backup
  assertion + the runner-pod appear/reap assertions, and the
  deprovision-unaffected net). The cnpg→ext orphan-adopt
  charts side (composition managementPolicies + `retainCnpgOnFlip` XRD field)
  lives in the `charts` repo and is covered by
  `charts/charts/crossplane-config/tests/composition_retain_cnpg_test.sh`.
  cnpg→ext positive path is unit-only (harness lacks the RDS password);
  cnpg→cnpg positive path needs a second mw-dev shard (follow-up).

## Trino Cells (customer-facing SQL over DuckLake, `kubernetes` tag)

An org enabled for Trino gets a catalog, a login, authorization and resource
limits on a shared multi-tenant Trino cluster called a **cell**. The control
plane is the only writer of that state: `provisioner/trino_provisioner.go`
projects it every controller tick from `duckgres_managed_warehouse_trino` +
the org's warehouse row + its Duckling CR. Enablement is env-inferred —
`DUCKGRES_TRINO_COORDINATOR_URL` identifies legacy (`controlplane/trino_inputs.go`).
The optional `DUCKGRES_TRINO_CELLS_FILE` adds namespace-isolated logical cells
with blue/green backends. It requires legacy unless
`DUCKGRES_TRINO_REGISTRY_ONLY=true` explicitly selects a registry-only deployment.
That mode requires a valid registry and forbids a legacy coordinator URL.
Registered reconcilers never claim unassigned warehouses. The optional
`DUCKGRES_TRINO_DEFAULT_CELL` assigns unowned tenants atomically at enablement;
see [the placement runbook](docs/trino-cells.md#automatic-placement-runbook).
With no Trino configuration, the branch never wires. **Trino is binary: if
you asked for it, a wiring failure is fatal at startup**, because silently
skipping leaves the cell's OPA sidecar serving a last-good bundle while
password/tenant/catalog changes never propagate.

- **The catalog is DuckLake and carries NO secret.** Per org:
  `connector.name=ducklake`, `ducklake.metadata.connection-url` (a JDBC URL
  whose `sslmode` follows the store kind — `disable` for in-cluster
  `cnpg-shard`, `require` for `external` and any future kind),
  `ducklake.metadata.connection-user`,
  **`ducklake.metadata.connection-password-file`**, `ducklake.data-path`,
  `fs.s3.enabled` (NOT `fs.native-s3.enabled` — that spelling was rejected at
  CREATE CATALOG and cost a release), `s3.region`, `s3.auth-type=IAM_ROLE`,
  `s3.iam-role` (the per-org duckling role: the tenant S3 boundary), and a
  small `s3.max-connections`. Every value comes from the `ManagedWarehouse`
  row's `metadata_store_*` / `s3_*` / `worker_identity_*` blocks. **Trino logs
  the full `CREATE CATALOG` statement, renders catalog properties in its web
  UI, and ships them to workers — a password in a property is readable by
  anyone who can see a query listing.** Hence the file indirection; never add
  `ducklake.metadata.connection-password`.
- **Tenant passwords live in one Secret, keyed by org id**
  (`TrinoTenantSecretName`, mounted at `TenantSecretMountPath`). The value is
  read from the org's Duckling CR status (`credentialSecretRef`) through the
  SAME resolver the worker activation path uses, so Trino and the DuckDB
  workers can never authenticate a tenant's metadata store with two different
  credentials. The projection is AUTHORITATIVE, not additive: a disabled org's
  key is removed on the next tick.
- **Tenant client coordinates are non-secret and readiness-gated.** The org
  detail endpoint returns `status.connection` only while the catalog is ready
  and the coordinator is reachable. `DUCKGRES_TRINO_CLIENT_URL` names the HTTPS
  endpoint clients should dial; when unset, the control plane falls back to its
  coordinator URL and TLS server name. The response contains host, port and the
  tenant principal, never a password or password hash. A client URL whose
  leading host label is `{database_name}` (registry `client_url` or the env
  var; `ResolveTrinoClientURL`) advertises each org's own
  `<database_name>.<domain>` with username `root`: the Trino fork qualifies a
  login with the org its host names
  (`http-server.authentication.password.host-qualified-user.domains`), so a
  user types the same username and host on both engines. Set it only once
  that host actually reaches Trino — advertising an unrouted host is worse
  than advertising the shared one.
- **`Reconcile` order is load-bearing**: cluster secrets → auth files →
  resource groups → OPA bundle → tenant passwords → catalogs, and the
  `globalErr` gate SKIPS the catalog step if any projection failed. A
  coordinator that just lost its `password.db` keys 401s every catalog REST
  call, which would surface as a misleading "catalog reconcile failed" masking
  the real problem.
- **`ensureClusterSecrets` is write-once with a sentinel.** The K8s Secret is
  the source of truth for each cluster credential; the configstore holds only
  a one-bit `duckgres_trino_cluster_bootstrap` row per namespace. Missing
  Secret + not bootstrapped ⇒ generate; missing Secret + already bootstrapped
  ⇒ **fail loud**, because regenerating the env-projected
  internal-communication shared secret would split-brain a running cluster.
  Admin and observer password/hash pairs are the deliberate exceptions:
  missing or incomplete pairs regenerate together. Each attempt reads both
  keys from one Secret snapshot and updates with that snapshot's resource
  version. A create/update conflict reloads and adopts the winner, with five
  attempts per call. Complete but invalid pairs fail without writing; transient
  API errors also fail without attempting repair. No advisory lock serializes
  these Kubernetes writes. Never overwrite an existing complete pair to repair
  a suspected bootstrap race. Confirm stored corruption before operator recovery.
- **Static catalog reconcile is `SHOW CATALOGS` first**: create only what's missing,
  drop only names matching `opa.ManagedCatalogPattern` that aren't wanted, so
  `system`, `jmx` and hand-made catalogs survive. An org whose password is
  momentarily unresolvable keeps its existing catalog (never dropped) but is
  NOT reported ready.
- **Shared catalog mode is explicit and fenced.** Registered cells can use
  `catalog_management: paused` for compatibility rollout or `gateway-shared`
  for sole-active-backend catalog writes. Freeze new admissions and DDL before
  warming the standby; bulk-check catalog states and admitted tenant credentials,
  cut over, then release before draining. See
  [docs/runbooks/trino-shared-catalogs.md](docs/runbooks/trino-shared-catalogs.md).
  Never replay CREATE across both slots. A remote FAILED query is not proof
  that synchronous DDL stopped; retain the durable intent until fenced recovery.
  Paused and non-owner replicas still refresh auth and OPA projections.
- **Provisioner catalog inventory is narrowly authorized.** The admin can read
  `catalog_name` and `state` from `system.metadata.catalogs`, in addition to
  the existing node inventory. The observer and customer principals cannot.
  `FAILING` startup placeholders are not usable catalogs.
- **Catalog naming is a THREE-way contract**: `TrinoCatalogName` (`org_` +
  sanitized org id, no `_iceberg` suffix — warehouses are DuckLake),
  `opa.ManagedCatalogPattern`, and the regex literal inside `policy.rego`.
  `TestTrinoCatalogNameMatchesManagedNamePattern` +
  `TestPolicyRegoContainsManagedNamePattern` fail if any one moves alone.
- **A Trino principal resolves to its org by EXACT lookup, never by splitting.**
  `configstore.NewTrinoPrincipalOwners` indexes the principals the auth files
  project (bare `<database_name>` → root, `<database_name>.<username>` → that
  login). Usage metering (`trino_usage_collector.go`, which also passes the
  duckgres username — not the principal — to the team resolver) and the admin
  console both use it, so a per-user login's queries are its org's usage and a
  principal outside the password file is never attributed to anyone.
- **Every duckgres login authenticates to Trino, under `<database_name>.<username>`.**
  `ListTrinoEnabledOrgs` returns the org's logins in `Users`, and
  `BuildTrinoAuthFiles` writes one `password.db` line per login with the
  bcrypt hash **copied through unchanged** — it is the same hash pgwire
  verifies, so one password works on both engines and nothing is re-hashed or
  minted. The bare `<database_name>` principal survives alongside them and
  authenticates with root's hash, so it follows root's login row exactly: no
  `root` row, a disabled root or a blank password leaves `RootPasswordHash`
  empty and drops ONLY the bare principal — from `password.db`, the pool's
  Gateway binding, `TrinoPrincipalOwners` and the advertised
  `status.connection` alike. (`TrinoPrincipalOwners` separately keeps a
  root-independent database -> org index, so minted `<db>.svc_…` service
  credentials are attributed whether or not root exists.) The org is still listed and its other logins
  still work; a root-less org is enabled normally (no root preflight). Three
  rules that are load-bearing rather than cosmetic: (1) usernames are
  projected through an **allowlist** (`trinoUsernamePattern`) because
  duckgres barely validates them and a `:`, `,` or newline would let whoever
  can create org users append lines to `password.db` — including an admin
  line; (2) `rejectPrincipalCollisions` now also holds back orgs that derive
  the same Trino username, since the password file is ONE flat namespace per
  cell and a duplicate line is a cross-tenant auth bug; (3) the resource-group
  selector's `orgCaptureRegex` captures only up to the first `.`, or every
  login gets a private leaf with the full per-tenant limits and an org with
  ten logins holds ten times its budget.
- **A project-scoped login (`project_reader` / `project_user`) joins
  `scope_<org>_team_<id>`, NOT the org group.** The scope group owns the same
  catalog in `group_catalogs` — so the cross-tenant check is the unchanged
  check — and carries a `group_scopes` document that narrows it to that
  team's schemas. Scopes only ever SUBTRACT; keep it that way if the rules are
  restructured. The scope comes from `OrgUserQueryAccess`, the same derivation
  the pgwire path uses, so Trino and DuckDB cannot disagree about it, and a
  scoped row whose scope will not resolve is DROPPED rather than projected
  unscoped. Scoped logins get **no write authority at all** — `project_user`
  is read/write on pgwire and read-only here, a narrowing; making it writable
  means gating writes per-schema, not per-catalog.
- **The Rego policy is the tenant-isolation boundary.** The cell can assume
  every per-org duckling role, so nothing below OPA stops org A reading org
  B's catalog. Treat `provisioner/opa/policy.rego` as security review.
  **Query visibility is same-org only**: `ViewQueryOwnedBy`,
  `FilterViewQueryOwnedBy` and `KillQueryOwnedBy` (the plugin's exact
  operation strings — the filter one is NOT `FilterViewQuery`) are allowed
  only when the requester and the query OWNER share a bundle-known
  `org_<sanitized>` group, derived from the same `data.group_catalogs`
  ownership map every other decision uses. This matters because
  `ExecuteQuery` is unconditionally allowed, so without it org A reads org
  B's SQL text — table names, filter literals, customer identifiers — via
  `system.runtime.queries` and the web UI, and can kill B's queries. The
  owner arrives as `input.action.resource.user.{user,groups}` (an
  `@JsonUnwrapped` `TrinoIdentity`); `ImpersonateUser` uses a different,
  groups-less shape for the same field — do not conflate them. **The admin
  principal deliberately gets NO cross-tenant query visibility** (only its
  own queries): the reconcile loop issues only `SHOW`/`CREATE`/`DROP
  CATALOG` and never reads `system.runtime.queries`, so the grant would buy
  nothing and leak every tenant's SQL. **Every filter op needs a `batch`
  rule as well as an `allow` rule.** `opa.policy.batched-uri` is enabled
  (without it, filtering a catalog with >1024 tables overruns the OPA
  client's queue), and `filterViewQueryOwnedBy` goes through it like the
  catalog filters do. A missing `batch` rule fails closed, which for query
  visibility looked like "org-mates' queries are missing" rather than an
  error — Trino short-circuits self-ownership before OPA, so each tenant
  still saw its own query. `TestBatchedQueryFilteringMatchesNonBatched`
  pins the two shapes to the same decision.
- **The observer principal (`__duckgres_observer`) is the admin console's
  read-only identity**, and is deliberately NOT the provisioner's admin.
  Split authority: admin does CREATE/DROP CATALOG and sees only its own
  queries; the observer gets cluster-wide `ViewQueryOwnedBy` /
  `FilterViewQueryOwnedBy` / `KillQueryOwnedBy` plus `ReadSystemInformation`
  (one op gating EVERY MANAGEMENT_READ resource — `/v1/node` and
  `/v1/resourceGroupState`, which the console reads, plus `/v1/thread`,
  `/v1/announce` GET, `/v1/maxActiveSplits`, `/v1/integrations/gateway`; all
  GETs of operational state, enumerated in policy.rego) and
  holds NO TENANT catalog — no `data.group_catalogs` entry, and the observer
  group is excluded from `tenant_owns_catalog` and from the same-org query
  match, so even a mistaken bundle entry grants no tenant data access. Its
  ONE data grant is `system.runtime.nodes`: `AccessCatalog` on `system` plus
  `SelectFromColumns` pinned to that single table, because a cell on the
  default `discovery.type` serves no `/v1/node` and `/v1/announce` carries
  neither health nor version. `AccessCatalog` alone opens nothing — every
  read still passes `SelectFromColumns` — so `system.runtime.queries`
  (tenant SQL), `system.metadata.*` and `system.jdbc.*` (which would
  enumerate every tenant catalog, schema, table and column) all stay denied;
  `TestObserverSystemGrantIsPinnedToTheNodesTable` pins that. `is_observer` is
  the same username-AND-group conjunction as `is_admin`. Its credential is
  a second regenerate-if-missing pair on `trino-auth`
  (`ensureCredentialPair`), projected into password.db/group.db and read by
  the console through `TrinoProvisioner.ObserverCredential` on every call
  (never captured — a self-heal would otherwise 401 forever). Console reads
  redact SQL before it leaves the CP. See `controlplane/admin/README.md`.
- **Resource groups must keep a selector for BOTH operational principals**
  (`root.admin.__admin_provisioner` and `root.admin.__duckgres_observer`).
  The final selector matches user `(?<org>.*)`, which matches anything, so a
  principal without its own lane is admitted as a tenant into
  `root.tenants.free.<principal>` — and those leaves are `JmxExport: true`,
  so it would appear as a phantom tenant in the per-tenant resource-group
  metrics.
- **Resource groups must keep the `root.admin.__admin_provisioner` selector.**
  Trino rejects a query matching no resource group, so dropping it silently
  breaks every reconcile tick's own DDL.
- **Cells, minimally**: `trino_cell_id` on the row names the owning cell
  (`DUCKGRES_TRINO_CELL_ID`, default `configstore.DefaultTrinoCellID`). A
  provisioner claims unassigned orgs (`ClaimTrinoCell`, conditional in SQL so
  no cell can steal another's tenant), reconciles its own, and ignores the
  rest — including writing NO state for them. Claims return an authoritative
  boolean: a lost claim must never project the losing cell's tenant. Only
  legacy claims unassigned warehouses. Registered logical IDs have the reserved
  storage prefix `registered:`; the old stored `cell-001` remains legacy.
  Admin-only initial selection runs before first enablement and refuses changes
  to any already owned warehouse, including a disabled one. An owned org moves
  only through `POST /orgs/:id/trino/cell/move` (`ConfigStore.MoveTrinoCell`): a
  compare-and-swap on the named source under the admission lock that resets
  the row to pending. The source's authoritative projection then cleans the org
  up and the destination provisions it, with NO overlap - the org has no Trino
  in between. Per-org state writes carry `TrinoStateUpdate.CellID` so a stale
  source tick cannot mark a moved row ready; keep that fence. No live
  migration, capacity model, rebalancer, drain, or Gateway routing controller
  is included.
  See [docs/trino-cells.md](docs/trino-cells.md) for configuration and recovery.
- **Blue/green projections share one logical cell's namespace**, but internal
  communication Secrets remain distinct, chart-owned read-only references.
  Stopped backends do not receive catalog calls or observer polling. Every
  running backend must reconcile its independent catalog set before the org is
  ready. Cell failures do not block sibling-cell projections. Configuration is
  startup-loaded and must agree across control-plane replicas.
- **The existing deployment's API identity is `legacy`.** The Trino console
  exposes this name in `cell.id` and in owned orgs' `status.cell` / `orgs[].cell`.
  `TrinoCell.StoredID` keeps the configured ownership ID private to the adapter.
  Match connection readiness against the raw persisted ID, never the alias.
  Do not rename org assignments, catalog-store keys or environment settings.
  The general org endpoint still exposes the original `trino.trino_cell_id`.
  Unknown and unassigned ownership values are not relabeled.
- **The bundle endpoint is mounted OUTSIDE `/api/v1`** (`/bundles/trino`) with
  its own bearer auth, and `buildTrinoWiring` bootstraps SYNCHRONOUSLY so the
  handler is constructed with the real token — there is no window where it
  serves under a placeholder. Registered cells use `/bundles/trino/<cell-id>`;
  each cell's token can read only its own bundle. Legacy retains the old path.
- Touching any of this → update `provisioner/trino_provisioner_test.go`,
  `provisioner/trino_cluster_secrets_test.go`, `provisioner/opa/*_test.go`,
  `provisioning/api_test.go`, `tests/configstore/trino_postgres_test.go` +
  `trino_cluster_secrets_test.go`, the migration asserts in
  `tests/configstore/migrations_postgres_test.go`, and — for anything the
  admin console reads — `controlplane/admin/trino{,_client}_test.go` plus
  the `ui/src/lib/trino.test.ts` derivations and
  `tests/mw-dev/e2e/trino.sh`. Cell moves → also
  `tests/configstore/trino_selection_postgres_test.go`,
  `controlplane/admin/trino_fleet_test.go` and the `trino_cell_move`
  assertion in `tests/mw-dev/e2e/harness.sh`.

## Shared Trino Compute Pool (`mode: "shared-pool"`, `kubernetes` tag) — LOAD-BEARING CONTRACT

Operator-managed pool of immutable Trino instances, replacing fixed blue/green
for a cell that opts in. **Ships disabled**: a cell without `mode:
"shared-pool"` in `DUCKGRES_TRINO_CELLS_FILE` behaves byte for byte as today.
Code: `controlplane/trinopool/` (pure: blueprint, phases, planner, catalog-version
port), `controlplane/trinocatalog/` (fenced catalog publisher),
`controlplane/trinogateway/` (Gateway protocol v1 client),
`controlplane/trino_pool_*.go` (config, effects, validation, operator, wiring,
publication barrier), migration `000041` (upstream owns `000040`,
the Trino backend selection).

- **Env, all default-off**: `DUCKGRES_TRINO_POOL_ENABLED` (a pooled cell with
  this off FAILS startup — falling back to blue/green would hand the operator a
  shape they did not ask for), `DUCKGRES_TRINO_POOL_OPERATOR_ENABLED` (off =
  desired state is recorded, nothing external is touched),
  `DUCKGRES_TRINO_POOL_GATEWAY_URL`, `DUCKGRES_TRINO_POOL_CATALOG_WRITER_ENABLED`
  / `_BOOTSTRAP` / `_DSN_FILE` / `_WRITER_IDENTITY`, and
  `DUCKGRES_TRINO_POOL_CONFIG_CONFIGMAP` / `_NAMESPACE` / `_REGISTRY_KEY` (the
  desired-state source, below — REQUIRED for a pooled cell), and
  `DUCKGRES_TRINO_POOL_CATALOG_SCHEMA` (REQUIRED with the catalog writer: the
  publisher credential names a database only, and the role's privileges are on
  the cell's schema, so unqualified SQL would resolve against `public` where it
  can neither create nor read), `DUCKGRES_TRINO_POOL_CATALOG_CELL_ID` (also
  REQUIRED with the catalog writer: the catalog store's PARTITION — see the
  identity bullet below; there is no default), and
  `DUCKGRES_TRINO_POOL_PUBLISHER_IMAGE` (this
  process's own digest-pinned image, rendered from the same chart helper as the
  container and as the ConfigMap's `publisher-image` key — the projection fence
  below compares the two).
- **The pool keeps the cell's identity**: routing group and namespace are
  unchanged. Replacing compute never rewrites which warehouse lives where.
- **The catalog store's partition is its OWN identity**
  (`DUCKGRES_TRINO_POOL_CATALOG_CELL_ID`, resolved by
  `trinoPoolCatalogCellID`): it is the value every coordinator of the cell
  carries in `catalog-store.cell-id`, and the rows the publisher writes must
  carry exactly it or each coordinator reconciles an EMPTY catalog set. It is
  not derived from the pool id — that id carries duckgres's reserved
  `registered:` prefix and doubles as the warehouse-ASSIGNMENT key, so deriving
  one from the other would force a cluster to spell its partition
  `registered:<cell>` and would silently re-key every coordinator's read the day
  a pool's logical id changes. It is **REQUIRED with the catalog writer and
  fails closed** — unset, blank and unusable are one refusal, and there is
  deliberately NO fallback to the pool id: a missing setting would otherwise
  publish a full catalog set under a partition nobody reads, which is
  indistinguishable from a store nothing has been published to yet. The chart
  renders it from the same value it renders the coordinators'
  `catalog-store.cell-id` from, and guards the pair the way it already guards
  the schema pair. The pool row's
  `publication_revision` checkpoint addresses the POOL
  (`RecordTrinoPoolPublicationRevision` refuses a pool id the held lease does
  not name), never the partition. Regressions:
  `TestCatalogWriterPublishesUnderTheConfiguredStorePartition` and
  `TestCatalogWriterRequiresAStorePartition`.
- **Missing/invalid desired config FREEZES the pool** at last-good state. It is
  never a desired count of zero (that would delete the fleet) and never a
  startup abort (a ConfigMap problem must not take the CP down). A pool that
  vanished from the registry is an error (⇒ freeze), never an empty desired
  state.
- **Desired state is published from the API OBJECT, not from the mounted copy
  of it.** The registry and blueprint files are the BOOT source — they are what
  tells the process a pool exists. Every publication is then derived from ONE
  read of `DUCKGRES_TRINO_POOL_CONFIG_CONFIGMAP`, taken immediately before the
  write, because a projected volume refreshes per pod on the kubelet's schedule
  and a `subPath` mount never refreshes at all: two replicas can hold different
  contents indefinitely, and an idle pod can publish what it read at boot. A
  pooled cell that does not name that object REFUSES TO START; a silent
  fallback would leave nobody able to say which configuration a live pool is
  driven from. The blueprint's key is the last element of `blueprint_file` (the
  mapping a ConfigMap volume performs), and both documents come from the SAME
  read — two reads can straddle an update and pair a new registry entry with a
  release the cluster has already replaced.
- **Fences, not leadership.** The janitor lease decides who executes; the
  pool's `authority_epoch` fences every durable write — desired-state
  publication, freeze and thaw included — every Kubernetes object (annotation,
  compared before any adopt) and every Gateway call (`controllerEpoch`). Seeding
  the pool row is the ONLY unfenced write and can only INSERT. A refused fenced
  write ENDS the leadership term: re-acquiring would let a superseded controller
  fence the valid leader straight back, and the two would trade the pool
  forever. The fence owner is per-PROCESS, not per control plane.
- **Desired state is ORDERED as well as fenced.** The blueprint carries a
  `generation`; a publication that would move it backwards is refused. Holding
  the fence proves who may write, not that what they hold is current — a replica
  carrying an older blueprint could otherwise publish it over a newer one as a
  legal fenced write. The ordinal must come from the config source; a content
  hash is not monotonic and would refuse valid configurations. It is a BACKSTOP,
  not the freshness mechanism (a settings-only edit need not move it, and two
  configurations can carry the same one — hence the API read above), and a
  backwards generation FREEZES the pool while KEEPING the lease: it is a
  configuration problem, not a lost fence, and reporting it as one ended the
  leadership term every tick and handed the pool to a replica that did the same.
- **The catalog writer's fence IS the pool authority.** It is claimed when the
  operator wins the lease, never at startup (where every replica would claim it
  and the fence would distinguish nobody), and a takeover needs a strictly
  higher epoch — an equal epoch held by another identity is a collision, not a
  handover.
- **The published catalog revision is read from the catalog STORE, not from
  whatever a publication last managed to write down.** The pool row's
  `publication_revision` is a cache of it, and the admission gate certifies
  members against that number. A catalog can commit while the follow-up write
  of its revision fails — and nothing republishes it, because that catalog now
  exists, so no later mutation carries the number forward. The gate would keep
  certifying members against a revision predating the tenant, admitting it and
  reporting the warehouse ready without its catalog. So: a failed checkpoint is
  RETURNED from the publish path (not logged and dropped), the writer's claim
  checkpoints the revision the takeover already read, and every tenant-admission
  tick reconciles the row against the store's own writer state and fails CLOSED
  when it cannot. Holding admissions does not hold the compute lifecycle —
  `reconcileOnce` isolates that step — so a pool still repairs and drains.
- **A pooled cell answers the managed-Hoglake connector question from the
  store.** Adoption of an existing catalog is refused unless its connector is
  confirmed (`CatalogConnectors`), which on a coordinator-mediated cell is a
  `system.metadata.catalogs` query. A pooled cell has no fixed coordinator, so
  the pool writer answers from the `trino_catalogs` rows the coordinators
  reconcile FROM. That is the PUBLISHED definition and is sound only for that
  decision — whether any member has applied it stays with pool admission, which
  proves it per member. Never read a published row as evidence that a catalog is
  operational.
- **Failure is not drain.** SUSPECT excludes a member from new work; LOST
  requires positive evidence that the admitted process ENDED and is
  reported as failed. A repair NAMES the instance it replaces, so the Gateway
  charges the repair budget instead of the single planned surge. There is NO
  local path back from SUSPECT: the Gateway excluded the member and only a
  fresh certified admission un-excludes it, so flipping the row back to SERVING
  would claim a member serves while nothing is routed to it. A suspected member
  always leaves — proven dead through the loss claim, or replaced through the
  planned drain after `trinoPoolSuspectDrainAfter` when it cannot be proven
  dead. A suspect is already excluded from serving capacity, so its drain does
  not reduce the Gateway's serving count again. Planned ACTIVE drains retain
  the serving-floor guard.
- **Capacity repair counts SERVING, not departing instances.** DRAINING,
  SEALED and RETIRING members retain physical capacity and pinned work, but do
  not satisfy desired serving capacity. Only one candidate may prepare at a
  time. A repair keeps its configured repair slot throughout its nonterminal
  lifetime, including SERVING. Its persisted `repair_for` reserves one target;
  a second live repair must not target that same instance. When no normal slot
  is free, an unreplaced departing member can be a repair target only while
  serving capacity is short. Gateway independently validates registration
  against active plus preparing capacity and its own repair budget. Deploy
  that Gateway support before this planner. No repair permits deletion of the
  original member, skips validation, lowers the serving floor, or exceeds the
  configured desired + surge + repair capacity. Automatic normalization of
  serving repair slots after their targets retire remains a separate TODO;
  never clear the local repair flag without matching Gateway semantics.
- **Two observations prove a process ended, and both are statements Kubernetes
  makes after the fact** (`processTerminationEvidence`): every recorded object
  is verifiably absent, or **the exact container instance that hosted the
  admitted process is no longer the one running in the pod that hosted it**.
  The second is what a coordinator that restarts IN PLACE produces: it keeps
  every object it had, so absence can never arrive, and the member was retained
  until the drain timer — where a drain cannot finish either, because the dead
  process's transactions stay pinned to it, holding the instance and its repair
  slot forever.
  **Exactness is the whole contract.** The container id is recorded at
  registration (migration `000041`) precisely so a termination record can be
  correlated with the admitted process. An earlier version accepted ANY
  termination record on the hosting pod plus a differing identity from the
  member's endpoint, and that is NOT evidence: a pod restarted once during
  startup carries such a record for its whole life, and a second coordinator
  pod makes the endpoint's answer ambiguous about which process replied —
  together they declared a live, serving member dead and wrote off its pinned
  work (regression:
  `TestALiveAdmittedProcessIsNotDeclaredLostByAnUncorrelatedTermination`).
  Everything ambiguous fails closed: an unreadable cluster, an unenumerated pod
  list, more than one live coordinator pod, a member with no recorded container
  id, a termination naming a different instance. A label-filtered listing
  omission is NOT deletion (labels are mutable), a timeout is never evidence,
  and nothing is deleted or restarted to manufacture proof.
  **Evidence boundary, stated honestly:** this rests on what the Kubernetes API
  reports. A force-deleted pod object, or a node partitioned from the API
  server, can leave a process running that the API no longer describes; neither
  observation can see that. Within that boundary "proven dead" means
  "Kubernetes reported it dead" — which is what the user accepts, as distinct
  from a timeout or routing ambiguity being classified as death.
- **A serving member's PROCESS is observed, not just its pod.** A container can
  restart inside a Pod and report ready with the same pod UID and a new Trino
  incarnation; the Gateway refuses to dispatch to anything but the boot identity
  it registered, so a readiness-only check reports SERVING while that capacity
  is gone. The identity is re-read against the admitted one, bounded to one
  request per member per `trinoPoolIdentityObserveEvery`. A DIFFERENT identity
  suspects the member; an unanswered probe is not evidence and changes nothing;
  the stored identity is never quietly updated to match.
- **A FAILED_PREPARING candidate is cleaned up, not abandoned.** It is NOT
  terminal. The Gateway must grant never-admitted retirement before Duckgres
  deletes a registered candidate's UID-bound objects. Verified resource absence
  then completes retirement and moves the local row to FAILURE_RETIRED, freeing
  its capacity. No loss claim is needed. A PREPARING candidate whose immutable
  release or blueprint digest differs from the fresh desired specification uses
  this path, including configuration-only changes with the same image. Compare
  the snapshot's blueprint digest, not the identity-bound instance SpecDigest.
  VALIDATING may have an unknown admission result and must not use this shortcut.
- **Admission is a durable step.** The intent is recorded before the call, so a
  lost response is resolved by read-back under the same identity. OK, FAILED
  (a decision) and UNKNOWN (no answer) stay distinct. The step identity is the
  business intent, never the authority envelope — hashing the whole request made
  the retry after a lost response a permanent conflict. A step is recorded
  UNKNOWN before the effect and UPDATED with the outcome after it; a decided
  outcome is never re-decided. A failed step records `attempts` and
  `next_attempt_at` (full jitter, 0.5s→30s) and is not retried until then — the
  schedule is durable so a restart cannot reset it to zero — and a step that
  succeeds, or an instance that reaches the end of its life, closes its
  operation so `terminal_at` is not NULL forever.
- **Instance identity is persisted BEFORE any Kubernetes object exists**, and
  names are deterministic, so a lost create is resolved by read-back rather
  than by creating a second instance. Identities and live endpoints are never
  reused (PK over terminal rows + partial unique index).
  New instance names are `cell-<eight random hexadecimal digits>`, independent
  of the logical pool ID. The suffix uses four cryptographically random bytes,
  not a UUID or sequence. Coordinator and worker Deployments retain their
  normal Kubernetes ReplicaSet and pod suffixes. Existing persisted instance
  names and endpoints are unchanged; the shorter names appear on new instances.
- **Blueprint is not a template engine.** Argo delivers a validated JSON with
  real `corev1.PodTemplateSpec`s; duckgres injects only object names, labels,
  selectors, replica counts and the three env vars the `identity_binding`
  declares. Images must be digest-pinned (sidecars included). Each instance
  keeps its OWN blueprint snapshot, so a new release cannot change what a
  running instance reads. Worker anti-affinity is NARROWED to the instance, not
  dropped — a pool-wide selector would make instances fight for nodes.
- **Serving instances roll on blueprint changes, not only image changes.**
  The planner parses each immutable blueprint snapshot and compares it with the
  desired blueprint using the same current representation on both sides.
  Worker replicas, pod resources, and configuration files use the existing
  surge, admission, drain, and retirement protocol. Stored `SpecDigest` values
  remain ownership and operation identities; encoder changes or uppercase hex
  do not trigger rollouts or rewrite them. A malformed digest with a readable
  snapshot nominates only that instance for guarded replacement. An unreadable
  snapshot blocks only that instance's planned replacement and reports an error;
  unrelated repairs, scaling, and eligible rollouts can continue. Repair the
  unreadable record before attempting its retirement; never infer its namespace
  from desired configuration. Registry-only identity changes such as node
  environment or service port are not blueprint rollout triggers.
- **Deletion requires an irreversible Gateway retirement claim** for that exact
  incarnation; the phase machine permits it only from `RETIRING` onwards (plus
  `FAILED_PREPARING`, which provably never admitted work). Delete verifies each
  object's UID itself before passing a precondition, and retirement completes
  only on verified absence INCLUDING terminating pods. Pool-shared objects
  named in the blueprint are refused by name in both apply and delete.
- **Drain has no deadline.** Sealing is driven by the Gateway's obligations
  endpoint; a timer would be a decision to lose open transactions. The serving
  floor refusal is surfaced, never overridden.
- **Candidate validation uses no canary.** The candidate is probed through its
  OWN Service with the existing observer credential: `/v1/catalog/sync` must be
  enabled, ready, zero failed catalogs, applied revision at least the pool's
  DURABLE `publication_revision`, a process identity that does not change during
  the pass, a worker count that agrees with the running pods, and the pods'
  RUNNING images equal to the blueprint's pinned release. The `auth-revision`
  check is claimed ONLY when every security component reported what it loaded
  AND each projected component reports exactly what THIS control plane is
  serving: the OPA bundle's revision (`opa.PolicyRevision`, published in the
  bundle as `data.trino.revision` and digesting the POLICY BYTES as well as the
  data — `policy.rego` lives in the duckgres binary, so the image check says
  nothing about it), and the `sha256:<hex>` fingerprints of the projected
  `password.db` and `group.db`. Every instance of a required kind must match: a
  second password authenticator reads a file duckgres does not write. "The
  component answered" is not evidence — a coordinator whose OPA still serves the
  projection from before a tenant existed answers perfectly. A coordinator
  without `opa.policy.revision-uri` reports OPA as unacknowledged, the check is
  absent, and admission fails closed — deliberate, so `opa.policy.revision-uri`
  is REQUIRED on a pooled coordinator, pointed at that revision document.
- **The authorization projection is FENCED for a pooled cell.** Every control
  plane builds it from its own view and every replica serves it, so two things
  are needed together. An ORDER: the config store records the accepted
  projection (digest + revision) under the pool's authority, built from source
  rows read INSIDE that transaction at `REPEATABLE READ`, with project scopes
  derived from the same read rather than the polled snapshot — a revision
  allocated for content read at another moment numbers bytes nobody can prove
  were current. And an eligible PRODUCER: only a process whose own
  `DUCKGRES_TRINO_POOL_PUBLISHER_IMAGE` — captured at STARTUP, from the same
  chart helper that renders the container — equals the `publisher-image` key of
  the same pool ConfigMap may advance it, compared after the authority is held.
  The startup value is used rather than the pod's spec or a runtime image ID: a
  pod specification can be edited under a running process, and the runtime
  identity is the node's platform-specific digest rather than the manifest the
  chart names. Both sides must be pinned by `@sha256:<64 hex>`; a floating tag
  on either side is refused, because two equal tags are not evidence of equal
  bytes. That is equality, never an ordering
  of image identities — an older binary that wins the lease would otherwise
  publish its own older `policy.rego` under a HIGHER revision, which no counter
  can detect, and a deliberate rollback moves the desired value so the older
  binary becomes eligible again. Every uncertainty fails closed, so a partial
  rollout PAUSES pooled publication (OPA keeps its last-good bundle) rather than
  regressing it. The auth Secret is written under that revision with the
  object's own resourceVersion as the CAS and refuses to replace a newer stamp;
  the bundle handler gates the bundle it CAPTURED before a 200 AND before a 304
  (a 304 preserves exactly the stale bundle the fence exists to retire), and an
  unreadable record is a refusal. That gate reads the durable record on EVERY
  request and must stay uncached: OPA's periodic downloader activates each
  bundle inline as it fetches it, so a cached "still accepted" answer does not
  merely delay the truth, it INSTALLS a superseded projection after a newer one
  was accepted and a tenant admitted against it. A replica that is not the
  advancer still BUILDS and SERVES its projection — refusing would strand every
  coordinator polling it on its last-good bundle — but writes the auth Secret
  only when what it built equals the accepted digest, and being un-advanceable
  is never reported as a reconcile failure (every replica but one is in that
  state at any moment, and failing would mark every pooled warehouse Failed).
  Candidate admission compares against the
  accepted projection, not against what the local process last published.
  Legacy cells install no fence and are byte-for-byte unchanged.
- **A retried step repeats the IDENTICAL request — every field, not just the
  generation.** The Gateway journals each step under its identity and hashes
  the WHOLE request body, minus the authority envelope, so a step whose effect
  landed while its response was lost can only be resolved by repeating exactly
  what was sent. Anything rebuilt from freshly read state reports a changed
  intent, and that member can never finish the transition — not on the next
  tick, not after a leader change, never. Three fields did that: the expected
  generation (sealing read it back from the obligations the seal had just
  moved), the loss claim's `observedAt` (re-stamped per attempt; it now carries
  the durable moment the member was excluded), and registration's boot identity
  (re-probed per attempt, so an in-place restart changed it). Where the request
  genuinely cannot be reproduced — registration's observed identity, a
  suspicion whose reason depends on which check fired first — the member is
  READ BACK and what the Gateway recorded is adopted, never re-derived. A
  read-back that FAILS stays an error: an unresolved operation identity is not
  permission to send a body this controller derived meanwhile. The accepted
  read-back states are explicit SETS, not a rank — the lifecycle is a graph
  (SUSPECT ↔ DRAINING), and a retirement counts as a recorded loss only when
  its kind is FAILED, because a DRAINED one completed a different transition.
  `observedAt` is omitted from the loss claim: it is optional, the Gateway
  stamps its own receipt time, and every value available here is either
  re-derived per attempt or not the termination-observation time at all.
  Regressions: `TestALostResponseDoesNotChangeTheRetriedRequest` and
  `TestALostResponseDoesNotChangeTheRetriedMemberRequest`, against a fake that
  journals the whole payload the way the Gateway does.
- **One instance never stops the pool.** `progressInstances` records a failing
  instance and carries on, and the planner still runs, so a member that cannot
  make progress no longer holds every repair, drain and replacement behind it.
  Only a lost fence ends the sweep. This is the isolation the tenant loop
  already applies.
- **Tenant admission is a committed publication, not a published binding.**
  Publishing principals (PUT `…/tenants/{t}/principals`) tells the Gateway which
  logins belong to a tenant; the Gateway dispatches work only for an ADMITTED
  one, and only the barrier (open → receipt per active member → commit) admits
  it. A receipt is EVIDENCE: each member's own `/v1/catalog/sync` must report
  the published catalog revision applied and all three projection revisions
  current before duckgres records it. A tenant that leaves the projection is
  REVOKED, once (the durable row is kept); with the gate on, a warehouse is held
  at Provisioning until its publication commits. Readiness then follows the
  COMMITTED target revision, not the state field: a tenant that adds a login
  opens a new attempt, and reading the state alone would flap a warehouse that
  has served for weeks back to Provisioning because somebody created a user.
  All of it reads the durable record (migration `000041`), never the
  leader's memory.
- **A refusal settles a tenant's occurrence only on the FIRST request of it.**
  The Gateway rolls a refused call back before it journals anything, so the step
  identity stays unclaimed, and publishing principals carries no revision
  ordering of its own. On a REISSUE an earlier copy of the same request may
  still be executing there, so closing the occurrence would let the controller
  advance to a newer intent, publish and checkpoint it, and then have that older
  copy land and overwrite the Gateway's binding with the superseded principal
  set — which duckgres never corrects, because it believes it already published
  the newer one. A removed login would stay dispatchable and a current one could
  not dispatch. `failTenantStep` therefore takes `firstAttempt`: true from
  `publishChangedBindings`/`revokeDepartedTenant`, false from
  `reissuePublication`/`reissueRevoke`. The property the tests pin is that what
  duckgres has checkpointed always names the set the Gateway actually binds
  (`TestACheckpointedBindingNeverDivergesFromTheGateway`), NOT that the newest
  desired set wins.
  **Known limitation (followup, not recovery):** a tenant whose first request
  was UNKNOWN and whose reissue is then refused for a condition that never
  clears stays HELD on that occurrence — it publishes no corrected binding and
  is not revoked until the refusal stops. That is the conservative direction
  (the alternative loses a binding silently), and resolving it needs either a
  durable "was this occurrence ever unknown" bit or a Gateway way to burn a step
  identity. Both are out of scope here; do not "fix" it by closing the
  occurrence on a reissue.
- **One authoritative boot identity.** The coordinator's `processId` is probed
  BEFORE member registration and is sent as `bootId` on both register and admit;
  the Gateway requires the receipt to carry the pair it recorded. A restart
  before admission fails the candidate (`FAILED_PREPARING`) rather than retrying
  against an incarnation that no longer exists.
- **Transport is plain internal HTTP** to each instance's ClusterIP Service; TLS
  terminates at the Gateway. Probes declare the forwarded HTTPS hop
  (`X-Forwarded-Proto`) rather than relaxing auth, and
  `allow-insecure-over-http` is never set. There is no pool certificate, no
  private CA and no `tls_server_name` for a pooled instance. Credentials and
  query data cross the cluster network unencrypted on this hop — accepted, with
  coordinator NetworkPolicies a deferred follow-up, so pool admission must NOT
  be described as non-bypassable by an in-cluster caller.
- **The Gateway client matches the Java source, not a design doc**
  (`GET members` is a bare array; admission is one `admit` call with a nested
  receipt; fields are `desiredMembers`/`maxRepair`; the Gateway computes the
  guard payload hash itself). Decoding is pinned by fixtures generated from the
  real records — regenerate with `tools/gatewaywire`. Pooled registration needs
  a Gateway backend record, created INACTIVE and never activated through the
  legacy route.
- **Gateway administration has two authentication layers when form auth is enabled.**
  `DUCKGRES_TRINO_MANAGED_GATEWAY_USERNAME` selects the existing API-role
  identity. The pool client sends Basic authentication with that username and
  the existing rollout token as password, plus
  `X-Gateway-Transaction-Admin-Token` for the pool capability check. An opaque
  token is not a form-session JWT. An unset username preserves the token-only
  Bearer mode for Gateways without the outer form-auth requirement. No new
  secret or API authorization exception is introduced.
- **Principal binding is duckgres-authoritative.** The tenant's principal set
  is derived from the SAME projection that writes `password.db`, asserted
  identical in test; the gate would otherwise block real users or admit
  principals Trino rejects.
- **Catalog writer**: one fenced transaction per mutation against the
  Trino-side catalog store's schema (writer-state row lock, exact epoch AND
  identity, journal replay resolution, `catalog_count` recomputed inside the
  transaction; the tables are created by `trinocatalog.EnsureSchema` and
  asserted against a real PostgreSQL in `tests/trinocatalog/`).
  Takeover is explicit; a mutation never claims a higher epoch implicitly. A
  lost COMMIT is resolved from the journal, never retried blind.
- Touching any of this → update `controlplane/trinopool/*_test.go`,
  `controlplane/trinogateway/*_test.go` (fixtures are generated from the real
  Java records — see `tools/gatewaywire/README.md`),
  `controlplane/trino_pool_*_test.go`,
  `controlplane/trino_pool_member_retries_test.go`,
  `controlplane/trino_pool_publication_order_test.go`,
  `tests/configstore/trino_pool_postgres_test.go`,
  `tests/trinocatalog/*_postgres_test.go`, AND the
  `trino_shared_pool_disabled` assertion in `tests/mw-dev/e2e/harness.sh`.
  The pool's migrations are numbered ABOVE upstream's, so adding one means
  extending `TestMigrationVersionsAreUnique`'s range implicitly (it reads the
  embedded directory) and updating the version asserts in
  `tests/configstore/migrations_postgres_test.go` — including
  `TestConfigStoreMigration40PinsExistingTrinoBackends`, which rewinds every
  version at or above 40 so goose can re-apply them in order.

## Logical Catalog Alias (`org_<database_name>` as the startup `database`)

A pgwire session may select its catalog by the name the org has on Trino
(`configstore.TrinoCatalogName`, `org_<sanitized database_name>`) instead of
`ducklake`. The catalog is the same one either way: the alias only renames it
on the wire. This exists so SQLMesh sees ONE catalog name across the Duckgres
and Trino engines and migrating between them needs no state rewrite.

- **What the startup `database` may be**: `""`, `ducklake`, or the
  SNI-resolved org's own catalog name. Everything else is 3D000, as before.
- **Identity is still SNI-only, and this must stay true.** The alias is
  compared against the catalog name of the org the managed hostname ALREADY
  resolved — it is never a key into `DatabaseOrg`, `Orgs`, or any other map,
  so it can neither discover nor select an org. A sibling tenant's catalog
  name is just an unrecognized string and fails closed. If a change here ever
  looks up an org BY the database name, it has reintroduced exactly what PR
  #651 removed. See the comment in `ResolvePostgresConnection`.
- **Physical vs. visible**: `EffectiveCatalog` stays `ducklake` and is what
  every statement executes against; `LogicalCatalog` (and
  `sessionMetadataResult.visibleCatalog`) is the name reported by
  `current_database()`, `pg_database`, `information_schema`, and the logs.
  `visibleCatalogName` is the one place that chooses between them.
- **SQL written against the alias**: the transpiler's `LogicalCatalogTransform`
  rewrites `<alias>.public.t` → `ducklake.main.t` (fed by `clientConn.database`
  in `newTranspiler`), and `rewriteDirectQuery` expands `USE <alias>` to
  `ducklake.main`.
- Opt-in per connection: a session that connects with `ducklake` or nothing is
  byte-for-byte unaffected.
- Touching any of this → update `controlplane/configstore/store_test.go`,
  `controlplane/session_search_path_test.go`, `server/direct_query_rewrite_test.go`,
  `server/logical_catalog_alias_test.go`, `server/session_database_metadata_test.go`,
  and `logical_catalog_alias` in `tests/mw-dev/e2e/harness.sh`.

## TODO Reference

`TODO.md` is a lightweight backlog for ideas that do not yet have a better
home. It is not the PostgreSQL compatibility source of truth; use
`docs/postgres-compatibility.md` for compatibility status, test citations, and
known gaps.