containers — images and services without surprises

Pinned: docker engine, compose — unpinned · authored 2026-07-18 · run /skill-update containers against the installed engine (docker --version)

On-demand: load this when the project needs an image (training on another box, serving) or local services (a real MLflow server, Postgres, Label Studio). The reproducibility rules are the repo's usual ones at image altitude: pin what you build from, build from the lockfile, and keep data/secrets out of layers. Compose is for local support services; running images in production is kubernetes — v1.0.0 un-parked orchestration (see memory/process/decision-log.md). Image security obligations are canon (policy/supply-chain.md C5, policy/platform-security.md P4); scanning, SBOM, and signing are supply-chain-security.

Runtime images are hardened by default: digest-pinned base, distroless or slim, multi-stage so build tooling never ships, USER set to a non-root uid. A container that runs as root fails Pod Security restricted and is blocked before it reaches a cluster (P1).

The training image (reproducibility is the point)

FROM nvidia/cuda:VERSION-cudnn-runtime-ubuntu22.04@sha256:DIGEST   # digest-pinned, CUDA ≤ host driver
COPY --from=ghcr.io/astral-sh/uv:latest /uv /usr/local/bin/uv
WORKDIR /app
COPY pyproject.toml uv.lock ./
RUN uv sync --frozen --no-dev                       # THE lockfile, not a requirements guess
COPY conf/ conf/
COPY src/ src/
RUN useradd -m runner && chown -R runner /app
USER runner                                        # never train as root
ENTRYPOINT ["uv", "run", "python", "src/PKG/train.py"]

• CUDA base vs host driver: the image's CUDA version must be ≤ what the host driver supports (nvidia-smi on the host shows the max). This is env-uv's wheel-matching problem wearing a container; same failure mode (torch.cuda.is_available() False inside the box).
• GPU runtime: host needs nvidia-container-toolkit; run with --gpus all (compose: deploy.resources.reservations.devices). Verify inside the container, not from the Dockerfile.
• Data is mounted, never copied: -v $DATA_ROOT:/data:ro (read-only for training). Copying a dataset into a layer makes a multi-GB image that leaks data with every push.
• .dockerignore is a security control, not tidiness: data/, models/, mlruns/, .env, .git/ — a COPY . . without it bakes datasets and secrets into layers that out-live the container. Same class of leak guard-secrets blocks in files.

The serving image

Slim base (no CUDA unless GPU inference is real), the same uv sync --frozen, and the model pulled at startup from the registry alias (models:/NAME@champion, per serving) — not baked into the image. Baking couples redeploys to retrains and hides which model is live; pulling keeps rollback = alias move. Health endpoint + model_version in responses as serving specifies.

Compose — the support services

One docker-compose.yml for the services the pipeline leans on, e.g. a real tracking backend:

services:
  postgres:
    image: postgres:16@sha256:DIGEST
    env_file: .env                       # POSTGRES_PASSWORD lives there, not here
    volumes: ["pgdata:/var/lib/postgresql/data"]   # named volume = the database's life
  mlflow:
    image: ghcr.io/mlflow/mlflow:VERSION
    command: mlflow server --backend-store-uri postgresql://... --host 0.0.0.0
    ports: ["5000:5000"]
    depends_on: [postgres]
volumes:
  pgdata:

• Named volumes hold the state that matters (the tracking DB!). docker compose down -v deletes them — the bash hook confirm-gates it; treat volumes like mlflow.db, not like cache.
• Secrets ride env_file: .env (gitignored, per the security canon) — never literal in YAML.
• Pin image tags with digests; latest is an unpinned dependency by another name.
• Every compose service gets a row in the resource matrix (.claude/memory/process/resources.md) + its env keys in .env.example, same commit — ports, volumes, and credential references live there, not in tribal memory.

Gotchas

• Rebuild triggers: uv.lock changes must rebuild the sync layer — order COPY of lockfile before source so code edits don't bust the dependency cache (and lockfile edits do).
• A container is not isolation from cost or data loss: mounted volumes and cloud creds passed in are as live as on the host — the same guards apply.
• Image provenance: record the image digest of a training run with the run (a tag/param) — "which container produced this checkpoint" is part of reproducibility.