Docker Debugging

Find out what the container actually ran, in what environment, and why it stopped.

Purpose

Use this when:

• A container exits immediately, or restarts repeatedly.
• A file that exists in the build is missing at runtime.
• The container cannot reach the network, or cannot be reached.
• Permission denied on a mounted volume.
• It works on one machine and not another.

Do not use this when:

• The container runs under Kubernetes and the symptom is scheduling or probes — use kubernetes-troubleshooting.
• The question is image quality, size or security — use container-image-review.

What this skill assumes: Docker or a compatible runtime on the host. Commands are shown for docker; podman and nerdctl accept the same arguments for everything used here.

Operating Procedure

1. Get the exit code and the reason. docker ps -a and docker inspect give both.
2. Read the logs, including from the exited container. A stopped container keeps its logs until it is removed.
3. Establish what actually ran. The effective entrypoint and command are frequently not what the Dockerfile appears to say.
4. Compare build-time and run-time. Most "missing file" reports are a multi-stage build that did not copy it, or a volume mounted over it.
5. Check the boundaries: network, mounts, users, and architecture.
6. Reproduce interactively with an overridden entrypoint — the fastest way to inspect a broken image.
7. Fix in the Dockerfile or the run configuration, then rebuild and verify.

What docker inspect reports is Observed. What the Dockerfile says is what was intended — they differ whenever a base image, a compose file, or a runtime flag overrides something.

Investigation

Step 1 - status and exit code

docker ps -a
docker inspect <container> --format '{{.State.Status}} {{.State.ExitCode}} {{.State.Error}} {{.State.OOMKilled}}'
docker logs <container> --tail=100
docker logs <container> --timestamps 2>&1 | tail -50

Exit codes carry specific meaning:

Exit 127 in a container that built successfully is almost always one of two things: a binary compiled against glibc running on Alpine (musl), or a multi-stage build whose final stage lacks a runtime dependency. Both are covered in container-image-review.

Step 2 - what actually ran

docker inspect <image> --format 'ENTRYPOINT={{.Config.Entrypoint}} CMD={{.Config.Cmd}} USER={{.Config.User}} WORKDIR={{.Config.WorkingDir}}'
docker history <image> --no-trunc | head -20

The effective command is ENTRYPOINT with CMD appended as arguments. Three rules explain most surprises:

• Specifying a command on docker run replaces CMD, not ENTRYPOINT.
• Exec form (ENTRYPOINT ["app", "--flag"]) runs the binary directly as PID 1.
• Shell form (ENTRYPOINT app --flag) runs it under /bin/sh -c, which means the shell is PID 1 and signals do not reach your process. The container then ignores SIGTERM and gets SIGKILLed after the grace period, which shows up as slow, ungraceful shutdowns and dropped connections.

Prefer exec form. When shell features are genuinely needed, exec the real process so it takes over PID 1.

Step 3 - inspect a broken image interactively

docker run --rm -it --entrypoint sh <image>
docker run --rm -it --entrypoint sh <image> -c 'ls -la /app; id; env | sort'
docker create --name tmp <image> && docker cp tmp:/app ./app-contents && docker rm tmp

Overriding the entrypoint is the fastest way into an image whose startup fails. If the image has no shell — distroless, scratch — use docker cp to extract the filesystem and inspect it on the host instead.

For a running container:

docker exec -it <container> sh
docker diff <container>

docker diff shows every file added, changed or deleted relative to the image. It is an underused command: it answers "did the application write where I expected?" directly, and often reveals a process writing to a path that a volume was supposed to cover.

Step 4 - volumes and permissions

docker inspect <container> --format '{{json .Mounts}}' | jq
docker exec <container> id
ls -ln <host-path>

Two distinct problems that produce the same error message:

1. A mount covers the image's content. Mounting a host directory at /app hides everything the image put there. If the host directory is empty, /app is empty at runtime. A bind mount always wins over image content.
2. UID mismatch. The container's user has a UID that does not match the host file ownership. Containers see numeric UIDs, not names — appuser inside the container and appuser on the host are unrelated unless the numbers match. Compare docker exec <container> id with ls -ln (numeric) on the host.

A named volume behaves differently from a bind mount on first use: Docker copies the image's content into a new, empty named volume. It does not do this for bind mounts, and it does not do it again once the volume has content — which is why a volume can keep serving old files long after the image changed.

Step 5 - networking

docker inspect <container> --format '{{json .NetworkSettings.Networks}}' | jq
docker port <container>
docker exec <container> sh -c 'command -v ss >/dev/null && ss -lntp || netstat -lntp'
docker exec <container> cat /etc/resolv.conf
docker exec <container> getent hosts <other-service>

The usual causes, in order:

Step 6 - architecture

docker image inspect <image> --format '{{.Architecture}}/{{.Os}}'
uname -m
docker run --rm <image> uname -m

An arm64 image on an amd64 host — or the reverse — fails with exec format error, or runs under emulation at a fraction of the speed. This has become common with Apple Silicon laptops building images for x86 servers. Build multi-architecture images with docker buildx, or set --platform explicitly and consistently.

Decision Tree

Container problem
├── Exits immediately with 0 → the process is not long-running; check ENTRYPOINT/CMD
├── Exit 127 → command not found
│   ├── glibc binary on Alpine → musl mismatch
│   └── Multi-stage build did not copy a dependency
├── Exit 126 → found but not executable → file mode or shebang
├── Exit 125 → the daemon rejected the run → bad flag
├── Exit 137 + OOMKilled → memory limit
├── Exit 143 → asked to stop; if unexpected, check the orchestrator
├── exec format error → architecture mismatch
├── File missing at runtime
│   ├── docker diff / exec shows it absent → build did not produce it
│   └── Present in image, absent at runtime → a mount covers it
├── Permission denied on a mount → compare numeric UIDs
├── Cannot connect to a published port
│   ├── Bound to 127.0.0.1 inside → bind 0.0.0.0
│   └── Not published → -p
└── Container cannot reach another → same user-defined network?

Common Causes

Safety

Read-only

docker ps, logs, inspect, history, diff, port, top, and image inspect. These read daemon state and change nothing.

docker exec is read-only only if the command you run is. A shell in a production container can change files, and on most setups it is an audited action.

Potentially disruptive

• docker restart / stop / kill — ends in-flight work. stop sends SIGTERM and then SIGKILL after the timeout (10s by default), which is frequently too short for a graceful shutdown; use -t to extend it.
• docker rm <container> — removes the container and its logs. Capture docker logs before removing anything you might need to explain.
• docker network disconnect — severs connectivity for a running container immediately.
• Changing a published port or a mount — requires recreating the container, not restarting it.
• Rebuilding with --no-cache — slow, and pulls fresh base images, which can introduce changes you did not intend to adopt during an incident.

Destructive

• docker volume rm <volume> — deletes the data in that volume permanently. Databases, uploads, and anything else that survives a container restart lives there. Required first: confirm what uses it (docker volume inspect, docker ps -a --filter volume=<name>), confirm a backup exists, and get explicit confirmation.
• docker system prune -a --volumes — removes all stopped containers, unused networks, all unused images, the build cache, and with --volumes every volume not attached to a running container. A stopped database container's volume qualifies as unused. This is the single most common way local and staging data is destroyed. Required first: run the non-destructive listing (docker volume ls, docker ps -a) and confirm what would be removed.
• docker rm -f on a container with an anonymous volume — the container goes, and -v would take the volume with it. Know which flags you are passing.

On a Kubernetes node, do not prune manually at all: the kubelet manages image garbage collection, and removing images by hand can leave workloads unable to restart.

Verification

Before: container exits with 127 immediately; no logs
After:  container Up 10 minutes; /healthz returns 200; SIGTERM stops it in 2s

docker ps --filter name=<name> --format '{{.Status}}'
docker inspect <container> --format '{{.State.Status}} {{.State.ExitCode}} {{.RestartCount}}'
docker exec <container> wget -qO- http://localhost:8080/healthz
time docker stop <container>

Three checks that matter beyond "it started":

1. It stays up. A restart count that keeps climbing means it is still failing, just more slowly.
2. It serves traffic, from outside the container as well as inside.
3. It stops gracefully. time docker stop should finish well inside the timeout. If it takes the full 10 seconds, the process is being SIGKILLed — signals are not reaching it, and in production that means dropped connections on every deploy.

Anti-Patterns

Bad: The container exits, so rebuild with --no-cache. Better: Read the exit code and the logs first. A cache rebuild takes minutes and fixes almost none of the failures listed above.

Bad: docker system prune -a --volumes to free disk space. Better: That deletes volumes belonging to stopped containers. Prune images specifically, and check what will be removed before running anything with --volumes.

Bad: Run as root to fix a permission error. Better: It works, and it removes a real security boundary. Align the UIDs or fix the ownership of the host directory.

Bad: Use shell-form ENTRYPOINT for convenience. Better: The shell becomes PID 1, signals do not reach the application, and every stop takes the full timeout and kills in-flight requests.

Bad: Add EXPOSE and expect the port to be reachable. Better: EXPOSE is metadata. Publishing needs -p or the orchestrator's equivalent.

Bad: Debug by adding RUN echo lines and rebuilding. Better: docker run --entrypoint sh gets you inside the exact image in seconds, without a rebuild cycle.

Composition

• container-image-review — image size, base image choice, non-root users, build correctness.
• kubernetes-troubleshooting — the same containers under an orchestrator.
• linux-troubleshooting — host-level disk, memory and process problems beneath the runtime.
• dns-troubleshooting — name resolution inside containers.

References

• Docker: ENTRYPOINT and CMD interaction
• Docker: Networking overview — bridge versus user-defined networks and their DNS behaviour.