Debug infra startup

A service can be in three different states that are easy to confuse:

1. Process started — the container is running, the PID exists
2. HTTP server up — the health or ready endpoint returns 200
3. Actually accepting traffic — the write path, the ring, the index, the queue is open

Most startup bugs live in the gap between 2 and 3. The readiness signal lies, or the probe checks the wrong thing, and the dependent service crashes into what looks like a mystery.

Ground rules

• Never trust a readiness signal until you have probed the actual work path. A /ready 200 proves the HTTP server is up. It does not prove the service can handle your request.
• Reproduce the probe the way the failing component uses the service. If the failing component writes, your probe must write. If it queries, your probe must query.
• Check the image before writing a healthcheck. Distroless and minimal images have no shell, no wget, no curl. A CMD-SHELL healthcheck silently fails or is never executed.
• Fix the probe before fixing the timeout. Increasing --interval or --retries on a broken probe only delays the same failure.
• One race, one fix. Do not add sleeps everywhere. Find the specific gap and close it with a real probe.

The three readiness levels

Always establish which level you are actually at before debugging further.

Workflow

1. Establish what the readiness signal actually checks

Read the service documentation or source for what the health and ready endpoints verify:

# What does the ready endpoint actually gate on?
curl -sf http://localhost:3100/-/ready
# "ready" != "ring joined" != "ingester accepting writes"

# For services that expose internal state, check it directly
curl -s http://localhost:3100/ring | jq '.ingesters[] | {addr, state}'
curl -s http://localhost:9090/-/status

If the ready endpoint only checks that the HTTP server started, it is not a readiness signal for the work path. Treat it as a liveness signal only.

2. Probe the actual work path

Construct the simplest possible request that exercises the real path:

# Loki: probe the push path, not just /ready
curl -sf -X POST http://localhost:3100/loki/api/v1/push \
  -H 'Content-Type: application/json' \
  -d '{"streams":[{"stream":{"probe":"1"},"values":[["'"$(date +%s%N)"'","probe"]]}]}'

# Postgres: probe a write, not just a connection
psql "$DATABASE_URL" -c "INSERT INTO _healthcheck (ts) VALUES (now()) ON CONFLICT DO NOTHING"

# Redis: probe a write
redis-cli SET _healthcheck 1 EX 5

# Elasticsearch: probe index availability
curl -sf "http://localhost:9200/_cluster/health?wait_for_status=yellow&timeout=10s"

If the work-path probe fails while /ready returns 200, the readiness signal is insufficient. You have found your bug.

3. Check the image for shell availability

Before writing any healthcheck, confirm what the image actually contains:

# Does it have a shell?
docker run --rm --entrypoint sh <image> -c "echo ok" 2>&1

# Does it have wget?
docker run --rm --entrypoint wget <image> --version 2>&1

# Does it have curl?
docker run --rm --entrypoint curl <image> --version 2>&1

If the image is distroless or scratch-based, none of these work. Use the exec form instead:

# WRONG on distroless: CMD-SHELL is never available
healthcheck:
  test: ["CMD-SHELL", "wget -qO- http://localhost:3100/-/ready || exit 1"]

# Correct: exec form, binary must exist in the image
healthcheck:
  test: ["CMD", "/usr/bin/wget", "-qO-", "http://localhost:3100/-/ready"]

# If no HTTP client exists, use a native TCP check via Docker's built-in
healthcheck:
  test: ["CMD", "/bin/grpc_health_probe", "-addr=:50051"]

For images with no probe binary at all, add a sidecar or use a purpose-built probe image:

# docker-compose: separate probe container that waits before declaring healthy
depends_on:
  loki:
    condition: service_healthy

4. Find where the wait logic belongs

Pick the right place based on what tools are available:

Do not add sleep N as the wait mechanism. A sleep either waits too long on a fast machine or not long enough under load. Use a probe loop:

# Portable retry loop: probe the real work path
wait_for_loki() {
  local retries=30
  until curl -sf -X POST http://localhost:3100/loki/api/v1/push \
    -H 'Content-Type: application/json' \
    -d '{"streams":[{"stream":{"probe":"1"},"values":[["'"$(date +%s%N)"'","probe"]]}]}' \
    > /dev/null 2>&1
  do
    retries=$((retries - 1))
    [ "$retries" -le 0 ] && echo "Loki never became ready" >&2 && return 1
    sleep 1
  done
}

5. Check config path assumptions

Startup failures that look like race conditions are sometimes config path mismatches — the process starts, but it is reading the wrong directory or file:

# Is the config being picked up from where the process expects it?
docker inspect <container> | jq '.[0].HostConfig.Binds'
docker exec <container> cat /etc/loki/config.yaml

# Is the working directory what the tool assumes?
# e.g. a build tool that expects to be run from the repo root
# but the CI job is already inside a subdirectory
pwd
ls -la

If a tool says a file or directory does not exist, confirm what the working directory actually is before assuming the config is wrong.

6. Read the startup logs in full

Do not skim. The real error is almost never on the last line:

docker logs <container> 2>&1 | head -80
docker logs <container> 2>&1 | grep -i "error\|warn\|fail\|ring\|leaving\|joining"

# For a service that crashed immediately
docker logs <container> --tail 50

# Follow until a pattern appears
docker logs -f <container> 2>&1 | grep -m 1 "started\|ready\|error"

Ring state messages, joining/leaving log lines, and "waiting for" messages appear early in the log. A service that looks ready from the outside may have logged its actual state several lines up.

7. Verify the fix end-to-end

After fixing the probe or the wait logic:

1. Tear down and restart from clean state: docker compose down -v && docker compose up -d
2. Wait for the probe to pass naturally — do not skip or bypass it
3. Confirm the work-path probe succeeds after the service is declared healthy
4. Run the dependent component and confirm it no longer fails at startup

Common patterns and their fixes