🐳 Docker Troubleshooting Playbook

Docker Troubleshooting Playbook: when the container will not start, talk, or behave
docker ps -a; docker logs --tail 50 NAME; docker inspect NAME

The Docker and Compose sheets show how to RUN containers. This playbook is for when they do not work: containers that exit at once, restart in a loop, cannot reach each other or the internet, fill the disk, get killed, or behave differently from your laptop. It goes in the order you should investigate: what state is it in, what did it say, what does Docker think its configuration is, then the network, storage and host.
The Compose file checks and the "reading the output" examples ran for real (Docker Compose v5 on this machine, on typed sample data, so the examples never touch a running container). Commands that act on live containers are shown without output.

$ docker ps -a --format 'table {{.Names}}\t{{.Status}}\t{{.Image}}\t{{.Ports}}'      # everything, including stopped: is it Up, Exited (N), Restarting, or (unhealthy)?
$ docker logs --tail 100 --timestamps NAME           # what the program said (stdout and stderr)
$ docker inspect NAME                                  # Docker's full record: config, state, mounts, network (very long: use --format, below)
$ docker events --since 30m --until 0s                 # what Docker did recently: start, die, oom, kill, health_status
$ docker compose ps -a; docker compose logs --tail 50  # the same for a Compose project (run in the project folder, or use -p NAME)
Read the STATUS column first:
  Up 3 hours                  running. If it still misbehaves, the problem is inside the application, its config, or the network.
  Up 3 hours (unhealthy)      the healthcheck fails: docker inspect --format '{{json .State.Health}}' NAME | jq
  Up 2 seconds / Restarting   crash loop: the program starts and dies, Docker restarts it. Read the logs and the exit code.
  Exited (0)                  the program finished normally (a one-shot job, or a program that daemonizes and returns: not what a container wants)
  Exited (1)                  the program failed: its error message is in the logs
  Exited (137) / (139) / (143) killed by a signal (see the exit-code table)
  Created                     it was created but never started: a start error (a port in use, a missing mount): docker start NAME shows it

Code   Meaning                                       What to do
0      success                                       normal for a job; for a service it means the main process ended: check CMD/ENTRYPOINT
1      the application reported an error             read the logs
2      shell or program misuse (bad arguments)       check the command line and CMD
125    docker itself failed to run the container     the error is printed by docker run/start: bad option, port already allocated, no such device
126    the command was found but is not executable   file permissions (chmod +x), wrong architecture, or a directory instead of a file
127    command not found                             typo in CMD/ENTRYPOINT, the binary is missing from the image, or a shebang points to an interpreter the image does not have (#!/bin/bash in an alpine image)
130    interrupted (Ctrl+C, SIGINT)                  someone stopped it in the foreground
137    killed (128+9, SIGKILL)                       OUT OF MEMORY, or docker kill, or the stop timeout expired: check OOMKilled below
139    segmentation fault (128+11)                   a crash in the program or a library, often a wrong-architecture binary or a musl/glibc mismatch
143    terminated (128+15, SIGTERM)                  docker stop: a normal shutdown if it exits quickly, but a service that ignores SIGTERM takes 10 seconds then gets 137
Rule: anything above 128 is 128 plus a signal number. Ask Docker directly instead of guessing:
$ docker inspect NAME --format 'exit={{.State.ExitCode}} oom={{.State.OOMKilled}} restarts={{.RestartCount}} error={{.State.Error}} finished={{.State.FinishedAt}}'
Here is what that kind of record looks like, and how to pull the useful fields out of it. (A typed-in sample of docker inspect output, trimmed; on your machine you run docker inspect NAME instead.)
$ jq -r '.[0] | "state=\(.State.Status) exit=\(.State.ExitCode) oom=\(.State.OOMKilled) restarts=\(.RestartCount) memory_limit_MB=\(.HostConfig.Memory/1048576)"' inspect.sample.json
state=restarting exit=137 oom=true restarts=7 memory_limit_MB=256
$ jq -r '.[0].State.Health | "health=\(.Status) streak=\(.FailingStreak) last=\(.Log[-1].Output)"' inspect.sample.json
health=unhealthy streak=5 last=curl: (7) Failed to connect to localhost port 8080
$ jq -r '.[0] | "user=\(.Config.User | if . == "" then "ROOT (no USER set)" else . end) privileged=\(.HostConfig.Privileged) published=\(.HostConfig.PortBindings | keys | join(","))"' inspect.sample.json
user=ROOT (no USER set) privileged=false published=8080/tcp
The story in that sample: a 256 MB limit, OOMKilled true and exit 137 mean the kernel killed the process for using more memory than allowed, and Docker keeps restarting it (7 times so far). The fix is more memory or a smaller program, not a restart policy.
Go-template version of the same (no jq needed, runs against a live container):
$ docker inspect NAME --format '{{.State.Status}} {{.State.ExitCode}} {{.State.OOMKilled}} {{.HostConfig.Memory}}'
$ docker inspect NAME --format '{{range .Mounts}}{{.Type}} {{.Source}} -> {{.Destination}}{{"\n"}}{{end}}'
$ docker inspect NAME --format '{{range $k, $v := .NetworkSettings.Networks}}{{$k}} {{$v.IPAddress}}{{"\n"}}{{end}}'
$ docker inspect NAME --format '{{json .Config.Env}}' | jq -r '.[]'
$ docker inspect NAME --format '{{json .State.Health}}' | jq

$ docker logs NAME                                      # the last lines before it died are usually the answer
$ docker run --rm -it --entrypoint sh myorg/web:1.4     # start the IMAGE with a shell instead of the real command: look around, run the command by hand
$ docker run --rm -it myorg/web:1.4 env                 # what environment does it really get? ;  docker run --rm myorg/web:1.4 ls -l /app
$ docker compose run --rm --entrypoint sh web           # the same for a Compose service (with its volumes, networks and environment)
$ docker diff NAME                                      # files the container changed: A added, C changed, D deleted (config written at start? a missing mount?)
$ docker cp NAME:/app/config.yml ./config.yml           # copy a file out of a stopped container to read it
$ docker export NAME | tar -t | head                    # list the container's file system
Common causes and the message that gives them away
  exec /app/start.sh: no such file or directory      (despite the file existing) the script has Windows CRLF line endings (#!/bin/sh^M) or its #! interpreter is missing: tr -d '\r' < start.sh > fixed.sh, and use the right shebang for the image (alpine has /bin/sh, not /bin/bash)
  exec format error                                    the image was built for another CPU architecture (arm64 vs amd64): docker image inspect IMAGE --format '{{.Architecture}}'; pull or build with --platform linux/amd64
  permission denied (on a file or a volume)            the container runs as a non-root user that cannot read the mounted folder: compare ls -ln on the host with docker run --rm IMAGE id; fix with chown, user:, or a group
  bind: address already in use / port is already allocated     something else holds the host port: sudo ss -tlnp | grep :8080; docker ps --filter publish=8080
  Error response from daemon: ... mount ... no such file or directory     a bind-mounted path does not exist on the host (Docker creates a FOLDER where you wanted a file)
  OCI runtime create failed ... executable file not found in $PATH         the command is not in the image: docker run --rm -it --entrypoint sh IMAGE, then which COMMAND
  Killed (in the logs) with exit 137                    out of memory: docker inspect --format '{{.State.OOMKilled}}' NAME, raise the limit or fix the leak
  Cannot connect to the database at startup             it started before the database was READY (depends_on only waits for "started" unless a healthcheck condition is used: see below)
$ docker update --restart=no NAME                       # stop the restart loop while you investigate (so the logs are not overwritten every few seconds)

docker compose config validates the file, resolves variables and prints the result without starting anything: the first thing to run when a project will not come up. These runs used a small project (a web service that waits for a healthy database):
$ docker compose config -q 2>&1 | grep -v 'level=warning'
error while interpolating services.db.environment.POSTGRES_PASSWORD: required variable DB_PASS is missing a value: DB_PASS must be set
$ DB_PASS=s3cret docker compose config -q && echo "valid"
valid
$ DB_PASS=s3cret docker compose config --services
db
web
$ DB_PASS=s3cret docker compose config --images
postgres:16
nginx:1.27
$ DB_PASS=s3cret docker compose config | sed -n '/^  db:/,/^  web:/p' | sed -n '1,12p'
  db:
    environment:
      POSTGRES_PASSWORD: s3cret
    healthcheck:
      test:
        - CMD-SHELL
        - pg_isready -U app
      interval: 5s
      retries: 5
    image: postgres:16
    networks:
      default: null
${VAR:?message} makes a missing variable an ERROR with your message (instead of silently becoming an empty string: the first run above also made Compose warn that DB_PASS was blank for the DB_URL line, which the grep hides, and then stopped at the required one). ${VAR:-default} supplies a default. Variable sources, in order of priority: your shell environment, then the .env file next to the compose file (or --env-file), then the defaults in the file. docker compose config prints what Compose really sees: use it when "it works on my machine".
Typical file mistakes, with the exact messages:
$ docker compose -f bad-indent.yaml config -q 2>&1 | grep -v 'level=warning'
go-yaml load error in parser (while parsing a block mapping) at L2.C3-L4.C4: did not find expected key
$ docker compose -f bad-key.yaml config -q 2>&1 | grep -v 'level=warning' | sed 's|validating .*/||'
bad-key.yaml: services.web additional properties 'prots' not allowed
$ docker compose -f bad-dep.yaml config -q 2>&1 | grep -v 'level=warning'
service "web" depends on undefined service "cache": invalid compose project
$ docker compose up -d --force-recreate web          # recreate one service even if nothing changed in its config
$ docker compose up -d --build --remove-orphans      # rebuild images and remove containers of services that are no longer in the file
$ docker compose down                                  # stop and remove containers and networks (volumes are KEPT);  down -v ALSO DELETES the named volumes: your data
$ docker compose logs -f --tail 20 db web              # follow several services
$ docker compose ps -a --format 'table {{.Name}}\t{{.Service}}\t{{.State}}\t{{.Health}}'
Waiting for readiness: depends_on with a plain list only orders the START. To wait until the database accepts connections, give it a healthcheck and use condition: service_healthy (as in the file above). Also make the application retry its connection: containers restart, networks blip.
$ docker compose up -d --wait                         # return only when every service is running AND healthy (fails if one is not)

Work out which of these four you have:
  1. Container to container by NAME fails: they are not on the same USER-DEFINED network. The default "bridge" network has NO name resolution; Compose creates a user-defined network per project, so service names work (db, redis) as hostnames.
  2. Host to container fails: the port is not published (-p / ports:), or published on 127.0.0.1 only, or the program inside listens on 127.0.0.1 instead of 0.0.0.0.
  3. Container to the internet fails: DNS, MTU, a proxy, an "internal" network, or the host firewall.
  4. From another machine fails: published on loopback only (by design?), the host firewall, or the cloud security group.
$ docker network ls; docker network inspect app_default --format '{{range .Containers}}{{.Name}} {{.IPv4Address}}{{"\n"}}{{end}}'
$ docker exec web getent hosts db                       # does the NAME resolve? (works in any image with libc; nslookup/dig need to be installed)
$ docker exec web sh -c 'nc -zv db 5432 || echo "cannot connect"'     # can I reach the port?   (or: wget -qO- http://db:8080/health)
$ docker exec web cat /etc/resolv.conf                  # inside a user-defined network this points at Docker's DNS, 127.0.0.11
$ docker port web                                       # what is published where
$ docker exec web ss -tlnp                              # what is the program LISTENING on inside? (0.0.0.0:8080 or [::]:8080 is reachable; 127.0.0.1:8080 is NOT, from outside the container)
$ sudo ss -tlnp | grep -E ':8080\b'                     # on the host: who holds the published port
$ curl -sv http://127.0.0.1:8080/ 2>&1 | head -15       # test from the host
A program that listens only on 127.0.0.1 inside the container is unreachable from anywhere else: the classic "works in the container, refused from outside". Configure it to bind 0.0.0.0 (and keep the exposure decision in the published port, e.g. 127.0.0.1:8080:8080 for loopback only).
Published ports and the host firewall: Docker adds its own firewall rules, which run BEFORE ufw and firewalld rules for published ports. "ufw deny 8080" does not hide a published port. Publish to 127.0.0.1 (ports: ["127.0.0.1:8080:80"]) or filter in the DOCKER-USER chain; see the Firewall guide.
Debug from the container's own point of view, even if its image has no tools: share its network namespace from the host:
$ sudo nsenter -t "$(docker inspect -f '{{.State.Pid}}' web)" -n ss -tlnp        # the container's sockets, using the HOST's ss
$ sudo nsenter -t "$(docker inspect -f '{{.State.Pid}}' web)" -n ip addr
$ docker run --rm -it --network container:web nicolaka/netshoot                  # a toolbox image (curl, dig, tcpdump, ss...) attached to web's network
$ sudo tcpdump -ni docker0 port 5432 -c 20                                       # watch the traffic on the default bridge; user networks have br-xxxx interfaces (ip -br link)
"No route to host" or odd timeouts when a VPN or LAN overlaps Docker's address range (172.17.0.0/16 and the 172.18-31 ranges, 192.168.x.x): choose different ranges.
$ docker network inspect bridge --format '{{(index .IPAM.Config 0).Subnet}}'
$ sudo tee /etc/docker/daemon.json <<'EOF'
{ "default-address-pools": [ { "base": "10.200.0.0/16", "size": 24 } ] }
EOF
$ sudo systemctl restart docker           # existing networks keep their old range until recreated: docker compose down && up
An "internal" network (internal: true in Compose) has NO route out by design: containers on it cannot reach the internet, which is exactly what you want for a database or a receive-only service. If a container "cannot reach the internet" and sits only on an internal network, that is the reason.
Slow or failing DNS inside containers is a frequent mystery: set explicit resolvers ("dns": ["10.0.0.2"] in daemon.json or dns: in Compose) when the host uses a stub resolver (127.0.0.53) that containers cannot use.

$ docker system df                                      # images, containers, volumes and build cache: how much, and how much is reclaimable
$ docker system df -v | head -40                        # per image, container and volume
$ docker ps -s                                          # the writable layer size of each container (a big SIZE = data written inside the container instead of a volume)
$ sudo du -sh /var/lib/docker/* 2>/dev/null | sort -rh | head      # where /var/lib/docker spends its space
$ sudo du -sh /var/lib/docker/containers/*/*-json.log | sort -rh | head -5    # runaway container logs
Clean in order of safety (read what each command will delete first):
$ docker image prune                                    # dangling images (untagged leftovers): safe
$ docker builder prune                                  # build cache: safe, builds get slower once
$ docker container prune                                # STOPPED containers
$ docker image prune -a --filter "until=720h"           # unused images older than 30 days
$ docker volume ls -f dangling=true                     # volumes no container uses: LOOK first (this may be someone's only copy of data)
$ docker system prune                                   # stopped containers, unused networks, dangling images, cache. NOT volumes
$ docker system prune -a --volumes                      # everything unused INCLUDING volumes: destructive, think twice
Container logs filling the disk: the default json-file driver never rotates. Set limits globally (new containers) or per service:
$ sudo tee /etc/docker/daemon.json <<'EOF'
{ "log-driver": "json-file", "log-opts": { "max-size": "10m", "max-file": "3" } }
EOF
$ cat <<'EOF'
> services:
>   web:
>     logging:
>       driver: json-file
>       options: { max-size: "10m", max-file: "3" }
> EOF
services:
  web:
    logging:
      driver: json-file
      options: { max-size: "10m", max-file: "3" }
Volumes and mounts
  Data disappeared after recreate      the data was in the container's writable layer or in an anonymous volume: mount a NAMED volume or a host folder at the path the program writes to
  Permission denied on a mount         the host folder's owner does not match the container user: docker run --rm IMAGE id, then chown on the host (or set user: in Compose). On SELinux hosts add :z to the mount (./data:/data:z)
  Folder created where a file was wanted   a bind mount to a path that did not exist: create the file first
$ docker volume inspect dbdata --format '{{.Mountpoint}}'
$ docker run --rm -v dbdata:/data -v "$PWD":/backup alpine tar czf /backup/dbdata-$(date +%F).tgz -C /data .      # back up a named volume with a helper container
$ docker run --rm -v dbdata:/data alpine ls -la /data    # look inside a volume

$ docker stats --no-stream --format 'table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}\t{{.MemPerc}}\t{{.PIDs}}'
$ docker inspect NAME --format 'oom={{.State.OOMKilled}} limit={{.HostConfig.Memory}}'
$ sudo dmesg -T | grep -iE 'out of memory|killed process' | tail         # the kernel's own record
$ docker events --filter event=oom --since 24h                            # Docker saw an OOM kill
$ docker top NAME                                                         # processes inside (as seen from the host)
$ docker run -d --memory 512m --memory-swap 512m --cpus 1.5 myorg/web:1.4  # limits at run time
$ cat <<'EOF'
> services:
>   web:
>     image: myorg/web:1.4
>     mem_limit: 512m
>     cpus: 1.5
>     pids_limit: 200
> EOF
services:
  web:
    image: myorg/web:1.4
    mem_limit: 512m
    cpus: 1.5
    pids_limit: 200
A memory limit that is too low shows as exit 137 with OOMKilled=true; a CPU limit shows as sluggishness without any error (the container is throttled: check the cgroup's cpu.stat for nr_throttled). Java, Node and other runtimes size themselves from the container limit only if they are recent versions: set heap sizes explicitly when in doubt.
Sample: audit published ports and root users across all containers (typed-in output, then the analysis):
$ awk '$2 ~ /0\.0\.0\.0|:::/ {print "PUBLIC on all interfaces: " $1}' ps.sample
PUBLIC on all interfaces: web
PUBLIC on all interfaces: admin
$ awk '$2 ~ /^127\.0\.0\.1/ {print "loopback only: " $1}' ps.sample
loopback only: cache
$ docker ps --format '{{.Names}} {{.Ports}}' | grep -E '0\.0\.0\.0|:::'               # the real command
$ docker ps -q | xargs -r docker inspect --format '{{.Name}} user={{if .Config.User}}{{.Config.User}}{{else}}ROOT{{end}} privileged={{.HostConfig.Privileged}}'

$ docker build --progress=plain --no-cache -t myorg/web:test .      # full output of every step, ignoring the cache
$ docker history --no-trunc myorg/web:1.4 | head -20                 # the layers and the commands that made them (and which one is huge)
$ docker build --platform linux/amd64 -t myorg/web:amd64 .           # build for a specific architecture
$ docker buildx du; docker buildx prune                              # build cache usage and cleaning
  COPY failed: file not found in build context       the file is outside the folder you passed (the "."), or is excluded by .dockerignore
  Build is slow every time                           put the lines that change rarely (dependency install) BEFORE the lines that change often (COPY . .): see the Docker sheet
  apt-get: Unable to locate package                   run apt-get update in the SAME RUN as apt-get install (a cached update layer goes stale)
  Image is huge                                       docker history to find the layer; clean caches in the same RUN (rm -rf /var/lib/apt/lists/*), use a multi-stage build, a slim or distroless base
  Works in the build, fails at run time               different user or environment at run time: check USER, WORKDIR, ENV, and that the files are readable by that user
  Behind a proxy                                       --build-arg HTTP_PROXY=... --build-arg HTTPS_PROXY=...  (and for the daemon: a systemd drop-in for docker.service)

$ systemctl status docker --no-pager; journalctl -u docker -n 50 --no-pager      # the daemon's own log: why did it not start, why did it kill a container?
$ docker info | grep -E 'Server Version|Storage Driver|Cgroup|Docker Root Dir|Rootless|Live Restore'
$ docker version                                       # client and server versions (a "version mismatch" or "Cannot connect" shows here)
$ sudo dockerd --validate --config-file=/etc/docker/daemon.json     # check daemon.json BEFORE restarting (a typo stops the daemon from starting)
  permission denied while trying to connect to the Docker daemon socket     your user is not in the docker group, or you have not logged in again since being added. Note: the docker group is ROOT-EQUIVALENT (a member can mount / into a container): treat it like sudo. Alternatives: sudo, or rootless Docker/podman.
  Cannot connect to the Docker daemon at unix:///var/run/docker.sock       the daemon is not running (systemctl status docker), or DOCKER_HOST points somewhere else (echo $DOCKER_HOST; docker context ls)
  Containers vanish after a daemon restart          no restart policy and live-restore off: add restart: unless-stopped (Compose) or --restart unless-stopped; "live-restore": true in daemon.json keeps containers running while the daemon restarts
  Docker ate the disk (/var/lib/docker)             see the disk section; consider moving "data-root" to a bigger volume
  Strange network problems after a Docker upgrade   restart docker and re-create the networks; check iptables backend (iptables --version shows nf_tables or legacy) and conflicting firewall managers
After a host reboot, containers come back only if they have a restart policy: restart: unless-stopped for services, none (or on-failure) for jobs.

$ docker ps -q | xargs -r docker inspect --format '{{.Name}} {{.HostConfig.CapAdd}} {{.HostConfig.SecurityOpt}} readonly={{.HostConfig.ReadonlyRootfs}}'
$ docker ps -q | xargs -r docker inspect --format '{{.Name}}{{range .Mounts}} {{.Source}}{{end}}' | grep -E 'docker.sock'      # who can control Docker? anything with the socket mounted can take over the host
Run containers as a non-root USER, with --read-only, --cap-drop ALL (add back only what is needed), --security-opt no-new-privileges, pinned image versions (not :latest), and ports published on loopback when a reverse proxy fronts them. The Container and Kubernetes Hardening guide has the full list.

Symptom                                          Most likely cause                                     First command
Exited (1) at once                               application error or missing config                  docker logs NAME
Exited (137), OOMKilled true                     memory limit too low, or a leak                      docker inspect --format '{{.State.OOMKilled}}' NAME
Restarting (1) ...                               crash loop                                           docker update --restart=no NAME, then docker logs
Up (unhealthy)                                   healthcheck fails: wrong URL/port, slow start        docker inspect --format '{{json .State.Health}}' NAME | jq
Cannot reach it from the host                    not published, wrong bind (127.0.0.1 inside), firewall   docker port NAME; docker exec NAME ss -tlnp
db: could not translate host name                not on the same user-defined network, or wrong service name   docker network inspect; docker exec NAME getent hosts db
"port is already allocated"                      another container or process holds the port         sudo ss -tlnp | grep :PORT; docker ps --filter publish=PORT
Disk full on the host                            images, build cache, volumes, container logs         docker system df; du -sh /var/lib/docker/containers/*/*-json.log
Slow container, host idle                        CPU or memory limits, throttling                      docker stats; cgroup cpu.stat
Works on my laptop, not on the server            architecture, env vars, file permissions, network ranges   docker compose config; docker image inspect --format '{{.Architecture}}'
Container data gone after update                 no volume at the data path                           docker inspect --format '{{json .Mounts}}' NAME
exec format error                                wrong CPU architecture                                docker image inspect IMAGE --format '{{.Architecture}}'
Image will not pull                              login, rate limit, wrong tag, DNS/proxy              docker pull -q IMAGE; docker logout/login; check DNS from the host

Read the status, the logs and the exit code, in that order, before you change anything. The answer is usually in the first twenty lines.
Stop the restart loop (docker update --restart=no) so the evidence stays put while you look.
docker compose config shows what Compose really understood: variables, defaults, the final file.
Debug from inside: a shell in the IMAGE (--entrypoint sh), nsenter for the container's network, a toolbox container attached to it.
Keep data in named volumes, put log limits in daemon.json, and prune in order of safety, volumes last.
A published port is public unless you say 127.0.0.1: and the host firewall will not save you.
Containers are cattle, but their volumes are pets: back up the volumes and test the restore.
"Restart it" is not a diagnosis. It is a coin flip that deletes the evidence.