🐳 Docker Incident Runbooks

Docker incident runbooks: restart loops, hangs, OOM kills, permissions and health checks
./docker-triage.sh CONTAINER

When a container misbehaves at night, you do not want a reference manual, you want a checklist: what do I see, what do I run first, what is the usual cause, how do I fix it, how do I know it is fixed, and how do I stop it from happening again. This page is a set of runbooks written that way, each one for an incident that really happens, each one reproduced on a real Docker engine so the commands, the messages and the numbers are real. It starts with a script that collects the evidence in one go (always do that BEFORE you restart anything), then one runbook per symptom.
The Docker Troubleshooting Playbook page is the reference (the five-command survey, Compose file problems, networks, disk space, image builds, the daemon, hardening) and the Disk Full runbook covers the host's disk. This page does not repeat them; it covers the incidents where the question is "why is THIS container behaving like that, right now?". The lab ran Docker Engine 29.8.2 on Ubuntu 24.04, using only images that were already on the machine (alpine:3.21, nginx:alpine and python:3.13-slim, run with --pull never so nothing was downloaded). Every lab container was named lab-rb-something and removed by its exact name afterwards; the machine's own twelve containers were checked before and after (a fingerprint of the container names was identical), so nothing else was touched. Not run: Compose-specific behaviour (depends_on conditions and so on), Kubernetes (where the same container problems appear as CrashLoopBackOff and OOMKilled in pod status), a stopped daemon, and anything that needs root on the host (kernel logs).

  1. CAPTURE BEFORE YOU CHANGE. A container that crashed still holds its exit code, its last log lines and its state (docker inspect, docker logs) until you remove it. docker rm, docker compose down and a new docker run all destroy that evidence. Run the evidence script first and save its output.
  2. Look at the CONTAINER, then the HOST. docker inspect and docker logs tell you what Docker knows; the host's disk, memory and ports (df, free, ss) explain a surprising number of container problems.
  3. Stop a restart loop before you diagnose it, without deleting anything: docker update --restart=no NAME (runbook 4), so the logs stop scrolling and the load stops.
  4. Change ONE thing at a time, and write down what you changed. A container is cheap to recreate and hard to reason about after five edits.
  5. Fix the CAUSE in the image, the Compose file or the host, not in the running container: a docker exec fix disappears when the container is recreated.
  6. Remember what you used: docker run flags that are not in a Compose file or a script are lost the next time somebody recreates the container.

This script collects what an incident needs, in one read-only pass: the image and command, the state (including exit code, whether the kernel's OOM killer fired and how many times it restarted), the health status and its last results, the memory limit and mounts, the published ports and networks, the last log lines, a resource snapshot if it is running, and the events of the last 15 minutes. It only runs docker inspect, logs, stats and events, so it is safe to run on a production host. It needs docker and jq.
$ cat docker-triage.sh
#!/usr/bin/env bash
# docker-triage.sh CONTAINER [LOG_LINES]: collect the evidence for a container incident in one report
# Read-only: it only runs docker inspect / logs / stats / events. Save the output before you restart anything.
set -uo pipefail
c=${1:?usage: docker-triage.sh CONTAINER [LOG_LINES]}; n=${2:-25}
docker inspect "$c" >/dev/null 2>&1 || { echo "no such container: $c" >&2; exit 2; }
echo "=== $c at $(date '+%F %T')"
docker inspect "$c" | jq -r '.[0] | [
  "image:       \(.Config.Image)",
  "command:     \(.Path) \(.Args | join(" "))",
  "state:       \(.State.Status)  exit=\(.State.ExitCode)  oom-killed=\(.State.OOMKilled)  restarts=\(.RestartCount)",
  "started:     \(.State.StartedAt)",
  "finished:    \(.State.FinishedAt)",
  "error:       \(if .State.Error == "" then "-" else .State.Error end)",
  "health:      \(.State.Health.Status // "no healthcheck")",
  "restart:     \(.HostConfig.RestartPolicy.Name)   user: \(if .Config.User == "" then "root" else .Config.User end)",
  "memory:      limit=\(.HostConfig.Memory)  (0 = unlimited)   read-only rootfs: \(.HostConfig.ReadonlyRootfs)",
  "ports:       \((.NetworkSettings.Ports // {}) | to_entries | map("\(.key)->\((.value // []) | map(.HostPort) | unique | join(","))") | join(" "))",
  "networks:    \(.NetworkSettings.Networks | keys | join(", "))",
  "mounts:      \(.Mounts | map("\(.Source):\(.Destination)(\(if .RW then "rw" else "ro" end))") | join(" "))"
] | .[]'
echo "--- last $n log lines"; docker logs --tail "$n" "$c" 2>&1 | cut -c1-200
if [[ $(docker inspect -f '{{if .State.Health}}yes{{end}}' "$c") == yes ]]; then
  echo "--- last health check results"; docker inspect "$c" | jq -r '.[0].State.Health.Log[-3:][] | "\(.Start[0:19]) exit=\(.ExitCode) \(.Output | gsub("\n";" ") | .[0:80])"'
fi
if [[ $(docker inspect -f '{{.State.Running}}' "$c") == true ]]; then
  echo "--- resource snapshot"; docker stats --no-stream --format 'cpu={{.CPUPerc}} mem={{.MemUsage}} ({{.MemPerc}}) pids={{.PIDs}} net={{.NetIO}}' "$c"
fi
echo "--- events in the last 15 minutes (exec_* noise from health checks left out)"
docker events --since 15m --until 0s --filter "container=$c" --format '{{.Time}} {{.Action}} {{index .Actor.Attributes "exitCode"}}' 2>/dev/null | awk '$2 !~ /^exec_/ { cmd="date -d @"$1" +%T"; cmd | getline t; close(cmd); printf "%s %s %s\n", t, $2, $3 }' | tail -n 12
It was run against three genuinely broken containers (the lab started each one with the fault built in, waited a few seconds, then ran the script). The output below is the real output, with log lines trimmed to four.
A container killed for using too much memory (limit 32 MB, the program tries to allocate 200 MB):
$ ./docker-triage.sh lab-rb-t1 4
=== lab-rb-t1 at 2026-10-02 18:57:55
image:       python:3.13-slim
command:     python -u -c print("loading the dataset...", flush=True); x = bytearray(200*1024*1024)
state:       exited  exit=137  oom-killed=true  restarts=0
started:     2026-10-02T22:57:46.861881871Z
finished:    2026-10-02T22:57:47.029834393Z
error:       -
health:      no healthcheck
restart:     no   user: root
memory:      limit=33554432  (0 = unlimited)   read-only rootfs: false
ports:
networks:    bridge
mounts:      /tmp:/scratch(ro)
--- last 4 log lines
loading the dataset...
--- events in the last 15 minutes (start, die, oom, health_status, restart, kill)
18:57:46 create
18:57:46 attach
18:57:46 start
18:57:47 oom
18:57:47 die 137
One glance says it all: exit 137, oom-killed=true, a limit of 33,554,432 bytes, and in the event list "oom" just before "die 137": the kernel killed it a second after it started because it asked for more memory than its limit. The log ends at "loading the dataset...", which tells you where the program was.
A container that runs but fails its health check (nginx whose check asks for a page that does not exist):
$ ./docker-triage.sh lab-rb-t2 4
state:       running  exit=0  oom-killed=false  restarts=0
health:      unhealthy
ports:       80/tcp->18391
--- last health check results
2026-10-02T18:57:51 exit=1 wget: server returned error: HTTP/1.1 404 Not Found
2026-10-02T18:57:53 exit=1 wget: server returned error: HTTP/1.1 404 Not Found
2026-10-02T18:57:55 exit=1 wget: server returned error: HTTP/1.1 404 Not Found
--- events in the last 15 minutes (exec_* noise from health checks left out)
18:57:47 start
18:57:51 health_status: unhealthy
Running, with exit=0 (it has not exited), restarts=0, and the reason right there: the health command gets "404 Not Found". The nginx log lines (not shown) say the same from the server's side. (The lines are condensed here: the script also prints the image, command, mounts and a resource snapshot.)
A container in a restart loop (it prints a fatal error and exits with 78):
$ ./docker-triage.sh lab-rb-t3 4
state:       restarting  exit=78  oom-killed=false  restarts=7
restart:     on-failure   user: root
--- last 4 log lines
FATAL: cannot read /etc/app/config.yml
FATAL: cannot read /etc/app/config.yml
FATAL: cannot read /etc/app/config.yml
FATAL: cannot read /etc/app/config.yml
--- events in the last 15 minutes
18:57:47 start
18:57:47 die 78
18:57:48 start
18:57:48 die 78
18:57:48 start
18:57:48 die 78
18:57:49 start
18:57:49 die 78
18:57:51 start
18:57:51 die 78
18:57:54 start
18:57:55 die 78
State "restarting", seven restarts so far, the same exit code each time (78), the error message repeated in the log, and the events show the gaps between attempts growing: that is the restart policy's back-off (next runbook).
Save the report with ./docker-triage.sh NAME > incident-NAME-$(date +%F-%H%M).txt before you touch the container.

The exit status of the container's main process is the first clue. These were run for real:
$ docker run --rm alpine:3.21 true; echo $?
0
$ docker run --rm alpine:3.21 sh -c 'exit 3'; echo $?
3
$ docker run --rm alpine:3.21 nosuchcmd; echo $?
docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: ...
127
$ docker run --rm alpine:3.21 /etc/passwd; echo $?
docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: ...
126
  0           the process ended normally
  1 to 125    the application's own error code (3 above; 78 in the restart loop is the "configuration error" convention). Read the application's logs, not Docker's
  125         docker itself failed (a bad flag, the daemon is down)
  126         the command was found but cannot be executed (the lab ran /etc/passwd: not an executable; also a script without the execute bit, or without a valid interpreter line)
  127         the command was not found in the image (a typo, a missing binary, a wrong PATH, or a script whose first line points to an interpreter that does not exist)
  137         killed by SIGKILL (128+9): out of memory (the kernel's OOM killer), or docker stop's last resort after the grace period, or docker kill (runbooks 5 and 6)
  139         segmentation fault (128+11): a crash in the program or a library
  143         ended by SIGTERM (128+15): a normal stop, shown when the program does not handle the signal itself (runbook 5)
For 126 and 127 Docker prints "OCI runtime create failed" and the container never ran, so docker logs is empty: the error is in the docker run output (or in docker inspect, State.Error).

Symptom: docker ps shows "Restarting (N) x seconds ago", the restart count rises, logs repeat the same few lines.
Triage:
$ docker inspect -f 'status={{.State.Status}} restarts={{.RestartCount}} exit={{.State.ExitCode}} restarting={{.State.Restarting}}' lab-rb-loop
$ docker ps -a --filter name=lab-rb-loop --format "{{.Names}} | {{.Status}}"
$ docker logs lab-rb-loop | sort | uniq -c
The lab started a container with --restart=on-failure whose command prints an error and exits with status 3, and watched it every two seconds:
t+2s   status=restarting restarts=4 exit=3 restarting=true
t+4s   status=restarting restarts=6 exit=3 restarting=true
t+6s   status=restarting restarts=6 exit=3 restarting=true
t+8s   status=restarting restarts=7 exit=3 restarting=true
t+10s  status=restarting restarts=7 exit=3 restarting=true
$ docker ps -a --filter name=lab-rb-loop --format "{{.Names}} | {{.Status}}"
lab-rb-loop | Restarting (3) 2 seconds ago
$ docker logs lab-rb-loop 2>&1 | sort | uniq -c
      7 starting, then crashing: cannot connect to db
The count went up fast at first (4 restarts in 2 seconds) and then slowed down: Docker waits longer between attempts (by its documentation the delay starts at 100 ms and doubles up to a limit; the lab did not measure the exact values, only the growing gaps). That back-off protects the host, and it also hides the problem: a loop that retries every few minutes looks like an occasional glitch. docker ps shows the exit code in brackets ("Restarting (3)"). The events show the same rhythm, with the gaps growing:
$ docker events --since 30s --until 0s --filter container=lab-rb-loop --filter event=start --filter event=die
18:54:48 start        18:54:49 start        18:54:53 start
18:54:48 die exitCode=3   18:54:49 die exitCode=3   18:54:53 die exitCode=3
(condensed). Stop the loop without deleting the evidence:
$ docker update --restart=no lab-rb-loop2
$ docker inspect -f 'status={{.State.Status}} exit={{.State.ExitCode}} restarts={{.RestartCount}}' lab-rb-loop2     (checked every 3 seconds)
update+3s:  status=exited exit=3 restarts=6
update+6s:  status=exited exit=3 restarts=6
update+9s:  status=exited exit=3 restarts=6
update+12s: status=exited exit=3 restarts=6
Within three seconds the container was "exited" and the restart count froze at 6. (In the very first run the status still read "restarting" three seconds after the update, because Docker was in the middle of waiting between attempts; if you see that, check again after a few seconds.) Now the logs and the container's file system are still there, and you can start it by hand with a shell instead of its real command: docker run --rm -it --entrypoint sh IMAGE (the Docker Troubleshooting Playbook shows the variations).
Usual causes, in the order to check: a missing or wrong configuration file or environment variable (the exit code is often 1 or a code the application defines, and the log says so), a dependency that is not ready (the database is not up yet: depends_on in Compose waits for the container to START, not to be READY: use a health check and condition: service_healthy, or make the application retry), a port or file already in use, a permission problem (runbook 7), out of memory (runbook 6), a wrong command or entrypoint (126 and 127), or an image built for another CPU architecture ("exec format error"). Fix the cause, then recreate the container with the fix (docker compose up -d --force-recreate SERVICE).
Verify: the restart count stops rising (run the first command twice, a minute apart), the status is "running" and stays so, and, if there is a health check, it turns healthy.
Prevent: a restart policy is for recovery from transient failures, not a substitute for a working configuration. Use unless-stopped or on-failure:5 (a maximum number of attempts) rather than a bare always for services that can be misconfigured, send the logs somewhere that survives container removal, and alert on the restart count (a container with more than a few restarts in an hour needs a person).

Symptom: every docker stop, docker compose down or deployment pauses for exactly ten seconds per container, and the exit code is 137, not 0. Deploys are slow, and the application never gets to flush its data or close its connections.
Cause: docker stop sends SIGTERM and waits ten seconds (the grace period), then sends SIGKILL. If the program ignores SIGTERM, it is killed after the wait. The usual reason is that the program is PID 1 inside the container: the Linux kernel gives PID 1 NO default signal handlers, so a process that has not installed its own SIGTERM handler simply ignores the signal. A shell script as the entrypoint is the classic case, and so is the "shell form" of CMD (CMD myapp --flag, which runs sh -c "myapp --flag" and leaves sh as PID 1 so the application never receives the signal). The lab timed four variants with the same container:
A. a shell loop as PID 1, no signal handling (docker run -d alpine sh -c 'while true; do sleep 1; done'):
   docker stop took 10.2 s, exit code 137
B. the same command with --init (a tiny init program, docker-init, becomes PID 1 and forwards signals):
   docker stop took 0.2 s, exit code 143
C. the script handles SIGTERM itself (sh -c 'trap "echo got TERM, cleaning up; exit 0" TERM; while true; do sleep 1 & wait $!; done'):
   docker stop took 0.1 s, exit code 0
D. variant A with a shorter grace period (docker stop -t 2):
   docker stop -t 2 took 2.2 s, exit code 137
Variant A is the problem: ten full seconds and 137. Variant B shows the cheapest fix: --init (in Compose, init: true) made the stop instant, with exit code 143 (docker-init forwarded SIGTERM to the shell; because the shell is no longer PID 1, the signal's default action applies and it ends, reported as 128+15). Variant C is the proper fix when you own the script: a trap that cleans up and exits 0. Variant D only shortens the wait; the program was still killed. The PID 1 in each case, from the lab:
$ docker run --rm alpine:3.21 sh -c 'ps -o pid,comm | head -3'
PID   COMMAND
    1 sh
    7 ps
$ docker run --rm --init alpine:3.21 sh -c 'ps -o pid,args | head -3'
PID   COMMAND
    1 /sbin/docker-init -- sh -c ps -o pid,args | head -3
    7 sh -c ps -o pid,args | head -3
Triage: docker exec NAME cat /proc/1/comm shows what PID 1 is (a shell, a wrapper script, or the real application), and docker inspect -f '{{.Config.Cmd}} {{.Config.Entrypoint}}' NAME how it was started.
Fix, in order of preference: use the EXEC form in the Dockerfile (CMD ["myapp", "--flag"], ENTRYPOINT ["myapp"]) so the application is PID 1 and gets the signal; in a wrapper script end with exec myapp "$@" so the shell replaces itself with the application; make sure the application handles SIGTERM (graceful shutdown: finish requests, flush, exit 0); add a trap to shell entrypoints; or use --init / init: true when you cannot change the program. Raise the grace period only for programs that really need time (docker stop -t 60, or stop_grace_period: 60s in Compose), after the signal is delivered.
Verify: time docker stop NAME takes a fraction of a second (or the time the application really needs to shut down) and the exit code is 0 or 143, not 137.

Symptom: the container disappears or restarts; exit code 137; docker inspect says OOMKilled=true; the application log just stops, often mid-line, with no error at all.
Triage and reproduction: a container with a 32 MB limit running a program that allocates 200 MB:
$ docker run --name lab-rb-oom --memory 32m --memory-swap 32m python:3.13-slim python -c 'print("allocating 200 MB in a 32 MB container..."); x = bytearray(200*1024*1024); print("allocated")'
(no output at all)
[docker run exit: 137]
$ docker inspect -f 'exit={{.State.ExitCode}} oomkilled={{.State.OOMKilled}} memory-limit={{.HostConfig.Memory}} status={{.State.Status}}' lab-rb-oom
exit=137 oomkilled=true memory-limit=33554432 status=exited
$ docker events --since 60s --until 0s --filter container=lab-rb-oom --filter event=oom
oom lab-rb-oom
Three things to notice. The program printed NOTHING, not even its first message: the output was in the process's buffer when the kernel killed it, and a SIGKILL gives no chance to flush it. That is why OOM kills look like "the log just stops": run Python with -u (as the triage example above did), flush explicitly, or log to stderr. Second, the truth is in two places: docker inspect (OOMKilled=true, exit 137) and the event stream (an "oom" event right before "die"). Third, the same program with a bigger limit works:
$ docker run --rm --memory 400m --memory-swap 400m python:3.13-slim python -c 'x = bytearray(200*1024*1024); print("allocated 200 MB fine")'
allocated 200 MB fine
Exit 137 WITHOUT OOMKilled=true means something else sent SIGKILL: docker stop's timeout (previous runbook), docker kill, an orchestrator, or a person. Exit 137 with OOMKilled=true means the container exceeded its limit. A different case: the HOST ran out of memory and the kernel killed a process of a container with no limit; the host's kernel log (sudo dmesg -T | grep -iE 'out of memory|killed process', needs root, not run here) names the victim and the amounts. Also check which limit is meant: --memory limits the container's memory (and the page cache counts towards it); --memory-swap equal to --memory means no swap.
Fix: raise the limit if the program legitimately needs more (look at its real use first: docker stats --no-stream, the memory peak in the application's metrics), reduce what the program uses (a smaller batch size, streaming instead of loading everything, a leak fix), or add swap on the host as a stopgap. Set the limit on purpose, not by guessing: about 25 percent above the observed peak for the busiest realistic day.
Verify: OOMKilled=false after a normal busy period, no "oom" events (docker events --filter event=oom --since 24h), and the memory use stays flat (docker stats). Prevent: set memory limits and reservations on every service (so one container cannot starve the host), alert on OOM events and on memory use above 80 percent of the limit, and test the real peak load.

Symptom: the application cannot write (or read) a bind-mounted folder or a volume: "Permission denied", "EACCES", "unable to create directory"; it works as root and fails as a normal user.
Cause: a bind mount shows the host's files with their real numeric user and group ids; the container's user is matched by NUMBER, not by name. A folder owned by uid 1000 on the host is just "uid 1000" inside, and a container running as uid 1001 is a stranger to it. The lab made a folder owned by the lab's own uid (1000, mode 755) and tried three container users:
$ stat -c '%u:%g %a' data
1000:1000 755
$ docker run --rm -v "$PWD/data:/data" alpine:3.21 sh -c 'id -u; touch /data/by-root && echo "root: wrote a file"'
0
root: wrote a file
$ docker run --rm -u 1001:1001 -v "$PWD/data:/data" alpine:3.21 sh -c 'id -u; touch /data/by-1001 || echo "exit code $?"'
1001
touch: /data/by-1001: Permission denied
exit code 1
$ ls -ln data | awk 'NR>1{print $3":"$4, $9}'
0:0 by-root
Root could write (it ignores the permissions) and left a file owned by uid 0 on the HOST; the user 1001 got "Permission denied". Two fixes, both tested:
$ docker run --rm -u "$(id -u):$(id -g)" -v "$PWD/data:/data" alpine:3.21 sh -c 'touch /data/by-owner && echo "uid $(id -u): wrote a file"'
uid 1000: wrote a file
$ docker run --rm -v "$PWD/data:/data" alpine:3.21 chown 1001:1001 /data; stat -c '%u:%g' data
1001:1001
$ docker run --rm -u 1001:1001 -v "$PWD/data:/data" alpine:3.21 sh -c 'touch /data/by-1001 && echo "uid 1001: wrote a file"'
uid 1001: wrote a file
The first runs the container as the folder's owner (-u in docker run, user: in Compose); the second changes the owner on the host so it matches the user the image uses (chown 1001:1001, or give a shared group and mode 775). A side effect shows the cost: after the chown the folder was no longer owned by the lab user, so deleting the test files from the host failed with "Permission denied" and had to be done from inside a container (docker run --rm -v ...:/data alpine rm -f /data/by-1001 /data/by-owner). Files created by a root container on a bind mount are root-owned on the host too: clean up the same way.
Triage: docker inspect -f '{{.Config.User}}' NAME (empty means root), docker exec NAME id, ls -ln on the host folder (numeric ids, as the container sees them), and for a named volume docker run --rm -v VOLUME:/v alpine ls -ln /v. SELinux hosts add a second cause: a bind mount needs the :z or :Z suffix (-v /host:/data:Z) or the container is denied although the owner matches (not run on the Ubuntu lab machine). Rootless Docker and user namespaces remap the ids; then the numbers differ from what ls shows (check the docs for your setup).
Prevent: decide the user explicitly (USER in the Dockerfile with a fixed uid, user: in Compose), create and chown the data folders with the same uid, prefer named volumes (Docker creates them with the right owner when the image has the folder) over bind mounts for application data, and never run application containers as root without a reason.

Symptom: docker ps shows "(unhealthy)", a load balancer or orchestrator stops sending traffic, but the container is still running and the application seems fine.
How Docker health works: the HEALTHCHECK command (or --health-cmd) runs inside the container at an interval; after the configured number of consecutive failures the status becomes unhealthy, and one success makes it healthy again. The lab created a container whose check looks for a file that does not exist:
$ docker run -d --name lab-rb-health --health-cmd 'test -f /tmp/ready' --health-interval 2s --health-timeout 1s --health-retries 2 --health-start-period 3s alpine:3.21 sleep 600
t+3s   starting (failing streak 1)
t+6s   unhealthy (failing streak 2)
t+9s   unhealthy (failing streak 4)
t+12s  unhealthy (failing streak 5)
$ docker ps --filter name=lab-rb-health --format "{{.Names}} | {{.Status}}"
lab-rb-health | Up 12 seconds (unhealthy)
The status moved from "starting" to "unhealthy" after two failures in a row, and the failing streak kept counting. The reason is in the health log, which holds the last five results (command output and exit code):
$ docker inspect lab-rb-health | jq -c '.[0].State.Health.Log[-1] | {ExitCode, Output, Start: (.Start|.[0:19])}'
{"ExitCode":1,"Output":"","Start":"2026-10-02T18:56:30"}
(The jq page explains the filters.) Here the output is empty because "test -f" prints nothing: when you write a health command, make it print WHY it failed (curl -fsS ... shows the HTTP error; the earlier triage example's wget printed "404 Not Found"). Then the important facts:
$ docker inspect -f 'status={{.State.Status}} restarts={{.RestartCount}}' lab-rb-health
status=running restarts=0
$ docker exec lab-rb-health touch /tmp/ready; sleep 3; docker inspect -f '{{.State.Health.Status}}' lab-rb-health
healthy
Docker does NOT restart an unhealthy container by itself: it only records the status (a restart policy reacts to the process exiting, not to a failed check). An orchestrator or a helper tool acts on it; plain Docker does not. And once the cause was fixed (the file now exists), the next check passed and the status returned to healthy by itself.
Triage: the health log above, then run the check command by hand inside the container (docker exec NAME sh -c 'THE COMMAND'; echo $?) to see its output, then look at the application's own log at the same time stamps. Typical causes: the check is wrong (a path, a port, a tool that does not exist in the image: curl is not in many slim images, so the check fails with "not found" while the application is perfectly fine), the application is slow to start (the start period is too short: failures during the start period do not turn the status to unhealthy, as the lab saw, where the failing streak counted while the status was still "starting"; so set --health-start-period to the real startup time), the check is too strict or too heavy (a deep database query every two seconds), the timeout is too short for a busy host, or the application really is stuck.
Prevent: a health check should test the thing that matters cheaply (a lightweight endpoint that checks the application and its essential dependencies, not the whole world), use a realistic interval, timeout and start period, print a reason on failure, and be tested the way this runbook did: break it on purpose and watch the status change.

Symptom: docker run or docker compose up fails with "Bind for 0.0.0.0:PORT failed: port is already allocated", or the application starts but another service answers on that port.
Reproduction: one container publishing host port 18390, then a second one asking for the same port:
$ docker run -d --name lab-rb-web1 -p 18390:80 nginx:alpine
$ docker run -d --name lab-rb-web2 -p 18390:80 nginx:alpine
docker: Error response from daemon: failed to set up container networking: driver failed programming external connectivity on endpoint lab-rb-web2 (ef124e29...): Bind for 0.0.0.0:18390 failed: port is already allocated
$ docker ps --filter publish=18390 --format '{{.Names}}  {{.Ports}}'
lab-rb-web1  0.0.0.0:18390->80/tcp, [::]:18390->80/tcp
$ ss -ltnp | grep ':18390 '
0.0.0.0:18390
[::]:18390
$ docker ps -a --filter name=lab-rb-web2 --format '{{.Names}} | {{.Status}}'
lab-rb-web2 | Created
Docker's own view answers "who": docker ps --filter publish=PORT names the container that holds it. The host's view (ss -ltnp, with sudo to see process names; the lab ran it without) shows the listener. If ss shows a listener but docker ps shows no container, something outside Docker holds the port (a service on the host, a stale docker-proxy, another tool): find it with sudo ss -ltnp 'sport = :18390' or sudo lsof -i :18390. Note the leftover: the failed container was created but never started ("Created"), and it keeps its name, so the next attempt fails with "name already in use" until you remove it (docker rm lab-rb-web2).
Fix: choose another host port (-p 18391:80), stop the container or service that holds the port, or publish to one address only (-p 127.0.0.1:18390:80 does not conflict with a service listening on the other addresses). In Compose, check the "ports:" of every service, including the stopped stack you forgot. A published port is a firewall decision too: -p 18390:80 listens on all interfaces; use 127.0.0.1: for anything that should not be reachable from outside, and note that Docker's own firewall rules bypass ufw for published ports (see the Docker Troubleshooting Playbook, hardening section).

Symptom: one container cannot connect to another by its name ("bad address", "could not resolve host", "no such host"), although both are running.
Reproduction: three attempts. On the DEFAULT bridge network, no name resolution:
$ docker run -d --name lab-rb-a nginx:alpine
$ docker run --rm alpine:3.21 sh -c 'wget -q -T 3 -O- http://lab-rb-a/ 2>&1 | head -1'
wget: bad address 'lab-rb-a'
On a USER-DEFINED network, the same works, through Docker's embedded DNS:
$ docker network create lab-rb-net; docker run -d --name lab-rb-b --network lab-rb-net nginx:alpine
$ docker run --rm --network lab-rb-net alpine:3.21 sh -c 'wget -q -T 3 -O- http://lab-rb-b/ | grep -o "<title>.*</title>"; nslookup lab-rb-b | grep -E "^Name|^Address" | tail -2'
<title>Welcome to nginx!</title>
Name:	lab-rb-b
Address: 172.21.0.2
$ docker run --rm --network lab-rb-net alpine:3.21 cat /etc/resolv.conf | grep -v '^#'
nameserver 127.0.0.11
search .
options edns0 trust-ad ndots:0
$ docker run --rm alpine:3.21 cat /etc/resolv.conf | grep -v '^#' | head -2
nameserver <the host's DNS server>
The default bridge (network "bridge") is a legacy network without name resolution: containers there see the host's DNS servers, which know nothing about container names (the second resolv.conf), and can only reach each other by IP address. A network you create yourself has a resolver at 127.0.0.11 that answers for every container on that network (the first resolv.conf). Fix: put the containers on one user-defined network (docker network create NAME and --network NAME, or a Compose project, which creates one automatically: services reach each other by SERVICE name). Check that both are on the SAME network: docker inspect -f '{{json .NetworkSettings.Networks}}' NAME | jq 'keys'. A container can be on several networks; names resolve only between containers that share a network. Use the container's PORT (80), not the host's published port, from container to container.
If names resolve but the connection fails, it is a different problem: the service is not listening on the address or port (check from inside: docker exec NAME ss -ltn, or the application's bind address: 127.0.0.1 inside a container means only that container), or a firewall rule. If NOTHING resolves, even external names, look at the host's DNS and the daemon's DNS settings (the Playbook's network section).

Symptom: the host's /var/lib/docker fills up, "no space left on device", and docker system df does not show where it all went; one container's log file is gigabytes.
Cause: the default json-file logging driver keeps EVERYTHING the container writes to stdout and stderr, without a size limit, until the container is removed. A noisy container can write many gigabytes. The lab started two containers that print 3,000 lines, one with the default and one with a limit of two files of 20 KB:
$ docker run -d --name lab-rb-logA alpine:3.21 sh -c "$LOGGER"
$ docker run -d --name lab-rb-logB --log-opt max-size=20k --log-opt max-file=2 alpine:3.21 sh -c "$LOGGER"
no limit:        3000 lines kept, first: line 0: so
max-size=20k x2: 220 lines kept, first: line 2780: s  <- the oldest lines were rotated away
$ docker inspect -f '{{.HostConfig.LogConfig.Type}} {{json .HostConfig.LogConfig.Config}}' lab-rb-logA lab-rb-logB
json-file {}
json-file {"max-file":"2","max-size":"20k"}
Container A kept all 3,000 lines; container B kept only the newest 220 (the last two files of up to 20 KB), and docker logs shows what survived. Triage on a real host: the log files are root-only (/var/lib/docker/containers/ID/ID-json.log; sudo du -sh /var/lib/docker/containers/*/*-json.log | sort -h | tail finds the large ones, not run here), and docker inspect -f '{{.LogPath}}' NAME gives the path of one. Fix now: truncate the file of a runaway container (sudo truncate -s 0 FILE: the container keeps running and the space comes back immediately; the Disk Full runbook explains why truncating is better than deleting), then fix the cause: find out why it logs so much.
Prevent: set a limit for ALL containers in /etc/docker/daemon.json and restart the daemon:
{"log-driver":"json-file","log-opts":{"max-size":"10m","max-file":"3"}}
This applies only to containers created AFTERWARDS: recreate the existing ones (docker compose up -d --force-recreate). Or per service in Compose (logging: driver: json-file, options: max-size: "10m", max-file: "3"). Or use a driver that ships the logs elsewhere (journald, syslog, a log collector), remembering that docker logs may then not work for some drivers (check the docs of the driver you pick).

Symptom: "Too many open files", EMFILE, "accept4: too many open files" or a server that stops accepting connections after running for a while.
Cause: each process has a limit on open file descriptors (files and network sockets). A container inherits its limit from the Docker daemon (often 1024 soft in these images, and much higher in some setups). The lab showed the limit and what a program sees when it opens files until it fails:
$ docker run --rm python:3.13-slim python -c "$PY"
opened 1021 files, then: [Errno 24] Too many open files: '/dev/null'
$ docker run --rm --ulimit nofile=64:64 python:3.13-slim python -c "$PY"
opened 61 files, then: [Errno 24] Too many open files: '/dev/null'
$ docker run --rm --ulimit nofile=64:64 alpine:3.21 sh -c 'echo "soft $(ulimit -Sn)  hard $(ulimit -Hn)"'
soft 64  hard 64
With the default of 1024 the program managed 1021 files (three descriptors were already in use: standard input, output and error), and with a limit of 64 it opened 61. A busy server (many clients, many log files, many database connections) hits 1024 easily. Triage: docker exec NAME sh -c 'ulimit -n' for the limit, ls /proc/PID/fd | wc -l (inside the container) for the use, and whether the use grows steadily (a descriptor leak: the application never closes files or sockets: fix the code) or jumps with load (the limit is just too low). Fix: raise the limit for the service (--ulimit nofile=65536:65536; in Compose ulimits: nofile: {soft: 65536, hard: 65536}), set a daemon-wide default (default-ulimits in daemon.json) and, for applications that read their own settings, their configuration (worker_connections in nginx, max connections in a database). Raising the limit hides a leak but does not cure it: watch the number over days.

Symptom: after hardening ("read_only: true", docker run --read-only) the application fails with "Read-only file system" when it writes logs, pid files, caches or temporary files.
Reproduction:
$ docker run --rm --read-only alpine:3.21 sh -c 'touch /tmp/x 2>&1; touch /etc/x 2>&1'
touch: /tmp/x: Read-only file system
touch: /etc/x: Read-only file system
$ docker run --rm --read-only --tmpfs /tmp alpine:3.21 sh -c 'touch /tmp/x && echo "/tmp is writable (tmpfs)"; touch /etc/x 2>&1'
/tmp is writable (tmpfs)
touch: /etc/x: Read-only file system
A read-only root file system is a good hardening step (an attacker or a bug cannot change the image's files), and it exposes every place the application writes. Find them: run the container normally and look at docker diff NAME (the files it added, changed and deleted), or run it read-only and read the errors. Then give each real write location a place: a tmpfs for temporary and runtime files (--tmpfs /tmp, --tmpfs /var/run, in Compose tmpfs:), a named volume for data that must survive (/var/lib/app), and leave the rest read-only. Typical needs: /tmp, /var/run or /run (pid files and sockets), the web server's cache and temp folders (/var/cache/nginx, /var/tmp), and the application's own data and log folders. A tmpfs lives in memory and disappears with the container, and it counts towards the memory limit.

Symptom: the container's process list fills with "defunct" or "Z" processes, the process count grows until the container cannot start new ones, or signals (stop, reload) do not reach the application.
Cause: when a process ends, its parent must collect its exit status (reap it); until then it stays as a zombie. If a process's parent dies, its children are adopted by PID 1, which is expected to reap them. In a container PID 1 is whatever the image starts, and most applications and shells do not do this job. The lab started two containers that each spawn five short-lived orphaned children and then sleep, one without and one with --init:
$ docker run -d --name lab-rb-zA alpine:3.21 sh -c 'for i in 1 2 3 4 5; do ( sleep 0.2 & ) ; done; sleep 60'
$ docker run -d --init --name lab-rb-zB alpine:3.21 sh -c 'for i in 1 2 3 4 5; do ( sleep 0.2 & ) ; done; sleep 60'
$ for n in lab-rb-zA lab-rb-zB; do printf '%s: ' $n; docker exec $n sh -c 'ps -o stat= | grep -c "^Z"' | tr '\n' ' '; echo "zombie(s)   PID 1 is: $(docker exec $n sh -c 'cat /proc/1/comm')"; done
lab-rb-zA: 5 zombie(s)   PID 1 is: sleep
lab-rb-zB: 0 zombie(s)   PID 1 is: docker-init
Five zombies in the container without an init, none with it. In the first container PID 1 was "sleep": the shell replaced itself with its last command, so the program that adopted the orphans was sleep, which never reaps anything. With --init, docker-init is PID 1; it reaps the children and also forwards signals (runbook 5). Triage: docker exec NAME ps -o pid,ppid,stat,comm (zombies have state Z and are children of PID 1), docker exec NAME cat /proc/1/comm for what PID 1 is. (docker top did not show the zombies in the lab: use ps inside the container.) A few zombies are harmless; the danger is a program that spawns children constantly (a CGI server, a build tool, a script that calls other commands in a loop), so that the process table fills.
Fix: run with --init (Compose: init: true), or use a container-friendly init in the image (tini, dumb-init: ENTRYPOINT ["/sbin/tini", "--"]), or make the application itself reap its children. Zombies are cleaned by restarting the container, which is a stopgap.

  Restarting (N)                  docker logs NAME; docker inspect -f '{{.State.ExitCode}} {{.State.OOMKilled}} {{.RestartCount}}' NAME; docker update --restart=no NAME
  exited with 137                 docker inspect -f '{{.State.OOMKilled}}' NAME: true = memory limit; false = something sent SIGKILL (a slow stop, docker kill)
  exited with 126 or 127          docker run output: "OCI runtime create failed": the command: path, executable bit, interpreter, CRLF line endings in scripts
  stop takes 10 seconds           docker exec NAME cat /proc/1/comm; exec form CMD, trap SIGTERM, or --init
  (unhealthy)                     docker inspect NAME | jq '.[0].State.Health.Log'; run the check command by hand
  Permission denied               docker exec NAME id; ls -ln on the host folder; -u or chown
  port is already allocated       docker ps --filter publish=PORT; sudo ss -ltnp 'sport = :PORT'
  bad address / no such host      docker inspect -f '{{json .NetworkSettings.Networks}}' NAME: same user-defined network?
  no space left on device         docker system df; sudo du -sh /var/lib/docker/containers/*/*-json.log | sort -h | tail; log limits
  too many open files             docker exec NAME sh -c 'ulimit -n'; --ulimit nofile
  Read-only file system           docker diff NAME; --tmpfs and volumes for the real write paths
  defunct processes               docker exec NAME ps -o pid,ppid,stat,comm; --init
  "it worked yesterday"           docker events --since 24h --filter container=NAME; docker image ls (did the tag move?); compare docker inspect with the Compose file

  Removing the container before reading its state and logs: the evidence is gone, and the next crash starts from zero
  docker system prune -a in the middle of an incident: it deletes stopped containers (with their logs and exit codes), unused images and networks. Read what it will delete (the Playbook's disk section) and never run it as a reflex
  Fixing a running container with docker exec and calling it done: the fix disappears at the next recreate
  "restart: always" on a container that crashes at start: it hides the failure behind a loop and eats CPU; use a limited policy and alerting
  Giving a container more memory without finding out why it needs it: the leak just takes longer to kill it
  Running everything as root and bind-mounting host folders: it works until the first ownership problem, and it is a security risk
  Trusting "running" as "healthy": define a health check and alert on it
  Using the host's published port from another container: use the service name and the container's own port on a shared network
  Editing the Dockerfile CMD in shell form and wondering why signals never arrive: use the exec form

Capture first: ./docker-triage.sh NAME > incident.txt before you restart, remove or recreate anything.
Read the exit code: 137 with OOMKilled=true is memory, 137 without it is a SIGKILL from a slow stop, 126 and 127 never ran, 143 is a clean SIGTERM, anything else is the application's own code.
Stop a loop with docker update --restart=no, not with docker rm.
Make PID 1 count: exec form for CMD and ENTRYPOINT, exec in wrapper scripts, a SIGTERM handler, or --init. It fixes slow stops, lost signals and zombies at once.
Set memory limits on purpose, and flush your output (python -u, line-buffered logs), or an OOM kill leaves no trace.
Put application containers on a user-defined network and refer to each other by service name and container port.
Match the user ids: a fixed uid in the image, the same owner on the volume, and prefer named volumes to bind mounts.
Give every container a health check that prints why it fails, a realistic start period, and an alert on "unhealthy".
Limit the logs (max-size and max-file) for every container, in daemon.json and in Compose.
Test your runbooks the way this page did: break a throwaway container on purpose, run the commands, and write down what you see.