Docker Troubleshooting

When a container misbehaves, a systematic approach beats guessing. Almost every Docker problem can be diagnosed with a handful of commands: check the status and exit code, read the logs, inspect the configuration, and get a shell inside. This lesson walks through that workflow and the most frequent failure patterns, including the exact commands to use.

A Step-by-Step Workflow

# 1. What is the state of the container?
docker ps -a --format 'table {{.Names}}\t{{.Status}}\t{{.Ports}}'

# 2. What did the application say?
docker logs --tail 100 myapp
docker logs -f --since 5m myapp

# 3. How is it configured and why did it stop?
docker inspect myapp --format '{{.State.Status}} exit={{.State.ExitCode}} oom={{.State.OOMKilled}} error={{.State.Error}}'

# 4. Look inside
docker exec -it myapp sh

# 5. Check resource usage and events
docker stats --no-stream myapp
docker events --since 10m

Understanding Exit Codes

Exit code	Typical meaning
0	Process finished successfully (normal for one-off commands)
1	Application error; check the logs
125	Docker daemon error: the docker run command itself failed (bad flag, invalid option)
126	Command found but not executable (permissions)
127	Command not found (wrong CMD/ENTRYPOINT or missing binary)
137	Killed with SIGKILL (128 + 9): out-of-memory kill or docker kill
139	Segmentation fault (128 + 11)
143	Terminated by SIGTERM (128 + 15), usually a normal docker stop

Exit code 137 together with OOMKilled=true in docker inspect means the container exceeded its memory limit.

Container Exits Immediately

A container lives only while its main process runs. Common causes: the command finishes, crashes, or starts a background daemon and returns.

docker ps -a                       # note the Exited (N) status
docker logs myapp                  # find the error message

# Start the image with a shell to investigate (ignores the normal ENTRYPOINT)
docker run -it --rm --entrypoint sh my-app:1.0

# Run the failing command manually to see the error
docker run --rm my-app:1.0 ls -l /app

Run services in the foreground (nginx -g 'daemon off;') so the process stays alive as PID 1.

Container Restarts in a Loop

With --restart always or unless-stopped, a crashing app restarts repeatedly.

docker ps                          # STATUS shows "Restarting (1) 5 seconds ago"
docker logs --tail 50 myapp
docker inspect myapp --format '{{.RestartCount}}'
docker update --restart=no myapp   # stop the loop while you investigate

Out-of-Memory Kills

docker inspect myapp --format '{{.State.OOMKilled}}'
# true
docker stats --no-stream

Fix by raising the memory limit, reducing the application's usage, or setting JVM flags like -XX:MaxRAMPercentage so the heap fits inside the limit.

Port Already in Use

Bind for 0.0.0.0:8080 failed: port is already allocated

docker ps --format '{{.Names}} {{.Ports}}' | grep 8080   # another container?
sudo lsof -i :8080                                        # or a host process (Linux/macOS)
docker run -p 8081:8080 my-app:1.0                        # use a different host port

Cannot Connect to Another Container

docker network ls
docker network inspect appnet                   # are both containers listed?
docker exec -it api sh -c "getent hosts db"    # does the name resolve?
docker exec -it api sh -c "nc -zv db 5432"     # is the port reachable? (if nc exists)

Typical causes: containers on different networks, using localhost instead of the service name, using the published host port instead of the container port, or the target not being ready yet. See the Docker Networking lesson.

Debugging Minimal Images Without a Shell

Distroless or scratch images have no shell. Options:

# Share the target's namespaces from a debugging container
docker run -it --rm --pid=container:myapp --network=container:myapp \
  busybox sh

# Copy files out to inspect them
docker cp myapp:/app/config.yml .

# Docker Desktop includes a debug helper
docker debug myapp

docker debug is a newer Docker CLI feature whose availability depends on your Docker version and subscription, so rely on the namespace-sharing approach above when it is not available.

Build Problems

docker build --progress=plain --no-cache -t my-app:test .   # full, uncached, verbose output
docker build --target build -t my-app:build .               # build only an intermediate stage
docker history my-app:test                                  # which layer is large

Frequent causes:

• COPY failed: file not found in build context: the path is outside the context or excluded by .dockerignore.
• Stale cache: a RUN apt-get update layer cached from long ago; rebuild with --no-cache.
• Wrong architecture: building on arm64 for amd64 servers; use --platform linux/amd64.

Disk Space Issues

docker system df
# TYPE            TOTAL   ACTIVE   SIZE      RECLAIMABLE
# Images          42      5        18.3GB    15.1GB (82%)
# Containers      12      3        1.2GB     900MB (75%)
# Local Volumes   8       4        5.0GB     2.1GB (42%)
# Build Cache     96      0        6.4GB     6.4GB

docker container prune
docker image prune -a
docker builder prune

Large container logs are another culprit. Configure log rotation in the daemon or per container:

docker run -d --log-opt max-size=10m --log-opt max-file=3 my-app:1.0

Permission Denied Errors

• permission denied while trying to connect to the Docker daemon socket: your user is not in the docker group (Linux), or the daemon is not running.
• Permission denied writing to a mounted volume: the container user's UID does not match the owner of the host directory. Fix ownership or run with a matching --user.

Common Mistakes

• Restarting or recreating the container before reading its logs. Evidence is lost when you remove the container.
• Debugging with docker attach. Use docker logs and docker exec instead.
• Assuming "Up" means healthy. Add a HEALTHCHECK and read docker inspect health status.
• Ignoring exit codes. 127, 137, and 143 point to very different root causes.
• Never pruning. Build servers silently fill up with images and cache.

Best Practices

• Log to stdout and stderr so docker logs works, and keep logs rotated.
• Add health checks and meaningful startup error messages.
• Keep a debug workflow documented: status, logs, inspect, exec, stats.
• Reproduce problems with docker run --rm -it --entrypoint sh <image> before changing the Dockerfile.
• Set memory limits and monitor docker stats to catch OOM issues early.

Interview Questions

1. How do you debug a container that keeps crashing?

Check docker ps -a for the status and exit code, read docker logs, and use docker inspect for ExitCode, OOMKilled, and Error. Then reproduce by running the image with --entrypoint sh and executing the start command manually.

2. What does exit code 137 mean?

The process received SIGKILL (128 + 9). Most often the container was OOM-killed because it exceeded its memory limit, which docker inspect confirms with OOMKilled=true. It can also result from docker kill or a timed-out docker stop.

3. A container starts and exits right away. Why?

The main process finished or failed, or it daemonized into the background. Look at the logs, check that CMD/ENTRYPOINT is correct, and run the service in the foreground.

4. Why can't my app container connect to the database container?

Check that both are on the same user-defined network, that the app uses the container or service name rather than localhost, that it uses the container port, and that the database is ready (healthcheck plus retry logic).

5. How do you find out why Docker is using so much disk space?

Run docker system df (and -v for details). Remove unused containers, images, volumes, and build cache with the prune commands, and check for large container log files and add log rotation.

6. How can you inspect a container that has no shell?

Use a separate debug container that joins its namespaces (--pid=container:<name> --network=container:<name>), copy files out with docker cp, or use docker debug where available. Alternatively, build a debug variant of the image.

Summary

• Follow the workflow: status and exit code, logs, inspect, exec, stats.
• Exit codes are clues: 127 means command not found, 137 usually OOM, 143 is SIGTERM.
• Most network issues come from wrong networks, localhost, or readiness.
• Watch disk usage and log growth, and prune regularly.

Continue Learning

Previous: Docker Registry | Next: Docker Production Best Practices