Skip to content
docktionary

Kubernetes

Troubleshooting

Almost every Pod problem shows up as one of a handful of status reasons. Find the one you're looking at, and start from there.

CrashLoopBackOff

The container starts, exits, and Kubernetes restarts it -- with a growing delay between attempts each time it keeps failing. The Pod is running the retry loop correctly; something in the container itself is the actual problem.

Likely causes, roughly in order of how often they're the answer:

  • The app crashes on startup -- bad config, a missing required environment variable, a dependency it can't reach yet.
  • A liveness probe is failing and repeatedly killing an otherwise fine container -- see readiness vs. liveness probes.
  • OOMKilled -- the memory limit is lower than what the app actually needs.
terminal
# The current container just restarted and has no logs yet --
# --previous gets you the logs from the crash itself.
kubectl logs <pod> --previous

kubectl describe pod <pod>   # Last State: shows the exit code and reason

The exit code in describe pod's output means exactly what it would for docker -- see exit codes.

ImagePullBackOff / ErrImagePull

The kubelet can't pull the image at all, so the container never even gets a chance to start.

  • A typo in the image name or tag.
  • A private image with no (or the wrong) imagePullSecrets on the Pod.
  • An image that doesn't exist for this node's architecture (arm64 node, amd64-only image).
  • Registry rate limiting.
terminal
kubectl describe pod <pod>   # Events section has the exact pull error

Pending

The API server accepted the Pod, but the scheduler hasn't placed it on a node yet.

  • No node has enough free CPU/memory to satisfy the Pod's requests -- this is exactly what should trigger the cluster autoscaler to add a node, if one's running.
  • A nodeSelector or affinity rule that no node matches.
  • A PVC the Pod needs hasn't bound to a volume yet.
terminal
kubectl describe pod <pod>
# Events: "Insufficient cpu", "node(s) didn't match node selector", ...

OOMKilled

The container exceeded its own memory limit and the kernel's OOM killer ended it -- the Pod-level reason for exit code 137. The fix is the limit, the app's actual memory use, or both -- see resource requests & limits.

Evicted

Different from OOMKilled: eviction is the node running low on memory or disk overall, and the kubelet proactively removing Pods to relieve pressure -- a node-level decision, not that specific container hitting its own limit.

terminal
kubectl describe pod <pod>   # Status.Reason / Message explain the pressure
kubectl get events --sort-by=.lastTimestamp

Stuck in ContainerCreating

  • A volume won't mount -- a PVC stuck Pending, or a StorageClass that doesn't exist.
  • A large image still pulling for the first time on this node.
  • The CNI plugin not ready on a newly joined node.
terminal
kubectl describe pod <pod>   # Events show exactly which step is stuck

Completed vs. Error

For a Job or any Pod with restartPolicy: Never or OnFailure: exit code 0 shows as Completed; anything else shows as Error, with that same exit code available in describe pod.

A quick triage order

terminal
kubectl get pods -o wide                       # which Pod, which node, which status
kubectl describe pod <pod>                       # Events -- almost always names the problem
kubectl logs <pod> --previous                    # if it already restarted
kubectl get events --sort-by=.lastTimestamp       # cluster-wide, newest last