Kubernetes CrashLoopBackOff: how to diagnose and fix it

You deploy, run kubectl get pods and see CrashLoopBackOff in the status column with a restart count that keeps climbing. The pod never becomes ready, and the deployment rollout stalls. CrashLoopBackOff is not the error itself. It is Kubernetes telling you that the container keeps exiting, and that it is waiting progressively longer (starting at 10 seconds and doubling up to a five-minute cap) before trying again.

The useful work is figuring out why the container exits. This is the order in which we check, from the cheapest signal to the most involved.

Why it happens

A container's lifetime is the lifetime of its main process. When that process exits, for any reason, the container stops, and with the default restartPolicy: Always the kubelet starts it again. Common reasons for the exit:

• Application errors at startup: an unhandled exception, a failed database connection, a missing environment variable or config file.
• Out of memory: the container exceeds its memory limit and the kernel kills it (OOMKilled, exit code 137).
• Failing liveness probes: the app is running but the probe fails, so the kubelet kills and restarts it. This is common with slow-starting apps.
• Wrong command or image: the entrypoint does not exist, is not executable, or the image was built for a different CPU architecture.
• The process finishes: a script that does its work and exits with code 0 still restarts forever in a Deployment. That workload should be a Job.

Step 1: Read the logs from the crashed container

Plain kubectl logs shows the current container, which may have just started and printed nothing. The --previous flag shows output from the run that crashed, which is usually where the answer is.

# Overview: status, restarts, node
kubectl get pods -n production -o wide

# Logs from the last crashed run (add -c <name> for multi-container pods)
kubectl logs api-7d9f8c -n production --previous

# Exit code, reason and timestamps of the last termination
kubectl get pod api-7d9f8c -n production \
  -o jsonpath='{.status.containerStatuses[0].lastState.terminated}'

# Events: probe failures, OOM kills, image and mount errors
kubectl describe pod api-7d9f8c -n production
kubectl get events -n production --sort-by=.lastTimestamp | tail -20

If the logs are empty, the process probably died before it could write anything: a missing binary, a permissions problem, or an immediate kill. Move on to the exit code.

Step 2: Interpret the exit code

The Last State section of kubectl describe pod shows a reason and an exit code. They narrow the search quickly:

• 0: the process completed successfully. Something in your image runs and finishes; you either need a long-running foreground process or a Job instead of a Deployment.
• 1 (or another small number): an application error. The logs from --previous should explain it.
• 126: the command was found but is not executable. Check file permissions and the shebang line in entrypoint scripts.
• 127: command not found. The command or args in your manifest, or the image's ENTRYPOINT, points at something that does not exist.
• 137: killed with SIGKILL. If the reason is OOMKilled, it is memory. Otherwise it is often a liveness probe kill or a process that ignored SIGTERM.
• 139: segmentation fault, usually a native dependency or an architecture mismatch.
• 143: terminated with SIGTERM, typically after a failed liveness probe or a normal shutdown.

Step 3: Fix the common causes

OOMKilled

Compare the container's memory limit with what the app actually uses. kubectl top pod gives a quick reading if metrics-server is installed. If the app legitimately needs more, raise the limit and set the request to a realistic baseline. For runtimes with their own heap settings, make sure the heap fits inside the limit: a Node.js process with --max-old-space-size larger than the container limit, or a JVM without container-aware sizing, will be killed by the kernel before its own allocator complains. If memory grows steadily until the kill, you have a leak, and raising the limit only delays the crash.

Liveness probes killing a healthy app

A liveness probe that fires before the app has finished starting will kill it every time, producing a loop that looks like a crash. Use a startupProbe to give the app time to boot, keep the liveness check cheap and local (it should not call your database), and use a readiness probe for dependency checks.

containers:
  - name: api
    image: registry.example.com/api:1.8.2
    envFrom:
      - secretRef:
          name: api-secrets
    resources:
      requests:
        memory: "256Mi"
        cpu: "250m"
      limits:
        memory: "512Mi"
    startupProbe:
      httpGet: { path: /healthz, port: 8080 }
      periodSeconds: 5
      failureThreshold: 30   # up to 150s to start
    livenessProbe:
      httpGet: { path: /healthz, port: 8080 }
      periodSeconds: 10
      failureThreshold: 3
    readinessProbe:
      httpGet: { path: /ready, port: 8080 }
      periodSeconds: 5

Missing config, secrets or files

If a referenced Secret or ConfigMap does not exist at all, the pod usually shows CreateContainerConfigError rather than CrashLoopBackOff. CrashLoopBackOff appears when the resource exists but the values are wrong: a key with a typo, a secret in the wrong namespace copied in empty, a connection string pointing at a host the pod cannot reach. Compare what the app expects with what is mounted: kubectl get secret api-secrets -o jsonpath='{.data}' lists the keys, and kubectl exec into a running replica (or a debug copy) lets you inspect the environment and mounted files.

Wrong command or architecture

An exec format error in the logs means the image was built for a different architecture, for example arm64 on a laptop and amd64 nodes in the cluster. Build multi-platform images with docker buildx build --platform linux/amd64,linux/arm64. For exit codes 126 and 127, run the image locally with the same command to confirm the path exists and is executable.

Step 4: Debug interactively when logs are not enough

You cannot exec into a container that exits after two seconds. Instead, create a copy of the pod with the command replaced by something that stays alive, then start the real process by hand and watch it fail.

# Copy the pod, override the container's command, and attach a shell
kubectl debug api-7d9f8c -n production -it \
  --copy-to=api-debug --container=api -- sh

# Inside: check env, files and connectivity, then run the real entrypoint
env | sort
ls -la /app
node dist/server.js

# Clean up afterwards
kubectl delete pod api-debug -n production

Prevention tips

• Validate configuration at startup and log a clear message naming the missing variable before exiting.
• Set memory requests and limits from observed usage, and alert on containers approaching their limit.
• Always pair slow-starting apps with a startupProbe; keep liveness checks free of external dependencies.
• Use Jobs or CronJobs for tasks that finish, not Deployments.
• Gate rollouts on readiness so a crashing version never replaces healthy pods, and keep kubectl rollout undo in your runbook.

Key takeaways

• CrashLoopBackOff is a symptom: the container keeps exiting and Kubernetes is backing off between restarts.
• kubectl logs --previous, kubectl describe pod and events answer most cases in a few minutes.
• The exit code tells you where to look: 1 for app errors, 137 for OOM or SIGKILL, 126/127 for command problems.
• Use kubectl debug --copy-to to keep a broken container alive long enough to inspect it.

If your cluster spends more time restarting pods than serving traffic, our DevOps and cloud team can review your manifests, probes and resource settings. Get in touch.