Debugging Pods Like a Senior Engineer

A repeatable order of operations — events, describe, logs, exec, ephemeral containers — turns 'the Pod is broken' into a root cause in minutes.

When a Pod misbehaves, flailing at random commands wastes time. Experienced operators follow a consistent funnel from cheapest signal to most invasive.

The debugging funnel

Events first. kubectl get events --sort-by=.lastTimestamp or the Events section of describe explains most scheduling and image failures (FailedScheduling, ImagePullBackOff, OOMKilled) instantly.
Describe the Pod. kubectl describe pod shows container states, restart counts, the last termination reason, and the exit code — the Last State: Terminated block is gold for crash loops.
Logs, including the previous run. kubectl logs POD -c CONTAINER --previous shows the output of the container before its last crash — the actual stack trace you need.
Exec in if the container is running: kubectl exec -it POD -- sh to poke at the live filesystem, env, and network.
Ephemeral containers when the image has no shell. kubectl debug attaches a temporary debug container (with your favorite tools) that shares the target's process and network namespaces — the modern way to debug slim/distroless images.

Decode the common statuses

Status	Usual cause
`ImagePullBackOff`	Bad image name/tag or missing registry credentials
`CrashLoopBackOff`	App exits on startup — read `logs --previous`
`OOMKilled`	Hit the memory limit — raise it or fix the leak
`Pending`	No node fits the requests / a taint blocks it

Example

# 1. What does the cluster say happened? (scheduling, pulls, OOM)
kubectl get events --sort-by=.lastTimestamp | tail -20

# 2. Container states, restart count, last exit reason
kubectl describe pod web-7d9f

# 3. The crash's actual stack trace (output before the last restart)
kubectl logs web-7d9f -c web --previous

# 4. Poke around a running container
kubectl exec -it web-7d9f -- sh

# 5. Distroless image with no shell? Attach a debug toolbox
kubectl debug -it web-7d9f --image=busybox:1.36 --target=web

# Bonus: live resource usage per Pod (needs metrics-server)
kubectl top pod web-7d9f --containers

When to use it

A developer uses kubectl debug to attach an ephemeral container with debugging tools to a distroless production pod that has no shell.
An SRE uses kubectl describe and events to quickly identify that a pod is stuck in Pending because of insufficient CPU on all available nodes.
A platform team uses kubectl get events --sort-by=.lastTimestamp to reconstruct the timeline of failures during a recent incident.

More examples

Ephemeral debug container

Attaches an ephemeral busybox container to a running pod, enabling interactive debugging of distroless or minimal images.

# Inject a debug container into a running pod
kubectl debug -it my-pod \
  --image=busybox \
  --target=app \
  -- /bin/sh

# Now you have shell access with shared process namespace
# even if the original image has no shell

Investigate Pending pods

Inspects scheduling failure events and compares node allocated resources with available capacity to find why a pod cannot be scheduled.

kubectl describe pod stuck-pod | grep -A10 Events:
# Events:
#   Warning  FailedScheduling  Insufficient cpu

# Check node capacity
kubectl describe nodes | grep -A5 'Allocated resources'

# Check resource requests vs available
kubectl top nodes

Sort and filter cluster events

Retrieves cluster events sorted by time and filtered by type or object name to reconstruct what happened during an incident.

# All events sorted by time
kubectl get events -A --sort-by=.lastTimestamp

# Only Warning events
kubectl get events -A --field-selector type=Warning

# Events for a specific pod
kubectl get events \
  --field-selector involvedObject.name=my-pod

Discussion

Be the first to comment on this lesson.