Observability & Troubleshooting

Wire up Container Insights for metrics and logs, then work a broken pod from Pending to Ready with a repeatable checklist.

You cannot fix what you cannot see. Get telemetry flowing first, then troubleshoot methodically instead of guessing.

Container Insights

The CloudWatch Observability add-on deploys the CloudWatch agent and Fluent Bit to collect node/pod/container metrics and ship container logs to CloudWatch — giving you Container Insights dashboards for CPU, memory, disk, and network per pod and per node, plus Container Insights with enhanced observability for control-plane and performance signals. It's the fastest way to get from zero to real visibility.

A troubleshooting checklist that always works

kubectl get pods -o wide — what state, on which node?
kubectl describe pod — read the Events at the bottom. Pending usually means unschedulable (no resources, no matching node, or no free IP); ImagePullBackOff is registry/permissions; CrashLoopBackOff is the app dying on start.
kubectl logs (add --previous for a crashed container) — the app's own words.
kubectl get events --sort-by=.lastTimestamp — the cluster's recent history.
Still stuck? Check the node: kubectl describe node for pressure/taints, and node logs via SSM.

Common EKS-specific culprits

Unauthorized from kubectl → IAM identity not mapped (access entry / aws-auth).
Pod Pending with IP errors → subnet exhaustion (see prefix delegation).
Pod can't reach AWS → missing or mis-scoped IRSA / Pod Identity role.

Example

# Turn on Container Insights + container logs in one add-on
aws eks create-addon --cluster-name demo \
  --addon-name amazon-cloudwatch-observability \
  --service-account-role-arn arn:aws:iam::111122223333:role/CloudWatchAgentRole

# The go-to triage sequence for a broken pod
kubectl get pods -o wide
kubectl describe pod my-app-7d9f-abcde | sed -n '/Events/,$p'
kubectl logs my-app-7d9f-abcde --previous --tail=100
kubectl get events --sort-by=.lastTimestamp | tail -20

When to use it

An SRE uses kubectl describe pod and the event log to discover a CrashLoopBackOff is caused by a missing ConfigMap that the pod mounts as a volume.
A platform engineer uses CloudWatch Container Insights to identify a pod consuming 4x its CPU limit, causing noisy-neighbor throttling on shared nodes.
A developer runs kubectl exec to open a shell in a running pod and curl the downstream service directly, isolating whether the problem is the app or the Service/Ingress layer.

More examples

Debug a Pending pod

Describes a Pending pod to read the Events section which shows exactly why the scheduler cannot place it — the first step in any pod scheduling debug.

kubectl describe pod stuck-pod-xyz
# Look for Events section — common causes:
# - Insufficient CPU/memory on nodes
# - No nodes match pod's node selector or taint tolerations
# - PVC not bound
kubectl get events --sort-by='.lastTimestamp' | tail -20

Check pod logs and previous logs

Retrieves current and previous container logs, useful for diagnosing CrashLoopBackOff where the current container has already restarted after the error.

# Current logs
kubectl logs my-app-7d9f8b-xyz --tail=50
# Previous container logs (after crash)
kubectl logs my-app-7d9f8b-xyz --previous --tail=50
# Multi-container pod
kubectl logs my-app-7d9f8b-xyz -c sidecar

Network connectivity debug pod

Launches a temporary netshoot pod loaded with network debugging tools to test DNS, HTTP connectivity, and routing from inside the cluster network.

kubectl run netdebug --image=nicolaka/netshoot \
  --rm -it --restart=Never -- bash
# Inside the debug pod:
# curl http://api-svc.default.svc.cluster.local/health
# nslookup api-svc.default.svc.cluster.local
# traceroute 10.0.1.45

Discussion

Be the first to comment on this lesson.