Troubleshooting: Logs, NotReady Nodes & crictl

Start at journalctl -u k3s, work the NotReady checklist (CNI, containerd, disk, clock), and inspect at the CRI level.

When k3s misbehaves, resist jumping straight to kubectl — if the node is down, kubectl cannot help. Start at the systemd journal, which shows why the service itself is unhappy.

First stop: the journal

journalctl -u k3s -f on a server (or -u k3s-agent on a worker) is the highest-signal source. Look for TLS/token errors, etcd election churn, and containerd startup failures.

The NotReady checklist

  1. Connectivity — can the agent reach the server on :6443, and is Flannel's UDP :8472 open? A blocked VXLAN port shows as Ready nodes but broken pod networking.
  2. containerd — use crictl to see if the runtime is even answering.
  3. Disk pressure — a full /var flips nodes to NotReady fast; check df -h and eviction events.
  4. Clock skew — TLS and etcd hate unsynced clocks; verify NTP.
  5. cgroups — on a Raspberry Pi, missing memory cgroups keeps the node from ever going Ready.

Look below Kubernetes

k3s crictl ps, crictl logs, and describe node events reveal problems the pod abstraction hides — image pull failures, sandbox errors, and OOM kills.

Example

Example · bash
# 1. Why is the service unhappy? (agents: use k3s-agent)
sudo journalctl -u k3s -f --no-pager

# 2. Is the container runtime answering?
sudo k3s crictl ps -a
sudo k3s crictl logs <container-id>

# 3. Node-level clues for NotReady
kubectl describe node <name> | sed -n '/Conditions/,/Events/p'
df -h /var/lib/rancher
timedatectl status        # clock in sync?

# 4. Common edge fix: confirm memory cgroups exist
grep memory /proc/cgroups

When to use it

  • An SRE investigates a NotReady node by checking journalctl -u k3s and finds a CNI network interface conflict caused by a previous failed install.
  • A developer uses k3s crictl ps to discover that a pod's container has crash-looped 15 times and checks crictl logs to read the application error.
  • An operator debugs a clock-skew warning on an edge node using timedatectl and fixes it by enabling chrony, resolving etcd leader election failures.

More examples

Stream k3s logs for errors

Streams the k3s systemd service logs in real time and then filters to error-level messages for quick triage.

Example · bash
journalctl -u k3s -f --no-pager
# Or filter for errors only:
journalctl -u k3s -p err --no-pager -n 50

Debug a NotReady node

Describes the node to read its conditions, checks for NodeNotReady events, and runs k3s's built-in config validator.

Example · bash
kubectl describe node <node-name> | grep -A10 Conditions
kubectl get events --field-selector reason=NodeNotReady
k3s check-config

Inspect containers with crictl

Lists all containers including exited ones, reads their logs, and inspects the exit code to diagnose crash-loop causes.

Example · bash
k3s crictl ps -a
k3s crictl logs <container-id>
k3s crictl inspect <container-id> | grep -E 'state|exitCode|reason'

Related

k3s pro edge tuningk3s flannelk3s requirements

Discussion

  • Be the first to comment on this lesson.