Safe Upgrades & Rollbacks

Snapshot first, upgrade servers before agents one at a time, respect skew, and know that rollback means restore.

Upgrading k3s is low-drama if you follow the order and take a safety net first. The golden rule: an upgrade is easy to move forward and hard to move back, so the snapshot comes before anything else.

The safe sequence

  1. Take an on-demand etcd (or SQLite) snapshot.
  2. Upgrade server nodes first, one at a time, waiting for each to return Ready and rejoin etcd quorum before touching the next.
  3. Upgrade agents afterwards.
  4. Never let agents run a newer minor version than the servers — respect the Kubernetes version-skew policy (agents may trail servers, not lead).

Rollback reality

There is no k3s downgrade. A minor-version upgrade migrates the datastore schema, so "rolling back" really means restore the pre-upgrade snapshot onto the old binary. That is exactly why step 1 exists.

Fleet upgrades

For many nodes, the system-upgrade-controller drains, upgrades, and uncordons nodes from a declarative Plan, honouring the server-then-agent order for you.

Example

Example · yaml
# system-upgrade-controller Plan: servers first, then agents
apiVersion: upgrade.cattle.io/v1
kind: Plan
metadata:
  name: server-plan
  namespace: system-upgrade
spec:
  concurrency: 1            # one node at a time
  version: v1.30.6+k3s1
  nodeSelector:
    matchExpressions:
      - {key: node-role.kubernetes.io/control-plane, operator: In, values: ["true"]}
  serviceAccountName: system-upgrade
  cordon: true
  upgrade:
    image: rancher/k3s-upgrade

When to use it

  • An ops team takes a named etcd snapshot before every k3s upgrade so they can run cluster-reset if the new version causes an incompatibility.
  • An operator uses the system-upgrade-controller to roll out k3s patch releases one node at a time with automatic cordon/drain, ensuring zero-downtime.
  • A developer pins k3s to a specific minor version in staging for three weeks before promoting the same version pin to production.

More examples

Snapshot before upgrading

Takes a named etcd snapshot as a rollback point, then upgrades the server binary using the install script.

Example · bash
k3s etcd-snapshot save --name pre-v1.30-upgrade
kubectl get nodes
# Then upgrade the first server:
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.30.0+k3s1 sh -

Drain and upgrade a node

Safely drains the agent node before upgrading its k3s binary, then restores scheduling after the upgrade.

Example · bash
kubectl drain worker-01 --ignore-daemonsets --delete-emptydir-data
curl -sfL https://get.k3s.io | \
  K3S_URL=https://server:6443 K3S_TOKEN=$TOKEN \
  INSTALL_K3S_VERSION=v1.30.0+k3s1 sh -
kubectl uncordon worker-01

System-upgrade-controller Plan

Defines an upgrade Plan that rolls out the stable k3s release to all agent nodes one at a time automatically.

Example · yaml
apiVersion: upgrade.cattle.io/v1
kind: Plan
metadata:
  name: k3s-agents
  namespace: system-upgrade
spec:
  concurrency: 1
  nodeSelector:
    matchLabels:
      node-role.kubernetes.io/agent: 'true'
  channel: https://update.k3s.io/v1-release/channels/stable
  upgrade:
    image: rancher/k3s-upgrade

Related

k3s upgradesk3s pro etcd hak3s backups

Discussion

  • Be the first to comment on this lesson.