HA Embedded etcd: Sizing, Snapshots & Restore

Run an odd number of servers, keep etcd on fast disk, and rehearse the cluster-reset restore before you ever need it.

Embedded etcd turns k3s into a real HA control plane, but etcd is unforgiving about two things: quorum and disk latency.

Sizing the control plane

  • Always run an odd number of servers (3 or 5). Three tolerates one failure; five tolerates two. Even counts add cost without adding fault tolerance.
  • Stop at 5 for almost everyone — more members means more Raft replication traffic, not more resilience.
  • etcd is fsync-bound. Put /var/lib/rancher/k3s/server/db on SSD/NVMe. A slow SD card or network disk causes leader elections and flapping.

Snapshots you can trust

k3s takes scheduled etcd snapshots automatically. Tune the cadence and retention, and ship them off the node to S3-compatible storage — a snapshot that only lives on the failed disk is not a backup.

Restoring

Restore is a --cluster-reset operation on a single surviving server: stop k3s everywhere, reset-restore on one node, then re-join the other servers fresh. Rehearse it in staging so the real incident is muscle memory.

Example

Example · bash
# First server: init etcd + scheduled snapshots to S3
sudo k3s server --cluster-init --token shared-secret \
  --etcd-snapshot-schedule-cron '0 */6 * * *' \
  --etcd-snapshot-retention 10 \
  --etcd-s3 --etcd-s3-bucket k3s-backups \
  --etcd-s3-endpoint s3.example.com

# Take an on-demand snapshot right before risky changes
sudo k3s etcd-snapshot save --name pre-change

# Disaster restore on one surviving server (k3s stopped first)
sudo systemctl stop k3s
sudo k3s server --cluster-reset \
  --cluster-reset-restore-path=/var/lib/rancher/k3s/server/db/snapshots/pre-change

When to use it

  • A production team provisions three server nodes on NVMe disks so embedded etcd has the low-latency I/O it needs to maintain leader elections reliably.
  • An SRE schedules hourly etcd snapshots to an S3 bucket and rehearses a cluster-reset restore quarterly so the team can recover within the RTO target.
  • An operator monitors etcd disk space with a Prometheus alert rule to catch storage exhaustion before it causes leader election failures.

More examples

Bootstrap HA cluster with etcd

Bootstraps a 3-node embedded etcd cluster: the first server uses --cluster-init, subsequent servers join with --server.

Example · bash
# Node 1 (first server):
curl -sfL https://get.k3s.io | \
  INSTALL_K3S_EXEC='--cluster-init --token my-secret' sh -
# Nodes 2 and 3:
curl -sfL https://get.k3s.io | \
  INSTALL_K3S_EXEC='--server https://node1:6443 --token my-secret' sh -
kubectl get nodes

Schedule automatic snapshots to S3

Configures k3s to upload hourly etcd snapshots to an S3 bucket, retaining 24 copies for a 24-hour restore window.

Example · yaml
# /etc/rancher/k3s/config.yaml
etcd-snapshot-schedule-cron: '0 * * * *'
etcd-snapshot-retention: 24
etcd-s3: true
etcd-s3-bucket: my-k3s-backups
etcd-s3-region: us-east-1
etcd-s3-access-key: AKIAIOSFODNN7EXAMPLE
etcd-s3-secret-key: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

Restore from snapshot with cluster-reset

Stops k3s, performs a point-in-time restore from a named snapshot using --cluster-reset, then restarts the server.

Example · bash
systemctl stop k3s
k3s server --cluster-reset \
  --cluster-reset-restore-path=/var/lib/rancher/k3s/server/db/snapshots/pre-upgrade
systemctl start k3s
kubectl get nodes

Related

k3s ha etcdk3s backupsk3s pro safe upgrades

Discussion

  • Be the first to comment on this lesson.