High Availability with etcd

Run three or more server nodes with embedded etcd so the control plane survives a node failure.

A single server is a single point of failure. For production, run an HA control plane: three server nodes, each holding a copy of the cluster state in an embedded etcd database.

Why three?

etcd uses a majority (quorum) to agree on data. With three members, the cluster keeps working even if one server is down. The number of servers should be odd (3, 5, ...).

How to start it

Launch the first server with --cluster-init to switch from SQLite to embedded etcd. The other servers then join using the shared token and the first server's URL.

High availability k3s with three embedded etcd serversFixed address (LB / VIP) :6443Server 1embedded etcdServer 2embedded etcdServer 3embedded etcdetcd quorum (Raft) — survives one server failureAgents connect through the fixed address
Three etcd members keep a majority even when one server goes offline.

Example

Example · bash
# Server 1 initialises the etcd cluster
sudo k3s server --cluster-init --token "shared-secret"

# Servers 2 and 3 join the existing control plane
sudo k3s server \
  --server https://192.168.1.10:6443 \
  --token "shared-secret"

# Confirm all three are control-plane members
kubectl get nodes

When to use it

  • A production team deploys three k3s server nodes with embedded etcd so the cluster stays up when one control-plane node is rebooted for maintenance.
  • An on-prem Kubernetes operator migrates from a SQLite single-server k3s cluster to an etcd HA setup to meet a 99.9% uptime SLA.
  • A disaster-recovery plan relies on k3s etcd snapshots stored on an NFS share, allowing cluster restoration after a full datacenter failure.

More examples

Bootstrap first HA server

Starts the first k3s server with --cluster-init to enable embedded etcd and begin an HA control-plane cluster.

Example · bash
curl -sfL https://get.k3s.io | \
  INSTALL_K3S_EXEC='--cluster-init' sh -
kubectl get nodes

Join additional server nodes

Adds a second or third server node to form a quorum, joining the embedded etcd cluster for HA.

Example · bash
TOKEN=$(cat /var/lib/rancher/k3s/server/node-token)
curl -sfL https://get.k3s.io | \
  K3S_TOKEN=$TOKEN \
  INSTALL_K3S_EXEC='--server https://192.168.1.10:6443' sh -

Take an etcd snapshot

Creates a named etcd snapshot on demand, useful before upgrades or destructive configuration changes.

Example · bash
k3s etcd-snapshot save --name pre-upgrade-$(date +%Y%m%d)
ls /var/lib/rancher/k3s/server/db/snapshots/

Related

k3s node tokenk3s backups

Discussion

  • Be the first to comment on this lesson.