Why does etcd require an odd-number quorum for HA Kubernetes control planes, and what happens when quorum is lost?
Quick Answer
etcd uses Raft consensus requiring a majority (quorum) of nodes to agree on writes. Odd member counts (3, 5, 7) maximize fault tolerance without wasting a node—a 4-node cluster tolerates only 1 failure, same as 3. Losing quorum freezes the Kubernetes API; the data plane may keep running but cannot be managed.
Detailed Answer
etcd is the consistent, distributed key-value store holding all Kubernetes cluster state: Pod specs, Secrets, ConfigMaps, lease data, and RBAC rules. Every API server read and write for mutable state ultimately interacts with etcd. High availability deploys etcd as a Raft cluster where one leader handles writes and replicates log entries to followers.
Raft requires a majority of members to acknowledge a write before it commits. For N members, quorum is floor(N/2)+1. A 3-member cluster tolerates 1 failure (2 of 3 = quorum). A 5-member cluster tolerates 2 failures. Critically, a 4-member cluster still requires 3 for quorum—also tolerating only 1 failure—so the fourth node adds latency and operational cost without improving resilience. This is why etcd clusters always use odd counts.
When quorum is lost—imagine 2 of 3 etcd members permanently fail—the remaining member cannot elect a leader or accept writes. kubectl commands fail. Controllers stop reconciling. Existing Pods on worker nodes continue running (kubelet operates from local cache), but no scheduling, scaling, or Service updates occur. This split-brain protection is preferable to serving stale or divergent cluster state.
Recovery options include etcd snapshot restore to a fresh cluster, disaster recovery procedures documented per distribution (kubeadm, RKE, EKS manages etcd), and preventive monitoring of etcd latency, db size, and leader elections. Regular automated snapshots to object storage are non-negotiable for self-managed clusters.
Analogy: etcd quorum is a panel of judges requiring majority agreement before publishing a verdict. With three judges, one may be absent and rulings continue. If only one judge remains, court adjourns—no new rulings until quorum is restored—even though convicted prisoners (running Pods) remain in their cells.
Architect-level discussions cover stacked vs external etcd topology, TLS peer encryption, defragmentation schedules, and performance tuning (SSD, network latency under 10ms between members).
Monitor etcd db size and set auto-compaction retention to prevent unbounded growth. Leader election metrics (etcd_server_leader_changes_seen_total) spiking indicate network instability. For managed Kubernetes, understand what the provider backs up and test restore into an isolated environment quarterly—assuming the control plane is someone else's problem until it isn't is a common organizational failure mode.