What happens when etcd loses quorum and how do you recover the cluster?

Imagine a board of directors that requires a majority vote to approve any decision. If the company has five board members and three resign suddenly, the remaining two cannot approve anything -- even if they agree -- because they lack the required majority. The company is paralyzed: no new hires, no budget changes, nothing. That is exactly what happens when etcd loses quorum: the remaining members know the current state but cannot authorize any changes.

In Kubernetes, etcd is the single source of truth for all cluster state -- every Pod definition, Service, ConfigMap, Secret, and controller state lives in etcd. The kube-apiserver reads from and writes to etcd exclusively. Etcd uses the Raft consensus algorithm, which requires a strict majority (N/2 + 1) of members to agree on writes. For a 3-member etcd cluster, quorum requires 2 members; for 5 members, it requires 3. When quorum is lost, etcd switches to a degraded mode where it can serve stale reads (depending on consistency settings) but rejects all write operations.

When quorum is lost, the chain of failure propagates quickly. The kube-apiserver begins returning errors for any mutating request (POST, PUT, DELETE) because etcd refuses writes. Controllers in the kube-controller-manager that rely on leader election through the apiserver may lose their leases. The scheduler cannot bind Pods to nodes. Existing workloads continue running because kubelets cache their Pod specs locally and container runtimes are independent of the control plane. However, no new Pods can be created, no scaling can occur, node heartbeats cannot be updated (which eventually triggers node NotReady conditions), and self-healing stops entirely. The cluster is alive but brain-dead.

Recovery depends on the failure scenario. If etcd members are down due to transient issues (network partition, disk pressure, or crashed processes), the fastest path is to bring enough members back online to restore quorum. Check each member with etcdctl endpoint status and etcdctl member list. If a member's data is corrupted, remove it from the cluster with etcdctl member remove, then re-add it as a new member with etcdctl member add and let it rejoin and replicate. For catastrophic failure where all members are lost, you must restore from an etcd snapshot. Take regular snapshots with etcdctl snapshot save, then restore with etcdctl snapshot restore to a new data directory on each member, updating the initial-cluster and initial-advertise-peer-urls flags. After restoration, restart etcd and verify the kube-apiserver reconnects.

In production, etcd failures at scale are often caused by slow disks, large key-value sizes from too many Kubernetes objects, or aggressive compaction settings. The write-ahead log (WAL) is sensitive to disk latency; etcd recommends dedicated SSDs with sub-10ms p99 latency. A non-obvious gotcha is that etcd v3 has a default storage limit of 2GB (configurable up to 8GB), and if the database exceeds this limit, etcd enters a maintenance mode that effectively looks like quorum loss. Another trap: during recovery, if you restore a snapshot to an odd number of members but start them with stale peer URLs, they may form split-brain scenarios. Always restore all members from the same snapshot simultaneously and use a fresh cluster token to prevent old members from rejoining.