How does etcd's Raft consensus affect Kubernetes API server write latency, and how would you debug a cluster experiencing slow or failed writes under load?

Kubernetes treats etcd as the single source of truth, and every mutating API request goes through Raft consensus: the etcd leader appends the entry to its write-ahead log, replicates it to followers, and only acknowledges the write once a majority (quorum) of members have persisted it to disk (fsync, not just memory — this is what makes etcd durable across crashes). This means write latency has a hard floor set by disk fsync latency on however many nodes are needed for quorum, and any added network latency between etcd members directly inflates write latency.

Under load, several distinct failure modes show up and need different fixes

1. Disk-bound latency: etcd is extremely fsync-sensitive. If etcd shares a disk with other I/O-heavy workloads, or runs on non-SSD storage, fsync latency spikes and every write in the cluster slows down. Diagnose with etcd_disk_wal_fsync_duration_seconds — Google/etcd's own guidance treats >10ms p99 as a red flag for production etcd. 2. Quorum loss / leader churn: if network partitions or resource starvation cause repeated leader elections (etcd_server_leader_changes_seen_total incrementing), every election pauses writes cluster-wide for the duration of the election, which can be a second or more under contention. 3. API server overload independent of etcd: large List/Watch requests (e.g., a controller doing an unbounded LIST of all Pods cluster-wide) can starve the API server's request queue even when etcd itself is healthy — distinguish this by comparing apiserver_request_duration_seconds against etcd's own latency metrics; if etcd is fast but the API server is slow, the bottleneck is API server-side (admission webhooks, conversion, or queue depth), not etcd. 4. Large key/value churn: high rates of resource updates (e.g., frequent Endpoints/EndpointSlice updates from a churny Service) inflate etcd's DB size and revision history, which slows compaction and can eventually trigger 'etcdserver: mvcc: database space exceeded' errors if compaction/defrag isn't running regularly.

The debugging path: check etcd's own metrics first (fsync latency, leader changes) to rule in/out etcd as the bottleneck, then check API server request latency by verb and resource to see if it's etcd-bound or API-server-bound, then check what's generating the write/read volume (a misbehaving controller is the most common real-world cause).