Google SRE Interview: Your team manages a 5000-node Kubernetes cluster and etcd latency has degraded from 10ms to 200ms at the 99th percentile. Walk through how you would diagnose this using NALSD principles and what remediation steps you would take.
Quick Answer
Apply Non-Abstract Large System Design (NALSD) by quantifying the etcd write throughput, identifying whether the bottleneck is disk I/O (WAL fsync), network latency between etcd members, or excessive API server watch/list operations saturating etcd's MVCC layer.
Detailed Answer
Understanding NALSD for etcd
Google's NALSD framework requires you to reason about large systems with concrete numbers. etcd is the Raft-based key-value store backing all Kubernetes state. At 5000 nodes, you're likely dealing with 100,000+ objects (pods, services, endpoints, configmaps). Each API server write becomes an etcd transaction, and etcd's performance is fundamentally bound by disk fsync latency for its Write-Ahead Log (WAL).
Diagnosis Path
1. Quantify the load: etcd exposes etcd_disk_wal_fsync_duration_seconds and etcd_disk_backend_commit_duration_seconds. If WAL fsync p99 > 10ms, the disk subsystem is the bottleneck. At 5000 nodes, you should expect ~2000-5000 writes/second. 2. Check Raft consensus: etcd_server_leader_changes_seen_total increasing means leader instability. etcd_network_peer_round_trip_time_seconds reveals cross-member latency. Raft requires majority quorum for every write, so a slow member drags down the entire cluster. 3. Identify hot keys: Large list/watch operations on namespaces with thousands of pods generate excessive range queries. Use etcd_debugging_mvcc_keys_total and audit API server request logs for expensive LIST operations.
Remediation at Scale
Move etcd to dedicated NVMe-backed instances (not shared with kubelet/container workloads). Implement API Priority and Fairness (APF) to throttle expensive LIST operations. Consider etcd defragmentation if the database has grown beyond 4GB due to historical revisions. For clusters this large, evaluate splitting etcd into separate clusters for events vs. core resources (the --etcd-servers-overrides flag).
NALSD Capacity Planning
Calculate: 5000 nodes x 30 pods avg = 150,000 pod objects. Each pod is ~2KB in etcd. Total data ~300MB, well within etcd limits. But the write rate during rolling updates (updating 10% of pods = 15,000 writes in minutes) can overwhelm disk I/O. Plan for sustained 5000 writes/sec with p99 fsync < 10ms, which requires dedicated NVMe with >10,000 IOPS.