What are the key Prometheus metrics for monitoring Kubernetes cluster health?

Think of monitoring a factory floor. You check the building infrastructure (nodes), the assembly lines (pods), the power supply (control plane), and the shipping dock (networking). Each level has its own vital signs, and a problem at one level cascades to others. A power outage (control plane issue) affects all assembly lines (pods), but a single broken machine (crashed pod) only affects one product.

Kubernetes cluster health metrics come from three sources. Node metrics from node-exporter measure hardware and OS resources: CPU usage, memory available, disk space, network throughput. Kubernetes state metrics from kube-state-metrics report the desired vs actual state of Kubernetes objects: pod phase, deployment replicas, node conditions. Control plane metrics from the API server, scheduler, and etcd report the health of cluster management components.

The critical metrics organized by layer are: Nodes — node_cpu_seconds_total (CPU usage per core), node_memory_MemAvailable_bytes (available memory), node_filesystem_avail_bytes (disk space), kube_node_status_condition (Ready/NotReady). Pods — kube_pod_container_status_restarts_total (restart count), kube_pod_status_phase (Pending/Running/Failed), container_memory_working_set_bytes (actual memory usage vs limits). Control plane — apiserver_request_total (API server request rate and errors), etcd_server_leader_changes_seen_total (leader elections indicate instability), scheduler_pending_pods (pods waiting for a node).

At production scale, alert on symptoms that indicate user impact: nodes in NotReady for more than 5 minutes, pods with more than 5 restarts in 15 minutes, pending pods for more than 10 minutes (scheduling failure), API server error rate above 1%, and etcd leader changes more than 3 in an hour. Avoid alerting on individual pod CPU usage — it fluctuates naturally. Instead, alert on the ratio of usage to request, which indicates right-sizing problems.

The non-obvious gotcha is the difference between container_memory_usage_bytes and container_memory_working_set_bytes. The usage metric includes filesystem cache that the kernel can reclaim, so it often looks higher than the actual limit. The working_set metric reflects non-reclaimable memory and is what the OOM killer uses. Alerting on usage_bytes causes false alarms; alert on working_set_bytes approaching the memory limit instead.