How do kube-state-metrics and node-exporter work together to monitor Kubernetes?

Think of monitoring a factory. kube-state-metrics is like the floor manager who tracks the status of every work order: how many assembly lines are running, which ones are behind schedule, which workers are idle. node-exporter is like the building engineer who monitors the physical plant: power consumption, HVAC temperature, water pressure, and structural integrity. You need both perspectives -- the logical work status and the physical infrastructure health -- to understand what is really going on.

kube-state-metrics (KSM) is a service that connects to the Kubernetes API server and turns Kubernetes object states into Prometheus metrics. It does not instrument applications or nodes -- it purely converts the API's data into time series. For example, it exposes kube_deployment_spec_replicas (how many replicas you asked for), kube_deployment_status_replicas_available (how many are actually available), kube_pod_status_phase (whether a pod is Pending, Running, Succeeded, Failed, or Unknown), and kube_pod_container_resource_requests (CPU/memory requests). KSM runs as a single deployment (or sharded for large clusters) and exposes metrics on port 8080. It puts zero load on individual nodes or pods.

node-exporter is a Prometheus exporter that runs as a DaemonSet on every Kubernetes node, exposing hardware and OS metrics. It reads from /proc and /sys on Linux to report things like node_cpu_seconds_total (CPU usage per core per mode), node_memory_MemAvailable_bytes (available memory), node_filesystem_avail_bytes (disk space), node_network_receive_bytes_total (network traffic), and node_disk_io_time_seconds_total (disk I/O). Each node-exporter instance exposes its local node's metrics on port 9100. On Kubernetes, it is deployed as a DaemonSet with hostPID, hostNetwork, and volume mounts to /proc and /sys.

In production, combining both exporters unlocks powerful correlation queries. For example, you can spot resource contention by comparing kube_pod_container_resource_requests_cpu_cores (what pods asked for) against node_cpu_seconds_total (what the node is actually using). You can build utilization dashboards showing requested versus actual versus capacity for every node. The kube-prometheus-stack Helm chart deploys both exporters along with Prometheus, Alertmanager, and pre-built Grafana dashboards and alert rules. Common alerts include KubePodNotReady (pods stuck in a non-ready state using KSM metrics), NodeMemoryHighUtilization (node running low on memory using node-exporter metrics), and KubeDeploymentReplicasMismatch (desired versus actual replicas using KSM).

A key gotcha with kube-state-metrics is that it does not show actual resource usage -- it only reports what was requested and what the limits are. To see real CPU and memory consumption, you need metrics-server or cAdvisor (container_cpu_usage_seconds_total, container_memory_usage_bytes), which are built into the kubelet. Another pitfall is node-exporter's filesystem metrics: by default, it mounts the host's /proc and /sys into the container, but if those mounts are set up wrong, metrics will reflect the container's filesystem instead of the host's. On managed Kubernetes services (EKS, GKE, AKS), some kernel-level metrics may be blocked by the cloud provider's security boundaries.