What runs on a Kubernetes worker node, and how does it communicate with the control plane?

Think of a worker node like a restaurant kitchen station. The kubelet is the line cook who receives tickets (Pod specs) from the head chef (API Server) and actually prepares the dishes (starts containers). The container runtime (containerd or CRI-O) is the set of pots, pans, and ovens that the cook uses — the actual tools that do the cooking. And kube-proxy is the waiter who knows which table ordered which dish, making sure the right plate gets to the right customer via the correct route. Without any one of these three, the kitchen cannot function.

The kubelet is the primary node agent. It registers the node with the API Server, then watches (via the API Server's watch mechanism) for PodSpecs assigned to its node. When a new Pod is scheduled to its node, the kubelet calls the container runtime through the Container Runtime Interface (CRI) to pull images and start containers. It then continuously monitors container health by executing liveness and readiness probes at configured intervals. If a liveness probe fails, the kubelet restarts the container. It reports Pod status and node conditions (memory pressure, disk pressure, PID pressure, network unavailable) back to the API Server every 10 seconds by default (the node-status-update-frequency). If the API Server doesn't receive heartbeats for 40 seconds (the node-monitor-grace-period), the node is marked NotReady.

The container runtime is the software that actually creates and runs containers using Linux kernel features like namespaces (isolation) and cgroups (resource limits). Kubernetes removed direct Docker support in v1.24 — it now requires a CRI-compatible runtime. The two production choices are containerd (lightweight, used by EKS, GKE, and most managed platforms) and CRI-O (purpose-built for Kubernetes, used by OpenShift). The kubelet communicates with the runtime via a Unix socket using the gRPC-based CRI protocol. The runtime handles image pulling, container lifecycle, and log management.

kube-proxy runs on every node and implements the Service abstraction. When you create a Service, kube-proxy watches the API Server for Service and Endpoint objects, then programs the node's networking stack to route traffic correctly. In the default iptables mode, it creates iptables rules that perform DNAT (destination NAT) to translate the virtual Service IP to a real Pod IP, with random selection for load balancing. In IPVS mode (better for clusters with thousands of Services), it uses the Linux kernel's IPVS load balancer which supports multiple algorithms (round-robin, least-connections, source-hash). kube-proxy does NOT proxy traffic through itself — it only configures networking rules; actual packets flow directly from source to destination Pod.

The communication between worker nodes and the control plane is strictly one-directional in terms of initiation: the kubelet always initiates connections to the API Server, never the reverse. This is a security design — worker nodes can be in untrusted networks and the API Server never needs to push data to them. The kubelet establishes a persistent watch connection to the API Server, which means it receives updates the instant they happen (Pod scheduled, Pod deleted, ConfigMap changed) without polling. For features like kubectl exec and kubectl logs, the API Server does establish a reverse connection to the kubelet's HTTPS endpoint (port 10250), which is why kubelet has its own TLS certificate.

A common production gotcha: if the container runtime's socket becomes unresponsive (containerd hung, disk full preventing image pulls), the kubelet cannot start new Pods or report accurate status. The node might show Ready because the kubelet process itself is fine, but Pods scheduled there will be stuck in ContainerCreating forever. Monitoring containerd/CRI-O process health separately from kubelet health is essential for catching this early.