How do CPU and memory requests and limits affect Pod scheduling, quality of service, and node stability?
Quick Answer
Requests drive scheduling—kube-scheduler sums requests and finds nodes with capacity. Limits cap maximum usage; exceeding CPU throttles, exceeding memory triggers OOMKill. QoS classes (Guaranteed, Burstable, BestEffort) derive from request/limit alignment and affect eviction order.
Detailed Answer
Resource requests and limits are the contract between workloads and the cluster. Requests declare what a container needs to run reliably. The scheduler uses requests—not limits—to decide whether a Pod fits on a node. If worker-node-3 has 2000m CPU allocatable and existing Pods request 1800m, only 200m remains—a new Pod requesting 250m CPU waits in Pending until capacity frees or Cluster Autoscaler adds nodes.
Limits define maximum consumption. CPU is compressible: exceeding the limit throttles the container (CFS quota), slowing it down but keeping it alive. Memory is incompressible: exceeding memory limit triggers OOMKill by the kernel—immediate container restart. This asymmetry makes memory limits dangerous if set without understanding working set size.
Quality of Service classes affect eviction under node pressure. Guaranteed: every container has limits equal to requests (and only limits/requests set). Burstable: requests set but limits differ or not all containers equal. BestEffort: no requests or limits—the first evicted under pressure. payments-api with requests 250m/512Mi and limits 500m/1Gi is Burstable.
Namespace ResourceQuotas and LimitRanges enforce defaults and caps per tenant. LimitRange can inject default requests for containers that specify nothing—preventing BestEffort Pods from sneaking into production.
Analogy: requests are the seat reservation on an airplane—scheduling ensures enough seats exist before boarding. Limits are carry-on size rules—exceed CPU and you walk slower (throttle); exceed memory and security confiscates your bag (OOMKill).
Right-sizing requires metrics: use Vertical Pod Autoscaler recommendations or Prometheus queries for actual usage. Over-requesting wastes capacity; under-requesting causes noisy neighbor instability. Align HPA CPU targets with requests—HPA divides observed usage by requests to compute utilization percentage.
Init containers inherit the effective requests/limits of the Pod's highest-consuming container for QoS classification purposes. Ephemeral storage requests and limits prevent log or emptyDir exhaustion from destabilizing nodes—a often-overlooked dimension in production incidents.
When payments-api runs 3 replicas each requesting 250m CPU, the Deployment consumes 750m of schedulable capacity cluster-wide regardless of actual usage. Node allocatable minus system-reserved minus daemonset overhead defines the ceiling. Use kubectl describe node to audit Allocated resources and identify fragmentation where no single node has room for a new 250m Pod despite cluster aggregate capacity appearing sufficient.