Explain how container resource limits work with cgroups v2. What happens when a container exceeds its memory limit vs. its CPU limit? How does CPU throttling manifest as application latency?
Quick Answer
Memory limits trigger OOM kill when RSS exceeds the cgroup memory.max (hard kill, no graceful shutdown). CPU limits cause CFS throttling - the process is paused for the remainder of its period when it exhausts its quota, causing periodic latency spikes rather than termination.
Detailed Answer
Cgroups v2 Resource Model
Cgroups v2 (unified hierarchy, default in modern kernels 5.8+) uses a single directory tree at /sys/fs/cgroup/. Each container gets a scope/slice with resource controllers: memory, cpu, io, pids.
Memory Limit Behavior
- memory.max: Hard limit. When total RSS + page cache (if accounted) exceeds this, the kernel's OOM killer terminates the process with SIGKILL (exit code 137). No graceful shutdown possible. - memory.high: Soft limit. When exceeded, the kernel aggressively reclaims memory from the cgroup (swapping if swap is enabled, dropping page cache). Causes latency but no kill. - memory.low: Memory protection. Guarantees this much memory won't be reclaimed under system pressure. - In Kubernetes: resources.limits.memory maps to memory.max. resources.requests.memory affects scheduling only (not cgroup enforcement).
CPU Limit Behavior (CFS Bandwidth Control)
- cpu.max format: quota period (e.g., 100000 100000 = 1 CPU, 50000 100000 = 0.5 CPU) - Quota: Microseconds of CPU time allowed per period - Period: Default 100ms (100000us) - When a container uses its entire quota within a period, it's THROTTLED (frozen) until the next period starts
How CPU Throttling Causes Latency
With 500m CPU limit (50ms quota per 100ms period), a request that needs 80ms of CPU time takes: 50ms work -> throttled 50ms -> 30ms work = 130ms wall-clock time instead of 80ms. This creates periodic latency spikes where p99 is much worse than p50. The container isn't killed - it's just periodically paused.
The CPU Limits Debate
Many organizations (including Google) recommend NOT setting CPU limits, only requests. Rationale: CPU is compressible (throttling degrades but doesn't kill), and limits cause unnecessary latency when the node has idle capacity. Memory limits are always needed because memory is incompressible (OOM kills are worse than throttling).