A containerized workload is being CPU-throttled even though node-level CPU utilization looks low. How do you diagnose cgroup CPU throttling and what actually causes it?

This is a direct consequence of how the Completely Fair Scheduler's (CFS) bandwidth controller enforces Kubernetes CPU limits via cgroups. A container with resources.limits.cpu: 2 gets a cgroup quota of 200000us per 100000us (100ms) period. If the process is single-threaded-bursty or has a thread pool that spikes briefly, it can consume its entire 200ms-equivalent of CPU time within the first few milliseconds of a 100ms period and then gets throttled for the remainder of that period — even though, averaged over a full second, it never exceeded its nominal limit, and even though the node as a whole is mostly idle. Multi-threaded workloads with limits set just above their typical-but-bursty thread count are especially prone to this because all threads draw from the same shared quota.

Diagnosis: every cgroup exposes cpu.stat with nr_periods, nr_throttled, and throttled_time. A high ratio of nr_throttled/nr_periods combined with low overall node CPU usage is the signature. cAdvisor/kubelet expose this as the Prometheus metric container_cpu_cfs_throttled_periods_total — graphing throttled-periods-ratio per container is the standard way to catch this in production before users report latency spikes.

Fixes: (1) raise the CPU limit so the quota window has more headroom for bursts, (2) remove CPU limits entirely for latency-sensitive workloads and rely on requests for scheduling plus node-level isolation — this is what many SRE teams do for latency-critical services, accepting some risk of noisy-neighbor in exchange for not falsely throttling, (3) for newer kernels, the kernel's CFS quota period-handling has improved (burst feature, cpu.cfs_burst_us in cgroup v2) which allows accumulating unused quota to absorb short bursts.