A pod keeps getting OOMKilled but the application's heap usage looks normal at 60% of the container memory limit. What is happening and how do you debug it?
Quick Answer
Off-heap memory consumption (native threads, file caches, JNI allocations) or sidecar containers are consuming memory outside the JVM heap, pushing total RSS beyond the cgroup limit.
Detailed Answer
When Kubernetes OOMKills a pod, it's the kernel's cgroup OOM killer acting on the total RSS (Resident Set Size) of ALL processes in the pod, not just the JVM heap.
Root Causes
1. Off-heap memory: JVM uses native memory for thread stacks, metaspace, direct byte buffers, JNI, and compiled code. A JVM with 512MB heap can easily use 800MB+ total. 2. Sidecar containers: Istio envoy, log collectors, or other sidecars share the pod's memory limit. 3. Kernel page cache: File I/O-heavy workloads inflate RSS through page cache accounting.
Debugging Steps
1. kubectl top pod <pod> --containers to see per-container memory 2. Check cat /sys/fs/cgroup/memory/memory.usage_in_bytes inside the container 3. For JVM: use -XX:NativeMemoryTracking=summary and jcmd <pid> VM.native_memory 4. Compare container memory limit vs actual RSS: cat /proc/<pid>/status | grep VmRSS 5. Review pod spec for all containers and their individual resource requests/limits
Production at Scale
Set JVM flags like -XX:MaxRAMPercentage=70.0 to leave headroom for non-heap memory. Always set memory limits on sidecar containers separately. Use VPA recommendations to right-size pods based on actual usage patterns.
Code Example
# Check native memory tracking kubectl exec -it <pod> -- jcmd 1 VM.native_memory summary # Check cgroup memory usage kubectl exec -it <pod> -- cat /sys/fs/cgroup/memory/memory.usage_in_bytes # Check all container memory in pod kubectl top pod <pod> --containers # Set JVM to use percentage of container limit java -XX:MaxRAMPercentage=70.0 -XX:+UseContainerSupport -jar app.jar
Interview Tip
Interviewers love this question because it tests whether you understand Linux cgroups, JVM memory model, and Kubernetes pod resource management together. Always mention that OOMKill is a kernel-level action on total RSS, not application-level.