How do you determine whether a process was OOM-killed, and what would you check before it happens again?
Quick Answer
Check `dmesg`/`journalctl -k` for "Out of memory: Killed process" entries and the OOM killer's score dump, confirm via the process exit code (137) or a crash-loop pattern, then look at cgroup/container memory limits and `/proc/<pid>/oom_score` to understand what actually got picked and why.
Detailed Answer
The kernel OOM killer logs directly to the kernel ring buffer, so the first place to confirm an OOM kill (as opposed to an application crash or a manual kill) is dmesg -T | grep -i "out of memory" or journalctl -k --since "-1h" | grep -i oom. The log includes the victim's PID, command, and the OOM score calculation, plus a full memory snapshot at kill time — free memory, per-zone stats, and the score of every candidate process, which tells you why that particular process was chosen over others (it is usually the one with the highest resident memory relative to its oom_score_adj).
At the application layer, an OOM-killed process typically exits with code 137 (128 + SIGKILL) — in Kubernetes this shows up as OOMKilled in the pod status and container exit code, which is a much faster signal than digging through kernel logs if you are running in a cluster. Before it happens again, look at whether the process has an explicit memory limit (cgroup memory.max, a container/pod limit, or a systemd MemoryMax=) that is tighter than its real working set, and whether that working set is growing linearly over time (a leak) or spiking (a burst from a specific request pattern, batch job, or GC not keeping up). Capture a heap/memory profile or a periodic RSS trend (via pidstat -r or cgroup memory.current) proactively so the next kill comes with evidence instead of just a kernel log line.
Code Example
# Confirm and inspect the OOM kill event
dmesg -T | grep -i "out of memory"
journalctl -k --since "-2h" | grep -iE "oom|killed process"
# Kubernetes: fast confirmation without touching the node
kubectl get pod <pod> -o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}'
# Track RSS trend for a specific process before the next kill
pidstat -r -p <pid> 5
# Check current cgroup memory limit and usage
cat /sys/fs/cgroup/memory.max /sys/fs/cgroup/memory.current # cgroup v2
# Per-process OOM score (higher = more likely to be killed)
cat /proc/<pid>/oom_score
cat /proc/<pid>/oom_score_adjInterview Tip
Mention both the kernel-level evidence (dmesg/journalctl OOM log) and the platform-level shortcut (exit code 137 / Kubernetes OOMKilled status) — knowing both shows breadth.