architectmonitoring

How do you systematically debug container host issues using strace, perf, /proc, and dmesg?

⚡

Quick Answer

Follow a top-down approach: dmesg for kernel and hardware events, /proc for process and system state, perf for CPU profiling, and strace for system call tracing. This layered method isolates whether a problem is in the application, kernel, or hardware.

Detailed Answer

Think of diagnosing a car problem. A good mechanic does not start by taking apart the engine. They check the dashboard lights first (dmesg), then read the diagnostic codes (/proc), listen to the engine under load (perf), and only then hook up a scope to a specific sensor (strace). Each tool works at a different level of detail, and using the wrong one first wastes time. The same discipline applies to container host troubleshooting: start broad, then narrow down.

Having a methodology matters because container problems can come from many layers that normal application debugging ignores. A slow payments-api container might be caused by bad application code, a noisy neighbor container hogging shared CPU, a kernel memory reclaim storm, a disk controller running in degraded mode, or a network driver bug. Without a plan, teams chase symptoms: they restart the container (which hides the problem), scale horizontally (which wastes money), or blame the network (the default scapegoat).

The diagnostic flow starts with dmesg and journalctl for kernel ring buffer messages. OOM kills, hardware errors, filesystem corruption, and driver problems show up here first. Run dmesg -T --level=err,warn for timestamped kernel errors and warnings. Next, /proc gives you live process and system state: /proc/[pid]/status shows memory and state, /proc/[pid]/fd shows open file descriptors, /proc/[pid]/stack shows the kernel stack trace (crucial for finding processes stuck waiting on I/O or locks), /proc/meminfo shows system-wide memory details, and /proc/pressure shows PSI (Pressure Stall Information) for CPU, memory, and I/O across the whole system. For container-specific investigation, cgroup files under /sys/fs/cgroup show per-container resource usage, limits, and pressure. Then perf analyzes CPU behavior: perf top shows live CPU hotspots, perf stat counts hardware events (cache misses, context switches, CPU migrations), and perf record/report generates CPU profiles you can turn into flame graphs. Finally, strace traces individual system calls: strace -c shows call counts and time breakdown, strace -T shows per-call latency, and strace -e trace=network filters to just network calls.

At production scale, the critical discipline is correlating timestamps across tools. A container that slows down at 14:32 might correlate with a dmesg OOM kill of a different container at 14:31 (which triggered kernel memory reclaim), a spike in /proc/pressure/memory at 14:31, and increased cache misses in perf starting at 14:32 as surviving containers fight for memory. Without timestamp correlation, each tool tells an incomplete story.

The big gotcha is that strace uses ptrace and adds massive overhead to the traced process (10-100x slowdown for system-call-heavy apps). Running strace on a production container can turn a latency problem into a full outage. For production systems, prefer eBPF-based tracing (bpftrace, bcc tools) which adds nanoseconds of overhead instead of milliseconds. Save strace for development, staging, or containers that are already broken where the extra overhead does not matter. Similarly, perf record with high-frequency sampling can itself cause CPU contention on busy hosts. Use 99 Hz or 999 Hz sampling, not the default, on production systems.

# Step 1: Check kernel messages for hardware/driver/OOM issues dmesg -T --level=err,warn | tail -50 # Shows recent kernel errors with human-readable timestamps dmesg -T | grep -i 'oom\|killed\|error\|fail\|hardware' | tail -20 # Filters for critical events journalctl -k --since '1 hour ago' --priority=warning # Kernel journal for the last hour # Step 2: Check system-wide resource pressure (PSI) cat /proc/pressure/cpu # CPU stall information across all cgroups cat /proc/pressure/memory # Memory stall: if avg10 > 10, system is memory-constrained cat /proc/pressure/io # IO stall: if avg10 > 20, storage is saturated # Step 3: Identify the problem container using cgroup metrics for cg in /sys/fs/cgroup/system.slice/docker-*.scope; do # Iterates all container cgroups name=$(basename $cg) # Container cgroup name mem_current=$(cat $cg/memory.current 2>/dev/null) # Current memory in bytes cpu_pressure=$(cat $cg/cpu.pressure 2>/dev/null | head -1) # CPU pressure echo "$name mem=$(( mem_current / 1048576 ))MB $cpu_pressure" # Reports per-container state done # Step 4: Deep-dive into a specific container process CONTAINER_PID=$(docker inspect --format '{{.State.Pid}}' payments-api) # Gets container init PID cat /proc/$CONTAINER_PID/status | grep -E 'VmRSS|VmSize|Threads|State' # Memory and thread state ls /proc/$CONTAINER_PID/fd | wc -l # Count open file descriptors (high count = possible leak) cat /proc/$CONTAINER_PID/stack # Kernel stack trace (shows if process is stuck in a syscall) cat /proc/$CONTAINER_PID/io # Process IO counters: read_bytes, write_bytes, cancelled_write_bytes # Step 5: CPU profiling with perf (low-overhead production safe) perf stat -e cycles,instructions,cache-misses,context-switches,cpu-migrations \ -p $CONTAINER_PID sleep 10 # Counts hardware events for 10 seconds perf record -F 99 -p $CONTAINER_PID -g -- sleep 30 # Records CPU profile at 99Hz for 30 seconds perf report --stdio --sort=dso,symbol | head -40 # Shows top CPU consumers by library and function # Step 6: Syscall analysis with strace (use in staging, or on already-failing production pods) nsenter -t $CONTAINER_PID -p -n -m -- strace -c -p 1 -f 2>&1 | head -30 # Syscall summary inside container nsenter -t $CONTAINER_PID -p -n -m -- strace -T -e trace=network -p 1 2>&1 | head -20 # Network syscall latency # Step 7: Network diagnostics from the container's network namespace nsenter -t $CONTAINER_PID -n -- ss -tlnp # Listening TCP sockets inside the container nsenter -t $CONTAINER_PID -n -- ss -s # Socket statistics: established, closed, timewait counts nsenter -t $CONTAINER_PID -n -- cat /proc/net/sockstat # Socket memory usage by protocol # Step 8: Disk IO analysis for the container's block device iostat -xz 1 3 # Per-device IO stats: await (latency), %util (saturation), r/s, w/s cat /sys/fs/cgroup/system.slice/docker-${CONTAINER_ID}.scope/io.stat # Per-device IO counters for the container

How do you systematically debug container host issues using strace, perf, /proc, and dmesg?

More Linux questions