A Linux server's load average is 50 but CPU usage shows only 20%. What could cause this and how do you diagnose it?
Quick Answer
High load average with low CPU indicates I/O wait or uninterruptible sleep (D state) processes. Use iostat, vmstat, and ps to identify processes blocked on disk I/O, NFS mounts, or kernel locks.
Detailed Answer
Understanding Load Average
Linux load average counts ALL processes in runnable (R) OR uninterruptible sleep (D) state. Unlike other OSes, Linux includes D-state (I/O waiting) processes in load average.
Root Cause Analysis
1. Check I/O wait: vmstat 1 5 — look at wa (I/O wait %) column 2. Identify D-state processes: ps aux | awk '$8 ~ /D/' — shows processes in uninterruptible sleep 3. Check disk I/O: iostat -xz 1 — look at %util, await, r/s, w/s 4. Check for NFS hangs: mount | grep nfs then nfsstat -c — stale NFS mounts cause D-state processes 5. Check memory pressure: vmstat 1 — high si/so (swap in/out) indicates swapping, which causes I/O wait
Common Causes
- Disk I/O saturation: Too many reads/writes for the disk to handle (check %util in iostat) - NFS mount stale/hung: Network file system timeout causes processes to enter D state indefinitely - Swap thrashing: Not enough RAM, constant page swapping to disk - Filesystem journaling: ext4/XFS journal commits blocking writes during heavy I/O - RAID rebuild: Background RAID reconstruction consuming all disk bandwidth - Kernel lock contention: BKL or filesystem lock contention (rare in modern kernels)
Resolution
- Identify the bottleneck device and the processes causing I/O - For disk: add IOPS (upgrade to SSD/NVMe, increase EBS IOPS) - For NFS: fix the mount, add soft mount option, or migrate to local storage - For swap: add RAM or tune vm.swappiness - For kernel: check dmesg for blocked task warnings
Code Example
# Check load vs CPU usage
uptime
top -bn1 | head -5
# Check I/O wait
vmstat 1 5
# Look at columns: r (runnable), b (blocked), wa (I/O wait)
# Find D-state processes
ps aux | awk '$8 ~ /D/ {print}'
# Check disk I/O per device
iostat -xz 1 3
# High %util + high await = disk saturated
# Check what processes are doing I/O
iotop -oP
# Check for stale NFS mounts
mount -t nfs
df -h # Will hang if NFS is stale
# Check kernel messages for blocked tasks
dmesg | grep -i 'blocked for more than'Interview Tip
This is the classic Google SRE question. The key insight is that Linux load average includes D-state (I/O waiting) processes, unlike other OSes. Always mention vmstat's 'b' column and iostat's '%util' as your first diagnostic steps.