How do you troubleshoot a slow Linux server using top, vmstat, iostat, and sar?

Troubleshooting a slow server is like being a detective at a crime scene. You need a method, not random guessing. Just as a detective secures the scene, interviews witnesses, and collects evidence in order, a good engineer follows a structured approach: get the big picture first, then drill into whichever resource (CPU, memory, disk, network) is the bottleneck. Brendan Gregg's USE method, which checks Utilization, Saturation, and Errors for each resource, gives you that structure.

The first tool to reach for is top (or the nicer htop). It gives you a live dashboard with load averages (1, 5, 15 minutes), a CPU breakdown (user, system, idle, I/O wait, steal), memory and swap usage, and a per-process list you can sort. Load average tells you how many processes are competing for CPU time: a load of 8 on a 4-core machine means processes are queuing up. The 'wa' (I/O wait) percentage is key: high wa means processes are stuck waiting for disk, not burning CPU. The 'st' (steal time) in VMs shows CPU cycles taken by the hypervisor for other VMs. Press 'P' to sort by CPU, 'M' by memory, and '1' to see each core separately.

vmstat (virtual memory statistics) packs system health into a single line that refreshes at whatever interval you set. Important columns include r (processes waiting for CPU), b (processes blocked on I/O), swpd (swap used), si/so (swap in/out per second), bi/bo (disk reads/writes), and the CPU percentages. If r is consistently higher than your core count, you have CPU saturation. If si/so are not zero, you are actively swapping, which causes brutal latency. Run vmstat 1 10 for ten one-second snapshots. The first line is always a since-boot average, so ignore it when looking at current behavior.

iostat (from the sysstat package) focuses on disk I/O. Run iostat -xz 1 to see per-device stats including r/s and w/s (operations per second), rkB/s and wkB/s (throughput), await (average wait time in milliseconds), and %util (how busy the device is). If await is consistently above 10ms on SSDs or 20ms on spinning disks, you have an I/O bottleneck. If %util is near 100%, the device is maxed out. Use iotop alongside it to see which processes are generating the I/O. For NVMe drives, %util can be misleading since these devices handle massive parallelism, so focus on await and queue depth (avgqu-sz) instead.

sar (System Activity Reporter) is the history tool. It collects data every 10 minutes via a cron job and stores it in /var/log/sa/ or /var/log/sysstat/. Use sar -u for past CPU usage, sar -r for memory, sar -b for disk I/O, and sar -n DEV for network stats. This is invaluable for connecting performance drops to specific events, like discovering that CPU spikes match a cron job at 2 AM or a deployment at 3 PM. Data is kept for up to a month by default.

A common trap is fixating on one metric without checking the others. High CPU usage might actually be I/O wait (check wa in top), making it a disk problem, not a CPU problem. A server with high load average but low CPU often has processes stuck waiting on disk or network. Another mistake is not checking for recent changes first. Always run last, history, and check deployment logs before diving deep. In production, setting up continuous monitoring with tools like Prometheus node_exporter and Grafana dashboards prevents the need for manual sar digging and gives you alerts before users notice problems.