How would you diagnose a server that becomes unresponsive after a sudden increase in disk I/O?
Quick Answer
I would inspect I/O wait, disk saturation, process queues, and the filesystems that are under pressure before changing any service configuration.
Detailed Answer
When a Linux server becomes unresponsive after a burst of disk activity, the first thing I check is whether the machine is actually CPU-bound, memory-bound, or blocked on storage. A sudden increase in disk I/O can slow the kernel, delay scheduling, and make the machine appear hung even though the CPU is still busy. I would begin by looking at iostat, vmstat, and the process table to see whether a single process or a small set of processes is driving the load.
The next step is to identify whether the pressure is coming from logs, a backup job, a database writer, or a package manager. A flood of writes can create a queue in the kernel and lead to long latency for every operation. If the system is running out of available memory as well, the kernel may start swapping aggressively, which compounds the slowdown.
In operations terms, I care about both the immediate mitigation and the underlying cause. I would limit the offending workload if possible, move the hot data to a faster disk or a different mount, and make sure that log rotation, backup schedules, and batch jobs are not colliding. I would also verify whether the workload is generating too much data for the available throughput or whether a filesystem issue such as corruption or a failing disk is making the situation worse.
The subtle pattern here is that a server can seem healthy at the process level while still being effectively unusable because the storage layer is saturated. Senior engineers look at the system as a stack: infrastructure, kernel queues, filesystems, and applications. They understand that the best fix is often to reduce the workload or rebalance the I/O rather than simply restarting services.
What sets strong candidates apart is the ability to translate system symptoms into an action plan. They talk about the ordering of checks, the likely causes, and the tradeoffs of each mitigation instead of giving a generic answer about checking logs.
Code Example
# Inspect I/O wait and disk throughput iostat -x 1 # Identify which processes are consuming the most I/O ps aux --sort=-%cpu | head # Check whether the server is swapping or running out of memory free -h && vmstat 1
Interview Tip
Candidates who answer well explain how they separate transient workload spikes from structural bottlenecks. Interviewers want to hear that the engineer understands the operating system and can reason about I/O, memory, and scheduler behavior under pressure.