An application suddenly begins failing with network connection errors. You check the server's CPU and Memory, and they are both completely idle (under 10%). What do you check next at the OS level?
Quick Answer
With CPU and memory ruled out, the next suspects are kernel-level networking resource limits that don't show up on standard resource dashboards: ephemeral port exhaustion (all local ports in TIME_WAIT from a high connection churn rate), the conntrack table filling up on a server doing NAT or running iptables/nftables rules, and the per-process or system-wide open file descriptor limit (ulimit -n), since every TCP socket consumes a file descriptor and Linux treats socket exhaustion as a resource error independent of CPU or RAM. Checking dmesg for kernel-logged network errors and ss -s for a summary of socket states across the system is the fastest way to identify which of these is actually happening.
Detailed Answer
Picture a busy restaurant that has plenty of empty tables (CPU) and plenty of food in the kitchen (memory), yet the host at the front still turns customers away — because the restaurant has exactly 50 phone lines for reservations, and all 50 are currently tied up with callers who've already hung up but whose lines haven't been released by the phone system yet. The restaurant looks completely idle by every visible measure, yet it's structurally unable to take a single new reservation until some of those stuck lines free up. Network connection failures with idle CPU and memory are almost always this exact shape of problem: a fixed-size resource that has nothing to do with compute capacity has quietly filled up.
Linux was designed with several networking-specific resource ceilings that exist independently of general compute resources, precisely because TCP connection handling has its own lifecycle and bookkeeping that CPU/memory monitoring was never built to surface. The most common one in production is ephemeral port exhaustion: every outbound TCP connection a server initiates (say, an application server making many short-lived calls to a database or an external API) consumes one ephemeral port from a finite range (typically 32768-60999 by default), and a closed connection doesn't immediately free its port — it lingers in the TIME_WAIT state for a configurable duration (default 60 seconds on Linux) specifically to handle any delayed packets from the old connection safely. Under high enough connection churn (many short connections opened and closed per second), a server can generate TIME_WAIT sockets faster than they expire, eventually exhausting the entire ephemeral port range, at which point new outbound connections fail immediately with 'Cannot assign requested address' or a generic connection error, despite the server otherwise being completely idle.
Internally, a second common cause on any server doing NAT, running iptables/nftables with stateful rules, or behind certain load balancer configurations is conntrack table exhaustion: the kernel's connection tracking subsystem maintains an entry for every tracked connection up to net.netfilter.nf_conntrack_max, and once that table fills, the kernel starts dropping new connection attempts at the netfilter layer before they ever reach the application, logging 'nf_conntrack: table full, dropping packet' to dmesg — a message that's easy to miss unless you specifically check kernel logs, since it produces no application-level error message of its own beyond a generic timeout.
A third, related resource ceiling is the file descriptor limit: Linux represents every open socket as a file descriptor, and both a per-process limit (ulimit -n, commonly 1024 by default unless explicitly raised) and a system-wide limit (fs.file-max) cap how many can be open simultaneously — an application under sustained load that isn't closing connections properly (a connection pool leak, or a downstream service that's slow to respond, causing connections to accumulate faster than they're released) can hit its file descriptor ceiling and start failing to open new sockets with 'Too many open files,' again with zero visible CPU or memory pressure, since a file descriptor table entry costs almost nothing in either.
The non-obvious gotcha: these three failure modes look nearly identical from the application's perspective — generic connection errors, timeouts, or 'address already in use' — and none of them show up on a standard CPU/memory/disk dashboard, which is exactly why a team can stare at seemingly perfect infrastructure health graphs while the server is structurally unable to make a new network connection; the fix is to build ss -s, conntrack -S, and file descriptor usage into the same monitoring dashboard as CPU and memory, since 'everything looks idle' is precisely the state these particular failures present as.