Linux

Checking your CPU setup is like looking under the hood of a car before a road trip. You want to know how many cylinders you have, how fast it can go, and whether it has turbo. Linux gives you three main tools to inspect your processors at different levels of detail, and knowing which one to use when saves you time in production.

The go-to tool is lscpu. It pulls data from /sys/devices/system/cpu and /proc/cpuinfo and presents it in a clean, organized format. You will see the architecture (like x86_64 or aarch64), how many CPU sockets the server has, cores per socket, threads per core, speed ranges, the cache layout (L1, L2, L3), and whether virtualization is supported. This is the command you run first when sizing a server or filing a support ticket. The nproc command is even simpler. It just tells you how many processing units are available. This is great in scripts where you need to set how many things run at once, like passing -j$(nproc) to make, or setting worker threads for Nginx or Gunicorn.

Under the hood, all CPU info comes from the kernel detecting processors at boot time. On x86 systems, the kernel uses CPUID instructions to probe each processor and fills in the /proc/cpuinfo virtual file. Each logical processor gets its own block in this file, with details like processor number, vendor, CPU family, model, stepping, microcode version, cache size, and a flags field listing supported instruction sets like sse4_2, avx2, and aes. The sysfs tree under /sys/devices/system/cpu/ adds runtime details like current clock speed, online/offline status, and how cores map to sockets.

In real-world production, knowing your CPU layout matters a lot for performance. On a 4-socket server with 24 cores per socket and hyperthreading, you get 192 logical CPUs. But memory access speed varies based on which NUMA node (memory bank tied to a socket) a process runs on. Tools like numactl and taskset let you pin processes to specific cores so they stay close to their memory. For Kubernetes workloads, CPU limits and requests map to these physical resources, and understanding the difference between real cores and hyperthreads affects how you plan capacity. The lstopo tool from the hwloc package can draw visual maps of your CPU layout.

A common mistake is confusing logical CPUs with physical cores. If nproc says 16, you might only have 8 real cores with hyperthreading turned on. This matters for CPU-heavy work because hyperthreading only gives you about 15-30% more performance, not double. Another trap is in virtual machines, where lscpu might show what the hypervisor presents rather than the real hardware, and CPU steal time (visible in top) tells you when other VMs are competing for your CPU cycles. Always check the flags in /proc/cpuinfo to confirm the server supports specific instruction sets like AVX-512 before deploying software that needs them.

Think of a hard link as two entries in a phone book that both dial the same person. Deleting one entry does not affect the other because both connect directly to the real person. A soft link is more like a sticky note saying 'call this other number.' If that number gets disconnected, the sticky note becomes useless. That is the core difference between these two link types.

A hard link adds another directory entry that maps a filename to an existing inode. The inode is the real data structure on disk that holds file metadata (permissions, ownership, timestamps) and pointers to the actual data. When you create a hard link, the inode's link count goes up by one. Both filenames are equally valid, and there is no 'original' versus 'copy.' A symbolic link (soft link or symlink) is a completely separate file with its own inode. That inode's data contains the text of the target path. When the kernel encounters a symlink, it reads the stored path and redirects to that location, adding a hop of indirection.

Inside the filesystem (ext4, xfs, etc.), hard links work through the inode reference count stored in the inode table. When you run ls -l, the second column shows this count. The file's actual data blocks are only freed when the link count drops to zero AND no running process has the file open. This is why you can delete a log file that a running process still has open and the disk space will not be reclaimed until that process closes the file. For symlinks, the kernel stores the target path either directly inside the inode itself (for short paths, usually under 60 bytes on ext4) or in a separate data block. The kernel follows symlinks automatically during system calls like open(), but provides lstat() and readlink() for inspecting the symlink itself without following it.

In production, both types serve important roles. Hard links are used by package managers like rpm and dpkg to save space when multiple packages include the same file. Backup tools like rsnapshot use hard links to create space-efficient snapshots where unchanged files across backups share the same data. Symlinks are everywhere in deployment setups: /opt/payments-api/current might point to /opt/payments-api/releases/v2.4.1, letting you do atomic deployments by switching the symlink to a new release folder. The /etc/alternatives system on Debian uses symlinks to manage multiple installed versions of the same program.

A key restriction is that hard links cannot cross filesystem boundaries because inode numbers are only unique within one filesystem. Trying to ln across mount points fails with 'Invalid cross-device link.' Hard links to directories are also blocked on most filesystems to prevent loops that would break tools like find and du. Symlinks can become dangling (broken) if the target is moved or deleted, causing confusing 'No such file or directory' errors. In Docker containers, symlinks pointing to absolute paths may resolve differently depending on the container's filesystem layout, leading to subtle deployment bugs. Always use readlink -f to resolve the final target of a symlink chain when debugging.

Creating a process in Linux is like a cell dividing in biology. The parent cell splits into two identical copies via fork(), and then one copy transforms itself into something completely different via exec(). The parent is responsible for acknowledging each child's death by calling wait(). If it does not, the dead child lingers as a zombie, a ghost entry in the system's process table. This two-step create-then-replace model is how every program launches on Linux.

Every process gets a unique Process ID (PID), a number assigned by the kernel from a counter that wraps around after hitting /proc/sys/kernel/pid_max (default 32768, up to about 4 million on 64-bit systems). Each process also records its Parent PID (PPID), forming a tree rooted at PID 1 (usually systemd). The fork() system call creates a child process that is a near-exact copy of the parent, including open files, signal handlers, memory layout, and environment variables. Modern Linux uses copy-on-write (COW), meaning parent and child share the same memory pages until one of them writes something, at which point only the changed page gets copied. After fork(), the child usually calls exec() (one of the execve family), which replaces the entire program in memory with a new one loaded from disk, while keeping the same PID and open files not marked with close-on-exec.

Inside the kernel, each process is represented by a task_struct, a large data structure holding the process state (running, sleeping, zombie, etc.), scheduling info, memory management details, file descriptor table, signal handlers, and credentials. When fork() is called, the kernel actually uses clone() internally with specific flags, allocates a new task_struct, copies or shares resources based on those flags, assigns a new PID, and places the child on the scheduler's run queue. When exec() is called, the kernel loads the new executable (typically an ELF binary), sets up fresh memory regions, maps the program and its shared libraries, and jumps to the new program's starting point.

In production, understanding this lifecycle is critical for debugging service failures. When a process exits, it enters the zombie state (shown as Z in ps output) until its parent calls wait() or waitpid() to collect the exit status. A zombie uses no CPU or memory but holds onto its PID and process table slot. A few zombies are harmless, but a parent that never reaps its children can pile up thousands of them, eventually using up all available PIDs and blocking new processes from starting. This often happens with badly written daemons or init scripts. The fix is to make the parent handle SIGCHLD properly. If the parent itself dies, init/systemd (PID 1) adopts the orphaned children and reaps them. In containers, PID 1 inside the container must handle zombie reaping, which is why lightweight init tools like tini or dumb-init are used in Docker containers.

A common trap is fork bombs, where a process recursively forks without limit, eating up all available PIDs and locking up the system. Production servers should set ulimit -u (max processes per user) to prevent this. Another subtle issue: fork() in a multithreaded program only copies the calling thread, which can leave mutexes in a broken state in the child. That is why it is safer to use posix_spawn() or to call exec() right after fork() in threaded programs. Also, you cannot kill zombie processes with SIGKILL because they are already dead. You have to fix or kill the parent process instead.

Picture virtual memory as each tenant in an apartment building believing they have the whole building to themselves. Each tenant (process) has their own floor plan (virtual address space), but behind the scenes the building manager (kernel) maps their rooms to actual physical spaces and sometimes stores overflow furniture in a basement locker (swap). If the building fills up completely and someone needs more space, the manager has to evict a tenant. That is the OOM killer.

Virtual memory is the foundation of Linux memory management. Every process gets its own virtual address space, usually spanning 128 TB on 64-bit systems. This space is split into sections: code (text), data, heap (which grows upward via brk/mmap), memory-mapped files, and stack (which grows downward). The kernel keeps page tables that translate virtual addresses to physical page frames, and the hardware MMU (Memory Management Unit) does this translation on every memory access. Pages are normally 4 KB, but huge pages (2 MB or 1 GB) can reduce pressure on the TLB (Translation Lookaside Buffer, which is a small cache of recent address translations) for memory-heavy applications like databases. Linux uses demand paging, which means pages are not actually allocated until you touch them, triggering a page fault that the kernel handles by assigning a physical frame.

Under the hood, the kernel organizes physical memory using the buddy allocator, which manages free pages in power-of-two blocks to reduce fragmentation. On top of that, the slab allocator (SLUB in modern kernels) handles small kernel objects like process structures and inodes efficiently. User-space memory is tracked through Virtual Memory Areas (VMAs) in each process's memory descriptor. The kernel also uses the page cache to keep recently read file data in RAM, which is the main reason a busy server's 'free' memory looks low. Linux aggressively caches file data but will give that memory back when applications need it. The kswapd daemon runs in the background, scanning for inactive pages and moving them to swap when free memory drops below set thresholds.

Swap space extends memory beyond physical RAM by using disk as overflow. When the kernel needs to free up RAM, it writes anonymous pages (heap, stack, anything not backed by a file) to the swap partition or swap file. The vm.swappiness setting (0-200, default 60) controls how aggressively the kernel swaps versus reclaiming file cache. In production, setting swappiness too high on a database server causes horrible latency spikes because disk I/O is thousands of times slower than RAM. Many production setups use vm.swappiness=1 or vm.swappiness=10 and rely on having enough RAM. Kubernetes environments often disable swap entirely (required before Kubernetes 1.22) or use newer swap-aware features. The mkswap and swapon commands manage swap devices, and /proc/swaps and free -h show current usage.

The OOM (Out-of-Memory) killer is the kernel's last resort when both RAM and swap are full. It calculates an oom_score for each process based on how much memory it uses, adjusted by oom_score_adj (a value from -1000 to 1000, where -1000 means 'never kill me'). The process with the highest score gets killed with SIGKILL. A tricky part is Linux's memory overcommit policy (vm.overcommit_memory), which lets processes request more memory than actually exists. The malloc() call succeeds, but if the memory is actually used later, the OOM killer can strike at any random moment, not just during allocation. Production systems should tune oom_score_adj to protect critical services: set -999 for your database and monitoring, and leave application servers at the default. Check dmesg or journalctl for OOM events. Another watch-out is transparent huge pages (THP), which can cause latency spikes during memory compaction. Many database vendors recommend disabling THP entirely.

Troubleshooting a slow server is like being a detective at a crime scene. You need a method, not random guessing. Just as a detective secures the scene, interviews witnesses, and collects evidence in order, a good engineer follows a structured approach: get the big picture first, then drill into whichever resource (CPU, memory, disk, network) is the bottleneck. Brendan Gregg's USE method, which checks Utilization, Saturation, and Errors for each resource, gives you that structure.

The first tool to reach for is top (or the nicer htop). It gives you a live dashboard with load averages (1, 5, 15 minutes), a CPU breakdown (user, system, idle, I/O wait, steal), memory and swap usage, and a per-process list you can sort. Load average tells you how many processes are competing for CPU time: a load of 8 on a 4-core machine means processes are queuing up. The 'wa' (I/O wait) percentage is key: high wa means processes are stuck waiting for disk, not burning CPU. The 'st' (steal time) in VMs shows CPU cycles taken by the hypervisor for other VMs. Press 'P' to sort by CPU, 'M' by memory, and '1' to see each core separately.

vmstat (virtual memory statistics) packs system health into a single line that refreshes at whatever interval you set. Important columns include r (processes waiting for CPU), b (processes blocked on I/O), swpd (swap used), si/so (swap in/out per second), bi/bo (disk reads/writes), and the CPU percentages. If r is consistently higher than your core count, you have CPU saturation. If si/so are not zero, you are actively swapping, which causes brutal latency. Run vmstat 1 10 for ten one-second snapshots. The first line is always a since-boot average, so ignore it when looking at current behavior.

iostat (from the sysstat package) focuses on disk I/O. Run iostat -xz 1 to see per-device stats including r/s and w/s (operations per second), rkB/s and wkB/s (throughput), await (average wait time in milliseconds), and %util (how busy the device is). If await is consistently above 10ms on SSDs or 20ms on spinning disks, you have an I/O bottleneck. If %util is near 100%, the device is maxed out. Use iotop alongside it to see which processes are generating the I/O. For NVMe drives, %util can be misleading since these devices handle massive parallelism, so focus on await and queue depth (avgqu-sz) instead.

sar (System Activity Reporter) is the history tool. It collects data every 10 minutes via a cron job and stores it in /var/log/sa/ or /var/log/sysstat/. Use sar -u for past CPU usage, sar -r for memory, sar -b for disk I/O, and sar -n DEV for network stats. This is invaluable for connecting performance drops to specific events, like discovering that CPU spikes match a cron job at 2 AM or a deployment at 3 PM. Data is kept for up to a month by default.

A common trap is fixating on one metric without checking the others. High CPU usage might actually be I/O wait (check wa in top), making it a disk problem, not a CPU problem. A server with high load average but low CPU often has processes stuck waiting on disk or network. Another mistake is not checking for recent changes first. Always run last, history, and check deployment logs before diving deep. In production, setting up continuous monitoring with tools like Prometheus node_exporter and Grafana dashboards prevents the need for manual sar digging and gives you alerts before users notice problems.

Think of a large office building where namespaces are the walls and doors that create private offices. Each tenant sees only their own space and has their own reception desk (hostname), phone system (network), and file cabinets (filesystem). Cgroups are the building management system that controls how much electricity (CPU), water (memory), and elevator bandwidth (I/O) each tenant can use. Containers are simply these two things combined with a filesystem image, wrapped behind a friendly interface like Docker.

Linux namespaces create isolated views of system resources that would normally be shared. There are eight types: PID (each container gets its own process numbering starting from 1), Network (separate network stack with its own interfaces and routing), Mount (isolated filesystem mounts), UTS (separate hostname), IPC (isolated inter-process communication), User (maps container root to an unprivileged host user for rootless containers), Cgroup (isolated view of the cgroup tree), and Time (separate system clock, added in kernel 5.6). When Docker starts a container, it calls clone() with flags like CLONE_NEWPID | CLONE_NEWNET | CLONE_NEWNS to put the new process in fresh namespaces. The unshare() call can move an existing process into new namespaces, and setns() lets a process join existing ones (this is how docker exec works).

Cgroups (control groups) v2 organize processes into a tree structure under /sys/fs/cgroup/. Each node in the tree has controller files that set resource limits. The memory controller (memory.max, memory.high) caps how much memory a group can use and triggers a cgroup-level OOM killer if the limit is exceeded. The CPU controller (cpu.max) throttles CPU time. For example, setting '100000 100000' gives the group 100% of one CPU core. The I/O controller (io.max) limits disk read/write speed and operations per second per device. The PID controller (pids.max) prevents fork bombs inside a container. When a process goes over its cgroup memory limit, the kernel kills a process inside that cgroup, not a random one system-wide like the global OOM killer would. Cgroups v2 (unified hierarchy) replaced v1 (separate hierarchies per controller) and is the default on modern distributions.

In production, knowing these building blocks is essential for debugging container resource issues. When a Kubernetes pod is OOMKilled, it is the cgroup memory controller firing, not the system OOM killer. The limits in your pod spec (like resources.limits.memory: 512Mi) translate directly to memory.max in the container's cgroup. CPU limits map to cpu.max with CFS throttling, and throttling metrics show up in cpu.stat (nr_throttled, throttled_usec). CPU requests map to cpu.weight (v2) or cpu.shares (v1), which provide proportional sharing rather than hard caps. Tools like systemd-cgtop show live cgroup resource usage, and looking directly at /sys/fs/cgroup/ reveals the actual kernel-level constraints.

A big gotcha is that many applications inside containers read /proc/meminfo and /proc/cpuinfo, which show the host's total resources, not the container's limits. This causes JVMs, Node.js, and other runtimes to auto-tune based on host memory, leading to OOM kills when they allocate more than the cgroup allows. Fixes include using tools like LXCFS to provide container-aware /proc files, or setting explicit runtime flags (like Java's -XX:MaxRAMPercentage). Another pitfall is cgroup v1 versus v2: Docker on older kernels uses v1 with separate hierarchies, while newer systems use v2's unified hierarchy, requiring different file paths when debugging. Check which version is active with stat -fc %T /sys/fs/cgroup/ (cgroup2fs means v2).

Think of systemd as an airport control tower managing everything needed to get the airport running. Each unit (service, mount, timer) is a department like baggage handling, fueling, or security. Targets are operational stages: 'pre-flight checks done' (multi-user.target), 'ready for passengers' (graphical.target). The control tower knows the dependencies between departments and starts them in parallel where possible, instead of calling each one sequentially like the old SysV init scripts did.

The basic building block in systemd is the unit, which is any resource systemd manages. There are twelve types: service (daemons), socket (for socket-based activation), target (grouping and sync points), mount (filesystem mounts), automount (mount on demand), timer (scheduled activation, like cron), path (file path monitoring), device (kernel devices), swap (swap space), slice (cgroup resource grouping), scope (externally started processes), and snapshot (saved states). Unit files live in /usr/lib/systemd/system/ (vendor defaults), /etc/systemd/system/ (admin overrides), and /run/systemd/system/ (runtime units). Each file has sections: [Unit] for description and dependencies (After=, Requires=, Wants=), [Service] for how to run it (ExecStart=, Restart=, Type=), and [Install] for boot enablement (WantedBy=).

Internally, systemd runs as PID 1 and builds a dependency graph of all units at boot. A clever feature is socket activation: a socket unit listens on a port, and when a connection comes in, systemd starts the matching service and hands it the connection. This eliminates startup ordering headaches because clients just connect and wait. The Type= directive tells systemd how to track when a service is ready: Type=simple means the main process is the service, Type=forking means it forks and the parent exits, Type=notify means the service sends a 'ready' signal via sd_notify(), and Type=oneshot is for scripts that run once and exit. systemd also puts each service in its own cgroup (system.slice/servicename.service), enabling per-service resource tracking and limits through directives like MemoryMax=, CPUQuota=, and IOWeight= right in the unit file.

In production, systemctl and journalctl are daily tools. Use systemctl start/stop/restart/reload for immediate control, enable/disable for boot persistence, status for a quick health check with recent logs, and list-units/list-unit-files for inventory. To override vendor unit files without editing them directly, use systemctl edit, which creates a drop-in file at /etc/systemd/system/servicename.service.d/override.conf. journalctl reads the binary journal maintained by systemd-journald and offers powerful filtering: journalctl -u payments-api.service for one service, -p err for error-level only, --since '1 hour ago' for time ranges, -f to follow like tail -f, and -o json for structured output. The journal automatically captures stdout/stderr, syslog messages, and kernel messages in one indexed store.

A critical gotcha is the difference between Requires= and Wants=. Requires= means if the dependency fails, your unit fails too. Wants= is a soft dependency that does not drag your unit down. Using Requires= carelessly can create cascading failures where a non-critical dependency takes down your main service. Another common mistake is forgetting to run systemctl daemon-reload after changing unit files; systemd caches the parsed config and will not see changes without it. Services with Restart=always will respawn forever after crashes, which can cause rapid restart loops burning CPU. Use StartLimitIntervalSec= and StartLimitBurst= to cap restarts (for example, 5 restarts within 60 seconds). Finally, journald defaults to storing logs in memory on some distros, meaning logs vanish on reboot unless you create /var/log/journal/ and set Storage=persistent in /etc/systemd/journald.conf.

Think of Linux file permissions like security badges in an office building. Each file is a room with three tiers of access: the room owner (user), the department (group), and everyone else (others). Each tier independently controls whether you can read documents (r), modify them (w), or run procedures (x). Special permissions act like master keys: setuid lets anyone run a program as if they were the owner, setgid makes new files inherit the department's group, and the sticky bit stops people from deleting each other's files in shared spaces.

The basic model uses nine bits in three groups of three: user (owner), group, and others. Each group has read (r/4), write (w/2), and execute (x/1). For regular files, read lets you view content, write lets you change it, and execute lets you run it as a program. For directories, the meanings shift slightly: read lets you list contents (ls), write lets you create or delete entries (but you also need execute), and execute lets you enter the directory (cd) and access files inside it. This directory execute bit is often misunderstood. A directory with r-- lets you list filenames but not access anything in them. chmod sets permissions using either octal (chmod 750 file) or symbolic notation (chmod u+rwx,g+rx,o-rwx file). The umask value (usually 022) controls default permissions for new files by masking bits from 666 (files) or 777 (directories).

Inside the filesystem, permissions are stored in the file's inode as a 16-bit mode field. The lower 12 bits hold the permissions: bits 0-8 for the nine rwx positions and bits 9-11 for the special permissions (sticky, setgid, setuid). When a process tries to access a file, the kernel's permission check compares the process's effective user and group IDs against the file's owner and group. If the user ID matches the owner, the owner bits are checked. If not but a group ID matches, the group bits apply. Otherwise, the 'others' bits are used. Root (or more precisely, the CAP_DAC_OVERRIDE capability) bypasses all of this. Access Control Lists (ACLs), managed with setfacl and getfacl, extend the basic model by letting you set permissions for specific individual users and groups.

In production, good permission management is critical for security. Web servers typically need the www-data user to read app files (644) and traverse directories (755) but should never be able to execute uploaded files. Database data directories should be owned by the database user with tight permissions (700). The setuid bit (chmod 4755) is used by system tools like passwd and sudo. When you run a setuid program, your process runs with the file owner's user ID (usually root), giving it elevated privileges by design. The setgid bit (chmod 2755) on directories causes new files created inside to automatically inherit the directory's group ownership, which is essential for shared project folders where multiple team members need access without manually running chgrp. The sticky bit (chmod 1777) on directories like /tmp stops users from deleting or renaming files they do not own, even if they have write permission on the directory.

A dangerous gotcha is that setuid on shell scripts is ignored by the kernel on most modern systems because of historical race condition exploits. Another common mistake is running chmod -R 777 to 'fix' permission issues, which opens everything to the world and creates serious security holes. In Docker containers, files created by root inside the container show up as root-owned on the host, complicating volume management. The fix is using the --user flag or setting up user namespace remapping. Also, moving files across filesystems with mv changes ownership to your user ID because mv does a copy-and-delete across filesystem boundaries, while mv within the same filesystem just updates directory entries and preserves ownership. Regularly audit setuid binaries with find / -perm -4000 -type f to catch unauthorized privilege escalation.

Think of a large office building where each company has its own reception desk, phone system, and mailroom. From inside their suite, each tenant thinks they have the whole building. But the structure itself, the walls, plumbing, and electrical systems, is shared. Linux namespaces create these virtual suites: each container gets its own view of system resources, but the kernel (the building's infrastructure) is shared and needs other protections.

Namespaces are the foundation that makes containers possible. Before namespaces, every process on a Linux system shared the same PID space, network stack, mount tree, and user database. Namespaces split these global resources into separate instances. When a container runtime like runc creates a container, it calls clone() or unshare() with flags like CLONE_NEWPID, CLONE_NEWNET, CLONE_NEWNS, and CLONE_NEWUSER. Each flag creates a new isolated instance of that resource type for the child process.

PID namespace gives a container its own process numbering starting from PID 1, so the container's main process cannot see or send signals to host processes. Network namespace creates a separate network stack with its own interfaces, routing tables, firewall rules, and port space; virtual ethernet pairs (veth) bridge the container's network to the host. Mount namespace gives the container its own filesystem view: it sees its root filesystem from the container image and any explicitly mounted volumes, but not the host's filesystem or other containers' files. User namespace maps UIDs inside the container to different UIDs on the host, so UID 0 (root) inside the container can actually be UID 100000 on the host, preventing privilege escalation if a process escapes the container.

At production scale on hosts running 50-100 containers, namespace isolation must be paired with cgroups for resource limits, seccomp for system call filtering, AppArmor or SELinux for mandatory access control, and read-only root filesystems. Monitoring should track namespace counts per host (visible in /proc/sys/user/max_*_namespaces), network namespace interface counts, mount point buildup after container restarts, and user namespace mapping configurations. Kubernetes uses all four major namespace types, and understanding them is key for debugging container networking issues, PID-based health checks, and volume mount permissions.

The non-obvious catch is that namespaces do not protect against kernel-level attacks. Every container shares the host kernel, so a kernel vulnerability (like a privilege escalation bug in a system call handler) can break through all namespace boundaries. User namespaces help prevent simple UID 0 escalation but cannot stop kernel exploits. Some resources are not namespaced at all: kernel modules, most /proc/sys tunable settings, and the system clock (until time namespaces arrived in kernel 5.6) are shared across all containers. A container that loads a kernel module or changes a shared sysctl setting affects every other container on the host.

Think of building utility management. In the old system (cgroups v1), electricity, water, and gas each had separate meters and billing departments. A tenant could be on different plans for each utility with no coordination between them. In the new system (cgroups v2), all utilities go through a single management system with one tenant record, coordinated limits, and the ability to detect when a tenant is struggling with any resource.

The move from v1 to v2 happened because v1 had fundamental design problems. In v1, each resource controller (cpu, memory, blkio, cpuset, etc.) had its own separate hierarchy. A process could be in /cgroup/cpu/payments and /cgroup/memory/checkout at the same time, creating inconsistent groupings. Controllers had different naming conventions: memory used memory.limit_in_bytes while CPU used cpu.shares, with different units and behaviors. Some controllers could not work together because they lived on separate hierarchies.

cgroups v2 enforces a single unified tree at /sys/fs/cgroup. Every process belongs to exactly one group, and all enabled controllers apply at that point. Resource files follow a consistent naming pattern: controller.parameter (like memory.max, cpu.weight, io.max). The memory controller uses memory.max for hard limits (the kernel kills processes if exceeded), memory.high as a slowdown point (the kernel throttles allocations before hitting max), and memory.low for best-effort protection from memory reclaim. The CPU controller uses cpu.weight (1-10000, default 100) for proportional CPU sharing and cpu.max for hard bandwidth limits in the format 'quota period' (for example, '200000 100000' means 200ms of CPU every 100ms period, which equals 2 cores). The IO controller uses io.max with per-device read/write bandwidth and IOPS limits.

At production scale, cgroups v2 introduced Pressure Stall Information (PSI), which reports how much time processes in a group spent waiting for CPU, memory, or I/O. This is a game-changer for monitoring because traditional metrics like CPU utilization and memory usage tell you what a container is using but not whether it is struggling. PSI tells you that a container spent 30% of the last 10 seconds stalled on I/O, which directly indicates resource saturation. Kubernetes uses cgroups v2 for memory quality-of-service (throttling via memory.high before OOM), swap management, and smarter eviction decisions.

The tricky part is migration. cgroups v1 and v2 cannot both control the same resource type at the same time. During the transition, some systems run in a hybrid mode where some controllers are on v1 and others on v2, which brings back the coordination problems v2 was built to solve. Container runtimes, systemd, and orchestrators all need to agree on which version to use. On older distributions, switching to v2 requires a kernel boot parameter (systemd.unified_cgroup_hierarchy=1) and checking that all your tools support v2. A misconfigured hybrid setup can result in containers that appear to have no memory limit because the controller is attached to the wrong hierarchy.

Think of a building's smart wiring system. Traditional monitoring means opening walls to install new sensors (kernel modules), which risks damage and needs permits. eBPF is like a modular sensor network already built into the walls: you plug in new sensors (eBPF programs) that a safety inspector (the verifier) checks before turning them on. If a sensor causes problems, it gets removed without touching the walls.

eBPF exists because getting visibility and control inside the kernel used to require either kernel modules (risky, complex, tied to specific kernel versions) or user-space tools reading /proc and /sys (limited and expensive for frequent sampling). eBPF provides a safe, fast middle ground: programs run inside the kernel at native speed but are checked for safety before they load. This enables capabilities that previously required custom kernel development.

Here is how it works internally. An eBPF program is written in C (or Rust via the Aya framework) and compiled to eBPF bytecode. This bytecode is loaded into the kernel through the bpf() system call. The kernel's verifier then does a static analysis to make sure the program will terminate (no infinite loops), will not access invalid memory, stays within the instruction limit (currently about 1 million verified instructions), and only calls approved helper functions. Once verified, the program is compiled to native machine code via JIT and attached to a hook point. Hook points include kprobes (attach to any kernel function), tracepoints (stable pre-defined instrumentation points), XDP (process network packets before the full network stack touches them), and cgroup hooks (resource and access control). eBPF maps provide shared data structures like hash maps, arrays, and ring buffers for passing data between eBPF programs and user-space applications.

At production scale, eBPF powers widely-used tools: bpftrace for ad-hoc tracing, bcc tools for performance analysis, Cilium for container networking, Falco for runtime security detection, and Pixie for application observability. For system call tracing, eBPF programs attach to sys_enter and sys_exit tracepoints to capture every system call with its arguments and return values, replacing strace without the heavy ptrace overhead. For network monitoring, XDP programs process packets at the driver level before the kernel even allocates memory for them, enabling wire-speed packet filtering and DDoS protection. For security, eBPF programs on LSM (Linux Security Module) hooks can block file access, network connections, or privilege changes based on custom rules.

The biggest gotcha is the kernel version dependency problem. eBPF features are added gradually across kernel versions: basic map types need kernel 4.x, BTF (BPF Type Format) for portable programs needs 5.2+, ring buffer maps need 5.8+, and LSM hooks need 5.7+. A program that works on kernel 5.15 might fail to verify on kernel 5.4 because a required helper function does not exist yet. Production environments with mixed kernel versions across nodes create a compatibility headache. Also, eBPF programs on hot paths (like every network packet in XDP or every system call entry) add nanoseconds of latency per run. This is negligible for most workloads but measurable for ultra-low-latency systems processing millions of events per second.

Think of tuning a race car. Factory settings are fine for city driving, but at 200 mph every detail matters: tire pressure, gear ratios, suspension, fuel mixture. Linux kernel defaults are safe for general use, not for maximum performance. High-performance workloads like payment processing, real-time bidding, or database containers need specific kernel adjustments matched to how they consume resources.

Kernel tuning for containers covers four areas: network stack, memory management, CPU scheduling, and I/O. Network-heavy services like API gateways need higher net.core.somaxconn (the listen backlog queue, default 4096 in modern kernels but often not enough during traffic bursts), net.ipv4.tcp_max_syn_backlog (queue for half-open connections), and net.core.netdev_max_backlog (packet queue before the kernel processes them). File descriptor limits (fs.file-max and per-process RLIMIT_NOFILE) must handle containers with thousands of concurrent connections. For Kubernetes hosts, net.ipv4.ip_local_port_range should be widened to 1024-65535 to prevent running out of ephemeral ports.

Huge pages reduce TLB (Translation Lookaside Buffer) misses for memory-heavy workloads. The TLB is a small cache that stores recent virtual-to-physical address translations. With standard 4KB pages, a 64GB database process needs 16 million page table entries. With 2MB huge pages, that drops to 32,768, dramatically reducing TLB pressure. Transparent Huge Pages (THP) try to merge small pages automatically but can cause latency spikes during defragmentation. Database workloads (PostgreSQL, Redis, Elasticsearch) often run better with THP turned off and explicit huge pages allocated via vm.nr_hugepages. In Kubernetes, huge pages are a first-class resource type (hugepages-2Mi, hugepages-1Gi) that pods can request.

NUMA (Non-Uniform Memory Access) awareness is crucial on multi-socket servers. A process running on CPU socket 0 that reads memory attached to socket 1 pays 1.5-2x higher latency. Container runtimes and Kubernetes support CPU pinning through cpuset cgroups and NUMA-aware memory allocation via the Topology Manager. The single-numa-node policy ensures all resources assigned to a pod (CPU, memory, devices) come from the same NUMA node. The CPU Manager's static policy gives exclusive CPU cores to high-priority pods, preventing the scheduler from bouncing processes between cores and trashing CPU caches.

The non-obvious trap is that kernel tuning is not one-size-fits-all. A sysctl value optimized for a 10Gbps network-heavy API gateway on a 96-core server can hurt performance on a 4-core VM running batch jobs. THP improves throughput for large sequential memory allocations but increases tail latency for latency-sensitive services because of compaction stalls. NUMA binding improves steady-state performance but reduces flexibility during load spikes, potentially leaving cores idle on one socket while containers queue up on another. The right approach is to profile first using tools like perf, mpstat, and numastat, then tune based on evidence rather than copying configuration snippets from blog posts.

Think of diagnosing a car problem. A good mechanic does not start by taking apart the engine. They check the dashboard lights first (dmesg), then read the diagnostic codes (/proc), listen to the engine under load (perf), and only then hook up a scope to a specific sensor (strace). Each tool works at a different level of detail, and using the wrong one first wastes time. The same discipline applies to container host troubleshooting: start broad, then narrow down.

Having a methodology matters because container problems can come from many layers that normal application debugging ignores. A slow payments-api container might be caused by bad application code, a noisy neighbor container hogging shared CPU, a kernel memory reclaim storm, a disk controller running in degraded mode, or a network driver bug. Without a plan, teams chase symptoms: they restart the container (which hides the problem), scale horizontally (which wastes money), or blame the network (the default scapegoat).

The diagnostic flow starts with dmesg and journalctl for kernel ring buffer messages. OOM kills, hardware errors, filesystem corruption, and driver problems show up here first. Run dmesg -T --level=err,warn for timestamped kernel errors and warnings. Next, /proc gives you live process and system state: /proc/[pid]/status shows memory and state, /proc/[pid]/fd shows open file descriptors, /proc/[pid]/stack shows the kernel stack trace (crucial for finding processes stuck waiting on I/O or locks), /proc/meminfo shows system-wide memory details, and /proc/pressure shows PSI (Pressure Stall Information) for CPU, memory, and I/O across the whole system. For container-specific investigation, cgroup files under /sys/fs/cgroup show per-container resource usage, limits, and pressure. Then perf analyzes CPU behavior: perf top shows live CPU hotspots, perf stat counts hardware events (cache misses, context switches, CPU migrations), and perf record/report generates CPU profiles you can turn into flame graphs. Finally, strace traces individual system calls: strace -c shows call counts and time breakdown, strace -T shows per-call latency, and strace -e trace=network filters to just network calls.

At production scale, the critical discipline is correlating timestamps across tools. A container that slows down at 14:32 might correlate with a dmesg OOM kill of a different container at 14:31 (which triggered kernel memory reclaim), a spike in /proc/pressure/memory at 14:31, and increased cache misses in perf starting at 14:32 as surviving containers fight for memory. Without timestamp correlation, each tool tells an incomplete story.

The big gotcha is that strace uses ptrace and adds massive overhead to the traced process (10-100x slowdown for system-call-heavy apps). Running strace on a production container can turn a latency problem into a full outage. For production systems, prefer eBPF-based tracing (bpftrace, bcc tools) which adds nanoseconds of overhead instead of milliseconds. Save strace for development, staging, or containers that are already broken where the extra overhead does not matter. Similarly, perf record with high-frequency sampling can itself cause CPU contention on busy hosts. Use 99 Hz or 999 Hz sampling, not the default, on production systems.

Root Cause Analysis

1. Check I/O wait: vmstat 1 5 — look at wa (I/O wait %) column 2. Identify D-state processes: ps aux | awk '$8 ~ /D/' — shows processes in uninterruptible sleep 3. Check disk I/O: iostat -xz 1 — look at %util, await, r/s, w/s 4. Check for NFS hangs: mount | grep nfs then nfsstat -c — stale NFS mounts cause D-state processes 5. Check memory pressure: vmstat 1 — high si/so (swap in/out) indicates swapping, which causes I/O wait

Complete Packet Path

1. NIC receives frame: Hardware NIC receives Ethernet frame, DMA copies to ring buffer in kernel memory, raises IRQ

2. Kernel softirq processing: NAPI polls the ring buffer, creates sk_buff structure, passes up the stack

3. IP layer: Kernel checks destination IP. If it's the host IP with a mapped port, continues processing.

4. Netfilter/iptables PREROUTING: Docker's DNAT rule matches: `` -A DOCKER -p tcp --dport 8080 -j DNAT --to-destination 172.17.0.2:80 ` Packet destination is rewritten from host_ip:8080 to container_ip:80`

5. Routing decision: Kernel routing table sees destination 172.17.0.0/16 → route via docker0 bridge

6. Bridge forwarding: docker0 Linux bridge looks up MAC address for 172.17.0.2 in its FDB (forwarding database), forwards frame to the correct veth interface

7. veth pair: Packet enters vethXXX (host side), appears on eth0 (container side). veth pairs act as a virtual ethernet cable between namespaces.

8. Container network namespace: Packet arrives in the container's isolated network namespace with its own routing table, IP address, and iptables rules

9. TCP/IP processing in container: Kernel (shared kernel, different namespace) delivers to the listening socket

10. Application reads from socket: accept() → read() → application processes the HTTP request

Return path: Response goes back through the same chain in reverse, with SNAT (MASQUERADE) rewriting the source address on outbound.

This is a direct consequence of how the Completely Fair Scheduler's (CFS) bandwidth controller enforces Kubernetes CPU limits via cgroups. A container with resources.limits.cpu: 2 gets a cgroup quota of 200000us per 100000us (100ms) period. If the process is single-threaded-bursty or has a thread pool that spikes briefly, it can consume its entire 200ms-equivalent of CPU time within the first few milliseconds of a 100ms period and then gets throttled for the remainder of that period — even though, averaged over a full second, it never exceeded its nominal limit, and even though the node as a whole is mostly idle. Multi-threaded workloads with limits set just above their typical-but-bursty thread count are especially prone to this because all threads draw from the same shared quota.

Diagnosis: every cgroup exposes cpu.stat with nr_periods, nr_throttled, and throttled_time. A high ratio of nr_throttled/nr_periods combined with low overall node CPU usage is the signature. cAdvisor/kubelet expose this as the Prometheus metric container_cpu_cfs_throttled_periods_total — graphing throttled-periods-ratio per container is the standard way to catch this in production before users report latency spikes.

Fixes: (1) raise the CPU limit so the quota window has more headroom for bursts, (2) remove CPU limits entirely for latency-sensitive workloads and rely on requests for scheduling plus node-level isolation — this is what many SRE teams do for latency-critical services, accepting some risk of noisy-neighbor in exchange for not falsely throttling, (3) for newer kernels, the kernel's CFS quota period-handling has improved (burst feature, cpu.cfs_burst_us in cgroup v2) which allows accumulating unused quota to absorb short bursts.

Think of a library where every book has both a card in the catalog and a spot on the shelf, but the two are managed separately. If a librarian rips the card out of the catalog while a patron is still sitting at a table reading that exact book, the book doesn't magically vanish or come back to the shelf — it's still sitting there, still taking up space in the room, right up until the patron finishes and gets up. Anyone checking the catalog would see no such book listed at all, yet the room is just as full as before, which looks like a contradiction until you realize the catalog and the physical space are two different bookkeeping systems.

Linux filesystems were designed with exactly this separation: an inode represents the actual data blocks on disk and tracks a link count (how many directory entries point to it) and, separately, the kernel tracks how many open file descriptors currently reference that inode across all running processes. rm only ever removes a directory entry, decrementing the link count by one — it does not touch the underlying data blocks directly at all. The kernel's rule is that data blocks are only actually reclaimed when both conditions are true simultaneously: the link count drops to zero AND no process has an open file descriptor referencing that inode. A process that opened the file before you ran rm keeps holding a valid file descriptor to that inode regardless of whether the file still has a name anywhere in the filesystem, so as far as that process (and the kernel's block allocator) is concerned, the file is very much still alive and still consuming its full 20GB.

Internally, this is precisely why an application's logging framework that opens a log file once at startup and keeps writing to that same file descriptor indefinitely creates exactly this trap: an admin (or a naive logrotate configuration without the copytruncate option or without sending the application a reload signal) deletes or rotates the file at the filesystem level, the application keeps writing into what is now an invisible, nameless inode, and df shows no improvement because from the block allocator's perspective, absolutely nothing has changed — the exact same amount of data is still allocated, just no longer reachable by any path a human can type.

At production scale, this is one of the most common on-call disk-full pages precisely because the standard first instinct — 'find the biggest file, delete it' — works perfectly for files nobody has open, and fails silently (no error, just no space freed) for files something does have open, which is disproportionately likely for exactly the largest log files on a busy server, since large log files are large because something is actively, continuously writing to them. The correct diagnostic is lsof +L1 (list open files with a link count of 1 or less, meaning deleted-but-open) or lsof | grep deleted, which surfaces exactly which process ID is holding the phantom file and how large it still is.

The non-obvious gotcha and the actual production-safe fix without a reboot: you do not need to kill or restart the process to reclaim the space immediately if you can instead truncate the file descriptor directly — since /proc/<pid>/fd/<fd> on Linux is a symlink to the actual open file, running something like : > /proc/12345/fd/7 (or cat /dev/null > /proc/12345/fd/7) truncates the underlying data to zero bytes while the process's file descriptor remains valid and open, meaning the application keeps writing new log data from that point forward without ever needing to be restarted or losing its handle — disk space is reclaimed instantly, with zero application downtime, which is why this specific technique is the preferred production fix over restarting a live service just to release disk space.

Picture a busy restaurant that has plenty of empty tables (CPU) and plenty of food in the kitchen (memory), yet the host at the front still turns customers away — because the restaurant has exactly 50 phone lines for reservations, and all 50 are currently tied up with callers who've already hung up but whose lines haven't been released by the phone system yet. The restaurant looks completely idle by every visible measure, yet it's structurally unable to take a single new reservation until some of those stuck lines free up. Network connection failures with idle CPU and memory are almost always this exact shape of problem: a fixed-size resource that has nothing to do with compute capacity has quietly filled up.

Linux was designed with several networking-specific resource ceilings that exist independently of general compute resources, precisely because TCP connection handling has its own lifecycle and bookkeeping that CPU/memory monitoring was never built to surface. The most common one in production is ephemeral port exhaustion: every outbound TCP connection a server initiates (say, an application server making many short-lived calls to a database or an external API) consumes one ephemeral port from a finite range (typically 32768-60999 by default), and a closed connection doesn't immediately free its port — it lingers in the TIME_WAIT state for a configurable duration (default 60 seconds on Linux) specifically to handle any delayed packets from the old connection safely. Under high enough connection churn (many short connections opened and closed per second), a server can generate TIME_WAIT sockets faster than they expire, eventually exhausting the entire ephemeral port range, at which point new outbound connections fail immediately with 'Cannot assign requested address' or a generic connection error, despite the server otherwise being completely idle.

Internally, a second common cause on any server doing NAT, running iptables/nftables with stateful rules, or behind certain load balancer configurations is conntrack table exhaustion: the kernel's connection tracking subsystem maintains an entry for every tracked connection up to net.netfilter.nf_conntrack_max, and once that table fills, the kernel starts dropping new connection attempts at the netfilter layer before they ever reach the application, logging 'nf_conntrack: table full, dropping packet' to dmesg — a message that's easy to miss unless you specifically check kernel logs, since it produces no application-level error message of its own beyond a generic timeout.

A third, related resource ceiling is the file descriptor limit: Linux represents every open socket as a file descriptor, and both a per-process limit (ulimit -n, commonly 1024 by default unless explicitly raised) and a system-wide limit (fs.file-max) cap how many can be open simultaneously — an application under sustained load that isn't closing connections properly (a connection pool leak, or a downstream service that's slow to respond, causing connections to accumulate faster than they're released) can hit its file descriptor ceiling and start failing to open new sockets with 'Too many open files,' again with zero visible CPU or memory pressure, since a file descriptor table entry costs almost nothing in either.

The non-obvious gotcha: these three failure modes look nearly identical from the application's perspective — generic connection errors, timeouts, or 'address already in use' — and none of them show up on a standard CPU/memory/disk dashboard, which is exactly why a team can stare at seemingly perfect infrastructure health graphs while the server is structurally unable to make a new network connection; the fix is to build ss -s, conntrack -S, and file descriptor usage into the same monitoring dashboard as CPU and memory, since 'everything looks idle' is precisely the state these particular failures present as.

On a loaded production host, an unfiltered tcpdump -i any is expensive: it copies every packet through the kernel packet filter and formats output in real time, which can add CPU and I/O pressure exactly when you can least afford it. Narrow the capture as much as possible before you start: bind to the specific interface (-i eth0), apply a BPF filter for the host/port/protocol you care about (host 10.0.1.5 and port 443), and cap the packet snap length with -s 96 (or -s 0 only when you truly need full payloads) so you are not copying full jumbo frames for a header-level investigation.

Always write to a file with -w capture.pcap rather than letting tcpdump format and print packets live — text formatting is far more expensive than writing raw packets to disk, and a pcap file lets you re-filter and analyze offline with Wireshark or tshark without re-capturing. Use -C and -W to rotate capture files so a long-running capture cannot fill the disk, and set a packet count limit (-c 100000) or a time-bounded window when you are debugging a specific incident rather than leaving a capture running indefinitely. If you only need connection setup/teardown behavior, filter on TCP flags instead of capturing full streams.

Resets are directional, so a single capture point only tells you who received the RST, not who sent it. Run tcpdump on both hosts at the same time, filtered to the specific 5-tuple (source IP/port, dest IP/port, protocol) so the two captures are directly comparable, and use -tt for high-resolution timestamps so you can line up events across hosts (NTP-synced clocks matter here). If the RST shows up in the server-side capture as an outbound packet but never appears at all in an in-between capture point (a load balancer, NAT gateway, or firewall host), the reset is being generated downstream of the server and something in the path is tearing down the connection — commonly an idle-timeout on a stateful firewall/LB, a conntrack table eviction, or a security appliance resetting on a signature match.

Also check sequence and ACK numbers around the reset: a RST with a sequence number that does not match the expected next byte suggests a stale/duplicate connection (e.g., after a NAT device reused a source port) rather than an intentional application-level close. Cross-reference the reset timing against connection idle time — if resets cluster right around a round number like 60s, 300s, or 900s of inactivity, suspect an idle timeout on a middlebox rather than the two endpoints themselves.

For connection-level failures (server never responds, or resets immediately), filter on the TCP flags rather than trying to parse TLS: SYN packets with no matching SYN-ACK indicate the server is unreachable or a firewall is dropping the connection silently; a RST shortly after SYN-ACK usually means the listener rejected the connection or an intermediate device is tearing it down. For handshake-level failures (TCP connects fine but TLS negotiation fails), you do not need to decrypt anything — tcpdump can show you how far the handshake gets by packet count and size: a ClientHello followed immediately by a RST or a TLS alert record (visible as a small packet right after the ClientHello) tells you the failure is in TLS negotiation, not the network path. For deeper protocol detail (SNI, cipher suites, alert codes) hand the same capture to tshark -Y tls or Wireshark rather than trying to decode it by hand in tcpdump.

Always correlate against certificate lifecycle: a cluster of resets right after a deploy or at a predictable time each day/week often lines up with a certificate rotation, an expired intermediate cert, or a config reload that briefly serves the wrong cert chain.

Linux load average is often misread as a pure CPU metric. It actually reflects the number of processes in the run queue (runnable, waiting for CPU) plus processes in uninterruptible sleep (D state, usually waiting on disk I/O, NFS, or certain kernel locks). So "high load, low %CPU" is a strong signal that something is stuck on I/O rather than compute-bound.

Start with ps -eo pid,stat,comm | grep " D" to find processes currently in uninterruptible sleep, then check what they are blocked on with cat /proc/<pid>/stack (if available) or ls -l /proc/<pid>/fd to see what files/sockets they hold. Cross-check with vmstat 1 for the b (blocked) column and wa (I/O wait) percentage, and iostat -x 1 to see if a specific block device has high %util or await. On modern kernels, /proc/pressure/io and /proc/pressure/cpu give a direct, load-average-independent view of how much time tasks actually spend stalled on each resource, which is a much cleaner signal than load average alone. Also check for NFS or other network filesystem mounts — a slow or hung NFS server is one of the most common causes of a pile of D-state processes with load average climbing while CPU stays idle.

The kernel OOM killer logs directly to the kernel ring buffer, so the first place to confirm an OOM kill (as opposed to an application crash or a manual kill) is dmesg -T | grep -i "out of memory" or journalctl -k --since "-1h" | grep -i oom. The log includes the victim's PID, command, and the OOM score calculation, plus a full memory snapshot at kill time — free memory, per-zone stats, and the score of every candidate process, which tells you why that particular process was chosen over others (it is usually the one with the highest resident memory relative to its oom_score_adj).

At the application layer, an OOM-killed process typically exits with code 137 (128 + SIGKILL) — in Kubernetes this shows up as OOMKilled in the pod status and container exit code, which is a much faster signal than digging through kernel logs if you are running in a cluster. Before it happens again, look at whether the process has an explicit memory limit (cgroup memory.max, a container/pod limit, or a systemd MemoryMax=) that is tighter than its real working set, and whether that working set is growing linearly over time (a leak) or spiking (a burst from a specific request pattern, batch job, or GC not keeping up). Capture a heap/memory profile or a periodic RSS trend (via pidstat -r or cgroup memory.current) proactively so the next kill comes with evidence instead of just a kernel log line.

Linux aggressively uses spare RAM for page cache (file-backed pages) and buffers, on the principle that unused memory is wasted memory — that cache is reclaimed automatically the instant an application needs it. This means free -h showing very little in the free column is normal and expected on a healthy, long-running Linux box; the column that matters is available, which is the kernel's own estimate of memory that can be given to a new allocation without swapping, already accounting for reclaimable cache and buffers. If available stays roughly stable over days while used grows, you are looking at cache growth, not a leak.

To confirm a real leak, stop looking at system-wide memory and track a specific process: use ps -o rss,vsz -p <pid> or pidstat -r -p <pid> 60 sampled over hours, and watch whether RSS grows monotonically with no corresponding drop after garbage collection (for managed runtimes) or after requests complete. smem or /proc/<pid>/smaps_rollup can break down how much of a process's RSS is private (its own leaked memory) versus shared (mapped libraries, shared cache) — a real leak shows up as private, dirty memory (Private_Dirty) climbing continuously. Also check /proc/meminfo for Slab and SReclaimable vs SUnreclaim — a leak in kernel-level allocations (e.g., a buggy driver or a container leaking dentries/inodes) shows up as SUnreclaim growing rather than in any single process's RSS at all.

Load average counts running + uninterruptible (D-state, usually I/O) processes. Use top/htop for per-process CPU, vmstat/mpstat for run queue and iowait, and iostat for disk. High load with low CPU and high iowait means storage is the bottleneck, not CPU — a common misdiagnosis.

Every open file and socket consumes an FD, capped per process by ulimit -n (and system-wide by fs.file-max). 'Too many open files' means the cap was hit — often a leak (unclosed sockets). Raise the soft/hard limits (limits.conf or the systemd unit LimitNOFILE) and fix the leak; monitor with lsof/ls /proc/<pid>/fd.

ss -tlnp lists listening TCP sockets with the owning process; lsof -i :PORT does the same. For resource hogs, top/htop sort by CPU or memory, and ps aux --sort=-%mem shows the top consumers. These are the first commands in any triage.

VSZ includes mapped-but-unused memory, shared libraries, and reservations, so it overstates real usage. RSS is what's actually resident in RAM (though shared pages are counted per process). For real memory pressure, watch RSS and cgroup memory, not VSZ.

find is exact and always current but scans the disk (slow on large trees), so scope it to a directory when you can (find /etc -name ...). locate queries a prebuilt index (near-instant, but may be stale until updatedb runs). For binaries on your PATH, which nginx is quickest.

The predicates combine: -type f (files only), -size +50M (larger than 50MB), -mtime +30 (modified more than 30 days ago), and -delete removes matches. Always dry-run with -print first to confirm the match set, and prefer logrotate for ongoing log management rather than ad-hoc find deletes.