A production Linux web server is running at 100% disk utilization (df -h shows 100%). You locate and delete a massive 20GB log file using rm -rf. However, running df -h still shows the drive is completely full. Why, and how do you free the space without rebooting the server?

Think of a library where every book has both a card in the catalog and a spot on the shelf, but the two are managed separately. If a librarian rips the card out of the catalog while a patron is still sitting at a table reading that exact book, the book doesn't magically vanish or come back to the shelf — it's still sitting there, still taking up space in the room, right up until the patron finishes and gets up. Anyone checking the catalog would see no such book listed at all, yet the room is just as full as before, which looks like a contradiction until you realize the catalog and the physical space are two different bookkeeping systems.

Linux filesystems were designed with exactly this separation: an inode represents the actual data blocks on disk and tracks a link count (how many directory entries point to it) and, separately, the kernel tracks how many open file descriptors currently reference that inode across all running processes. rm only ever removes a directory entry, decrementing the link count by one — it does not touch the underlying data blocks directly at all. The kernel's rule is that data blocks are only actually reclaimed when both conditions are true simultaneously: the link count drops to zero AND no process has an open file descriptor referencing that inode. A process that opened the file before you ran rm keeps holding a valid file descriptor to that inode regardless of whether the file still has a name anywhere in the filesystem, so as far as that process (and the kernel's block allocator) is concerned, the file is very much still alive and still consuming its full 20GB.

Internally, this is precisely why an application's logging framework that opens a log file once at startup and keeps writing to that same file descriptor indefinitely creates exactly this trap: an admin (or a naive logrotate configuration without the copytruncate option or without sending the application a reload signal) deletes or rotates the file at the filesystem level, the application keeps writing into what is now an invisible, nameless inode, and df shows no improvement because from the block allocator's perspective, absolutely nothing has changed — the exact same amount of data is still allocated, just no longer reachable by any path a human can type.

At production scale, this is one of the most common on-call disk-full pages precisely because the standard first instinct — 'find the biggest file, delete it' — works perfectly for files nobody has open, and fails silently (no error, just no space freed) for files something does have open, which is disproportionately likely for exactly the largest log files on a busy server, since large log files are large because something is actively, continuously writing to them. The correct diagnostic is lsof +L1 (list open files with a link count of 1 or less, meaning deleted-but-open) or lsof | grep deleted, which surfaces exactly which process ID is holding the phantom file and how large it still is.

The non-obvious gotcha and the actual production-safe fix without a reboot: you do not need to kill or restart the process to reclaim the space immediately if you can instead truncate the file descriptor directly — since /proc/<pid>/fd/<fd> on Linux is a symlink to the actual open file, running something like : > /proc/12345/fd/7 (or cat /dev/null > /proc/12345/fd/7) truncates the underlying data to zero bytes while the process's file descriptor remains valid and open, meaning the application keeps writing new log data from that point forward without ever needing to be restarted or losing its handle — disk space is reclaimed instantly, with zero application downtime, which is why this specific technique is the preferred production fix over restarting a live service just to release disk space.