rsync

Imagine you and a friend each have a nearly identical thousand-page manuscript, and you want your friend's copy updated to match yours, but you're only allowed to mail a postcard's worth of text at a time. Instead of mailing the whole manuscript, your friend chops their copy into paragraphs, writes a short unique fingerprint for each paragraph, and mails you just the fingerprints. You then scan your own manuscript looking for paragraphs whose fingerprint matches one your friend already has — for those, you just say "keep paragraph 47 as-is" — and only mail the actual text for paragraphs that don't match anything on your friend's list.

This exists because naive file syncing tools re-transfer an entire file whenever any byte changes, which is wasteful for large files with small changes, think a 4GB virtual machine image where only a configuration file inside changed, or a log file that's mostly appended to. rsync's algorithm was designed specifically to make incremental sync bandwidth-efficient over slow or expensive links, and remains valuable today for large datasets over the internet, without requiring a full previous copy of the file to exist somewhere accessible to both machines simultaneously — only checksums need to cross the wire, not file content, for the parts that already match.

The receiving side splits its existing file into fixed-size blocks, commonly a few kilobytes tuned based on file size, and computes two checksums per block: a fast, weak rolling checksum, cheap to recompute as a sliding window moves forward one byte at a time, and a slower, strong checksum to avoid false-positive matches from the weak checksum alone. It sends this list of block checksums to the sender. The sender then scans its own version of the file with a sliding window, computing the same rolling checksum at every byte offset; when a rolling checksum matches one in the received list, it double-checks with the strong checksum before treating it as a real match, then instructs the receiver to copy its existing block instead of transmitting those bytes, and only sends the literal bytes for the byte ranges that never matched anything.

In production, rsync is commonly run over SSH for encrypted transport, with flags like --archive to preserve permissions, timestamps, and symlinks, --checksum to force full-file checksum comparison instead of relying on size and modification time, useful when clocks are unreliable, --bwlimit to cap bandwidth so a large backup sync doesn't saturate a shared link, and --exclude patterns to skip files that shouldn't be synced. What to monitor for large-scale sync jobs: transfer duration trending upward, a sign the delta algorithm is finding fewer matching blocks than expected, often because the destination file layout has drifted more than anticipated, and exit codes, since rsync returns distinct non-zero codes for partial transfer, connection failure, and vanished source files, and treating all non-zero exits identically hides very different failure classes.

The non-obvious gotcha experienced engineers hit is that the delta algorithm's efficiency depends entirely on byte-alignment surviving between versions — if a single byte is inserted or removed near the beginning of a large file rather than bytes just being overwritten in place, every block boundary downstream shifts by that amount, and the rolling checksum's whole purpose is to still find those shifted-but-unchanged blocks by sliding byte-by-byte rather than jumping in fixed increments. This works well for most real-world edits, but for files with structural reformatting, such as a JSON file re-serialized with different whitespace, the byte patterns can differ enough that rsync falls back to transferring nearly the entire file even though the logical content barely changed — a case where the algorithm's block-matching assumptions simply don't hold.

This is like a mail-forwarding service that's supposed to only forward new letters, but suddenly every previous letter you ever received starts getting re-forwarded too — either someone re-addressed the entire filing cabinet by changing all the metadata, someone dumped a huge new stack of mail into the cabinet, which is real data growth, or the delivery truck itself is just moving slower than usual due to network degradation even though the amount of mail is normal.

rsync decides whether to skip a file entirely by default using a "quick check" of file size and modification time — if those match between source and destination, it assumes the file is identical and skips it without even reading the contents. This default exists because reading and checksumming every file's contents on every run would be extremely slow for routine incremental backups, so rsync optimizes for the common case where files that haven't been touched also haven't changed. But this same optimization is exactly why a bulk metadata change, such as an anti-virus scan that touches file access times, a permissions-fixing script, or a filesystem migration that resets modification times, can silently defeat the quick check and force rsync to re-transfer the entire tree even though the actual file content is unchanged.

To diagnose live without killing the job, first check what rsync is currently transferring using progress output or by inspecting the process's open file descriptors, and compare the current transfer rate against the historical baseline — a rate that's roughly normal but running for way longer points to volume growth, while a rate that's crawling points to network or disk I/O contention. Next, independently compare source and destination directory sizes to check whether the source genuinely grew, which rules in or out the new-or-large-files theory. Finally, spot-check a handful of files that should have been skipped, old, untouched files, to see if their modification time actually changed recently on the source side, which would confirm the quick check was defeated by a metadata-wide change rather than real data growth.

In production, mature backup pipelines log per-run summary stats, files transferred, bytes transferred, duration, so that three hours instead of ten minutes is immediately visible as an anomaly against history rather than something a human notices only when the disk fills up. Teams should also alert on destination disk usage independently from job duration, since a job that's still running normally but transferring far more data than usual can fill a disk well before the job itself would be flagged as abnormally slow. Using --stats on every run and shipping that output to a metrics system turns rsync into an observable pipeline component instead of a black box.

The non-obvious gotcha is that killing a long-running rsync job mid-transfer is often safe for the destination, since rsync uses temporary files and atomic renames by default, so a killed job leaves the destination in its last-known-good state rather than corrupted, but only if you haven't used --inplace, which writes directly to the destination file instead of a temp file for performance reasons — under --inplace, killing the job mid-write can leave a partially-overwritten, corrupted destination file, which is a dangerous trade-off many engineers make for speed without realizing the safety cost.

This is like a fire extinguisher inspection program that only checks whether the extinguisher is physically present on the wall, never whether it still has pressure or whether anyone knows how to use it — the checklist gets marked "pass" every month right up until the day of an actual fire, when everyone discovers the inspection was checking the wrong thing entirely.

rsync's exit code is the first and cheapest signal, but it's frequently misused as a single boolean of success or failure when it actually carries specific meaning — the rsync manual defines distinct codes for partial transfer due to error, partial transfer due to vanished source files, a timeout waiting for data, and a protocol-level connection failure, among others. Treating a partial-failure code identically to a full connection-failure code means an alerting system either over-pages on minor partial failures or, worse, under-pages by suppressing all non-zero exits as "probably just a few flaky files," missing a real connectivity problem.

A properly instrumented rsync pipeline captures the exit code, parses --stats output, files transferred, total bytes, transfer rate, speedup ratio compared to a full copy, into structured metrics, and records duration per run so anomalies are visible as trend deviations, not just binary success or failure. Beyond the transfer job itself, the only way to actually know a backup is useful is to periodically restore a sample of files, or the whole dataset, into a scratch location and verify checksums against the known-good source — this is fundamentally different information from "the rsync command exited zero," because rsync exiting cleanly only proves the bytes it attempted to send were sent, not that the destination filesystem, storage backend, or retention policy hasn't silently corrupted or deleted them since.

In mature backup operations, teams run three tiers of checks: transfer-level, exit code classification and duration/byte trend alerting on every run, integrity-level, periodic full-content comparison runs since day-to-day syncs typically rely on the faster but less rigorous quick check, and restore-level, scheduled disaster-recovery drills that actually restore data to a scratch environment and validate it's usable, run monthly or quarterly depending on how critical the dataset is. Skipping the restore-level check is the single most common gap, because it's the only one of the three that would have caught a case where backups "succeeded" against a destination that was quietly misconfigured or running out of retained snapshot space.

The non-obvious gotcha is that rsync succeeding with a clean exit code says nothing about whether the destination itself is healthy — a destination filesystem silently running low on inodes, a cloud storage bucket lifecycle policy that's been deleting "old" backup files sooner than intended, or a destination that's actually a stale network mount serving cached directory listings can all make rsync report a clean, fast, successful transfer while writing into a location that isn't actually persisting the data correctly. Alerting purely on the rsync process's own exit code, without independently verifying the destination's actual state, misses exactly this class of failure.

A naive backup setup that just mirrors production into one destination folder is like having a single photocopier that instantly shreds the old copy the moment you photocopy a new page — if someone tears a page out of the original book by accident, your "backup" copier dutifully shreds that page from its copy too, because it was only ever designed to make the copy match the original exactly, not to remember history.

rsync --delete exists to keep a destination as an exact mirror of the source, which is genuinely useful for things like mirroring a website or a build artifact directory where you want the destination to reflect exactly what's on the source, nothing more and nothing less. But when the same flag is used for what's supposed to be a backup, a copy meant to survive mistakes made on the source, it defeats the entire purpose — a backup that mirrors the source in real time, including deletions, is not a backup against accidental deletion at all, only a backup against total destination loss.

A safer automation design uses rsync --link-dest pointed at the previous backup directory to create space-efficient historical snapshots: each run creates a new destination directory, and for any file that hasn't changed since the last snapshot, rsync creates a hardlink to the existing file in the previous snapshot directory instead of copying it again, so unchanged files cost zero additional disk space while still being independently protected per snapshot generation. Before any run that would use --delete, for cases where mirroring truly is intended, like a CDN origin sync, the automation should first run the exact same command with --dry-run and diff the planned deletions against an expected threshold — if a routine sync is suddenly about to delete 40 percent of files instead of the usual near-zero, that's a signal something upstream is wrong and the run should abort rather than proceed.

In production, the automation itself should be idempotent, safe to re-run after a partial failure without duplicating work or corrupting state, which for rsync is largely free since re-running a sync against a partially-completed destination just resumes where it left off. What needs explicit engineering is bounding blast radius: cap the maximum percentage of files a single run is allowed to delete or overwrite before requiring manual confirmation, retain a fixed number of historical snapshot generations with automatic pruning of the oldest rather than unlimited growth, and emit a structured audit record per run, source path, destination path, files added/changed/deleted, byte counts, exit code, so a security or platform team can reconstruct exactly what happened to any given file across time.

The non-obvious gotcha is that --link-dest snapshots only protect against file content being lost — if an attacker or a bad script targets the backup destination itself rather than the source, all those hardlinked snapshot directories share the same underlying inode data for unchanged files, so a deletion or ransomware-style encryption pass that touches the shared file across snapshots can silently corrupt what looks like multiple independent historical backups simultaneously. True protection against destination-side compromise requires either read-only or immutable storage for older snapshots, filesystem-level immutability flags or object storage with versioning and write-once-read-many policies, or replicating snapshots to a genuinely separate, less-privileged destination, not just relying on directory-level snapshot naming for safety.