rsync only transfers the parts of a file that changed instead of the whole file — how does its delta-transfer algorithm actually detect which bytes changed without both sides needing a full copy of the old file in memory at once, and what happens when that assumption breaks down?

Imagine you and a friend each have a nearly identical thousand-page manuscript, and you want your friend's copy updated to match yours, but you're only allowed to mail a postcard's worth of text at a time. Instead of mailing the whole manuscript, your friend chops their copy into paragraphs, writes a short unique fingerprint for each paragraph, and mails you just the fingerprints. You then scan your own manuscript looking for paragraphs whose fingerprint matches one your friend already has — for those, you just say "keep paragraph 47 as-is" — and only mail the actual text for paragraphs that don't match anything on your friend's list.

This exists because naive file syncing tools re-transfer an entire file whenever any byte changes, which is wasteful for large files with small changes, think a 4GB virtual machine image where only a configuration file inside changed, or a log file that's mostly appended to. rsync's algorithm was designed specifically to make incremental sync bandwidth-efficient over slow or expensive links, and remains valuable today for large datasets over the internet, without requiring a full previous copy of the file to exist somewhere accessible to both machines simultaneously — only checksums need to cross the wire, not file content, for the parts that already match.

The receiving side splits its existing file into fixed-size blocks, commonly a few kilobytes tuned based on file size, and computes two checksums per block: a fast, weak rolling checksum, cheap to recompute as a sliding window moves forward one byte at a time, and a slower, strong checksum to avoid false-positive matches from the weak checksum alone. It sends this list of block checksums to the sender. The sender then scans its own version of the file with a sliding window, computing the same rolling checksum at every byte offset; when a rolling checksum matches one in the received list, it double-checks with the strong checksum before treating it as a real match, then instructs the receiver to copy its existing block instead of transmitting those bytes, and only sends the literal bytes for the byte ranges that never matched anything.

In production, rsync is commonly run over SSH for encrypted transport, with flags like --archive to preserve permissions, timestamps, and symlinks, --checksum to force full-file checksum comparison instead of relying on size and modification time, useful when clocks are unreliable, --bwlimit to cap bandwidth so a large backup sync doesn't saturate a shared link, and --exclude patterns to skip files that shouldn't be synced. What to monitor for large-scale sync jobs: transfer duration trending upward, a sign the delta algorithm is finding fewer matching blocks than expected, often because the destination file layout has drifted more than anticipated, and exit codes, since rsync returns distinct non-zero codes for partial transfer, connection failure, and vanished source files, and treating all non-zero exits identically hides very different failure classes.

The non-obvious gotcha experienced engineers hit is that the delta algorithm's efficiency depends entirely on byte-alignment surviving between versions — if a single byte is inserted or removed near the beginning of a large file rather than bytes just being overwritten in place, every block boundary downstream shifts by that amount, and the rolling checksum's whole purpose is to still find those shifted-but-unchanged blocks by sliding byte-by-byte rather than jumping in fixed increments. This works well for most real-world edits, but for files with structural reformatting, such as a JSON file re-serialized with different whitespace, the byte patterns can differ enough that rsync falls back to transferring nearly the entire file even though the logical content barely changed — a case where the algorithm's block-matching assumptions simply don't hold.