What specific rsync exit codes, log fields, and duration metrics should you alert on to catch a failing backup pipeline before a restore is attempted and found to be broken, rather than discovering it during an actual disaster recovery?

This is like a fire extinguisher inspection program that only checks whether the extinguisher is physically present on the wall, never whether it still has pressure or whether anyone knows how to use it — the checklist gets marked "pass" every month right up until the day of an actual fire, when everyone discovers the inspection was checking the wrong thing entirely.

rsync's exit code is the first and cheapest signal, but it's frequently misused as a single boolean of success or failure when it actually carries specific meaning — the rsync manual defines distinct codes for partial transfer due to error, partial transfer due to vanished source files, a timeout waiting for data, and a protocol-level connection failure, among others. Treating a partial-failure code identically to a full connection-failure code means an alerting system either over-pages on minor partial failures or, worse, under-pages by suppressing all non-zero exits as "probably just a few flaky files," missing a real connectivity problem.

A properly instrumented rsync pipeline captures the exit code, parses --stats output, files transferred, total bytes, transfer rate, speedup ratio compared to a full copy, into structured metrics, and records duration per run so anomalies are visible as trend deviations, not just binary success or failure. Beyond the transfer job itself, the only way to actually know a backup is useful is to periodically restore a sample of files, or the whole dataset, into a scratch location and verify checksums against the known-good source — this is fundamentally different information from "the rsync command exited zero," because rsync exiting cleanly only proves the bytes it attempted to send were sent, not that the destination filesystem, storage backend, or retention policy hasn't silently corrupted or deleted them since.

In mature backup operations, teams run three tiers of checks: transfer-level, exit code classification and duration/byte trend alerting on every run, integrity-level, periodic full-content comparison runs since day-to-day syncs typically rely on the faster but less rigorous quick check, and restore-level, scheduled disaster-recovery drills that actually restore data to a scratch environment and validate it's usable, run monthly or quarterly depending on how critical the dataset is. Skipping the restore-level check is the single most common gap, because it's the only one of the three that would have caught a case where backups "succeeded" against a destination that was quietly misconfigured or running out of retained snapshot space.

The non-obvious gotcha is that rsync succeeding with a clean exit code says nothing about whether the destination itself is healthy — a destination filesystem silently running low on inodes, a cloud storage bucket lifecycle policy that's been deleting "old" backup files sooner than intended, or a destination that's actually a stale network mount serving cached directory listings can all make rsync report a clean, fast, successful transfer while writing into a location that isn't actually persisting the data correctly. Alerting purely on the rsync process's own exit code, without independently verifying the destination's actual state, misses exactly this class of failure.