Which MongoDB metrics actually predict replication lag turning into a customer-visible incident, versus metrics that just confirm it already has?

Think of the oplog like a ship's rolling log book that only has room for the last 200 pages — once full, the oldest page is torn out to make room for the newest entry. If a crew member (a secondary node) falls behind reading the log and the ship keeps tearing out old pages, at some point the crew member arrives at a page that's already been destroyed, and they have no way to catch up incrementally anymore — they need an entirely fresh copy of the ship's current state instead.

MongoDB's oplog (operations log) is a capped collection — a fixed-size collection that overwrites its oldest entries once full — that every secondary reads from to replicate changes from the primary. The oplog window is the amount of time, in minutes or hours, that currently sits in the oplog before the oldest entry gets overwritten; it is a moving target that shrinks under high write volume and grows during quiet periods. This design exists because keeping an unbounded log of every write forever would consume unbounded disk, so MongoDB trades log completeness for a rolling window sized to comfortably outlast normal replication lag under expected load.

Replication lag itself — the difference between the primary's latest operation timestamp and a given secondary's last applied operation timestamp — is a lagging measurement in the sense that by the time it's nonzero, replication is already behind, but it's a leading indicator relative to the real failure mode: if replication lag ever exceeds the current oplog window, the lagging secondary can no longer resume from where it left off, because the entries it needs have already been overwritten. At that point, that secondary requires a full initial sync (copying the entire dataset fresh from another member), which can take hours on a large collection and puts additional load on the source node it's syncing from during that time.

In production, mature monitoring tracks all three signals together: oplog window size (in minutes of retained history), replication lag (in seconds), and the ratio between them — because a replica set with a 24-hour oplog window and 30 seconds of lag has enormous headroom, while the same 30 seconds of lag against a 5-minute oplog window is one bad batch job away from a required resync. WiredTiger cache eviction rate is a related leading indicator, since a secondary whose cache is thrashing (evicting and re-reading pages repeatedly) applies oplog entries slower, which is often the actual root cause behind lag that otherwise looks like 'network is slow.'

The gotcha that catches teams sizing their oplog: oplog window shrinks non-linearly during predictable spike events — a nightly bulk import job, a Black Friday traffic surge, a backfill script — and a window that comfortably covers 24 hours of normal traffic can shrink to under 30 minutes during a two-hour bulk write job, and if any secondary was already a few minutes behind going into that window, the bulk job itself can be the direct cause of that secondary needing a full resync, turning a routine batch job into an unplanned, hours-long, high-IO resync event on the exact day the cluster is already under the most load.