During a MongoDB replica set failover, why can writes that the application already considers 'successful' disappear after the new primary is elected?

Imagine a manager who approves an expense reimbursement the instant they read the request, before the finance team has actually recorded it in the ledger. If the manager gets hit by a bus that afternoon and someone else takes over with only the finance team's ledger to go on, the approval that only existed in the manager's head is simply gone — the new manager has no way to know it ever happened, and there's no record to roll back because there was never a shared record in the first place.

MongoDB replica sets have one primary node accepting all writes and multiple secondaries replicating from it asynchronously via the oplog (operations log, a capped collection recording every write for secondaries to replay). Write concern controls how many nodes must confirm a write before MongoDB reports success back to the application: w:1 (the historical default in some driver configurations) means only the primary needs to have it in memory; w:majority means a majority of voting members must have replicated it. This tiered design exists because forcing every write to wait for full replication would hurt latency, so MongoDB lets applications choose their durability-versus-speed tradeoff per operation.

When a primary fails, the remaining secondaries detect the loss via missed heartbeats and hold an election, using the same core idea as other consensus protocols: the candidate whose oplog is most caught-up (compared via each secondary's last applied operation) has the best chance of winning, and a majority of voting members must agree. If the old primary had accepted a w:1 write that no secondary had replicated yet, that write simply never existed anywhere except the now-dead primary's local state — the newly elected primary has no way to know about it, and the cluster moves forward without it.

In production, this becomes actively destructive when the old primary rejoins the replica set after recovering: MongoDB runs a rollback procedure that finds where the rejoining node's oplog diverges from the new primary's history and discards the divergent writes from the old primary, writing them to a rollback file on disk rather than replaying them, specifically to keep all replica set members consistent. Engineers monitor replication lag (how far behind secondaries are), oplog window (how much time of history the oplog currently holds before it starts overwriting itself), and election frequency, because a replica set with growing lag is both more likely to lose more data on a failover and more likely to hit rollback conflicts.

The gotcha many teams learn only after an incident: switching every write to w:majority does eliminate this class of data loss, but it does not eliminate the underlying election pause — writes are still blocked for the 2-12 second window while a new primary is elected, and application code that isn't using retryable writes (a driver feature that automatically retries a write once against the new primary) will see hard errors during that window rather than a graceful delay, which looks identical to a data-loss bug from the application's error logs until you correlate the timestamps against the replica set's election log.