MongoDB

Picture a chef working in a kitchen with a small countertop (memory) and a much larger pantry down the hall (disk). As long as all the ingredients for today's menu fit on the counter, everything is fast — grab and go. The moment the menu needs more ingredients than the counter can hold, the chef has to keep walking to the pantry and back mid-recipe, and every dish takes longer, even though the chef is following the exact same recipe (query plan) as before.

MongoDB's default storage engine, WiredTiger, keeps a cache of frequently accessed data and indexes in memory, sized by default to roughly 50% of total RAM minus 1GB (with a floor of 256MB), specifically because leaving the rest of RAM for the OS file system cache and other processes was found to perform better than giving WiredTiger everything. This design assumes the 'hot' working set — the data actually being read and written by current traffic — fits inside that cache; MongoDB was never designed to require your entire dataset to fit in RAM, only the actively used portion of it.

When the working set exceeds the cache, WiredTiger has to evict pages (remove cached data to make room for new reads) using an LRU-like (least recently used) policy, and any subsequent access to evicted data triggers a page fault requiring a read from disk. This happens even for a query that is using a perfectly appropriate index, because the index itself is also subject to caching — a large index that doesn't fit in cache means even the index lookup incurs disk reads, not just the document fetch. This is the key diagnostic distinction from a query-plan problem: explain() still reports IXSCAN with a tight docsExamined-to-nReturned ratio, because the query planner did its job correctly; the slowdown is purely a storage-layer cache-miss cost that explain() doesn't directly surface.

In production, engineers watch wiredTiger.cache statistics from db.serverStatus() — specifically "bytes currently in the cache" against the configured cache size, and "pages read into cache" as a rate — alongside OS-level disk read IOPS and page fault counts. A cluster whose cache hit ratio (tracked in db.serverStatus().wiredTiger.cache) is dropping while query plans remain unchanged is a strong signal of a memory-sizing problem rather than an indexing problem, and the fix is either scaling to a larger instance with more RAM, sharding to spread the working set across more nodes, or reducing the working set itself (e.g., archiving cold data out of the hot collection).

The gotcha that catches teams sizing their clusters: adding more indexes to 'fix' slow queries can make this problem worse, not better, because every index also competes for the same WiredTiger cache space as the actual document data. A collection with five large secondary indexes might have a working set that technically includes documents plus all five indexes, and if that combined footprint exceeds the cache, you can end up in a state where adding the index that was supposed to speed up one query pushes another previously-fast query's data out of cache, trading one latency problem for a different one that shows up on a completely unrelated endpoint.

Imagine a hospital intake system where every new patient is assigned to a bed strictly by arrival timestamp, always filling ward A first, no matter how many other wards are open and empty. No amount of moving already-assigned patients between wards afterward (the balancer) fixes the fact that every single new patient keeps arriving into the same ward, because the assignment rule itself, not the redistribution process, is what's broken.

MongoDB shards a collection by splitting its documents into contiguous chunks based on a shard key's value, and distributes those chunks across shards; the balancer's job is only to migrate existing chunks to even out chunk count, not to change how new documents get assigned to chunks in the first place. If the shard key is monotonically increasing — an ObjectId (which embeds a timestamp) or an auto-incrementing counter — every new document's key value is always higher than every previous one, so it always lands in the single chunk currently representing the 'highest' range, which lives on one shard at a time. This was a known, deliberate tradeoff in range-based sharding: it makes range queries efficient (because a range of values sits contiguously on one shard) at the direct cost of write distribution when the key increases monotonically.

Internally, MongoDB does split an overfull chunk into two once it crosses a size threshold, and the balancer does eventually migrate one of those split chunks to a less-loaded shard — but by the time that migration completes, all of the new incoming writes have already moved on to an even newer chunk, which is still hashing or ranging onto the same original 'hot' shard. The balancer is fixing yesterday's imbalance while today's writes recreate a new one, so the hot shard's disk usage keeps climbing net of whatever the balancer manages to move away.

In production, engineers monitor per-shard disk usage, chunk count per shard, and specifically watch for a shard whose chunk count is roughly proportional to the others but whose disk usage or operation rate is disproportionately higher — that pattern (balanced chunk count, imbalanced load) is the signature of a hot chunk problem rather than a raw chunk-distribution problem, and it means adding more shards won't help until the key itself changes. The standard production fix is either switching the shard key to a hashed shard key (MongoDB hashes the key value before assigning it to a chunk range, spreading monotonic values pseudo-randomly across shards) or choosing a compound shard key that combines a coarser, evenly-distributed field with the monotonic one.

The gotcha that catches teams doing this migration: you cannot change a collection's shard key in place in most MongoDB versions — fixing this requires creating a new, correctly-sharded collection and migrating data into it (or using MongoDB 5.0+'s reshardCollection, which does this online but still consumes significant IO and time proportional to collection size), so 'switch the shard key' is not a quick config change, it's closer to a full data migration that has to be planned with the same care as any other zero-downtime resharding project, including a cutover step for in-flight writes during the migration window.

Think of the oplog like a ship's rolling log book that only has room for the last 200 pages — once full, the oldest page is torn out to make room for the newest entry. If a crew member (a secondary node) falls behind reading the log and the ship keeps tearing out old pages, at some point the crew member arrives at a page that's already been destroyed, and they have no way to catch up incrementally anymore — they need an entirely fresh copy of the ship's current state instead.

MongoDB's oplog (operations log) is a capped collection — a fixed-size collection that overwrites its oldest entries once full — that every secondary reads from to replicate changes from the primary. The oplog window is the amount of time, in minutes or hours, that currently sits in the oplog before the oldest entry gets overwritten; it is a moving target that shrinks under high write volume and grows during quiet periods. This design exists because keeping an unbounded log of every write forever would consume unbounded disk, so MongoDB trades log completeness for a rolling window sized to comfortably outlast normal replication lag under expected load.

Replication lag itself — the difference between the primary's latest operation timestamp and a given secondary's last applied operation timestamp — is a lagging measurement in the sense that by the time it's nonzero, replication is already behind, but it's a leading indicator relative to the real failure mode: if replication lag ever exceeds the current oplog window, the lagging secondary can no longer resume from where it left off, because the entries it needs have already been overwritten. At that point, that secondary requires a full initial sync (copying the entire dataset fresh from another member), which can take hours on a large collection and puts additional load on the source node it's syncing from during that time.

In production, mature monitoring tracks all three signals together: oplog window size (in minutes of retained history), replication lag (in seconds), and the ratio between them — because a replica set with a 24-hour oplog window and 30 seconds of lag has enormous headroom, while the same 30 seconds of lag against a 5-minute oplog window is one bad batch job away from a required resync. WiredTiger cache eviction rate is a related leading indicator, since a secondary whose cache is thrashing (evicting and re-reading pages repeatedly) applies oplog entries slower, which is often the actual root cause behind lag that otherwise looks like 'network is slow.'

The gotcha that catches teams sizing their oplog: oplog window shrinks non-linearly during predictable spike events — a nightly bulk import job, a Black Friday traffic surge, a backfill script — and a window that comfortably covers 24 hours of normal traffic can shrink to under 30 minutes during a two-hour bulk write job, and if any secondary was already a few minutes behind going into that window, the bulk job itself can be the direct cause of that secondary needing a full resync, turning a routine batch job into an unplanned, hours-long, high-IO resync event on the exact day the cluster is already under the most load.

Picture a bank trying to photograph every teller window in a large branch to create a single 'state of the vault' record. If each photo is taken a few seconds apart instead of simultaneously, and money is moving between windows during that gap, the resulting set of photos can show a transaction's withdrawal at one window but not yet its matching deposit at another — a snapshot that looks internally consistent per-photo but describes a moment in time that never actually existed across the whole branch.

In a sharded MongoDB cluster, data lives across multiple independent shards, each potentially its own replica set, and a single logical transaction or a cross-shard read can touch several of them. A backup taken by running mongodump or a filesystem snapshot against each shard independently, at slightly different wall-clock moments, produces a set of per-shard backups that are each locally consistent but collectively describe a cluster state that's inconsistent — some shards reflect a slightly later or earlier moment than others. MongoDB's cluster-wide backup tooling (like MongoDB Atlas's cluster snapshots, or Percona Backup for MongoDB) was built specifically to solve this by coordinating a single logical timestamp across all shards and the config servers, ensuring every shard's snapshot represents the exact same moment.

The correct automated approach captures three things atomically: a snapshot or dump of every shard's data, a snapshot of the config servers (which track which chunks live on which shard — restoring shard data without matching config server state can point queries at the wrong shard for a given key range), and enough oplog history around the snapshot point to allow point-in-time recovery to an exact moment rather than only to the last full snapshot. For non-sharded replica sets, this reduces to coordinating one consistent snapshot per replica set member instead of per shard, but the same core requirement — one consistent point in time — still applies.

In production, restore automation should never restore directly over the live production cluster as its first action; the disciplined pattern is to always restore into a freshly provisioned, isolated cluster first, run automated integrity and row-count validation checks against it, and only then proceed to the actual cutover (which itself should be a separate, explicit, approved step — often involving a brief write freeze and a DNS or connection-string switch, not an in-place overwrite). Engineers track restore drill frequency and restore time objective (RTO) as operational metrics, not just backup success rate, because a backup that has never been test-restored is not a verified backup, it's an unverified file.

The gotcha most teams learn only during an actual incident: restoring a sharded cluster's data without also restoring the config server metadata to the exact matching point in time can succeed technically — the restore command reports success and the cluster comes up — while silently returning wrong or incomplete query results, because the config servers' chunk-to-shard mapping no longer matches where the data actually landed. This kind of failure doesn't throw an error; it just quietly returns fewer documents than expected for range queries that now believe certain chunk ranges live on shards that no longer have that data, which can go unnoticed for days until someone notices a report total looks slightly off.

Think of embedding like stapling a customer's entire receipt history into one folder, versus referencing like keeping a customer file that just lists ticket numbers pointing to receipts stored in a separate filing cabinet. A stapled folder is fast to grab because everything is in one place. But if that customer generates thousands of receipts — a subscription billing daily for ten years — the folder becomes unmanageably thick, slows down every clerk who lifts it, and eventually can't fit in the drawer at all.

MongoDB stores related data as a single BSON document instead of joining across tables at query time like a relational database, and it enforces a hard 16MB per-document size limit (BSON is the binary JSON-like format MongoDB stores documents in) specifically to keep documents fitting comfortably in memory and network payloads. Embedding was designed for the common case where an application always reads a parent with its children in one request — an order and its line items — because a single document read is one disk seek and one round trip, versus a reference model's multiple round trips or a $lookup join. Referencing exists for the opposite case: data accessed independently of its parent, or that would make the parent balloon without bound.

When you embed, MongoDB stores the child data as a nested array inside the parent's BSON document. Every write to that document — even one touching a single embedded item — requires the storage engine to read and rewrite the relevant parts of the whole document, so a document with a 50,000-element array pays a growing tax on every write, not just reads. Referencing instead stores just an ObjectId (MongoDB's unique document identifier) pointing to another collection, and the application or an aggregation $lookup stage does a second query to join them; this keeps each document small and its write cost constant regardless of how many children exist, at the cost of an extra index and round trip.

In production, teams use the bucket pattern for genuinely high-write, time-growing data like IoT readings or activity events: instead of one document per device with an ever-growing array, they create a new document every N items or every time window (one document per device per hour), each with a bounded embedded array. This caps document size while keeping writes append-only. Engineers monitor average and p99 document size via db.collection.stats(), watch for documents approaching the 16MB ceiling well before they hit it, and alert on write latency creeping up on documents with large embedded arrays, since that's usually the earliest symptom of a design that's aging badly.

The gotcha that bites teams who designed correctly on day one: 'bounded' arrays are often only bounded by current product usage, not by anything structural. An order.line_items array is fine at 5-20 items until a bulk-ordering feature ships and some orders have 3,000 items, or a user.notifications array grows forever because nobody added a cleanup job. The array being unbounded doesn't throw an error until the exact moment a document crosses 16MB, so the failure is invisible in code review and in most of production, then appears suddenly as a hard write rejection on exactly the customer with the most data — usually your biggest account.

Picture a phone book sorted by last name. If you search for everyone with last name 'Chen' born after 1990, using the phone book (the index) is obviously faster than reading every entry in the city (a collection scan). But if the phone book is instead sorted by birth year first, you'd have to flip through every single page checking for 'Chen' on each one — you're still using the book, just not efficiently, because the index's sort order doesn't match how you're actually searching.

MongoDB's explain("executionStats") reports a stage called IXSCAN when the query planner chose to walk an index rather than scan the whole collection (COLLSCAN). Teams often stop there, assuming 'IXSCAN means it's fixed,' but the real signal for efficiency is the ratio between totalDocsExamined (how many documents the index scan pulled up as candidates) and nReturned (how many of those actually matched the full query and were returned). A healthy index keeps that ratio close to 1:1; a ratio of 500:1 means the index narrowed the search barely at all before MongoDB had to fetch and manually filter hundreds of documents per result.

This mismatch happens because compound indexes (indexes on multiple fields) are ordered structures — MongoDB can only use an index's sort order left-to-right, meaningfully filtering on the leading fields and merely following the rest as a means to a sort or avoid a separate sort pass. This is why the ESR rule exists: put Equality-match fields first (so MongoDB jumps straight to the matching range), then Sort fields (so results already come out in the needed order without a separate in-memory sort), then Range fields last (since range conditions can only narrow, not jump to, a position in the index). An index built in the wrong order — say, range field first — forces MongoDB to scan a huge swath of the index checking each entry individually.

In production, engineers watch db.currentOp() and the slow query log (via profiling level 1 or 2) for exactly this totalDocsExamined-to-nReturned gap, not just the presence of COLLSCAN, because dashboards that only alert on 'any collection scan' miss the more common production issue of a technically-indexed but poorly-ordered query silently burning CPU and disk IO under real traffic. Index selectivity (how much a field narrows the result set) also matters — an index on a boolean 'is_active' field barely helps if 95% of documents share the same value, since the planner still has to examine a huge fraction of the index.

The gotcha that catches experienced engineers too: MongoDB's query planner caches the winning query plan and reuses it for subsequent similar queries, so a plan that was efficient when a collection was small can keep getting reused even after data distribution changes dramatically (for example, once 90% of orders become status: "completed" instead of an even split) — until the plan cache is invalidated by a write threshold or a manual db.collection.getPlanCache().clear(), the query keeps running the now-inefficient plan without any error, just quietly worse latency.

Imagine a manager who approves an expense reimbursement the instant they read the request, before the finance team has actually recorded it in the ledger. If the manager gets hit by a bus that afternoon and someone else takes over with only the finance team's ledger to go on, the approval that only existed in the manager's head is simply gone — the new manager has no way to know it ever happened, and there's no record to roll back because there was never a shared record in the first place.

MongoDB replica sets have one primary node accepting all writes and multiple secondaries replicating from it asynchronously via the oplog (operations log, a capped collection recording every write for secondaries to replay). Write concern controls how many nodes must confirm a write before MongoDB reports success back to the application: w:1 (the historical default in some driver configurations) means only the primary needs to have it in memory; w:majority means a majority of voting members must have replicated it. This tiered design exists because forcing every write to wait for full replication would hurt latency, so MongoDB lets applications choose their durability-versus-speed tradeoff per operation.

When a primary fails, the remaining secondaries detect the loss via missed heartbeats and hold an election, using the same core idea as other consensus protocols: the candidate whose oplog is most caught-up (compared via each secondary's last applied operation) has the best chance of winning, and a majority of voting members must agree. If the old primary had accepted a w:1 write that no secondary had replicated yet, that write simply never existed anywhere except the now-dead primary's local state — the newly elected primary has no way to know about it, and the cluster moves forward without it.

In production, this becomes actively destructive when the old primary rejoins the replica set after recovering: MongoDB runs a rollback procedure that finds where the rejoining node's oplog diverges from the new primary's history and discards the divergent writes from the old primary, writing them to a rollback file on disk rather than replaying them, specifically to keep all replica set members consistent. Engineers monitor replication lag (how far behind secondaries are), oplog window (how much time of history the oplog currently holds before it starts overwriting itself), and election frequency, because a replica set with growing lag is both more likely to lose more data on a failover and more likely to hit rollback conflicts.

The gotcha many teams learn only after an incident: switching every write to w:majority does eliminate this class of data loss, but it does not eliminate the underlying election pause — writes are still blocked for the 2-12 second window while a new primary is elected, and application code that isn't using retryable writes (a driver feature that automatically retries a write once against the new primary) will see hard errors during that window rather than a graceful delay, which looks identical to a data-loss bug from the application's error logs until you correlate the timestamps against the replica set's election log.

Think of an aggregation pipeline like an assembly line sorting mail. If you make every single piece of mail in the country pass through the 'sort by zip code' station before you even filter out mail that isn't headed to your region, you've wasted enormous effort sorting things you were going to throw away anyway. The efficient approach is to filter out everything irrelevant at the very first station, so every downstream station only ever touches the mail that actually matters.

MongoDB's aggregation pipeline processes documents through an ordered sequence of stages, and only the first stage in the pipeline can use a collection index efficiently — after that, documents are already in an in-memory working set being transformed stage by stage. Each individual stage (other than a handful of exceptions) is also subject to a 100MB memory limit; if a single $group or $sort stage needs to hold more than 100MB of intermediate data in RAM, MongoDB throws an error unless allowDiskUse is enabled, which lets that stage spill to temporary files on disk. The pipeline model was designed this way so each stage can be reasoned about independently, but that same modularity means a badly-ordered pipeline pays the full cost of every stage on the full dataset rather than a shrinking one.

The practical failure pattern: a pipeline that starts with $group (aggregating totals across an entire multi-million-document collection) before ever applying a $match filtering down to, say, the last 24 hours of orders, forces that $group stage to build accumulator state across every document in the collection. If that intermediate state exceeds 100MB — which happens quickly when grouping by a high-cardinality field like customer_id across millions of orders — the stage fails outright in strict mode, or silently spills to slow disk-based temp files if allowDiskUse is on, either way at far higher cost than if $match had run first and used an index to narrow the working set from millions of documents to a few thousand.

In production, teams profile pipelines using db.collection.explain() on the aggregate call (not just find queries) to see whether early stages report IXSCAN and how many documents survive each stage, and they add a $project immediately after the initial $match to drop unneeded fields before expensive stages like $group or $sort run, shrinking the per-document memory footprint carried through the rest of the pipeline. allowDiskUse is treated as an emergency valve for pipelines that must legitimately process large intermediate result sets (like a full monthly reporting rollup), not a default flag to slap on every pipeline to paper over bad ordering, because disk-spilled stages run dramatically slower under real traffic than the same pipeline properly filtered upfront.

The gotcha that surprises even experienced engineers: only the very first stage of a pipeline can use an index, and only if it's a $match (or certain $sort placements) with no preceding transformation — inserting even a harmless-looking $addFields before your $match silently disables the index for that $match, since MongoDB no longer sees an untouched collection scan target at that point in the pipeline. Staging environments with small seeded datasets never surface this because the 100MB limit and index-usage difference are invisible at low document counts, which is exactly why this class of bug reliably passes code review and staging tests, then fails only under production data volume.