A MongoDB aggregation pipeline works fine in staging but fails with an out-of-memory-style error in production — what stage ordering mistake usually causes this, and how do you fix it without just enabling allowDiskUse everywhere?

Think of an aggregation pipeline like an assembly line sorting mail. If you make every single piece of mail in the country pass through the 'sort by zip code' station before you even filter out mail that isn't headed to your region, you've wasted enormous effort sorting things you were going to throw away anyway. The efficient approach is to filter out everything irrelevant at the very first station, so every downstream station only ever touches the mail that actually matters.

MongoDB's aggregation pipeline processes documents through an ordered sequence of stages, and only the first stage in the pipeline can use a collection index efficiently — after that, documents are already in an in-memory working set being transformed stage by stage. Each individual stage (other than a handful of exceptions) is also subject to a 100MB memory limit; if a single $group or $sort stage needs to hold more than 100MB of intermediate data in RAM, MongoDB throws an error unless allowDiskUse is enabled, which lets that stage spill to temporary files on disk. The pipeline model was designed this way so each stage can be reasoned about independently, but that same modularity means a badly-ordered pipeline pays the full cost of every stage on the full dataset rather than a shrinking one.

The practical failure pattern: a pipeline that starts with $group (aggregating totals across an entire multi-million-document collection) before ever applying a $match filtering down to, say, the last 24 hours of orders, forces that $group stage to build accumulator state across every document in the collection. If that intermediate state exceeds 100MB — which happens quickly when grouping by a high-cardinality field like customer_id across millions of orders — the stage fails outright in strict mode, or silently spills to slow disk-based temp files if allowDiskUse is on, either way at far higher cost than if $match had run first and used an index to narrow the working set from millions of documents to a few thousand.

In production, teams profile pipelines using db.collection.explain() on the aggregate call (not just find queries) to see whether early stages report IXSCAN and how many documents survive each stage, and they add a $project immediately after the initial $match to drop unneeded fields before expensive stages like $group or $sort run, shrinking the per-document memory footprint carried through the rest of the pipeline. allowDiskUse is treated as an emergency valve for pipelines that must legitimately process large intermediate result sets (like a full monthly reporting rollup), not a default flag to slap on every pipeline to paper over bad ordering, because disk-spilled stages run dramatically slower under real traffic than the same pipeline properly filtered upfront.

The gotcha that surprises even experienced engineers: only the very first stage of a pipeline can use an index, and only if it's a $match (or certain $sort placements) with no preceding transformation — inserting even a harmless-looking $addFields before your $match silently disables the index for that $match, since MongoDB no longer sees an untouched collection scan target at that point in the pipeline. Staging environments with small seeded datasets never surface this because the 100MB limit and index-usage difference are invisible at low document counts, which is exactly why this class of bug reliably passes code review and staging tests, then fails only under production data volume.