A MongoDB collection with sharding enabled has one shard at 90% disk usage while the others sit at 30% — what causes this imbalance despite the balancer running, and how do you fix it without downtime?
Quick Answer
This is almost always a shard key with poor cardinality or a monotonically increasing value (like a timestamp or auto-incrementing id), which routes most new writes to the same chunk range and therefore the same shard regardless of how many shards exist. The balancer can only move whole chunks between shards, so it can't fix ongoing imbalance caused by every new document hashing or ranging into the same hot chunk — the real fix is choosing a better shard key or switching to hashed sharding, not just waiting for the balancer.
Detailed Answer
Imagine a hospital intake system where every new patient is assigned to a bed strictly by arrival timestamp, always filling ward A first, no matter how many other wards are open and empty. No amount of moving already-assigned patients between wards afterward (the balancer) fixes the fact that every single new patient keeps arriving into the same ward, because the assignment rule itself, not the redistribution process, is what's broken.
MongoDB shards a collection by splitting its documents into contiguous chunks based on a shard key's value, and distributes those chunks across shards; the balancer's job is only to migrate existing chunks to even out chunk count, not to change how new documents get assigned to chunks in the first place. If the shard key is monotonically increasing — an ObjectId (which embeds a timestamp) or an auto-incrementing counter — every new document's key value is always higher than every previous one, so it always lands in the single chunk currently representing the 'highest' range, which lives on one shard at a time. This was a known, deliberate tradeoff in range-based sharding: it makes range queries efficient (because a range of values sits contiguously on one shard) at the direct cost of write distribution when the key increases monotonically.
Internally, MongoDB does split an overfull chunk into two once it crosses a size threshold, and the balancer does eventually migrate one of those split chunks to a less-loaded shard — but by the time that migration completes, all of the new incoming writes have already moved on to an even newer chunk, which is still hashing or ranging onto the same original 'hot' shard. The balancer is fixing yesterday's imbalance while today's writes recreate a new one, so the hot shard's disk usage keeps climbing net of whatever the balancer manages to move away.
In production, engineers monitor per-shard disk usage, chunk count per shard, and specifically watch for a shard whose chunk count is roughly proportional to the others but whose disk usage or operation rate is disproportionately higher — that pattern (balanced chunk count, imbalanced load) is the signature of a hot chunk problem rather than a raw chunk-distribution problem, and it means adding more shards won't help until the key itself changes. The standard production fix is either switching the shard key to a hashed shard key (MongoDB hashes the key value before assigning it to a chunk range, spreading monotonic values pseudo-randomly across shards) or choosing a compound shard key that combines a coarser, evenly-distributed field with the monotonic one.
The gotcha that catches teams doing this migration: you cannot change a collection's shard key in place in most MongoDB versions — fixing this requires creating a new, correctly-sharded collection and migrating data into it (or using MongoDB 5.0+'s reshardCollection, which does this online but still consumes significant IO and time proportional to collection size), so 'switch the shard key' is not a quick config change, it's closer to a full data migration that has to be planned with the same care as any other zero-downtime resharding project, including a cutover step for in-flight writes during the migration window.