How do you handle OpenSearch cluster yellow/red status and unassigned shards?
Quick Answer
Yellow means every primary shard is assigned but at least one replica isn't, so data is safe but under-replicated; red means at least one primary shard itself is unassigned, meaning some data is genuinely unavailable for reads and writes on that index. Use GET _cluster/allocation/explain to see OpenSearch's own reasoning for why a specific shard won't allocate — usually disk watermark limits, an incompatible allocation filter, or not enough distinct nodes to satisfy the replica count — and fix the underlying constraint rather than forcing allocation blindly.
Detailed Answer
Imagine a library with a rule: every book, the primary, needs at least one photocopy stored in a different building, the replica, in case a fire destroys the original building. Yellow status is like discovering some books don't have their backup copy made yet — mildly concerning, but the original book is still on the shelf and readable right now. Red status is like discovering the original book itself is missing from every building — nobody can check it out at all, because there's no copy anywhere to fall back on.
OpenSearch was designed around this primary/replica shard model specifically so data survives individual node failures, and cluster health is a deliberately simple three-color signal, green, yellow, red, layered on top of much more granular shard-level state, so operators get an at-a-glance severity assessment without parsing every shard's status during a fast-moving incident. Yellow is explicitly designed to be safe-but-degraded, you still have all your data, just less redundancy, while red is designed to be unambiguous: real, present data unavailability that needs immediate attention.
Under the hood, the cluster manager node, the elected coordinator formerly called master, continuously tries to allocate any unassigned shard to an eligible data node, subject to allocation rules: disk-based shard allocation watermarks, where a low watermark stops new shard allocation onto a node, a high watermark starts relocating shards away, and a flood-stage watermark makes indices read-only to prevent a full disk; shard allocation awareness and filtering, for example keeping replicas in different availability zones; and simply whether enough distinct nodes exist to hold a primary and all its replicas without doubling up on the same node. GET _cluster/allocation/explain surfaces exactly which of these rules is currently blocking a specific shard from allocating, turning 'why is my cluster yellow' from a guessing game into a direct, actionable answer.
In production, teams alert on cluster status transitioning to yellow at a ticket level, especially if it lingers beyond a normal maintenance window, and page immediately on red, while separately monitoring disk usage per node against the configured watermarks. The single most common cause of unassigned shards in real fleets is nodes crossing the low or high disk watermark during heavy indexing from something like a payments-api-logs index, which blocks new shard placement and can cascade into more shards becoming unassigned as remaining nodes also fill up in turn.
The gotcha: naively raising or disabling the disk watermarks to make yellow status go away, a common quick workaround under pressure, doesn't fix anything — it just lets OpenSearch keep placing shards onto nodes that are genuinely running low on disk, setting up a much worse red-status, out-of-disk-space incident later. The correct fix is almost always adding capacity, deleting or rolling over old indices via an ILM or ISM policy, or rebalancing shard distribution across existing nodes, not moving the watermark goalposts to hide the underlying pressure.