OpenSearch

Imagine a large apartment complex's management company decides upfront to divide all tenant mail into exactly ten mailroom bins, using a formula like the last digit of the apartment number. Every mail carrier and every tenant relies on that formula to know which bin holds which mail. If the company suddenly switched to seven bins instead of ten, the old formula's results become meaningless — mail correctly sorted into bin six under the ten-bin formula might belong somewhere completely different under the seven-bin formula, so every single piece of mail in the building would have to be physically re-sorted to make the new bin count correct.

OpenSearch indices work the same way. When you create an index, you specify number_of_shards, the primary shard count, defaulting to one in modern versions but historically five, and every document written to that index is deterministically routed to one specific primary shard using a hash of its _id, or a custom routing value, modulo the shard count. This fixed, pre-committed shard count exists because that routing formula must stay constant for OpenSearch to reliably find a document later — if the shard count could silently change, the same formula applied to the same document would compute a different shard, breaking lookups for anything indexed before the change took effect.

Internally, each primary shard is a fully independent Lucene index, Lucene being the underlying single-node search library OpenSearch is built on, with its own inverted index, segments, and merge policy. Replica shards are exact copies of a primary, kept in sync as new documents are written — the primary indexes first, then replicates the operation to its replicas — and OpenSearch can route read and search requests to either a primary or any of its replicas to spread query load, while writes always go through the primary first. The cluster manager tracks which node holds which shard copy and handles reassignment if a node holding a shard goes down unexpectedly.

In production, choosing the primary shard count upfront matters enormously: too few shards for a large, growing dataset, like a year of payments-api-logs, means each shard grows huge and becomes slow to search and rebalance; too many small shards across a cluster wastes memory on per-shard overhead, since each shard consumes file handles and heap for its own Lucene structures, and can itself become a cluster stability problem at scale. Teams commonly use rollover-based index patterns, creating a new index every day or week and aliasing them together, specifically to sidestep the 'I picked the wrong shard count forever' problem, since they're setting the shard count for one time-bounded chunk of data rather than for all data forever.

The gotcha: engineers coming from relational databases often assume you can just add more shards the way you'd add more database partitions dynamically, but OpenSearch has no live operation for primary shards at all — the only path is the _reindex, _split, or _shrink APIs, which physically copy or reorganize data into a brand-new index with a different shard count. That means correcting a sizing mistake after the fact requires enough temporary disk and cluster capacity to hold both the old and new index simultaneously during the migration, which is often the real operational blocker rather than the reindex logic itself being complicated.

Imagine three different reasons one specific highway toll booth causes traffic jams only during rush hour. Maybe that one booth happens to be the only one open on a stretch everyone needs, a hot booth getting disproportionate traffic. Maybe all booths periodically pause for a few seconds to do maintenance, and it's simply more noticeable when traffic is already heavy, a system-wide periodic pause like garbage collection. Or maybe certain cars, oversized trucks needing manual inspection, take much longer at any booth, and rush hour just has more of them passing through, expensive individual transactions. All three look identical from a distance, 'slow at rush hour,' but the fix for each is completely different.

OpenSearch surfaces node- and shard-level metrics precisely because these three failure modes, hot-spotting, JVM GC pressure, and expensive query patterns, look identical from a black-box 'average latency went up' dashboard but require completely different fixes. This is a direct consequence of it being a distributed system built on the JVM: work is spread across shards and nodes by design, not perfectly by usage pattern; memory management is handled by JVM garbage collection, which can pause application threads; and it accepts arbitrary user-composed queries whose cost varies enormously depending on shape.

To isolate the cause, GET _cat/shards?v combined with GET _nodes/stats/indices/search shows per-node search thread pool queue size and latency — if two nodes show dramatically higher search latency and larger queues while others sit near-idle, those nodes likely host a disproportionately hot shard, common when one heavily-queried index, like a current day's checkout-worker index in a daily-rollover pattern, isn't evenly distributed or a single shard receives most of the traffic. GET _nodes/stats/jvm shows GC pause counts and duration per node — if pause times spike in lockstep across most nodes correlating with the latency spike, it's memory pressure, often from oversized aggregations or too many open shards consuming heap. Finally, the _search profile API run against actual slow queries shows exactly where time is spent inside a single query's execution, revealing patterns like leading-wildcard queries that can't use the index efficiently, or deep from/size pagination that forces scanning and sorting far more documents than are actually returned to the client.

In production, the business-hours-only pattern is itself diagnostic: if latency is purely proportional to query volume, both hot-shard and GC-pressure theories remain plausible, since more load worsens both. But if latency is disproportionately worse than the increase in query volume would suggest, that points toward a resource ceiling being hit, like heap approaching its limit and triggering more frequent or longer GC pauses, rather than simple linear load. Teams typically build dashboards correlating query rate, GC pause time, and per-node search latency on the same timeline, specifically to make this kind of correlation-based diagnosis fast during an actual incident instead of guessing under pressure.

The non-obvious gotcha: a hot shard problem can be entirely self-inflicted by well-intentioned index design — using a low-cardinality routing key, like routing all documents for one large customer to the same shard for locality, creates a whale shard that gets far more traffic than its siblings. This only becomes visible under real business-hours load, never during off-peak testing, meaning a routing strategy that looked perfectly fine in staging can create a production-only hot-spotting incident that's genuinely hard to reproduce outside peak traffic windows.

Think of a shared office kitchen with a small trash can emptied every hour on the hour. If the can is already 90 percent full right after being emptied, you know it will overflow again well before the next hour is up — the fact that it's nearly full immediately post-cleanup is what tells you there's a real capacity problem, not just a matter of timing when you happened to check. If instead the can was only 10 percent full right after emptying, you'd know there's plenty of headroom even if it fills up 90 percent again just before the next scheduled cleaning.

OpenSearch, running on the JVM, exhibits exactly this pattern with heap memory: raw heap usage sampled at any random moment is nearly meaningless, since it depends heavily on whether a garbage collection cycle just ran. But heap usage sampled immediately after a garbage collection tells you how much memory is genuinely still in use by things that can't be freed, and a rising trend there means real memory pressure, not normal churn. OpenSearch's circuit breakers exist specifically because engineers learned that letting the JVM run out of heap causes an ungraceful OutOfMemoryError that can corrupt in-progress operations, so circuit breakers proactively reject a request that would push memory over a configured threshold, trading 'this one query fails cleanly' for 'the whole node doesn't crash.'

Internally, several circuit breakers are tracked separately: the parent breaker, an overall memory budget across all breakers combined; the field data breaker, memory used loading field values into memory for sorting or aggregating on fields not backed by doc_values; the request breaker, a per-request memory estimate for something like a large aggregation; and the in-flight requests breaker, memory held by requests currently being processed. Each increments a trip counter in _nodes/stats/breaker when it rejects an operation. Meanwhile _nodes/stats/jvm/mem/pools/old shows the old-generation heap pool specifically, where long-lived objects like shard-level caches accumulate — a steady upward trend in that value, sampled right after each GC cycle over hours or days, is the clearest predictive signal of an eventual crash.

In production, dashboards graph old-gen heap-after-GC as a time series, never raw instantaneous heap, alongside GC pause duration and frequency, since a rising trend in time spent in GC per minute means the JVM is working harder and harder to reclaim less and less memory, and circuit breaker trip counts per breaker type, since a rising field-data breaker trip count specifically often points to aggregations or sorts running on text fields that should have doc_values-backed keyword fields instead. Alerting typically pages on sustained old-gen usage above roughly 75 to 85 percent of max heap measured post-GC, since that's the zone where a single burst of memory-heavy queries can tip a node into a full OOM crash.

The non-obvious gotcha: circuit breakers tripping frequently isn't just an inconvenience to tune away — teams sometimes respond to breaker trips by raising the breaker's limit purely to stop the errors, which removes the exact safety margin that was preventing an actual OOM crash. The trips are a symptom, something is requesting too much memory per operation, often a poorly-designed aggregation or a script-based sort, and the correct fix is almost always addressing the query pattern or adding heap and nodes, not loosening the breaker that's actively protecting the cluster.

Think about an office building's shredding service that comes weekly to destroy documents past a set retention period. A well-run office doesn't let the shredding truck take anything past the cutoff date automatically and irreversibly — there's a process where anyone who knows a specific box of documents is part of an active legal matter can tag it 'hold, do not shred,' and the shredding service respects that tag even if the box is technically past its normal retention date on paper.

OpenSearch's Index State Management plugin automates exactly this kind of lifecycle for time-series log indices, moving indices through states like hot, actively written; warm, read-only and less resourced; cold; and eventually delete, based on age or size thresholds. It exists because manually rolling over and deleting daily log indices across dozens of services doesn't scale, and disk exhaustion from unbounded log retention is one of the most common self-inflicted OpenSearch outages. But a fully automated deletion pipeline is dangerous without a safety valve, since incident investigations frequently need log data older than the 'normal' retention window, especially for a slow-building issue that took days or weeks to even notice.

A safe ISM setup defines a policy per index pattern, like payments-api-logs-*, with rollover conditions based on index age or size, for example rolling over daily or at 50GB, a warm-phase transition after some days to move less-queried indices to cheaper, denser nodes, and a delete phase after a defined retention, say 30 or 90 days depending on compliance needs. The safety mechanism layered on top: before the delete action actually executes, ISM can be configured to check a metadata flag or a separate protected-indices allowlist, maintained via a small automation script responding to an incident-hold tag applied by on-call engineers, and skip deletion for any index matching that flag, plus send a notification through the policy's notification action a set number of days before deletion actually happens, giving humans a real window to intervene before data is gone.

In production, teams monitor ISM policy execution history — GET _plugins/_ism/explain on an index shows its current state and any errors in transitioning — to catch policies stuck retrying a failed transition, often due to an allocation problem preventing a warm-phase relocation, and track disk usage trends to confirm the retention policy is actually keeping pace with data growth. A policy that's technically running but consistently behind due to failed transitions provides a false sense of safety while disk quietly fills anyway, undetected until it becomes a real incident of its own.

The non-obvious gotcha: teams often set retention purely based on typical operational needs, like 'we only ever look back 14 days,' without accounting for the fact that security or compliance incidents are frequently discovered weeks or months after the fact. By the time someone realizes a suspicious authentication pattern from user-auth-service six weeks ago needs investigating, a 30-day ISM delete policy has already permanently destroyed the evidence. Retention windows need to be set based on the slowest realistic detection time for the kinds of incidents that matter most, not just typical day-to-day debugging needs that rarely look back more than a couple of weeks.

Imagine a library with a rule: every book, the primary, needs at least one photocopy stored in a different building, the replica, in case a fire destroys the original building. Yellow status is like discovering some books don't have their backup copy made yet — mildly concerning, but the original book is still on the shelf and readable right now. Red status is like discovering the original book itself is missing from every building — nobody can check it out at all, because there's no copy anywhere to fall back on.

OpenSearch was designed around this primary/replica shard model specifically so data survives individual node failures, and cluster health is a deliberately simple three-color signal, green, yellow, red, layered on top of much more granular shard-level state, so operators get an at-a-glance severity assessment without parsing every shard's status during a fast-moving incident. Yellow is explicitly designed to be safe-but-degraded, you still have all your data, just less redundancy, while red is designed to be unambiguous: real, present data unavailability that needs immediate attention.

Under the hood, the cluster manager node, the elected coordinator formerly called master, continuously tries to allocate any unassigned shard to an eligible data node, subject to allocation rules: disk-based shard allocation watermarks, where a low watermark stops new shard allocation onto a node, a high watermark starts relocating shards away, and a flood-stage watermark makes indices read-only to prevent a full disk; shard allocation awareness and filtering, for example keeping replicas in different availability zones; and simply whether enough distinct nodes exist to hold a primary and all its replicas without doubling up on the same node. GET _cluster/allocation/explain surfaces exactly which of these rules is currently blocking a specific shard from allocating, turning 'why is my cluster yellow' from a guessing game into a direct, actionable answer.

In production, teams alert on cluster status transitioning to yellow at a ticket level, especially if it lingers beyond a normal maintenance window, and page immediately on red, while separately monitoring disk usage per node against the configured watermarks. The single most common cause of unassigned shards in real fleets is nodes crossing the low or high disk watermark during heavy indexing from something like a payments-api-logs index, which blocks new shard placement and can cascade into more shards becoming unassigned as remaining nodes also fill up in turn.

The gotcha: naively raising or disabling the disk watermarks to make yellow status go away, a common quick workaround under pressure, doesn't fix anything — it just lets OpenSearch keep placing shards onto nodes that are genuinely running low on disk, setting up a much worse red-status, out-of-disk-space incident later. The correct fix is almost always adding capacity, deleting or rolling over old indices via an ILM or ISM policy, or rebalancing shard distribution across existing nodes, not moving the watermark goalposts to hide the underlying pressure.