OpenSearch query latency spiked 10x during business hours only — how do you tell apart a hot shard, GC pauses, and bad query patterns?

Imagine three different reasons one specific highway toll booth causes traffic jams only during rush hour. Maybe that one booth happens to be the only one open on a stretch everyone needs, a hot booth getting disproportionate traffic. Maybe all booths periodically pause for a few seconds to do maintenance, and it's simply more noticeable when traffic is already heavy, a system-wide periodic pause like garbage collection. Or maybe certain cars, oversized trucks needing manual inspection, take much longer at any booth, and rush hour just has more of them passing through, expensive individual transactions. All three look identical from a distance, 'slow at rush hour,' but the fix for each is completely different.

OpenSearch surfaces node- and shard-level metrics precisely because these three failure modes, hot-spotting, JVM GC pressure, and expensive query patterns, look identical from a black-box 'average latency went up' dashboard but require completely different fixes. This is a direct consequence of it being a distributed system built on the JVM: work is spread across shards and nodes by design, not perfectly by usage pattern; memory management is handled by JVM garbage collection, which can pause application threads; and it accepts arbitrary user-composed queries whose cost varies enormously depending on shape.

To isolate the cause, GET _cat/shards?v combined with GET _nodes/stats/indices/search shows per-node search thread pool queue size and latency — if two nodes show dramatically higher search latency and larger queues while others sit near-idle, those nodes likely host a disproportionately hot shard, common when one heavily-queried index, like a current day's checkout-worker index in a daily-rollover pattern, isn't evenly distributed or a single shard receives most of the traffic. GET _nodes/stats/jvm shows GC pause counts and duration per node — if pause times spike in lockstep across most nodes correlating with the latency spike, it's memory pressure, often from oversized aggregations or too many open shards consuming heap. Finally, the _search profile API run against actual slow queries shows exactly where time is spent inside a single query's execution, revealing patterns like leading-wildcard queries that can't use the index efficiently, or deep from/size pagination that forces scanning and sorting far more documents than are actually returned to the client.

In production, the business-hours-only pattern is itself diagnostic: if latency is purely proportional to query volume, both hot-shard and GC-pressure theories remain plausible, since more load worsens both. But if latency is disproportionately worse than the increase in query volume would suggest, that points toward a resource ceiling being hit, like heap approaching its limit and triggering more frequent or longer GC pauses, rather than simple linear load. Teams typically build dashboards correlating query rate, GC pause time, and per-node search latency on the same timeline, specifically to make this kind of correlation-based diagnosis fast during an actual incident instead of guessing under pressure.

The non-obvious gotcha: a hot shard problem can be entirely self-inflicted by well-intentioned index design — using a low-cardinality routing key, like routing all documents for one large customer to the same shard for locality, creates a whale shard that gets far more traffic than its siblings. This only becomes visible under real business-hours load, never during off-peak testing, meaning a routing strategy that looked perfectly fine in staging can create a production-only hot-spotting incident that's genuinely hard to reproduce outside peak traffic windows.