Which JVM and cluster-level metrics in OpenSearch actually predict an impending OOM crash or circuit breaker trip?

Think of a shared office kitchen with a small trash can emptied every hour on the hour. If the can is already 90 percent full right after being emptied, you know it will overflow again well before the next hour is up — the fact that it's nearly full immediately post-cleanup is what tells you there's a real capacity problem, not just a matter of timing when you happened to check. If instead the can was only 10 percent full right after emptying, you'd know there's plenty of headroom even if it fills up 90 percent again just before the next scheduled cleaning.

OpenSearch, running on the JVM, exhibits exactly this pattern with heap memory: raw heap usage sampled at any random moment is nearly meaningless, since it depends heavily on whether a garbage collection cycle just ran. But heap usage sampled immediately after a garbage collection tells you how much memory is genuinely still in use by things that can't be freed, and a rising trend there means real memory pressure, not normal churn. OpenSearch's circuit breakers exist specifically because engineers learned that letting the JVM run out of heap causes an ungraceful OutOfMemoryError that can corrupt in-progress operations, so circuit breakers proactively reject a request that would push memory over a configured threshold, trading 'this one query fails cleanly' for 'the whole node doesn't crash.'

Internally, several circuit breakers are tracked separately: the parent breaker, an overall memory budget across all breakers combined; the field data breaker, memory used loading field values into memory for sorting or aggregating on fields not backed by doc_values; the request breaker, a per-request memory estimate for something like a large aggregation; and the in-flight requests breaker, memory held by requests currently being processed. Each increments a trip counter in _nodes/stats/breaker when it rejects an operation. Meanwhile _nodes/stats/jvm/mem/pools/old shows the old-generation heap pool specifically, where long-lived objects like shard-level caches accumulate — a steady upward trend in that value, sampled right after each GC cycle over hours or days, is the clearest predictive signal of an eventual crash.

In production, dashboards graph old-gen heap-after-GC as a time series, never raw instantaneous heap, alongside GC pause duration and frequency, since a rising trend in time spent in GC per minute means the JVM is working harder and harder to reclaim less and less memory, and circuit breaker trip counts per breaker type, since a rising field-data breaker trip count specifically often points to aggregations or sorts running on text fields that should have doc_values-backed keyword fields instead. Alerting typically pages on sustained old-gen usage above roughly 75 to 85 percent of max heap measured post-GC, since that's the zone where a single burst of memory-heavy queries can tip a node into a full OOM crash.

The non-obvious gotcha: circuit breakers tripping frequently isn't just an inconvenience to tune away — teams sometimes respond to breaker trips by raising the breaker's limit purely to stop the errors, which removes the exact safety margin that was preventing an actual OOM crash. The trips are a symptom, something is requesting too much memory per operation, often a poorly-designed aggregation or a script-based sort, and the correct fix is almost always addressing the query pattern or adding heap and nodes, not loosening the breaker that's actively protecting the cluster.