What JVM metrics beyond raw heap usage should you scrape from a production payments-api service to predict a GC storm or OOM before it actually happens?

Watching only current heap usage on a JVM is like judging a bathtub's risk of overflowing purely from how full it looks at random glances, without ever checking whether the drain still works. The water level naturally goes up and down as someone runs the tap and lets it drain — normal, healthy behavior. But if the drain is slowly clogging, the water level *after each drain cycle* creeps up a little more every time, even though a random glance mid-fill still looks identical to a healthy tub. The metric that actually predicts overflow is the level right after draining, not the level at some arbitrary instant — and JVM memory health works the same way: the meaningful signal is heap occupancy immediately after a Full GC (once the collector has done everything it can to reclaim memory), not the current heap usage number, which fluctuates constantly and tells you almost nothing on its own.

This is why raw "heap used" dashboards, while common, are actually one of the weaker signals available — they were designed as a simple, always-available number, not as a leak detector. Real leak detection requires isolating the *floor* the heap settles at after garbage collection has done its best possible cleanup, because a genuine leak means that floor rises release over release, hour over hour, even while the ceiling (peak usage between GCs) looks perfectly normal and unremarkable to a casual dashboard glance.

Step by step, a well-instrumented JVM service exposes these via JMX (Java Management Extensions, the JVM's built-in interface for exposing runtime metrics) which tools like the Prometheus JMX exporter scrape and turn into time series: jvm_memory_used_bytes post-GC (specifically the value right after a GC event, which most APM tools can isolate), jvm_gc_pause_seconds histograms (frequency and duration of both Minor and Full GCs), jvm_memory_pool_bytes_used broken out per generation (Eden, Survivor, Old, Metaspace individually, not just aggregate heap), and jvm_threads_current (a slowly climbing thread count with no matching climb in traffic is its own leak signature, usually a thread pool that isn't returning threads or an executor that's never being shut down). Alerting logic should compare the post-Full-GC floor over a rolling window (say, day-over-day) rather than alerting on any single high-usage reading, since instantaneous heap usage during a legitimate traffic spike is completely normal and not itself a problem.

At production scale, the actionable pattern is: promotion rate (how many bytes per second are being copied into Old generation) correlated against Full GC frequency gives you a leading indicator days before an actual OOM, because a leak surfaces first as "we're promoting more into Old gen than we used to, and doing more Full GCs to compensate," long before the Old generation actually fills completely and throws OutOfMemoryError. Metaspace deserves its own separate alert, since it fills through an entirely different mechanism (class loading, not object allocation) and a Metaspace leak — commonly from an app framework that dynamically generates proxy classes without ever unloading old classloaders during hot-redeploys — won't show up in heap-used metrics at all, only in Metaspace-specific ones.

The gotcha: teams that only alert on absolute heap-used percentage (like "page if heap > 90%") get paged constantly during completely normal, healthy traffic spikes that GC handles fine, and eventually tune that alert's threshold up or silence it out of alert fatigue — which means the one time it's a real leak slowly creeping toward OOM over several days, the noisy absolute-threshold alert has already been desensitized or ignored, and the actual leak is caught only when the service finally OOMs in production rather than days earlier when the post-GC floor first started its slow, steady climb.