Java/JVM Ops

Picture a busy office building that sorts new hires into two kinds of desks: a temp desk near the door for people who might only be there a few days, and a permanent office deeper inside for people who prove they're sticking around long-term. Most temp workers leave within days and their desk gets cleared out constantly with almost no effort — a quick check at the door. Only after someone has stuck around through several rounds of that quick check does the company bother giving them a permanent office, because moving someone into a permanent office is expensive and disruptive, and you don't want to do it for people who are about to leave anyway. The JVM heap works on exactly this principle: it's split into a Young generation (the temp desks, further divided into Eden and two Survivor spaces) and an Old generation (the permanent offices), because most objects in real applications die young — a decades-old observation called the 'generational hypothesis' that this entire design is built around.

The JVM was designed this way because scanning the entire heap on every garbage collection would be prohibitively slow as heaps grow into gigabytes. By betting that most objects die quickly, the collector can do frequent, cheap Minor GCs that only scan the small Young generation (most objects there are already garbage and get reclaimed almost for free), and only occasionally do the much more expensive full-heap scan. Splitting Young generation into Eden and two Survivor spaces (commonly called S0 and S1) lets the collector use a 'copying' algorithm: live objects get copied out of Eden and one Survivor space into the other Survivor space, which is cheap because it only touches objects that survived, not the (usually much larger) set that died.

Step by step: new objects are allocated in Eden. When Eden fills up, a Minor GC runs — every object still referenced gets copied into an empty Survivor space, and each copied object's age counter (stored in its object header) increments by one. Objects that die (nothing references them) are simply left behind and the space is reused, no explicit deletion needed. This repeats on every Minor GC, with objects ping-ponging between the two Survivor spaces (S0 and S1 swap roles) each time they survive. Once an object's age counter crosses the JVM's tenuring threshold (a configurable value, -XX:MaxTenuringThreshold, defaulting around 15 for most collectors) it gets promoted — copied out of Young generation entirely into the Old generation, on the theory that anything that's survived this many collection cycles is probably going to live a long time and doesn't need to keep being copied back and forth.

At production scale, the thing to actually watch is the ratio of Minor GCs (cheap, sub-millisecond to low-tens-of-milliseconds pauses) to Major/Full GCs (expensive, potentially hundreds of milliseconds to seconds, because they scan the entire Old generation and sometimes the entire heap). A healthy service does frequent, fast Minor GCs and rare Full GCs. What goes wrong: if Survivor spaces are too small for a workload's actual allocation rate — say, a request-processing burst that allocates far more temporary objects than the Survivor space can hold — objects that would normally die within a few Minor GCs get force-promoted directly into Old generation simply because there's nowhere else to put them during a Minor GC, a pathology called premature promotion. This fills up Old generation with what should have been short-lived garbage, which triggers Full GCs far more often than the workload should actually need, and each Full GC pause directly shows up as p99 latency spikes on live user traffic.

The gotcha: engineers who only tune -Xmx (max total heap) without also considering the ratio between Young and Old generation sizing (-XX:NewRatio or explicit -Xmn) often make premature promotion worse, not better, when they increase heap size — a bigger heap with the same proportionally-small Survivor space just delays when the problem shows up, it doesn't fix the underlying object-survival pattern, and the service continues to pause unpredictably under bursty load even after a 'more memory' fix that looked successful in initial testing.

This kind of clockwork-regular slowdown is like a toll booth that, once every ninety cars, suddenly closes every lane for four seconds to count the cash drawer before reopening — from a distance the traffic flow looks normal almost all the time, but anyone stuck at exactly that moment experiences a hard stop, and it happens on a predictable rhythm rather than randomly. A stop-the-world GC pause behaves exactly like this: for the small fraction of a second (or, in a bad case, seconds) that a Full GC runs, every application thread is frozen so the collector can safely walk and compact memory, and then everything resumes as if nothing happened — until the next time the Old generation fills up again.

The reason CPU monitoring often misses this is a sampling and averaging problem: top typically refreshes every 1-3 seconds and shows an average utilization over that window, so a GC pause that fully saturates all cores for 500ms can get smoothed into an unremarkable-looking average if the sampling window doesn't line up precisely, especially on a multi-core box where the GC threads are a small fraction of total cores. This is exactly why 'no CPU spike' and 'GC pause' are not contradictory — the pause is real and CPU-intensive, it's just short and thread-count-limited enough to hide inside typical monitoring granularity.

Step by step to confirm: first, enable (or check if already enabled) unified JVM GC logging with -Xlog:gc*, which timestamps every GC event including its type (Minor vs Full) and duration; if a Full GC event's timestamp lines up with the observed 4-second latency spike, that's near-certain confirmation. Second, if you can't restart the process to add logging, use jstat -gcutil <pid> 1000 to stream live generation occupancy and GC counts once per second — watching the Old generation (O column) climb steadily and then drop sharply, paired with the FGC (full GC count) column incrementing at that exact moment, confirms the same thing without needing a log file. Third, correlate the 90-second period itself: a suspiciously regular interval like this usually means a fixed-size cache, a scheduled batch job, or a metrics-flushing routine that allocates a large, similarly-sized batch of objects on a fixed timer, filling Old generation to its trigger threshold on a near-identical cadence every time.

At production scale, once GC is confirmed as the cause, the fix path branches: if pause duration is the core problem, moving from a throughput-oriented collector to a low-pause collector like G1 (or ZGC/Shenandoah for very large heaps needing sub-millisecond pauses) directly targets pause time instead of total throughput. If the *root cause* is the 90-second periodic allocation spike itself, the better fix is addressing that pattern directly — batching smaller, or spreading a bulk cache refresh across a window instead of doing it all in one allocation burst — since a differently-tuned GC still has to deal with the same fundamental memory churn, just with a different pause profile.

The gotcha: engineers who see 'no CPU spike' and rule out GC based on that alone will spend hours chasing network flakiness, downstream latency, or application code before ever checking the GC log — when the actual fix would have taken two minutes if GC logging had been on by default. The deeper lesson experienced engineers learn the hard way is that GC logging should be enabled unconditionally in every production JVM service from day one, because it's nearly free to leave running and it's the single fastest way to rule GC in or out during exactly this kind of investigation, instead of needing a restart mid-incident just to turn logging on.

Watching only current heap usage on a JVM is like judging a bathtub's risk of overflowing purely from how full it looks at random glances, without ever checking whether the drain still works. The water level naturally goes up and down as someone runs the tap and lets it drain — normal, healthy behavior. But if the drain is slowly clogging, the water level *after each drain cycle* creeps up a little more every time, even though a random glance mid-fill still looks identical to a healthy tub. The metric that actually predicts overflow is the level right after draining, not the level at some arbitrary instant — and JVM memory health works the same way: the meaningful signal is heap occupancy immediately after a Full GC (once the collector has done everything it can to reclaim memory), not the current heap usage number, which fluctuates constantly and tells you almost nothing on its own.

This is why raw "heap used" dashboards, while common, are actually one of the weaker signals available — they were designed as a simple, always-available number, not as a leak detector. Real leak detection requires isolating the *floor* the heap settles at after garbage collection has done its best possible cleanup, because a genuine leak means that floor rises release over release, hour over hour, even while the ceiling (peak usage between GCs) looks perfectly normal and unremarkable to a casual dashboard glance.

Step by step, a well-instrumented JVM service exposes these via JMX (Java Management Extensions, the JVM's built-in interface for exposing runtime metrics) which tools like the Prometheus JMX exporter scrape and turn into time series: jvm_memory_used_bytes post-GC (specifically the value right after a GC event, which most APM tools can isolate), jvm_gc_pause_seconds histograms (frequency and duration of both Minor and Full GCs), jvm_memory_pool_bytes_used broken out per generation (Eden, Survivor, Old, Metaspace individually, not just aggregate heap), and jvm_threads_current (a slowly climbing thread count with no matching climb in traffic is its own leak signature, usually a thread pool that isn't returning threads or an executor that's never being shut down). Alerting logic should compare the post-Full-GC floor over a rolling window (say, day-over-day) rather than alerting on any single high-usage reading, since instantaneous heap usage during a legitimate traffic spike is completely normal and not itself a problem.

At production scale, the actionable pattern is: promotion rate (how many bytes per second are being copied into Old generation) correlated against Full GC frequency gives you a leading indicator days before an actual OOM, because a leak surfaces first as "we're promoting more into Old gen than we used to, and doing more Full GCs to compensate," long before the Old generation actually fills completely and throws OutOfMemoryError. Metaspace deserves its own separate alert, since it fills through an entirely different mechanism (class loading, not object allocation) and a Metaspace leak — commonly from an app framework that dynamically generates proxy classes without ever unloading old classloaders during hot-redeploys — won't show up in heap-used metrics at all, only in Metaspace-specific ones.

The gotcha: teams that only alert on absolute heap-used percentage (like "page if heap > 90%") get paged constantly during completely normal, healthy traffic spikes that GC handles fine, and eventually tune that alert's threshold up or silence it out of alert fatigue — which means the one time it's a real leak slowly creeping toward OOM over several days, the noisy absolute-threshold alert has already been desensitized or ignored, and the actual leak is caught only when the service finally OOMs in production rather than days earlier when the post-GC floor first started its slow, steady climb.

This is like requiring a black box flight recorder on every aircraft, but also making sure someone actually retrieves and stores that black box before the wreckage gets cleared away and the runway reopens for the next flight. Simply configuring -XX:+HeapDumpOnOutOfMemoryError is like installing the flight recorder — necessary but not sufficient, because if the dump file lands on the container's own ephemeral filesystem and the container then restarts (or gets rescheduled to a different node entirely), that evidence can vanish before anyone gets a chance to analyze it, exactly like a black box that never gets pulled from the wreckage.

The reason this needs actual automation rather than a single JVM flag is that Kubernetes' default behavior — restart the container immediately on crash — actively works against evidence preservation unless you design for it. Kubernetes was built to prioritize availability (get the workload running again fast), which is the right default for the *service*, but it's the wrong default for the *one specific crash* an engineer actually needs to debug, so the automation has to reconcile these two competing goals: keep restarting so users aren't impacted, while ensuring at least one full diagnostic artifact survives the restart cycle for analysis.

Step by step, a robust setup looks like this: first, the heap dump path must point at a volume that survives container restarts — either a persistent volume mounted into the pod, or (more commonly at scale) a local emptyDir large enough to hold one dump, paired with a sidecar or init-container-triggered script that watches for a new .hprof file and immediately uploads it to object storage (S3, GCS) the moment it appears, since emptyDir volumes are wiped if the pod is rescheduled to a different node. Second, dump retention needs a cap — a service crash-looping every few minutes could otherwise fill even a generously-sized volume within an hour, so the automation should keep only the most recent one or two dumps locally and treat everything past that as already shipped and safe to delete. Third, the pod's restart policy needs a backoff and a cap on restart attempts (Kubernetes' default exponential backoff for CrashLoopBackOff helps, but pairing it with an alert on repeated OOM-driven restarts within a short window ensures a human gets paged rather than the pod silently crash-looping at length while consuming node resources and repeatedly generating (and potentially losing) diagnostic dumps).

At production scale, teams often add a resource-aware nuance: heap dumps for a multi-gigabyte heap can themselves be multiple gigabytes, so writing one to disk on a memory-constrained or disk-constrained node during an already-degraded moment can itself cause a secondary failure (disk pressure evicting other pods on the same node) — which is why the dump target volume size and the node's available disk both need headroom sized specifically for the worst-case heap size, not just 'whatever's left over.'

The gotcha: many teams enable HeapDumpOnOutOfMemoryError, feel confident they're covered, and then discover during their first real incident that the dump landed in the container's writable layer or an emptyDir that Kubernetes wiped the instant it rescheduled the crashed pod to recover service — meaning the flag fired correctly, the dump was written correctly, and it was still lost forever because nobody built the retrieval half of the pipeline. The lesson experienced engineers learn is that heap-dump-on-OOM is not 'done' until there's a tested, automated hand-off that gets the file off the node before the next restart, not just a JVM flag in the deployment manifest.

An OutOfMemoryError is like a warehouse manager who keeps accepting new inventory shipments but never sends anything back out — eventually every aisle, shelf, and even the loading dock itself is full, and the next truck literally cannot be unloaded anywhere. The JVM's heap is that warehouse: it's a fixed-size region of memory the JVM reserves for your application's objects, and when garbage collection (the process of finding and clearing out objects nothing references anymore, like a crew removing spoiled inventory) can no longer free enough space to satisfy a new allocation, the JVM has no choice but to throw OutOfMemoryError and, usually, kill the request or the whole process.

The reason -XX:+HeapDumpOnOutOfMemoryError matters so much is that OOM is a terrible time to start investigating — the process is either dead or about to be, and by the time someone notices the alert, restarts the pod, and tries to reproduce it, the exact memory state that caused the failure is gone forever. This flag was designed to solve that specific problem: the instant the JVM decides it's out of memory, before it fully unwinds, it dumps the entire heap to a .hprof file on disk, freezing the crime scene exactly as it was at the moment of failure, so you can analyze it hours or days later without needing to reproduce anything live.

Step by step, the diagnostic flow looks like this: first, confirm the flag was actually set (a shocking number of production incidents get zero heap dump because nobody enabled it ahead of time, which is why it should be a default on every JVM service, not an opt-in you remember after the fact). Second, once you have the .hprof file, load it into Eclipse Memory Analyzer (MAT), which builds a full object reference graph and runs a 'Leak Suspects' report — this report works by finding the dominator tree, essentially asking 'if I deleted this one object, how much other memory would become collectible with it,' which surfaces the actual root cause object rather than just the biggest individual object. Third, you classify the OOM type from the error message itself: 'Java heap space' means actual application objects filled the heap; 'Metaspace' means too many classes were loaded (common with dynamic class generation, like some ORM proxies, or classloader leaks in app servers that reload webapps without unloading old classloaders); 'GC overhead limit exceeded' means the JVM is spending over 98% of its time garbage collecting and reclaiming under 2% of the heap each time, meaning it's still technically not 'out' but effectively thrashing; and 'unable to create new native thread' means the OS ran out of memory to allocate new OS-level thread stacks, which is an OS/native memory problem, not a Java heap problem at all.

In production, this gets more complicated inside containers, because the JVM historically read the *host's* total memory rather than the container's cgroup limit, and would size its default heap as a fraction of memory that didn't actually exist inside the container's boundary, guaranteeing an OOM kill by the kernel before the JVM's own heap ever felt pressure. What to monitor: heap usage after each full GC (a steadily rising post-GC floor across many GC cycles is the earliest sign of a genuine leak, as opposed to a temporary spike that GC successfully reclaims), GC pause frequency and duration, and container OOM-kill events from the orchestrator, which are a different failure mode entirely from a JVM-level OutOfMemoryError.

The gotcha: a container getting OOM-killed by the kernel and a JVM throwing its own OutOfMemoryError look similar from the outside (the process dies) but have completely different fixes. If -Xmx is set higher than the container's memory limit, the kernel's cgroup OOM killer terminates the entire JVM process abruptly with SIGKILL — no heap dump, no graceful shutdown, no stack trace, just an exit code 137 — while the JVM itself never got the chance to detect memory pressure and throw its own catchable OutOfMemoryError. Engineers who only tune -Xmx without also setting -XX:MaxRAMPercentage or ensuring -Xmx sits comfortably below the container's cgroup limit end up debugging a phantom 'crash with no logs,' when the real fix was simply giving the JVM enough headroom below the hard container ceiling to fail gracefully instead of being killed abruptly.

Diagnosing a hung application from one thread dump is like judging whether traffic on a highway is truly gridlocked from a single photograph — a photo can't tell you if that car has been sitting still for twenty minutes or just happened to be stopped for a split second when the shutter clicked. You need at least two photographs a few seconds apart to see whether the same cars are still in the exact same spots, which is the only real evidence of a genuine jam rather than normal, momentary traffic flow. A Java thread dump is exactly that snapshot — it captures the state of every thread in the JVM at one instant, and a thread showing as WAITING or BLOCKED in a single dump might resolve itself half a second later as part of completely normal operation.

This is why the standard practice is to always take multiple dumps in quick succession — typically three, ten seconds apart — rather than relying on one. jstack (or its modern equivalent, jcmd <pid> Thread.print) was designed to be a cheap, safe, repeatable operation specifically so engineers could do this comparison without meaningfully perturbing the running application; unlike a heap dump, which can pause the JVM for a noticeable stop-the-world moment on a large heap, a thread dump is comparatively lightweight.

Step by step: capture dump one, wait ten seconds, capture dump two, wait another ten seconds, capture dump three. Then look at each thread's state across all three: RUNNABLE means the thread is actively executing (or waiting on native I/O, which JVM thread states can't distinguish from real CPU work) — a thread pegged in RUNNABLE across all three dumps at the same stack frame is a hot spin loop or expensive computation, not a hang. BLOCKED means the thread is waiting to acquire a Java monitor (a lock) that another thread currently holds — if the same threads show BLOCKED on the same lock across all three dumps, that's a real, sustained contention problem. WAITING or TIMED_WAITING usually means a thread is idle in a pool waiting for work, which is completely normal and not evidence of a hang by itself. Critically, jstack automatically runs deadlock detection as part of every dump and will print an explicit 'Found one Java-level deadlock' section with the full cycle of threads and locks involved, which is the single most valuable piece of automatically-derived evidence you can get.

At production scale, the most common real-world pattern behind a 'hung' service isn't a classic deadlock at all — it's every HTTP-handling thread ending up BLOCKED waiting on a shared resource like a database connection pool, because the pool itself is exhausted (all connections checked out, none returned, due to a slow or leaking downstream call). This looks exactly like a hang from the outside (the app stops responding to new requests) but the actual root cause is upstream capacity, not a code-level deadlock, and the thread dump reveals this clearly once you notice dozens of threads all BLOCKED at the exact same stack frame inside the connection pool's acquire method.

The gotcha experienced engineers hit: a thread reported as RUNNABLE isn't necessarily doing useful CPU work — the JVM's thread state model can't distinguish 'genuinely computing' from 'blocked in a native OS call like a socket read,' so a thread stuck waiting on a slow network call from a misbehaving downstream service will show as RUNNABLE, not WAITING, in the dump. Engineers who only search dumps for BLOCKED or WAITING states and ignore RUNNABLE threads that never move across multiple dumps miss exactly this class of hang, which is common when a downstream service silently stops responding without closing the TCP connection.

Xms/Xmx bound the heap where your objects live and GC runs. Metaspace (post-Java 8, replacing PermGen) grows with loaded classes and lives in native memory. Container OOM-kills often come from non-heap growth the heap flags don't cover — size the whole process, not just -Xmx.

Older JVMs saw host memory and over-allocated heap, causing OOM-kills. Modern JVMs are container-aware; set -XX:MaxRAMPercentage=75 (leaving headroom for metaspace, threads, and off-heap) instead of a fixed -Xmx that ignores limit changes. Always set a container memory limit too.

G1 balances pause and throughput for most services. For latency-sensitive apps with large heaps, ZGC or Shenandoah keep pauses sub-millisecond. Parallel GC maximizes throughput for batch jobs where pauses don't matter. Measure with GC logs before switching.

For rising memory, take a heap dump (jcmd <pid> GC.heap_dump) and analyze in Eclipse MAT to find dominators. For high CPU, take repeated thread dumps (jstack) or use async-profiler to produce a flame graph showing hot methods. Enable GC logging to distinguish a leak from GC thrash.