A Logstash grok filter is pegging CPU at 100% and events are backing up behind it — how do you diagnose and fix a runaway regex pattern?

Imagine an overzealous proofreader given one sloppy instruction: check every possible way this sentence could be parsed before deciding what it means. For a short, clear sentence, that's instant. But for a long, ambiguous paragraph full of repeated words and delimiters, the number of 'possible ways to read it' the proofreader tries to consider can explode combinatorially, and they get stuck rereading the same paragraph for minutes, burning all their attention on one sentence while a stack of other work piles up behind them, untouched.

That's what happens inside Logstash's grok filter, built on the Oniguruma and Joni regex engines, when a pattern is too permissive. Grok patterns are named regex shortcuts, like %{IP} or %{GREEDYDATA}, composed together, designed to make regex-based log parsing approachable without hand-writing raw regular expressions. But grok doesn't protect you from writing a combination of patterns that triggers catastrophic backtracking, where the regex engine, faced with ambiguous or nested repetition, tries an exponential number of ways to match before giving up or succeeding on a single line.

The diagnostic sequence starts with the monitoring API: GET _node/stats/pipelines/<name> shows which pipeline's filter stage has abnormally high duration_in_millis per event, correlated with top -H showing Logstash's JVM threads pegged at CPU. From there, isolate the specific pattern by testing candidate log lines against each configured grok pattern individually, using the Grok Debugger or by temporarily commenting out patterns and watching for _grokparsefailure tags disappearing or CPU dropping — looking specifically for combinations of GREEDYDATA or .* sitting next to other loosely-bounded patterns, especially where the log line contains repeated delimiter characters that create many equally-valid ways to split the string.

In production this shows up as every pipeline.workers thread pinned at 100% CPU, events.duration_in_millis climbing while events.out throughput craters, and the persistent queue (if configured) filling because filters can't drain it fast enough — for a payments-api log pipeline this cascades quickly since transaction logs are both high-volume and time-sensitive. Ongoing prevention means routing any new grok pattern through a staging pipeline fed with representative production log volume before it ever reaches the main pipeline, and preferring dissect over grok wherever the log format has a fixed, predictable structure, since dissect does simple string splitting with zero backtracking risk.

The non-obvious gotcha: a grok pattern that has run fine for months can suddenly start pegging CPU not because the pattern itself changed, but because an upstream application changed its log format slightly — checkout-worker might start emitting a longer, more repetitive error message inside a field that a GREEDYDATA pattern was matching — and the same 'safe' pattern that worked fine on short messages now triggers exponential backtracking on the new, longer ones. Grok performance is never purely a property of the pattern in isolation; it depends on the actual data shape flowing through it, which is why any pattern audit needs representative production samples, not unit-test-style short strings that never expose the edge case.