A MySQL primary suddenly shows Threads_running climbing and query latency spiking cluster-wide, but CPU usage looks normal — what's your diagnostic sequence to find whether it's lock contention, a runaway query, or connection exhaustion?

Picture a highway with a single stalled car in one lane during rush hour. From a satellite view measuring total fuel burned (CPU usage), everything looks almost normal — most cars are idling, not accelerating. But every driver on the road is experiencing enormous delay, because they're all queued up behind one blockage, not because the road network itself is overloaded. Looking at aggregate 'engine usage' across all the idling cars would completely mislead you about where the actual problem is.

MySQL's Threads_running counts connections actively executing a statement, as opposed to Threads_connected which counts all open connections whether idle or busy. When Threads_running climbs while CPU stays flat, it means threads are executing but spending their time blocked — waiting on a row lock, a table lock, an internal mutex, or I/O — rather than burning CPU cycles computing something. This is a critical distinction because a CPU-bound problem and a lock-contention problem look completely different in every other regard but can both present as 'the database is slow,' and applying a CPU-bound fix (like adding more compute) to a lock-contention problem does nothing.

The diagnostic sequence starts with SHOW FULL PROCESSLIST or querying performance_schema.threads and performance_schema.data_lock_waits, looking specifically at the State column for values like 'Waiting for table metadata lock,' 'Waiting for row lock,' or 'statistics' — different wait states point to entirely different root causes. If many threads show 'Waiting for row lock' or similar, SHOW ENGINE INNODB STATUS's TRANSACTIONS section reveals exactly which transaction is holding the lock everyone else is waiting on, often a single long-running or forgotten-open transaction (an application connection that started a transaction and never committed or rolled back, perhaps due to a bug in error handling) blocking dozens of otherwise fast queries.

In production, engineers correlate Threads_running against Threads_connected and max_connections, because a separate but easily confused failure mode is connection exhaustion — the application's connection pool holding onto more connections than the database allows, causing new connection attempts to queue or fail outright, which superficially also presents as 'everything is slow' but requires a completely different fix (connection pool sizing or max_connections tuning) than lock contention does. Distinguishing the two quickly during an incident saves critical minutes, since Threads_running high with Threads_connected near max_connections points at exhaustion, while Threads_running high with Threads_connected comfortably below max_connections but many threads in a lock-wait state points at contention.

The gotcha experienced engineers watch for: the actual root-cause transaction holding a lock is often not itself slow or expensive — it might be a completely ordinary UPDATE that ran fine, except the application connection that ran it never called COMMIT or ROLLBACK afterward due to a bug, leaving the transaction open (and its locks held) indefinitely while the connection sits idle. SHOW ENGINE INNODB STATUS and performance_schema will show this as a transaction with a long Trx_started time but essentially zero recent activity — it looks idle, not busy, which is exactly why teams scanning for 'the slow query' during an incident often miss it, since the actual blocking transaction isn't running anything slow at all, it's just never finishing.