SHOW REPLICA STATUS reports Seconds_Behind_Source climbing steadily on one replica — how do you tell whether the bottleneck is single-threaded apply, network, or the replica's own disk, and how does multi-threaded replication actually fix it?
Quick Answer
Check Slave_SQL_Running_State and the replica's IO thread versus SQL thread progress separately — if the IO thread (receiving data from the source) is caught up but the SQL thread (applying it) isn't, the bottleneck is apply speed, which multi-threaded replication fixes by applying independent transactions from different schemas or table groups in parallel instead of one row at a time. If both threads are behind, the bottleneck is upstream — network bandwidth or the source's binlog generation rate.
Detailed Answer
Imagine a single cashier at a busy store re-ringing up every receipt from a much larger store's entire day of sales, one line item at a time, in the exact order they originally happened. Even if the register tape (network transfer) arrives instantly, the cashier can only process as fast as one person typing — replaying history one transaction at a time is fundamentally slower than the original store, which had dozens of registers running simultaneously.
MySQL replication historically applied every change from the binary log (binlog, the source's record of every data-changing statement or row event) using a single SQL thread on the replica, strictly in commit order, to guarantee the replica ends up in exactly the same state as the source. This was the safe default because naively parallelizing could apply conflicting changes out of order and corrupt data — but it meant a replica's apply capacity was capped at whatever a single CPU core and one thread of I/O could sustain, no matter how many cores or disks the replica actually had available.
Replication in MySQL is split into two independent threads: the IO thread, which streams binlog events from the source over the network and writes them to the replica's local relay log, and the SQL thread, which reads the relay log and actually applies the changes to the replica's data. Seconds_Behind_Source measures the SQL thread's lag relative to the source's current time, so diagnosing the bottleneck means checking whether the IO thread is caught up (relay log receipt is current) while the SQL thread lags — that isolates the problem to apply speed. Multi-threaded replication (enabled via replica_parallel_workers) fixes exactly this by allowing the SQL thread's job to be split across multiple worker threads, applying transactions from different logical partitions (by default, schema/database) concurrently as long as MySQL can prove they don't touch overlapping data.
In production, engineers watch Seconds_Behind_Source as the headline metric but drill into Slave_SQL_Running_State and performance_schema replication tables to see whether workers are actually parallelizing effectively or serializing anyway — a common trap is that if every write goes to a single schema or a small number of hot tables, replica_parallel_workers has almost nothing to parallelize across, because the default parallelization mode groups work by schema, and one busy schema is still effectively one lane. WRITESET-based parallelization (available in newer MySQL versions) parallelizes based on which actual rows a transaction touches rather than which schema, which unlocks real concurrency even for single-schema workloads, at the cost of needing row-based binlog format.
The gotcha that catches teams enabling multi-threaded replication for the first time: increasing replica_parallel_workers doesn't help, and can even make ordering-sensitive lag worse, if the workload is dominated by large single transactions (a big batch update or a long-running migration) — a multi-gigabyte transaction still applies as one atomic unit on one worker thread regardless of how many other workers are configured, so the real fix for large-transaction-driven lag is breaking up application-side batch jobs into smaller transactions, not tuning replication thread count, and no amount of replica-side configuration can parallelize a single indivisible transaction.