How does ClickHouse replication work with ReplicatedMergeTree and ClickHouse Keeper, and what are common failure modes?

Think of a legal document signing system where multiple offices must maintain identical copies of all contracts. A central registry (ClickHouse Keeper) records which contracts exist and which office has which version. When Office A receives a new contract, it registers it in the central registry and other offices fetch a copy. If the central registry goes down, no new contracts can be registered, but existing copies remain accessible. If an office loses network, it falls behind but catches up when reconnected.

ReplicatedMergeTree is the replicated variant of MergeTree that provides data redundancy and high availability. Each replica is a full copy of the table data, and all replicas are peers (no primary/secondary distinction for reads). When an INSERT hits any replica, it writes the data as a new part and registers that part in ClickHouse Keeper's replication log. Other replicas watch this log and fetch the new part from the inserting replica. The replication is at the part level (entire files transferred), not row-by-row like streaming replication in PostgreSQL.

Internally, ClickHouse Keeper (or ZooKeeper) stores the replication log (an ordered list of operations: new parts, merges, mutations), the current parts list for each replica, and deduplication block hashes. Insert deduplication works by hashing each inserted block; if the same hash appears again (from a retry after a network timeout), the duplicate is silently ignored. This provides exactly-once INSERT semantics even with at-least-once delivery from clients. Merges are coordinated: one replica decides which parts to merge (becomes the merge leader), performs the merge, and other replicas fetch the merged result rather than independently merging.

At production scale, a typical setup has 2-3 replicas per shard with a 3-node ClickHouse Keeper ensemble. The Keeper ensemble requires a quorum (2 of 3 nodes) to accept writes. If Keeper loses quorum, no new inserts can be registered (INSERT queries fail or queue in client buffers), but SELECT queries on existing data continue working. Replication queue lag is the most common operational issue: a slow network between replicas or a replica that was down for extended time must catch up by fetching many parts, which consumes network bandwidth and can further slow the cluster.

The non-obvious gotcha is split-brain during network partitions between Keeper and ClickHouse nodes. If a replica cannot reach Keeper but can still receive client connections, it will accept INSERTs into local parts but cannot register them in the replication log. When connectivity restores, these orphan parts must be reconciled. ClickHouse handles this with the insert_quorum setting: set it to 2 to ensure INSERTs only succeed when at least 2 replicas confirm the write, preventing orphan parts at the cost of higher INSERT latency. Also, replication queue entries for very large parts (hundreds of GB) can take hours to transfer, during which the replica serves stale data for those partitions.