How do you handle the split-brain problem in an Elasticsearch cluster? What is the minimum number of master-eligible nodes and why?
Quick Answer
Use an odd number of master-eligible nodes (minimum 3) with discovery.zen.minimum_master_nodes set to (N/2)+1. In ES 7+, the cluster coordination layer handles this automatically.
Detailed Answer
Split-Brain Problem
If a network partition divides the cluster, both sides could elect their own master, leading to two independent clusters accepting writes. When the partition heals, data is inconsistent and potentially lost.
Solution (ES 7+)
Elasticsearch 7+ uses a new cluster coordination subsystem based on the Raft consensus algorithm. It automatically calculates the quorum requirement — you just need to configure cluster.initial_master_nodes during bootstrap. The cluster won't elect a master unless a majority of master-eligible nodes agree.
Minimum Configuration
- 3 master-eligible nodes (quorum = 2) - Dedicated master nodes (don't use them as data nodes in production) - Spread across 3 availability zones for network partition resilience
Pre-7.0 (Legacy)
Required manual setting: discovery.zen.minimum_master_nodes: 2 for a 3-node cluster. If set incorrectly (e.g., 1), split-brain is possible.
Production Best Practice: Use 3 dedicated master nodes, separate from data nodes. This keeps cluster management lightweight and prevents resource contention from heavy indexing/searches affecting master stability.
Code Example
# elasticsearch.yml for dedicated master node node.name: master-1 node.roles: [master] cluster.name: production cluster.initial_master_nodes: ["master-1", "master-2", "master-3"] discovery.seed_hosts: ["master-1:9300", "master-2:9300", "master-3:9300"] # Check cluster health GET _cluster/health # Look for: status (green/yellow/red), number_of_nodes, active_shards # Check master election GET _cat/master?v # Shows current master node # Check node roles GET _cat/nodes?v&h=name,node.role,master
Interview Tip
Mention the Raft-based coordination in ES 7+ — it shows you know the modern approach, not just the legacy minimum_master_nodes setting.