What changed with Kafka KRaft mode replacing ZooKeeper, and why does it matter?

Think of a company that used to outsource its HR department to an external firm (ZooKeeper). Every hiring decision, org chart change, and payroll update had to go through the external firm, adding latency and a dependency. KRaft mode brings HR in-house — the company manages its own employee records directly, which is faster, simpler, and eliminates the external dependency.

Historically, Kafka relied on Apache ZooKeeper for metadata management: tracking which brokers are alive, which broker is the controller, partition leadership assignments, topic configurations, and ACLs. This meant operating two distributed systems — Kafka and ZooKeeper — each with their own deployment, monitoring, scaling, and failure modes. ZooKeeper required its own ensemble of 3 or 5 nodes, its own storage, and its own expertise to troubleshoot.

KRaft (Kafka Raft) replaces ZooKeeper with a built-in metadata quorum. A subset of Kafka nodes run as controllers using the Raft consensus protocol to manage cluster metadata. The controller quorum elects a leader, and all metadata changes (topic creation, partition reassignment, broker registration) go through this leader. The metadata is stored as a replicated log within Kafka itself, using the same storage engine as regular Kafka topics. This means controller failover happens in milliseconds instead of the seconds it took with ZooKeeper leader election.

At production scale, KRaft simplifies operations significantly. You deploy and manage one system instead of two. Cluster startup is faster because brokers do not need to wait for ZooKeeper to be available. Scaling is simpler because you do not need to resize the ZooKeeper ensemble separately. Monitoring is unified — you no longer need separate dashboards and alerts for ZooKeeper health. Strimzi supports KRaft mode since Kafka 3.5, and ZooKeeper support is being deprecated.

The non-obvious gotcha is that KRaft changes how you think about controller nodes. In ZooKeeper mode, any broker could become the controller. In KRaft mode, you explicitly designate controller nodes (or run combined controller+broker nodes for smaller clusters). For production, dedicated controller nodes are recommended because they avoid resource contention between metadata operations and message processing. Teams migrating from ZooKeeper mode should test KRaft in staging first, as some older Kafka clients may not support KRaft-specific metadata protocols.