How would you automate Kafka topic configuration changes (like partition count or retention increases) so that a bad change can't silently break every consumer, and what has to be idempotent about that automation?
Quick Answer
Topic config automation must diff desired-state config against live broker state before applying anything, since some changes (like lowering partition count) are impossible and others (like changing a partitioning-relevant setting) have irreversible side effects on consumers. The automation itself must be idempotent — safe to re-run — and must gate destructive or irreversible operations like partition increases behind an explicit dry-run and approval step rather than applying them automatically on every pipeline run.
Detailed Answer
Think of topic configuration automation like a building permit process for renovating a house that's currently occupied. Some changes are reversible and low-risk — repainting a room (adjusting a retention value) can be done and undone freely. Others are irreversible the moment you do them — knocking down a load-bearing wall (increasing partition count, which reshuffles which key maps to which partition forever) can't be undone by just reversing the automation, because the house is still occupied while you do it and residents (consumers) get disrupted mid-change.
Kafka's AdminClient API and CLI tools (kafka-configs.sh, kafka-topics.sh --alter) expose topic configuration as declarative key-value pairs, which makes it tempting to manage them the same way as any other infrastructure-as-code resource — diff desired state against live state, apply the delta. This works cleanly for reversible settings like retention.ms, cleanup.policy, or min.insync.replicas. It does not work safely for partition count, because Kafka only supports increasing partitions, never decreasing them, and increasing partitions changes the hash-to-partition mapping for every existing key going forward, silently breaking any consumer logic that assumed a key always lands on the same partition it always has.
A safe automation pipeline therefore treats topic config changes as two distinct categories internally: mutable-safe settings get a standard plan-diff-apply flow, similar to Terraform, run automatically in CI on every merge. Partition count changes (and any other irreversible operation) get routed through a separate path that requires a human-reviewed dry-run showing exactly which topics and consumer groups are affected, and typically a scheduled maintenance window, because the blast radius touches every consumer of that topic simultaneously, not just the automation's direct target.
In production, idempotency for this kind of automation means the pipeline can be re-run safely after a partial failure without making things worse — for example, if a run to bump retention.ms on 40 topics fails after updating 22 of them, re-running it should update only the remaining 18, not error out or reapply to the already-correct ones. This requires the automation to always re-fetch live config immediately before diffing, never trust a cached 'last known state,' since another process or engineer could have changed the topic out of band between runs.
The gotcha experienced platform teams learn the hard way: kafka-configs.sh --alter with --add-config fully replaces the config map for the keys specified but silently leaves unspecified keys untouched — which sounds safe, but means an automation script that generates its config payload from a slightly stale template can silently fail to remove a deprecated setting that a previous manual change added, because the automation only ever adds and never explicitly reconciles the full config against a source of truth, leaving configuration drift that only shows up during the next incident when someone assumes a setting isn't there anymore.