What happens during a Kubernetes rolling update, and how do you rollback a bad deployment?

Think of a rolling update like replacing light bulbs in a theater while the show is still running. You can't turn off all the lights at once (that's a Recreate strategy — total blackout). Instead, you unscrew one old bulb, screw in a new one, verify it lights up (readiness probe), and only then move to the next one. At every moment during the replacement, the audience still has enough light to see the show. If a new bulb turns out to be defective (bad deployment), you stop the replacement and screw the old bulbs back in (rollback).

When you update a Deployment (change the container image, environment variables, or any field in the Pod template), the Deployment controller creates a brand new ReplicaSet with the updated Pod template. It then orchestrates a carefully choreographed transition between the old and new ReplicaSets. The two parameters that control this dance are maxSurge (how many extra Pods above the desired count are allowed during the update — controls speed) and maxUnavailable (how many Pods can be offline simultaneously — controls safety margin). With replicas=4, maxSurge=1, maxUnavailable=1: Kubernetes can have up to 5 Pods running (4+1 surge) and at minimum 3 Pods available (4-1 unavailable) at any point.

The update proceeds in cycles. In each cycle: (1) the new ReplicaSet is scaled up by maxSurge number of Pods, (2) Kubernetes waits for those new Pods to pass their readiness probe (this is the critical gate — without a readiness probe, Kubernetes considers Pods ready immediately, potentially sending traffic to uninitialized applications), (3) once new Pods are Ready, the old ReplicaSet is scaled down by maxUnavailable Pods. This cycle repeats until all old Pods are terminated and all new Pods are running. The entire process is recorded as a new revision in the Deployment's rollout history.

Rollback is elegantly simple because Kubernetes keeps old ReplicaSets around (scaled to 0 replicas). When you run kubectl rollout undo deployment/payments-api, the Deployment controller doesn't create anything new — it simply scales up the previous ReplicaSet (which still has the old Pod template with the known-good image) and scales down the current one. This means rollback is typically faster than the original deployment because: the old image may still be cached on nodes (no pull needed), and the old ReplicaSet already exists (no creation delay). You can also rollback to a specific revision with kubectl rollout undo --to-revision=3.

By default, Kubernetes keeps the last 10 old ReplicaSets (controlled by revisionHistoryLimit). Setting this too low (like 1) means you can only undo one step back. Setting it too high wastes API Server memory with stale ReplicaSet objects. For most teams, 5 revisions is the sweet spot.

The most critical production gotcha: if you don't have a readiness probe, Kubernetes considers Pods ready the instant the container process starts — even if your Spring Boot app needs 45 seconds to initialize. During a rolling update, traffic gets routed to these half-started Pods, causing 500 errors for real users. The second gotcha: if your readiness probe never passes (bug in health endpoint, wrong port, misconfigured path), the rollout hangs forever — new Pods stay in a NotReady state, old Pods never get terminated, and the Deployment reports 'waiting for rollout to finish'. Use progressDeadlineSeconds (default 600s) to automatically mark a stuck rollout as Failed after a timeout.