How do you decide whether to roll back, scale up, or restart services during a Kubernetes production incident?

Imagine a restaurant suddenly falling behind during dinner service. If the chef changed the recipe ten minutes ago and every dish is being returned, the fix is to go back to the old recipe. If the recipe is fine but twice as many customers arrived, add staff or open more stations. If one oven controller is stuck in a bad state, restart that oven carefully. Production Kubernetes decisions are similar: rollback, scale, and restart solve different classes of failure.

Kubernetes gives teams several recovery levers, but each lever changes the system in a different way. A rollout undo changes the running application version back to a previous ReplicaSet. Scaling replicas increases concurrency and spreads load across more pods. Restarting pods replaces running containers while keeping the same declared configuration. These actions are powerful because Kubernetes controllers continuously reconcile desired state, but they can also worsen an incident if chosen for the wrong failure mode.

The decision sequence begins with timing. If 5xx errors, failed probes, or latency spikes started immediately after a deployment, check deployment history, image tag, ReplicaSet age, and application logs. A rollback is usually safest if the previous version was healthy and no incompatible database migration was applied. If metrics show CPU saturation, queue depth growth, or request concurrency exceeding capacity while the release is unchanged, scaling replicas or nodes can restore headroom. If pods have memory leaks, stuck threads, expired connections, or repeated CrashLoopBackOff after a transient dependency outage, a controlled restart can help once downstream systems are stable.

At scale, mitigation must account for dependencies. Scaling an API from 10 to 50 replicas might increase database connections fivefold and turn partial latency into a full database outage. Rolling back after a schema migration may break writes if the old app does not understand new columns or constraints. Restarting all pods at once can drop warm caches, reset connection pools, and create synchronized traffic spikes. Mature teams use PodDisruptionBudgets, maxUnavailable, readiness probes, connection limits, and canary rollback procedures to avoid turning a small failure into a cascading one.

The subtle gotcha is that Kubernetes success messages do not guarantee business recovery. A rollout can be "successfully rolled back" while the ingress is still returning 503 because endpoints are not ready. A scale-up can create running pods that fail readiness because a ConfigMap is wrong. A restart can clear application logs from the previous container unless you capture --previous logs first. Always verify with external health checks, error rate, latency percentiles, and dependency metrics before declaring the incident resolved.