How would you investigate a production rollout that causes a spike in 5xx errors after a new Deployment is applied?

When a deployment introduces rising 5xx errors, the safest first move is to treat the issue as a blast-radius problem rather than a simple app bug. I would compare the current rollout status with the last known good revision and ask whether the new image, config, or resource requests changed. In practice the fastest path is to inspect the Deployment rollout, the ReplicaSet history, and the pod-level readiness and restart state so I can tell whether the service is failing to become ready, crashing on startup, or simply receiving traffic before it is healthy.

The next step is to look at Kubernetes events and logs for the affected pods. A common pattern is that the image starts but the application never becomes ready because the readiness probe is too strict, the app binds to the wrong port, or dependencies are still warming up. If traffic is already hitting the new pods, I would verify service selectors, endpoint slices, and ingress routing to make sure the workload is actually reachable.

At scale, the important question is whether the rollout is creating a partial outage or a full incident. I would check whether the new replicas are landing on saturated nodes, whether a resource request or limit mismatch causes throttling, or whether a configmap or secret mount changed and broke startup. The right response depends on whether the failure is from a rollout bug, a capacity issue, or a dependency issue.

During incident response I focus on reducing impact quickly: pause the rollout, revert to the previous revision if necessary, and keep enough capacity for the old version to serve traffic while the new version is stabilized. I would also review the telemetry pipeline to ensure alerting reflects what operators actually need to see, because many incidents are escalated by bad dashboards rather than by missing data.

The non-obvious trap in this scenario is that a healthy-looking Deployment can still fail in production if the readiness gate is misconfigured or if the service depends on a slow-starting dependency. Senior engineers treat these situations as a topology problem: they verify endpoints, traffic routing, and runtime dependencies before assuming the image itself is broken.