How do you triage a broad production DevOps incident when the first report only says the application is unhealthy?

Think of this like arriving at a busy hospital emergency desk where someone only says, "a patient is sick." A good doctor does not immediately prescribe medicine. They ask whether the patient is breathing, when symptoms started, what changed recently, which vital signs are abnormal, and whether the issue affects one person or a whole group. Production incident response works the same way: the first job is to classify the problem so the team stops guessing and starts making targeted decisions.

In Kubernetes, a vague production issue can come from many layers: an application bug, a bad ConfigMap, a broken rollout, pod eviction, node pressure, ingress routing, DNS failure, database saturation, or cloud load balancer health checks. The designed advantage of Kubernetes is that it exposes state through consistent APIs. Pods show runtime health, Deployments show rollout status, Events show scheduling and probe failures, Services and Ingress show traffic paths, and metrics reveal pressure. The operator's job is to read those signals in the right order.

A disciplined triage sequence starts with impact: which endpoint is failing, what is the 5xx rate, what latency percentile changed, and when did it begin? Next, inspect the cluster state with kubectl get pods, events, rollout status, and logs for the affected namespace. Then check resource pressure with kubectl top nodes and kubectl top pods. If the issue lines up with a recent deployment, compare the new ReplicaSet, image tag, environment variables, and readiness probe behavior. If traffic is reaching pods but failing downstream, inspect dependencies such as database connection pools, queues, caches, and external APIs.

At production scale, the main mistake is treating all failures as application failures. A pod can be healthy but unreachable because a Service selector changed. A Deployment can be complete but still serving errors because readiness probes only check process liveness. A database can be the real bottleneck while Kubernetes only shows pods restarting from connection timeouts. Engineers monitor request rate, error rate, latency, saturation, pod restarts, node conditions, ingress controller errors, and dependency health together because no single metric tells the full story.

The non-obvious gotcha is that the safest technical action depends on the diagnosis confidence. Rolling back is often correct after a bad deploy, but it can make a database migration mismatch worse. Scaling replicas helps CPU saturation, but it can overwhelm a database with more connections. Restarting pods can clear transient bad state, but it can also destroy evidence and amplify a thundering herd. Strong incident responders separate mitigation from root cause analysis: stabilize users first, preserve enough evidence, and record every action with timestamps.