How do you troubleshoot a production incident in Kubernetes and maintain RCA documentation to prevent the same issue from recurring?

Think of a hospital emergency room. When a patient arrives, the team follows a triage protocol: check vitals first, then run diagnostics, then treat. They do not start with the most complex test — they start with the quickest indicators and narrow down. After the patient recovers, the team documents what happened, why, and how to improve response for future cases. Kubernetes incident response works the same way.

In Kubernetes, production incident triage starts with the broadest view and narrows. First, check cluster health: are nodes Ready, is the control plane responsive? Then check the affected namespace: are pods Running, are services reachable? Then drill into the specific failing component: what do the pod events say, what do the container logs show, what are the resource use numbers? This top-down approach prevents wasting time on application logs when the real problem is a node running out of disk space.

The diagnostic sequence uses specific commands at each level. For cluster health: kubectl get nodes and kubectl top nodes. For namespace health: kubectl get pods -n payments and kubectl get events -n payments --sort-by=lastTimestamp. For pod-level diagnosis: kubectl describe pod, kubectl logs with --previous for crashed containers, and kubectl exec for connectivity tests. For resource issues: kubectl top pods and checking against resource requests and limits. For networking: checking Service endpoints, NetworkPolicy rules, and DNS resolution from inside the pod.

At production scale, RCA documentation follows a blameless post-mortem template. The document captures: incident summary (what happened, when, impact), timeline (when detected, when triaged, when resolved), root cause (the specific technical failure), contributing factors (what made detection or resolution slower), resolution steps (what was done to fix it), and preventive actions (what changes will prevent recurrence). Each preventive action should be a concrete ticket with an owner and deadline — not vague statements like 'improve monitoring.' Examples: 'Add PDB for payments-api with minAvailable=2' or 'Create OOMKilled alert with threshold at 80% memory use.'

The non-obvious gotcha is that teams often fix the immediate symptom without addressing the root cause. A pod restarting due to OOMKilled might be fixed by increasing memory limits, but the real cause might be a memory leak in the application, an unbounded cache, or a goroutine leak. The RCA should distinguish between the immediate fix (increase limits) and the long-term fix (patch the memory leak). Without this distinction, the same incident recurs at a larger scale.