What Kubernetes evidence would you collect before diagnosing whether a production outage is caused by the application, the cluster, networking, or the database?

Think of a package delivery problem. A customer says the package never arrived, but the failure could be at the warehouse, the truck, the street address, the building front desk, or the customer's mailbox. Looking only at the final missing package does not identify the failed handoff. In production Kubernetes, a request travels through DNS, load balancer, ingress, service, endpoints, pods, application code, and dependencies such as databases or queues. Evidence must follow that path.

Kubernetes separates concerns across objects. Deployments and ReplicaSets show what version should be running. Pods show container state, restarts, probes, and scheduling. Events explain why Kubernetes could not do something, such as pull an image, mount a volume, or schedule onto a node. Services and Endpoints show whether traffic can reach ready pods. Ingress controllers and load balancers show whether external traffic is accepted and routed correctly. Logs and metrics show what the application did once traffic arrived.

A practical diagnosis starts with the user-facing endpoint and works inward. If curl to the public health endpoint returns a TLS, DNS, or 503 error, inspect ingress and service routing. If endpoints are empty, check readiness probes, selectors, and pod labels. If endpoints exist and pods are ready but responses are 500, inspect application logs, recent deploys, and dependency errors. If logs contain database timeouts or connection pool exhaustion, move to database metrics such as active connections, locks, slow queries, or CPU. If many pods are Pending, Evicted, or OOMKilled, investigate node pressure and resource requests.

At production scale, correlation matters more than one-off errors. A single pod restart may be normal during rolling deploys, while simultaneous restarts across many pods indicate a shared dependency or node-level issue. A spike in 5xx errors at the ingress with no pod logs may mean traffic never reached the app. High p95 latency with normal CPU can point to downstream waiting, not compute saturation. Good dashboards combine golden signals: request rate, error rate, duration, saturation, restarts, readiness, queue depth, and database pool usage.

The non-obvious gotcha is that Kubernetes readiness can hide failures in both directions. If readiness is too shallow, broken pods remain in service and return errors. If readiness depends on a fragile downstream check, a database blip can remove every pod from endpoints and cause a total outage. Evidence collection should include the probe definitions themselves, not just their pass/fail status. A production-ready diagnosis asks whether the health model matches the actual business path.