What common issues do you observe during CI/CD deployments to Kubernetes, and how do you systematically resolve them?
Quick Answer
Common CI/CD deployment issues include image pull failures (wrong tag or registry auth), resource quota exhaustion, failing readiness probes blocking rollout, ConfigMap or Secret mismatches between environments, and RBAC permission errors. Systematic resolution involves checking Events, pod logs, describe output, and rollout history.
Detailed Answer
Think of moving into a new apartment. Common problems are the moving truck arriving at the wrong address (image pull errors), the apartment not having enough power outlets (resource limits), the door lock not matching your key (RBAC errors), and the furniture not fitting through the doorway (resource quota). Each problem looks different but follows a predictable troubleshooting pattern.
In Kubernetes CI/CD, deployment failures cluster around a few categories. Image-related failures happen when the image tag does not exist in the registry, registry credentials are expired, or the image was pushed to a different repository than the manifest references. Resource failures occur when the namespace has a ResourceQuota and the new deployment exceeds CPU or memory limits. Configuration failures happen when a ConfigMap or Secret referenced by the pod does not exist in the target namespace or has different keys than the application expects.
The troubleshooting sequence is predictable. First, check the Deployment rollout status with kubectl rollout status. If the rollout is stuck, describe the Deployment to see the events. Then check the ReplicaSet events to see why new pods are not being created. If pods exist but are not ready, check pod events with kubectl describe pod, then check container logs with kubectl logs. For image pull errors, the events will show ErrImagePull or ImagePullBackOff with a specific error message. For resource issues, the events will show FailedScheduling or quota exceeded messages.
At production scale, the most impactful issues are deployments that pass in lower environments but fail in production. This usually happens because of environment-specific differences: different resource quotas, different network policies blocking connectivity, different secrets or certificates, or different node configurations. Teams prevent this by making environments as similar as possible, using the same Helm chart with different values, and running integration tests in a staging environment that mirrors production networking and security policies.
The non-obvious gotcha is that a deployment can appear successful — kubectl rollout status reports completion — but the application is still broken. This happens when readiness probes are too lenient (checking only that the HTTP port is open, not that the application can actually serve requests). Teams should use deep health checks that verify database connectivity, downstream service availability, and application-specific readiness before marking a pod as ready.