Your Kubernetes cluster autoscaler is not scaling up even though pods are stuck in Pending state. Walk through your complete debugging process.
Quick Answer
Check node group limits, pod resource requests vs available instance types, PDB constraints, scheduling constraints (affinity/taints), and cloud provider quotas.
Detailed Answer
The Cluster Autoscaler (CA) simulates scheduling to determine if a new node would allow pending pods to be placed. Several conditions can prevent scale-up:
Debugging Flowchart
1. Check CA logs: kubectl logs -n kube-system deployment/cluster-autoscaler — look for 'scale up' or 'unschedulable' messages 2. Check CA status: kubectl get configmap cluster-autoscaler-status -n kube-system -o yaml 3. Verify node group limits: CA won't scale beyond maxSize in the ASG/MIG configuration 4. Pod scheduling constraints: Pods with nodeAffinity, nodeSelector, or tolerations for taints that no node group satisfies 5. Resource fit: If pods request 8Gi memory but the largest instance type in your node group only has 7.5Gi allocatable, CA can't find a fit 6. Cloud provider quotas: AWS EC2 vCPU limits, GCP quotas silently prevent instance creation 7. PodDisruptionBudgets: If existing PDBs prevent node draining during rebalancing 8. Scale-down cooldown: After a scale-down, there's a default 10-minute cooldown before scale-up
Production at Scale
Use multiple node groups with different instance types. Enable expander=least-waste for cost optimization. Set --balance-similar-node-groups=true for multi-AZ clusters. Monitor CA metrics via Prometheus: cluster_autoscaler_unschedulable_pods_count.
Code Example
# Check autoscaler logs kubectl logs -n kube-system -l app=cluster-autoscaler --tail=100 # Check pending pods and their events kubectl get pods --field-selector=status.phase=Pending -A kubectl describe pod <pending-pod> | grep -A 10 Events # Check node group status kubectl get configmap cluster-autoscaler-status -n kube-system -o yaml # Verify allocatable resources on existing nodes kubectl describe nodes | grep -A 5 'Allocatable'
Interview Tip
This is a Google/Amazon favorite. Walk through it systematically — logs first, then constraints, then cloud-level issues. Mention that CA simulates scheduling, not just counts pods.