You are using Karpenter for node provisioning on EKS and notice that during a traffic spike, new nodes are provisioned but pods remain Pending for 3-4 minutes. What is causing the delay and how do you optimize node startup time?
Quick Answer
Node startup latency is composed of EC2 instance launch time (~60s), kubelet registration (~15s), CNI plugin initialization (~10s), and container image pull time (~60-120s). Optimize with warm pools, bottlerocket AMIs, image caching, and Karpenter's consolidation settings.
Detailed Answer
Breakdown of Node Startup Timeline
Karpenter's advantage over Cluster Autoscaler is direct EC2 fleet API calls (no ASG intermediary), but total pod-ready time still has multiple phases:
1. Karpenter scheduling decision (~5s): Karpenter evaluates pending pods, selects instance types, and calls EC2 RunInstances 2. EC2 instance launch (30-90s): Depends on instance type availability and capacity. Spot instances may take longer due to capacity rebalancing 3. Node bootstrap (15-30s): Cloud-init/userdata execution, kubelet startup, node registration with API server 4. CNI initialization (5-15s): VPC CNI assigns ENI and IP addresses. With prefix delegation, this is faster but warm IP pool needs tuning 5. Container image pull (30-180s): The biggest variable. Large images (>1GB) on cold nodes dominate total startup time
Optimization Strategies
- Bottlerocket OS: Purpose-built container OS with faster boot time (~20s vs ~45s for Amazon Linux 2) - Image caching with Spegel: P2P image distribution across nodes eliminates registry pull time for common images - Karpenter NodePool configuration: Use kubelet.maxPods and instance type constraints to ensure optimal bin-packing - VPC CNI warm pool: Set WARM_IP_TARGET=5 to pre-allocate IPs on existing nodes - Drift detection: Use Karpenter's spec.disruption settings to proactively replace nodes before they become stale
Production Configuration
Set Karpenter's consolidationPolicy: WhenEmptyOrUnderutilized with consolidateAfter: 30s for cost optimization. Use nodeClassRef to specify fast-launching AMIs. Configure pod topology spread constraints to prevent all pods landing on a single new node.
Code Example
# Karpenter NodePool optimized for fast startup
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
name: fast-startup
spec:
template:
spec:
requirements:
- key: karpenter.sh/capacity-type
operator: In
values: ["on-demand", "spot"]
- key: node.kubernetes.io/instance-type
operator: In
values: ["m6i.xlarge", "m6i.2xlarge", "m5.xlarge"]
nodeClassRef:
group: karpenter.k8s.aws
kind: EC2NodeClass
name: bottlerocket-fast
disruption:
consolidationPolicy: WhenEmptyOrUnderutilized
consolidateAfter: 30s
---
apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
name: bottlerocket-fast
spec:
amiSelectorTerms:
- alias: bottlerocket@latest
blockDeviceMappings:
- deviceName: /dev/xvdb
ebs:
volumeSize: 100Gi
volumeType: gp3
iops: 5000
throughput: 250
userData: |
[settings.kubernetes]
max-pods = 110
[settings.kubernetes.node-labels]
pool = "fast-startup"Interview Tip
Break down the startup timeline with actual numbers. Interviewers want to see you can identify the bottleneck (usually image pull) and propose specific solutions. Mention Bottlerocket for OS-level optimization and Spegel/Dragonfly for P2P image distribution.