How would you design automatic self-healing for EKS worker nodes so that systemd/kubelet failures are detected and unhealthy nodes are replaced without manual intervention?
Quick Answer
Layer node-level health checks (systemd watchdog on kubelet/containerd, EC2 instance status checks) with cluster-level remediation: enable EC2 Auto Scaling Group health checks plus a node-problem-detector + draino/Karpenter combo (or AWS's own node auto-repair) so a node that fails liveness gets cordoned, drained, and terminated, letting the ASG/Karpenter replace it automatically.
Detailed Answer
Self-healing for EKS nodes operates at two layers and you need both
1. Process-level (systemd): configure systemd watchdogs (WatchdogSec=) on kubelet and containerd units so that if either hangs (not crashes — hangs, which is the harder case), systemd restarts the unit automatically. Combine with Restart=on-failure and a sane RestartSec to avoid restart storms.
2. Node-level: a process-level restart doesn't help if the underlying EC2 instance itself is degraded (hardware fault, kernel panic, network partition). For that you need node-problem-detector (NPD) running as a DaemonSet, watching kernel logs, NTP drift, filesystem corruption, and conntrack table exhaustion. NPD surfaces these as Node conditions and Events; combine it with a remediation controller — draino (cordons and drains nodes matching a bad condition) or Karpenter's built-in node disruption/health features, or AWS's managed node auto-repair if you're on EKS managed node groups.
3. Infrastructure-level: the ASG (or Karpenter NodePool) needs EC2 health checks and ELB health checks enabled so that an instance failing AWS-level status checks gets terminated and replaced automatically, independent of what's happening inside Kubernetes.
The key design point interviewers probe for: don't just restart kubelet blindly — distinguish 'process is wedged, restart it' from 'node is bad, evacuate workloads first, then terminate.' A drain-before-terminate path matters because killing a node with stuck kubelet but otherwise-healthy pods can cause unnecessary pod disruption if a simple kubelet restart would have fixed it.
Code Example
# systemd watchdog for kubelet (in kubelet.service override)
[Service]
WatchdogSec=30
Restart=on-failure
RestartSec=5
# Deploy node-problem-detector
kubectl apply -f https://raw.githubusercontent.com/kubernetes/node-problem-detector/master/deployment/node-problem-detector.yaml
# Check node conditions NPD has surfaced
kubectl get nodes -o json | jq '.items[].status.conditions[] | select(.type=="KernelDeadlock" or .type=="FrequentKubeletRestart")'
# Karpenter: node disruption budget for automated unhealthy-node replacement
apiVersion: karpenter.sh/v1
kind: NodePool
spec:
disruption:
consolidationPolicy: WhenEmptyOrUnderutilized
budgets:
- nodes: "10%"Interview Tip
Structure the answer in layers (process / node / infrastructure) instead of naming one tool. The strongest signal is distinguishing a wedged-process restart from a node-level evacuation, since that's the actual design decision behind self-healing automation, not just 'use a watchdog.'