Explain the differences between kube-proxy's iptables mode, IPVS mode, and eBPF-based alternatives like Cilium. When would you choose each, and what are the performance implications at 10,000+ services?
Quick Answer
iptables mode uses O(n) linear chain traversal per packet, IPVS uses O(1) hash table lookups, and eBPF (Cilium) bypasses iptables entirely for O(1) lookups with additional features like transparent encryption. At 10,000+ services, iptables becomes a significant bottleneck.
Detailed Answer
iptables Mode (Default)
kube-proxy creates iptables rules for every Service/Endpoint combination. For a ClusterIP service with 3 endpoints, it creates ~8 rules (KUBE-SERVICES chain -> KUBE-SVC-xxx -> KUBE-SEP-xxx with probability-based load balancing). At 10,000 services with 3 endpoints each, you get ~240,000 iptables rules. Every packet traverses these chains linearly. Rule update time is O(n) because iptables does a full table replacement (iptables-restore). During updates, there's a brief period where connections can be dropped.
IPVS Mode
IPVS (IP Virtual Server) uses kernel-level hash tables for service-to-endpoint mapping. Lookup is O(1) regardless of service count. It supports multiple load balancing algorithms (round-robin, least-connection, source-hash). Connection tracking is built-in. At 10,000 services, IPVS adds negligible latency (<0.1ms) while iptables adds 1-5ms per connection. However, IPVS still uses iptables for SNAT/masquerade operations, so it doesn't fully eliminate iptables dependency.
eBPF with Cilium
Cilium attaches eBPF programs to network interfaces, bypassing both iptables and IPVS entirely. Service routing happens in the kernel at the socket/TC level before packets enter the Linux networking stack. Benefits include: true O(1) lookups, no conntrack dependency (optional), transparent encryption (WireGuard/IPsec), L7 policy enforcement, and observability via Hubble. At 10,000+ services, Cilium maintains consistent <0.1ms overhead.
Decision Framework
- Small clusters (<200 services): iptables is fine, simple to debug - Medium clusters (200-5000 services): IPVS for performance without major architecture change - Large clusters (5000+ services) or need L7 visibility: Cilium/eBPF - Regulatory requirements for encryption in transit: Cilium with WireGuard