How does Cilium use eBPF to replace kube-proxy and traditional service mesh sidecars, and what architectural tradeoffs should an architect evaluate before adopting it?

Think of a highway toll system. Traditional kube-proxy is like a toll booth where every car stops, gets checked, and is directed to its lane. EBPF with Cilium is like an electronic pass reader embedded in the road surface — the car never stops, the toll is processed at wire speed, and the road itself knows which lane to direct traffic into without a booth.

Cilium replaces the iptables-based kube-proxy and the user-space proxy model used by traditional service meshes. Instead of maintaining thousands of iptables rules that the kernel evaluates linearly, Cilium attaches eBPF programs to network hooks inside the kernel. These programs handle service IP translation, load balancing across endpoints, network policy enforcement, and even some L7 protocol parsing without packets ever leaving kernel space. This eliminates the context switches between kernel and user space that Envoy-based sidecars require for every connection.

Internally, Cilium uses several eBPF map types to store service endpoints, identity labels, policy rules, and connection tracking state. When a packet arrives, the eBPF program attached to the network interface or socket looks up the destination service, selects a backend Pod using consistent hashing or round-robin, rewrites headers, and forwards the packet — all within a single kernel function call chain. Hubble, the observability layer built on top of Cilium, taps into these eBPF data paths to provide flow logs, DNS visibility, and HTTP metrics without injecting any proxy.

At production scale, Cilium handles over 5,000 production deployments as of 2025, including platforms at Adobe, Bell Canada, and multiple hyperscalers. Teams should monitor eBPF program load errors, map memory usage, endpoint synchronization latency, dropped flow events in Hubble, and kernel version compatibility. Cilium requires Linux kernel 5.10 or later for full feature support, and some advanced features like bandwidth manager or BBR congestion control need even newer kernels.

The non-obvious gotcha is that Cilium does not fully replicate every L7 feature of Envoy-based meshes. While it handles mTLS via SPIFFE identities, basic HTTP routing, and L7 policy, complex traffic management like retries with budgets, circuit breaking with outlier detection, or gRPC-aware load balancing may still require a sidecar or gateway proxy. Architects should map their actual L7 requirements before declaring a full service mesh unnecessary, because removing sidecars and then re-adding them later is a painful migration.