How do you tune Linux kernel settings for high-performance containers (sysctl, huge pages, NUMA, scheduler)?

Think of tuning a race car. Factory settings are fine for city driving, but at 200 mph every detail matters: tire pressure, gear ratios, suspension, fuel mixture. Linux kernel defaults are safe for general use, not for maximum performance. High-performance workloads like payment processing, real-time bidding, or database containers need specific kernel adjustments matched to how they consume resources.

Kernel tuning for containers covers four areas: network stack, memory management, CPU scheduling, and I/O. Network-heavy services like API gateways need higher net.core.somaxconn (the listen backlog queue, default 4096 in modern kernels but often not enough during traffic bursts), net.ipv4.tcp_max_syn_backlog (queue for half-open connections), and net.core.netdev_max_backlog (packet queue before the kernel processes them). File descriptor limits (fs.file-max and per-process RLIMIT_NOFILE) must handle containers with thousands of concurrent connections. For Kubernetes hosts, net.ipv4.ip_local_port_range should be widened to 1024-65535 to prevent running out of ephemeral ports.

Huge pages reduce TLB (Translation Lookaside Buffer) misses for memory-heavy workloads. The TLB is a small cache that stores recent virtual-to-physical address translations. With standard 4KB pages, a 64GB database process needs 16 million page table entries. With 2MB huge pages, that drops to 32,768, dramatically reducing TLB pressure. Transparent Huge Pages (THP) try to merge small pages automatically but can cause latency spikes during defragmentation. Database workloads (PostgreSQL, Redis, Elasticsearch) often run better with THP turned off and explicit huge pages allocated via vm.nr_hugepages. In Kubernetes, huge pages are a first-class resource type (hugepages-2Mi, hugepages-1Gi) that pods can request.

NUMA (Non-Uniform Memory Access) awareness is crucial on multi-socket servers. A process running on CPU socket 0 that reads memory attached to socket 1 pays 1.5-2x higher latency. Container runtimes and Kubernetes support CPU pinning through cpuset cgroups and NUMA-aware memory allocation via the Topology Manager. The single-numa-node policy ensures all resources assigned to a pod (CPU, memory, devices) come from the same NUMA node. The CPU Manager's static policy gives exclusive CPU cores to high-priority pods, preventing the scheduler from bouncing processes between cores and trashing CPU caches.

The non-obvious trap is that kernel tuning is not one-size-fits-all. A sysctl value optimized for a 10Gbps network-heavy API gateway on a 96-core server can hurt performance on a 4-core VM running batch jobs. THP improves throughput for large sequential memory allocations but increases tail latency for latency-sensitive services because of compaction stalls. NUMA binding improves steady-state performance but reduces flexibility during load spikes, potentially leaving cores idle on one socket while containers queue up on another. The right approach is to profile first using tools like perf, mpstat, and numastat, then tune based on evidence rather than copying configuration snippets from blog posts.