Kubernetes Networking Deep Dive

Everything about how traffic flows inside and into a Kubernetes cluster.

The Four Networking Rules

Kubernetes mandates these rules (implemented by the CNI plugin):

Every Pod gets a unique cluster-wide IP — no NAT between pods
Pods on any node can talk to Pods on any other node without NAT
Nodes can communicate with Pods without NAT
A Pod's self-view IP is the same as what other Pods see (no masquerading)

Network Layers in Kubernetes

Internet
   │
   ▼
Cloud Load Balancer (L4 TCP)
   │
   ▼
NodePort (30000-32767)
   │
   ▼
kube-proxy (iptables / IPVS rules on every node)
   │
   ▼
Service ClusterIP (virtual IP — lives in iptables)
   │
   ▼
Pod IP (assigned by CNI plugin — e.g., 10.244.2.5)
   │
   ▼
Container port

CNI Plugins Compared

Plugin	Dataplane	NetworkPolicy	Encryption	Best for
Flannel	VXLAN overlay	❌ No	❌ No	Simple clusters, dev
Calico	eBPF or iptables	✅ Yes	✅ WireGuard	Production, policy-heavy
Cilium	eBPF	✅ Yes (L3-L7)	✅ WireGuard	High perf, observability
Weave	VXLAN mesh	✅ Yes	✅ AES	Simple multi-cluster
AWS VPC CNI	Native VPC	✅ (Security Groups)	AWS managed	EKS
Azure CNI	Native VNet	✅ Yes	Azure managed	AKS

Flannel does NOT support NetworkPolicies. Always use Calico or Cilium in production.

Service Types Deep Dive

ClusterIP (default)

                  ClusterIP: 10.96.45.23:80
                        │
              iptables DNAT rule (kube-proxy)
             /           |           \
       Pod 10.0.1.5  Pod 10.0.2.3  Pod 10.0.3.7

spec:
  type: ClusterIP
  clusterIP: 10.96.45.23   # auto-assigned; set to "None" for headless
  ports:
  - port: 80
    targetPort: 8080

NodePort

Exposes the Service on every node's IP at a static port (30000-32767).

External client → <any-node-ip>:31000
                        │
                    kube-proxy (iptables)
                        │
               forwards to any matching Pod
               (even if Pod is on another node)

spec:
  type: NodePort
  ports:
  - port: 80
    targetPort: 8080
    nodePort: 31000      # omit for auto-assignment

LoadBalancer

Creates an external cloud load balancer (AWS NLB, GCP LB, Azure LB) that forwards to NodePorts.

spec:
  type: LoadBalancer
  ports:
  - port: 80
    targetPort: 8080
  # Cloud-specific annotations:
  # AWS: service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
  # GCP: cloud.google.com/load-balancer-type: "Internal"

Headless Service (StatefulSets / DNS discovery)

clusterIP: None — no virtual IP. DNS returns individual Pod IPs directly.

spec:
  clusterIP: None   # headless
  selector:
    app: postgres

DNS resolves:

postgres.default.svc.cluster.local → A records for each Pod IP
postgres-0.postgres.default.svc.cluster.local → Pod-0 IP (StatefulSet)

ExternalName

DNS alias to an external hostname — no proxy, just CNAME.

spec:
  type: ExternalName
  externalName: my-database.rds.amazonaws.com

DNS in Kubernetes

CoreDNS runs in kube-system and handles all DNS for the cluster.

DNS record formats

# Service
<svc-name>.<namespace>.svc.cluster.local

# Pod (rarely used)
<pod-ip-dashed>.<namespace>.pod.cluster.local
# e.g. 10-244-2-5.default.pod.cluster.local

# StatefulSet Pod
<pod-name>.<headless-svc>.<namespace>.svc.cluster.local
# e.g. postgres-0.postgres.default.svc.cluster.local

ndots and search domains

Every pod gets /etc/resolv.conf:

search default.svc.cluster.local svc.cluster.local cluster.local
nameserver 10.96.0.10   # CoreDNS ClusterIP
options ndots:5

ndots:5 means: if the name has fewer than 5 dots, try appending search domains first. This is why kubectl exec pod -- wget my-svc works (resolves to my-svc.default.svc.cluster.local).

Debugging DNS

# Test DNS from inside a pod
kubectl run dns-test --image=busybox --rm -it -- nslookup kubernetes.default

# Check CoreDNS pods
kubectl get pods -n kube-system -l k8s-app=kube-dns

# Check CoreDNS logs
kubectl logs -n kube-system -l k8s-app=kube-dns

# Check CoreDNS config
kubectl get configmap coredns -n kube-system -o yaml

# Test cross-namespace resolution
nslookup my-svc.other-namespace.svc.cluster.local

kube-proxy Modes

kube-proxy runs on every node and programs network rules for Services.

Mode	How it works	Scale	Notes
iptables (default)	Installs iptables DNAT rules	O(n) rules, degrades at 10k+ services	Most widely used
IPVS	Linux kernel LB, hash table lookup	O(1)	Better for large clusters; `--proxy-mode=ipvs`
nftables	Modern iptables replacement (K8s 1.29 beta)	Better than iptables	Replacing iptables in future
eBPF (Cilium)	Bypass iptables entirely	Near line rate	Requires Cilium CNI

# Check current mode
kubectl get configmap kube-proxy -n kube-system -o yaml | grep mode

Ingress Deep Dive

How Ingress works

Internet → Cloud LB → Ingress Controller Pod → Service → Pods
                             │
                         Reads Ingress objects
                         Configures NGINX/Envoy/Traefik

The IngressController is a Deployment that:

Watches all Ingress objects in the cluster
Dynamically configures its proxy (nginx, Envoy, etc.)
Exposes itself via a LoadBalancer Service

Popular Ingress Controllers

Controller	Based on	Features
ingress-nginx	NGINX	Most widely used, battle-tested
Traefik	Traefik proxy	Auto Let's Encrypt, dashboard
AWS ALB Controller	AWS ALB	Native AWS integration, WAF
Kong	NGINX + Kong	API gateway features
Istio Gateway	Envoy	Service mesh integration

# Install ingress-nginx
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm install ingress-nginx ingress-nginx/ingress-nginx \
  --namespace ingress-nginx \
  --create-namespace \
  --set controller.replicaCount=2

Ingress with TLS, path routing, rewriting

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app-ingress
  namespace: production
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$2
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/proxy-body-size: "50m"
    cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - api.example.com
    secretName: api-tls-cert
  rules:
  - host: api.example.com
    http:
      paths:
      - path: /v1(/|$)(.*)
        pathType: Prefix
        backend:
          service:
            name: api-v1
            port:
              number: 80
      - path: /v2(/|$)(.*)
        pathType: Prefix
        backend:
          service:
            name: api-v2
            port:
              number: 80
      - path: /
        pathType: Prefix
        backend:
          service:
            name: frontend
            port:
              number: 80

NetworkPolicy Deep Dive

By default: all pods can talk to all other pods. NetworkPolicy restricts this.

Default deny all (start here)

# Deny ALL ingress and egress in a namespace
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: production
spec:
  podSelector: {}    # selects ALL pods
  policyTypes:
  - Ingress
  - Egress

Allow specific traffic

# Allow: frontend → api on port 8080
# Allow: api → postgres on port 5432
# Allow: all pods → DNS on port 53
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-policy
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: api
  policyTypes:
  - Ingress
  - Egress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: frontend
    ports:
    - protocol: TCP
      port: 8080
  - from:
    - namespaceSelector:
        matchLabels:
          name: monitoring    # allow Prometheus scraping
    ports:
    - protocol: TCP
      port: 9090
  egress:
  - to:
    - podSelector:
        matchLabels:
          app: postgres
    ports:
    - protocol: TCP
      port: 5432
  - to: []                    # allow DNS
    ports:
    - protocol: UDP
      port: 53
    - protocol: TCP
      port: 53

Service Topology & Traffic Distribution

# K8s 1.32+ — route to nearest endpoint first (same zone, then cross-zone)
apiVersion: v1
kind: Service
metadata:
  name: api
spec:
  trafficDistribution: PreferClose   # stable in K8s 1.32
  selector:
    app: api
  ports:
  - port: 80

TopologySpreadConstraints — spread pods evenly across zones:

spec:
  topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
      matchLabels:
        app: api
  - maxSkew: 1
    topologyKey: kubernetes.io/hostname
    whenUnsatisfiable: ScheduleAnyway
    labelSelector:
      matchLabels:
        app: api

Common Interview Questions

Q: How does a Service know which Pods to send traffic to?

The Service has a selector (label map). The Endpoints controller watches for Pods matching that selector and updates the Endpoints object. kube-proxy reads Endpoints and programs iptables/IPVS rules. Only Ready pods (passing readiness probe) are added.

Q: What happens when you hit a ClusterIP?

ClusterIP is a virtual IP — no process listens on it. kube-proxy on each node installs iptables DNAT rules that intercept packets destined for the ClusterIP and rewrite the destination to a random Pod IP.

Q: What is the difference between port, targetPort, and nodePort on a Service?

port = what clients connect to on the Service (ClusterIP:port). targetPort = the port on the Pod the traffic is forwarded to. nodePort = the port opened on every node's IP for NodePort/LoadBalancer Services.

Q: Why would you use a Headless Service?

When you need direct Pod IPs (not load-balanced). StatefulSets use headless Services so each pod gets a stable DNS name (pod-0.svc.ns.svc.cluster.local). Also used for client-side load balancing in databases like Cassandra.

Q: What is an Ingress vs a Gateway API?

Ingress is the older API (stable since K8s 1.19) but limited — one spec for all use cases. Gateway API (GA in K8s 1.31) is more expressive with separate Gateway, HTTPRoute, GRPCRoute objects and supports role-based ownership (infra team owns Gateway, app team owns Routes).