Chaos Mesh: Zero to Hero

Chaos engineering for Kubernetes — deliberately inject failures (kill pods, add network latency, stress CPU, corrupt I/O) to prove your system survives them before a real incident does. You'll install Chaos Mesh, run pod/network/stress experiments, target precisely with selectors, chain them into workflows, and practise it safely. Hands-on throughout.

The journey:

1. What chaos engineering is (and isn't)
2. Install Chaos Mesh
3. Your first experiment — PodChaos
4. Selectors — targeting precisely
5. NetworkChaos — latency, loss, partition
6. StressChaos, IOChaos, and more
7. Scheduling and workflows
8. Running it safely
9. Production practices

---

Prerequisites

Start in a non-production cluster. Confirm access: kubectl get nodes.

---

Part 1 — What chaos engineering is (and isn't)

Chaos engineering is the practice of injecting controlled failures to build confidence that a system tolerates them. It's not random breakage — it's a scientific method:

1. Define steady state (the metric that says "healthy" — e.g. p99 latency, success rate).
2. Hypothesize it holds under a specific failure.
3. Inject that failure on a small blast radius.
4. Observe: did steady state hold? If not, you found a weakness on your terms.

Chaos Mesh makes step 3 easy on Kubernetes: it defines faults as CRDs (PodChaos, NetworkChaos, StressChaos…), a controller injects them, and a dashboard drives it. The goal is learning, not destruction — always with a hypothesis and a way to stop.

---

Part 2 — Install Chaos Mesh

helm repo add chaos-mesh https://charts.chaos-mesh.org
helm repo update
helm install chaos-mesh chaos-mesh/chaos-mesh \
  --namespace chaos-mesh --create-namespace --version 2.6.3

kubectl get pods -n chaos-mesh          # controller-manager, daemon, dashboard

The chaos-daemon runs on each node to inject faults into pod namespaces; the controller-manager reconciles experiment CRDs. Access the dashboard with a port-forward to visualize and launch experiments.

---

Part 3 — Your first experiment — PodChaos

PodChaos kills or disrupts pods. The classic experiment: does your app survive a pod dying?

apiVersion: chaos-mesh.org/v1alpha1
kind: PodChaos
metadata:
  name: kill-one-api-pod
  namespace: app
spec:
  action: pod-kill            # also: pod-failure, container-kill
  mode: one                   # kill exactly one matching pod
  selector:
    namespaces: [app]
    labelSelectors:
      app: api
  duration: "30s"

kubectl apply -f podchaos.yaml
kubectl get podchaos -n app          # experiment status
kubectl describe podchaos kill-one-api-pod -n app

If your Deployment has ≥2 replicas and a PodDisruptionBudget, killing one pod should be invisible to users — the Service routes around it and the ReplicaSet replaces it. If it isn't invisible, you just found a gap. pod-failure makes a pod unavailable without deleting it; container-kill kills a container inside a pod.

---

Part 4 — Selectors — targeting precisely

Every experiment's blast radius is set by its selector and mode:

• selector — namespaces, labelSelectors, annotationSelectors, pods (explicit), nodes. Narrow it tightly.
• mode — one, all, fixed: N, fixed-percent: 20, random-max-percent.

spec:
  mode: fixed-percent
  value: "20"                 # affect 20% of matching pods
  selector:
    namespaces: [app]
    labelSelectors: { tier: backend }

Getting the selector right is the safety mechanism — mode: all across a namespace is a self-inflicted outage. Start with mode: one, a labelled subset, and a short duration.

---

Part 5 — NetworkChaos — latency, loss, partition

Network failures are the most instructive — most outages are partial, not total. NetworkChaos injects delay, loss, duplication, corruption, and partitions:

apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata: { name: db-latency, namespace: app }
spec:
  action: delay
  mode: all
  selector:
    namespaces: [app]
    labelSelectors: { app: api }
  delay:
    latency: "200ms"
    jitter: "50ms"
  direction: to
  target:                     # only traffic to the database
    selector:
      namespaces: [app]
      labelSelectors: { app: postgres }
    mode: all
  duration: "2m"

This adds 200ms±50ms to API→DB traffic — revealing whether timeouts, retries, and circuit breakers behave. action: partition splits two groups entirely (a split-brain test); loss drops a percentage of packets. This is how you validate resilience assumptions that only show up under degraded networks.

---

Part 6 — StressChaos, IOChaos, and more

Chaos Mesh covers the failure spectrum:

apiVersion: chaos-mesh.org/v1alpha1
kind: StressChaos
metadata: { name: cpu-hog, namespace: app }
spec:
  mode: one
  selector: { namespaces: [app], labelSelectors: { app: api } }
  stressors:
    cpu: { workers: 2, load: 90 }
  duration: "1m"

Each maps to a real production failure mode — noisy neighbours (StressChaos), a slow disk (IOChaos), clock drift breaking TLS/tokens (TimeChaos), DNS outages (DNSChaos).

---

Part 7 — Scheduling and workflows

Run experiments on a schedule, or chain them into a scenario:

# Schedule: kill a pod every 10 minutes (game-day cadence)
apiVersion: chaos-mesh.org/v1alpha1
kind: Schedule
metadata: { name: periodic-kill, namespace: app }
spec:
  schedule: "*/10 * * * *"
  type: PodChaos
  podChaos:
    action: pod-kill
    mode: one
    selector: { namespaces: [app], labelSelectors: { app: api } }

A Workflow sequences and parallelizes experiments (add latency, then kill a pod, while stressing CPU) to reproduce compound failures. Combined with a load generator and your dashboards, this is a repeatable game day you can run in CI or on a cadence.

---

Part 8 — Running it safely

Chaos is only useful if it's controlled:

• Non-prod first, then production only with buy-in, monitoring, and an abort plan.
• Small blast radius — mode: one, labelled subsets, short duration; widen gradually.
• Define steady state and watch it — run experiments with your dashboards/alerts open; the experiment is the hypothesis test.
• Have an off switch — kubectl delete <chaos-kind> <name> stops it immediately; experiments auto-recover after duration.
• Announce game days — people shouldn't mistake a planned experiment for a real incident.

---

Part 9 — Production practices

• Every experiment is a hypothesis about a steady-state metric — measure it, don't just break things.
• Blast radius small, then widen — start one/short, escalate deliberately.
• HA is a prerequisite — chaos on single-replica, PDB-less workloads just causes outages, teaching nothing.
• Pair with observability — the Prometheus/Grafana and OpenTelemetry stack is how you see the effect.
• Automate game days with Schedules/Workflows; treat findings as reliability backlog.
• Lock down access (RBAC) — fault injection is powerful; not everyone should wield it.
• Always have a rollback/abort and communicate before running in prod.

---

Common mistakes to avoid

• Random breakage with no hypothesis. That's an outage, not an experiment — define steady state first.
• mode: all / no duration. A cluster-wide, open-ended fault is a self-inflicted incident.
• Chaos on non-HA workloads. Killing the only replica proves nothing except that one replica isn't HA.
• No monitoring during the run. If you can't measure steady state, you learn nothing.
• Starting in production. Build confidence in staging first.
• No abort plan. Always know how to stop it (delete the CR) and that it auto-recovers.
• Unrestricted access. Fault injection needs RBAC guardrails.

---

Practice: try it yourself

1. Survive a kill. Deploy an HA app (≥2 replicas + PDB), run a pod-kill PodChaos mode: one, and confirm users see nothing while a pod is replaced.
2. Add latency. Inject 200ms NetworkChaos on app→DB traffic and watch whether your timeouts/retries cope.
3. Stress it. Run StressChaos CPU on one pod and observe whether the HPA/KEDA reacts and neighbours suffer.
4. Break a non-HA app on purpose. See the outage, then add a second replica + PDB and re-run to prove the fix.
5. Schedule a game day. Add a Schedule that kills a pod every 10 minutes and watch your dashboards.

Self-check:

• What are the four steps of a chaos experiment?
• How do selector and mode control blast radius?
• Why is HA a prerequisite for meaningful chaos?
• Name three failure modes beyond pod-kill and the Chaos Mesh kind for each.

---

Where to go next

• See the effect — the Prometheus and OpenTelemetry tutorials give you the steady-state signals chaos tests.
• Autoscale under stress — the KEDA tutorial to react to the load chaos creates.
• Practice with SRE/reliability interview questions in the question bank.

You now have the loop: hypothesis → inject (pod/network/stress) → narrow blast radius → observe steady state → schedule game days → run safely. That's chaos engineering that builds real confidence.