Chaos Engineering: Breaking Your Cluster to Make It Stronger

TL;DR Chaos engineering is breaking your own cluster on purpose, with the hypothesis written down first and a rollback ready. I run it on the homelab with Litmus: a weekly pod-delete, a monthly node drain, a game day every quarter. So far it has found four things the dashboard would never have shown me.

My dashboard is a wall of green. Pods running, replicas matched, CPU comfortable, no alerts firing. I look at it and get that small hit of “everything’s fine.” And mostly it is. The cluster’s been up for weeks and nothing has fallen over.

That wall is also the most dangerous thing in my homelab. It tells me nothing about what happens when something goes wrong. It only tells me that, right now, nothing has.

So picture the other version of that morning. A node dies at the one moment of the day when load matters. A DNS lookup starts timing out and takes every readiness probe down with it. A service I deployed eight months ago and forgot about eats memory until the kernel’s OOM killer steps in. None of that shows up on a healthy dashboard. It shows up while I’m asleep, and the person debugging it is me, badly rested and annoyed.

So how do I find out before production does? I break things myself. On a weekday afternoon, with a coffee and a rollback ready. That’s the whole pitch for chaos engineering, and I’d rather find my weak spots on my own schedule than let production pick the moment.

What breaking things on purpose means

A distributed system doesn’t fail in one place. You can read every line of code, draw the prettiest architecture diagram in the world, and still have no idea how the thing behaves when a node drops mid-request. That behaviour lives in the gaps between components. The only way to see it is to cause the failure and watch.

There’s a reason I keep coming back to this on the blog. I want to understand what I run. A green dashboard asks me to trust it, and I don’t fully trust anything I haven’t seen fail. After a chaos experiment I’ve watched it recover, instead of assuming it would.

A clarification, because the name scares people off. Chaos engineering doesn’t mean randomly destroying things to see what happens, and it doesn’t mean yeeting fault injection into prod without a plan. Every experiment has three things attached to it:

A hypothesis. “If one node fails, pods reschedule within five minutes.” You’re testing a belief you already hold.
A blast radius you control. A known target, a set duration, a rollback you can hit straight away.
Something you learn. Either the belief holds or you’ve got a bug you didn’t know about. I’ll take either.

Netflix made this famous with Chaos Monkey, which killed random instances so engineers had no choice but to build services that survived it. The tooling has grown up a lot since then. The idea hasn’t changed. You cause the failure at a time you picked, instead of waiting for it.

The chaos engineering cycle

1. Write down the hypothesis

Write down what you believe is true before you touch anything. You’re committing to a prediction, so the result can surprise you:

“If one node fails, pods will reschedule within 5 minutes”
“If the database becomes slow, the API will degrade gracefully, not crash”
“If DNS fails, cached responses will serve requests for 30 seconds”

2. Design the experiment

Now scope it. Four questions:

Blast radius. What can be affected?
Duration. How long does the fault persist?
Rollback. How do you stop it right now?
Metrics. What tells you it passed or failed?

People skip the rollback question. It’s the one that keeps an experiment from turning into an incident.

3. Run it somewhere it can’t hurt

Start small. Dev first, then staging, then production, and production only with safeguards. I skipped that order exactly once. I haven’t since.

4. Observe and measure

This is where your observability stack pays for itself. If you can’t see what happened, all you did was cause damage. Watch the obvious things. Did the right alerts fire? Did latency move? Did the error rate spike? How long did recovery take, next to what you guessed?

5. Learn and fix

Write down what you found, fix the weak spot, and update the runbook so the next person doesn’t start from scratch. The next person is usually me, six months on. Then design the next experiment.

Litmus, chaos the Kubernetes way

Litmus is a CNCF project that does chaos engineering the way Kubernetes does everything else. You declare an experiment as YAML and apply it. That’s why I picked it for the homelab. The experiment lives in Git next to the workload it attacks, so it’s reviewable and reproducible, and it isn’t a clever command someone ran once and never wrote down.

Installing Litmus

# Add Litmus helm repo
helm repo add litmuschaos https://litmuschaos.github.io/litmus-helm/
helm repo update

# Install Litmus
helm install litmus litmuschaos/litmus \
  --namespace litmus \
  --create-namespace \
  --set portal.frontend.service.type=ClusterIP

Litmus architecture

Declaring an experiment with ChaosEngine

apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: nginx-chaos
  namespace: default
spec:
  appinfo:
    appns: 'default'
    applabel: 'app=nginx'
    appkind: 'deployment'
  engineState: 'active'
  chaosServiceAccount: litmus-admin
  experiments:
    - name: pod-delete
      spec:
        components:
          env:
            - name: TOTAL_CHAOS_DURATION
              value: '30'
            - name: CHAOS_INTERVAL
              value: '10'
            - name: FORCE
              value: 'false'

This kills nginx pods every 10 seconds for 30 seconds in total. Small, contained, easy to watch. Start here.

The experiments worth knowing

You don’t need all of these on day one. It helps to know the menu though, because each one probes a different assumption you’re quietly making about your cluster.

Pod-level chaos

Pod delete. Kills pods to test restart behaviour.

experiments:
  - name: pod-delete
    spec:
      components:
        env:
          - name: TOTAL_CHAOS_DURATION
            value: '60'
          - name: CHAOS_INTERVAL
            value: '10'
          - name: PODS_AFFECTED_PERC
            value: '50'  # Kill 50% of pods

Container kill. Kills one container inside a pod.

experiments:
  - name: container-kill
    spec:
      components:
        env:
          - name: TARGET_CONTAINER
            value: 'sidecar'
          - name: CHAOS_INTERVAL
            value: '10'

Node-level chaos

Node drain. Simulates node maintenance.

apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: node-drain-chaos
spec:
  engineState: 'active'
  auxiliaryAppInfo: ''
  chaosServiceAccount: litmus-admin
  experiments:
    - name: node-drain
      spec:
        components:
          env:
            - name: TOTAL_CHAOS_DURATION
              value: '60'
            - name: TARGET_NODE
              value: 'worker-1'

Node CPU hog. Simulates CPU pressure.

experiments:
  - name: node-cpu-hog
    spec:
      components:
        env:
          - name: TOTAL_CHAOS_DURATION
            value: '60'
          - name: NODE_CPU_CORE
            value: '2'  # Consume 2 cores

Network chaos

Pod network loss. Simulates a network partition.

experiments:
  - name: pod-network-loss
    spec:
      components:
        env:
          - name: NETWORK_INTERFACE
            value: 'eth0'
          - name: NETWORK_PACKET_LOSS_PERCENTAGE
            value: '100'  # Total loss
          - name: TOTAL_CHAOS_DURATION
            value: '30'

Pod network latency. Injects latency.

experiments:
  - name: pod-network-latency
    spec:
      components:
        env:
          - name: NETWORK_LATENCY
            value: '300'  # 300ms latency
          - name: JITTER
            value: '100'  # 100ms jitter

Storage chaos

Disk fill. Tests how the workload handles disk pressure.

experiments:
  - name: disk-fill
    spec:
      components:
        env:
          - name: FILL_PERCENTAGE
            value: '90'
          - name: TOTAL_CHAOS_DURATION
            value: '60'

Running your first experiment

Enough menu-reading. Here’s a full experiment against a throwaway app, start to finish, so you can copy it and watch something break.

1. Deploy a test application

apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo-app
  labels:
    app: demo
spec:
  replicas: 3
  selector:
    matchLabels:
      app: demo
  template:
    metadata:
      labels:
        app: demo
    spec:
      containers:
        - name: nginx
          image: nginx:alpine
          ports:
            - containerPort: 80
          readinessProbe:
            httpGet:
              path: /
              port: 80
            initialDelaySeconds: 5
            periodSeconds: 5

2. Create a ServiceAccount for Litmus

apiVersion: v1
kind: ServiceAccount
metadata:
  name: litmus-admin
  namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: litmus-admin
rules:
  - apiGroups: [""]
    resources: ["pods", "pods/exec", "pods/log", "events"]
    verbs: ["create", "delete", "get", "list", "patch", "update", "watch"]
  - apiGroups: ["apps"]
    resources: ["deployments", "replicasets"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: litmus-admin
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: litmus-admin
subjects:
  - kind: ServiceAccount
    name: litmus-admin
    namespace: default

3. Run the pod-delete experiment

apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: demo-chaos
  namespace: default
spec:
  appinfo:
    appns: 'default'
    applabel: 'app=demo'
    appkind: 'deployment'
  engineState: 'active'
  chaosServiceAccount: litmus-admin
  experiments:
    - name: pod-delete
      spec:
        components:
          env:
            - name: TOTAL_CHAOS_DURATION
              value: '30'
            - name: CHAOS_INTERVAL
              value: '10'
            - name: FORCE
              value: 'false'
        probe:
          - name: "check-endpoint"
            type: "httpProbe"
            httpProbe/inputs:
              url: "http://demo-app.default.svc:80"
              insecureSkipVerify: false
              method:
                get:
                  criteria: "=="
                  responseCode: "200"
            mode: "Continuous"
            runProperties:
              probeTimeout: 5
              interval: 2
              retry: 2

4. Observe the results

# Watch chaos engine status
kubectl get chaosengine demo-chaos -w

# Check chaos result
kubectl get chaosresult demo-chaos-pod-delete -o yaml

# Watch pod behavior
kubectl get pods -l app=demo -w

Game days, the same idea aimed at the team

Here’s the part the YAML can’t fix. A chaos experiment tells you whether the system survives. It says nothing about whether the humans behind it know what to do when the pager goes off. The first time anyone opens a runbook shouldn’t be during a real outage. A game day is how you avoid that.

You take the same controlled failure and point it at the team instead of the cluster.

Planning a game day

Scope. Which systems, which failure scenarios.
Team. The engineers who’d respond to the real thing.
Runbooks. Written before the day, so there’s something to test.
Success criteria. What does “handled well” look like?
Time. A block in the calendar, not a gap between meetings.

Game day structure

09:00 - Briefing: Explain scope, rules, how to abort
09:15 - Inject failure (team doesn't know which one)
09:15 - Team responds as if real incident
10:00 - Failure revealed, additional scenarios introduced
11:00 - Debrief: What worked, what didn't, what to improve

Example scenarios

Scenario	What You Learn
Kill primary database	Failover speed, data integrity
DNS outage	Caching behavior, timeout handling
Certificate expires	Monitoring gaps, renewal process
Cloud region unavailable	Multi-region readiness
Secrets manager down	Application behavior without secrets

Keeping it in Git

The experiments are YAML, so they belong in version control with everything else. I keep mine managed with ArgoCD, defined but dormant, ready for when I want to run one:

# chaos-experiments/pod-delete.yaml (in Git)
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: scheduled-pod-delete
  namespace: chaos-testing
spec:
  engineState: 'stop'  # Manually activated
  # ... experiment config

The flow is the one I use for everything else. Define the experiment in Git, let ArgoCD sync it, flip engineState to active when I want it to run, read the result, commit whatever fix or finding came out of it. The experiment and its history live next to the workload it tests.

Staying safe while doing this

Done carelessly, chaos engineering is an outage you caused yourself. The difference is discipline, so here’s mine.

Start in non-production and stay there until you trust the experiment. Have the rollback ready before you start, not after things go sideways. Watch the experiment live instead of kicking it off and walking away. Tell anyone who might get paged that you’re about to do this. And write down what you found, because next month nobody will remember what it found.

The mistakes are just as predictable. Running during your real peak window when you weren’t trying to test peak behaviour. Injecting a failure without warning the on-call. Skipping the hypothesis, so you can’t tell signal from noise. Ignoring the finding because the cluster recovered “well enough.”

Most important of all, decide your abort conditions up front. Error rate past a threshold, latency past a ceiling, real users complaining, or any behaviour you didn’t predict. Litmus can enforce part of this with a probe that kills the experiment for you:

# Abort on high error rate
probe:
  - name: "error-rate-check"
    type: "promProbe"
    promProbe/inputs:
      endpoint: "http://prometheus:9090"
      query: "rate(http_requests_total{status=~'5..'}[1m])"
      comparator:
        type: "float"
        criteria: "<"
        value: "0.1"  # Abort if > 10% errors

Why not everyone does this already

If chaos engineering is this useful, why isn’t it everywhere? The obstacles are real, and if I don’t name them this reads like an easy sell. It isn’t one.

The big one is fear, and it’s reasonable fear. Deliberately breaking a system you’re responsible for feels insane the first time, especially if you’ve never seen it fail safely. “It works” is a strong argument against “let’s go break it.” Then there’s the time. Designing a real experiment, watching it, documenting it, fixing what it found, none of that is free, and it competes with shipping features. And there’s culture. On a team that assigns blame, causing a failure on purpose is a career risk, even when it’s the responsible thing to do.

None of those go away on their own. What worked for me was starting so small that there was nothing to be afraid of, a pod-delete against a throwaway nginx in a dev namespace.

What I run in the homelab

So here’s the version I run on my own cluster. Nothing heroic. A weekly pod-delete against random pods in non-critical namespaces, a monthly node drain to rehearse losing a machine, and roughly once a quarter a game day with a couple of stacked failures.

That small cadence has already paid for itself. What it taught me:

Longhorn handles a node disappearing far more calmly than I expected. I used to hope that. Now I’ve seen it.
Several of my workloads had a useless terminationGracePeriod and were getting hard-killed mid-request, and I’d never noticed in normal operation.
My alerting fires later than it should, consistently. I’d rather learn that on a weekday afternoon than during an incident.
Recovery takes longer than I guess. Always.

Each of those four was a page I’d have got in the middle of the night, traded for an afternoon of mild discomfort. I’d take that trade again.

The dashboard is green again this morning. I trust it a bit more than I did, because I’ve watched it go red on my own schedule and come back.