Graceful Degradation in Kubernetes: What Happens When Components Fail

Everyone repeats the line that Kubernetes is self-healing. Pods die, they come back. Nodes drop, workloads reschedule. The system reconciles itself toward the state you declared, and most days you never have to think about it.

Then one day the thing doing the healing is the thing that broke. The API server is down. etcd won’t respond. The scheduler is wedged. Now what?

This is the question I actually care about, because “self-healing” is only useful if I understand its edges. I want to know what degrades gracefully and what takes the whole cluster with it. So I’ve run my clusters through a lot of failures on purpose: planned, unplanned, and a few “hold my beer” experiments on hardware I didn’t mind losing. Here is what actually happens when each piece breaks, and why most of it matters less than people fear.

The components that can fail

Before the scenarios, here is the map. You can’t reason about blast radius if you don’t know which part owns which job.

Control Plane:

• kube-apiserver: The API that everything talks to
• etcd: The database storing all cluster state
• kube-scheduler: Decides where pods run
• kube-controller-manager: Runs controllers (ReplicaSet, Deployment, etc.)
• cloud-controller-manager: Cloud provider integrations (if applicable)

Node Components:

• kubelet: Manages pods on each node
• kube-proxy: Handles network rules for Services
• Container runtime: Actually runs containers

The thing worth internalising up front: most of these are control-plane brains, and your running workloads do not consult the brain to keep serving traffic. That single fact explains almost every scenario below.

Scenario 1: API server down

The kube-apiserver is the single front door for every Kubernetes API request. Kill it and you’d assume the cluster falls over. It doesn’t.

Immediate impact:

• kubectl commands fail
• No new deployments or updates possible
• No new pod scheduling
• Existing pods keep running

What keeps working:

• Running pods continue to run
• Containers stay alive
• Network connectivity between pods
• Services continue to route traffic

What breaks:

• No new pods can be created
• Failed pods won’t be replaced
• Horizontal Pod Autoscaler stops working
• No changes to any resources

flowchart TD
    subgraph api_down["API Server Down"]
        subgraph control["Control Plane"]
            etcd["etcd ✓"]
            sched["scheduler<br/>idle"]
            ctrl["controller-mgr<br/>idle"]
            apiX["✗ API Server"]
        end
        control -->|"no updates"| nodes
        subgraph nodes["Worker Nodes"]
            N1["Node 1<br/>Pods ✓"]
            N2["Node 2<br/>Pods ✓"]
            N3["Node 3<br/>Pods ✓"]
        end
    end
    note["Pods keep running - they don't need the API"]

This is the whole design philosophy in one picture. Kubernetes expects the API server to be unavailable sometimes, so it degrades gracefully: your workloads keep serving, you just lose the ability to change anything. The control plane is the management layer, not the runtime. That separation is what makes the system survivable.

For HA setups you want multiple API servers behind a load balancer. See Kubernetes High Availability: stacked vs external etcd for architecture options.

Scenario 2: etcd down

etcd is the brain of Kubernetes. Every bit of cluster state lives here. This is the failure that actually scares me, and the one I back up religiously.

Immediate impact:

• API server can’t read or write state
• Effectively same as API server down
• Existing pods keep running

What keeps working:

• Running pods continue to run
• Containers stay alive
• Network connectivity
• Services work

What breaks:

• Same as API server down, plus:
• Risk of state inconsistency on recovery
• Split-brain scenarios in partial failures

flowchart TD
    subgraph etcd_down["etcd Down"]
        subgraph control["Control Plane"]
            etcdX["etcd ✗"] --> apiX["API ✗"]
            apiX --> other["other components"]
        end
    end
    note["Without etcd, API server cannot function<br/>Nodes continue running - they cache their assignments"]

Notice the runtime still holds. Nodes cache their assignments, so the pods that were running stay running even though the brain is offline. The danger isn’t the outage, it’s recovery: bring etcd back inconsistent or from a stale snapshot and you can corrupt your view of the world. That’s why backups here aren’t optional. See etcd Deep Dive for how etcd works and how to back it up properly.

Scenario 3: scheduler down

The kube-scheduler decides where pods run. When it’s down:

Immediate impact:

• New pods stay in Pending state
• No scheduling decisions made

What keeps working:

• Existing pods keep running
• API server functions normally
• You can create resources (they just won’t be scheduled)

What breaks:

• New pods can’t be scheduled
• Rescheduling after node failure doesn’t happen
• HPA creates pods that stay Pending

flowchart TD
    subgraph sched_down["Scheduler Down"]
        apply["kubectl apply deployment"] --> api["API accepts"]
        api --> pending["Pod: Pending..."]
        subgraph queue["Pending Queue"]
            web["Pod: web"]
            apiPod["Pod: api"]
            job["Pod: job"]
            waiting["... waiting"]
        end
    end
    note["Nodes have capacity but nobody assigns pods to them"]

The tell here is a pile of Pending pods on a cluster with plenty of free capacity. The API accepts your manifests, etcd stores them, the nodes have room, but nothing places the pods on a node. In HA setups you run multiple schedulers with leader election: one active, the rest waiting to take over.

Scenario 4: controller manager down

The controller manager runs the controllers that earn Kubernetes the “self-healing” label.

Immediate impact:

• ReplicaSet controller stops
• Deployment controller stops
• Node controller stops
• All reconciliation loops stop

What keeps working:

• Existing pods keep running
• Scheduling still works
• API server still works

What breaks:

• Failed pods don’t get replaced
• Deployments don’t roll out
• Node failures aren’t handled
• Orphaned resources aren’t cleaned up

flowchart TD
    subgraph ctrl_down["Controller Manager Down"]
        rs["ReplicaSet: 3 desired, 2 running → no action taken"]
        deploy["Deployment: rollout in progress → stuck"]
        node["Node marked NotReady → pods not evicted"]
    end
    note["The 'self-healing' part of Kubernetes stops"]

This is the scenario where the word “magic” finally falls away. The self-healing is just software running reconciliation loops, comparing desired state to actual state and acting on the gap. Stop the loops and the gap stops closing. A ReplicaSet that wants three replicas will happily sit at two forever, because nobody is watching. I find this reassuring rather than alarming: it means I can read the source, predict the behaviour, and trust it for the right reasons instead of treating it as a black box.

Scenario 5: kubelet down on a node

The kubelet is the Kubernetes agent on each node. When it fails:

Immediate impact (on that node):

• Node marked as NotReady after timeout (default ~40 seconds)
• Pods on that node get evicted (after another timeout)
• No new pods scheduled to that node

What keeps working:

• Containers keep running (they don’t need kubelet)
• Network might still work (depends on CNI)
• Other nodes unaffected

What breaks:

• No pod lifecycle management on that node
• Health checks stop
• Resource updates stop
• Eventually pods are rescheduled elsewhere

flowchart LR
    subgraph kubelet_down["Kubelet Down on Node 2"]
        subgraph N1["Node 1 ✓ Ready"]
            P1["pod ✓"]
        end
        subgraph N2["Node 2 ✗ NotReady"]
            P2["pod ?<br/>orphaned"]
        end
        subgraph N3["Node 3 ✓ Ready"]
            P3["pod ✓<br/>rescheduled"]
        end
        P2 -.->|"rescheduled"| P3
    end
    note["After pod-eviction-timeout, pods get rescheduled"]

The part that surprises people: the containers keep running even with the kubelet dead. The container runtime is what keeps them alive, the kubelet just manages their lifecycle. Kill the manager and the workers keep working until something decides to evict and reschedule them somewhere with a healthy kubelet.

Scenario 6: container runtime down

If the container runtime (containerd, CRI-O) fails:

Immediate impact:

• Running containers might die
• New containers can’t start
• Health checks fail

What happens next:

• kubelet detects failures
• Pods marked as Failed
• Pods get rescheduled to other nodes

This is the one node-level failure that does take your workloads with it, because the runtime is the thing actually running them. It triggers the same eviction-and-reschedule dance as a dead kubelet, just from the other direction.

Scenario 7: network partition

Network partitions are the nastiest failures, because nothing has actually crashed. A node loses its path to the control plane while happily continuing to run containers.

What happens:

• Node marked NotReady (can’t reach API server)
• Pods eventually evicted
• But they might still be running on the partitioned node
• Potential for “split brain”: same pod running in two places

flowchart LR
    subgraph partition["Network Partition"]
        subgraph control["Control Plane"]
            cp["Node 2 is NotReady<br/>Evicting pods..."]
        end
        control x--x|"✗"| partitioned
        subgraph partitioned["Partitioned Node"]
            pn["I'm fine, running<br/>these pods..."]
        end
    end
    note["Pod 'web-abc123' now runs on Node 1 AND Node 2<br/>Both think they're the real one"]

The control plane gives up on the unreachable node and reschedules its pods elsewhere. The partitioned node never got the memo and keeps running the originals. Now the same pod exists twice, each convinced it’s the only one. For stateless web pods that’s a shrug. For a stateful database it’s a data-corruption incident waiting to happen, which is why stateful workloads need fencing and quorum, not just “reschedule and hope.”

Failure timeouts to know

These timeouts control how fast Kubernetes reacts to failures:

You can tune these down for faster failover, but there’s a real trade-off: tighten them too far and a two-second network blip evicts healthy pods. Fast detection and false positives are the two ends of the same dial.

The blast radius principle

Every failure has a blast radius. Knowing the radius tells you where to spend your redundancy budget.

Read that table bottom to top and your HA priorities write themselves. etcd and the API server have the widest radius, so they get the most redundancy. A single container failing is noise the system already handles.

What this means for you

1. Running workloads are resilient: existing pods survive most control plane failures
2. Management operations aren’t: you need control plane HA for continuous deployment
3. etcd is the critical path: protect it, back it up, monitor it
4. Failures cascade: API server down looks like everything is down, even when workloads are fine
5. Timeouts matter: know your failure detection times before you tune them

The reason I trust Kubernetes isn’t that it never fails. It’s that it fails in ways I can predict and reason about. The control plane and the runtime are separated on purpose, so losing the brain doesn’t kill the body. The real question for your cluster is never whether a component will fail. It’s whether you’ve designed for the radius when it does.

Testing failures

Don’t wait for production to teach you how your cluster behaves. Break it yourself, on hardware you don’t mind losing:

# Simulate API server failure (on a test cluster!)
kubectl exec -it -n kube-system kube-apiserver-xxx -- kill 1

# Simulate scheduler failure
kubectl scale deployment kube-scheduler -n kube-system --replicas=0

# Simulate kubelet failure on a node
ssh node-1 'sudo systemctl stop kubelet'

For controlled, repeatable experiments, reach for a chaos engineering tool like Litmus Chaos instead of kill 1 by hand.

Understanding failure modes isn’t pessimism, it’s how you earn the right to call a system resilient. Every system fails. The only choice you get is whether you designed for it or got surprised by it.