Reliability

As a kid I had a word for the things that fascinated me: unbreakable. Indestructible was never quite right, because indestructible means something that never breaks. Unbreakable is the better word. It means something that keeps working even after it breaks. I remember exactly when the fascination started. A photo of an A-10 Thunderbolt II that came back from a mission with half a wing gone, the tail in tatters, and the fuselage full of holes. That thing still brought its pilot home. ...

Everyone repeats the line that Kubernetes is self-healing. Pods die, they come back. Nodes drop, workloads reschedule. The system reconciles itself toward the state you declared, and most days you never have to think about it. Then one day the thing doing the healing is the thing that broke. The API server is down. etcd won’t respond. The scheduler is wedged. Now what? This is the question I actually care about, because “self-healing” is only useful if I understand its edges. I want to know what degrades gracefully and what takes the whole cluster with it. So I’ve run my clusters through a lot of failures on purpose: planned, unplanned, and a few “hold my beer” experiments on hardware I didn’t mind losing. Here is what actually happens when each piece breaks, and why most of it matters less than people fear. ...