As a kid I called the things that fascinated me unbreakable. Indestructible never fit, because indestructible means it never breaks. Unbreakable means it keeps working after it breaks.
I know exactly when it started. A photo of an A-10 Thunderbolt II back from a mission with half a wing gone, the tail in tatters and the fuselage full of holes. It still brought its pilot home.
Luck had nothing to do with it. Somebody sat at a drawing board years before that mission and decided the plane had to come home looking like that.
Things that refuse to die
You know the Camel Trophy stories? Land Rover Defenders chased through the Indonesian jungle, chassis bent, parts snapped off, the whole thing caked in mud. And they still arrived. Things went wrong the entire trip. They arrived because they’d been built to break and keep driving anyway.
Now take a modern car. One sensor dies and you’re on the hard shoulder waiting for a tow truck.
At some point “robust” and “complex” became the same word. More redundancy, more failover, systems on top of systems. But complexity fails all by itself. Every extra component is one more thing that can break, and one more thing you have to understand when it does.
Why I do this for work
I build platforms for a living. Kubernetes clusters, GitOps pipelines, that kind of thing. And every time I catch myself asking the same question. What happens when this falls over?
That’s a different question from how we stop it falling over. Prevention matters, but it’s a separate problem. I want to know what happens when the thing goes down anyway, because it will. Servers crash and code has bugs. Networks do things nobody planned for. Call that pessimism if you like. From where I sit it’s just the job.
Most of the systems I run into are built for the happy path. Requests come in, responses go out, the logs are green. Then something goes sideways, and it turns out nobody ever decided what should happen at that moment. There’s no plan. There’s a stack trace, and a Slack channel that’s starting to fill up.
Kubernetes and the illusion of self-healing
Kubernetes gets sold as self-healing. Pod crashed? A fresh one starts. Node dropped off the cluster? The workloads land somewhere else.
All true. Up to a point.
The catch is that the cluster is itself a system that can fail. etcd can corrupt. The control plane can go down. A network partition can split your nodes into two halves that both think they’re in charge, and none of the self-healing above helps you with any of that.
Healing inside a cluster is great, and the cluster itself is still a single point of failure. That’s what I’ve been experimenting with lately, clusters that are unbreakable together, where the whole keeps serving when an entire cluster drops out.
That one’s still an experiment, so it gets its own post once it holds up.
The test I run before I build anything
Before I build anything, I hold it against that childhood word. Perfect is the wrong bar, and so is indestructible. Would this thing keep doing its job while half of it is on fire?
Most of the time the honest answer is no. So the next question is what would have to change to make it yes.
Sometimes the fix is small. Sometimes the answer is “this whole design has to be different”. And sometimes it’s “more effort than this problem is worth”, which is fine too, as long as I made that call on purpose instead of finding out during an outage.
Not everything has to be unbreakable. What I want is that somebody thought about what happens when it breaks, before it breaks, while changing their mind was still cheap.
That’s why I like chaos engineering so much. Break it on purpose, on a day I picked, and watch what happens. I’d much rather learn that from an experiment I scheduled than from an outage I didn’t see coming. And that photo of the A-10 is still in my head every time I run one.