As a kid I had a word for the things that fascinated me: unbreakable. Indestructible was never quite right, because indestructible means something that never breaks. Unbreakable is the better word. It means something that keeps working even after it breaks.
I remember exactly when the fascination started. A photo of an A-10 Thunderbolt II that came back from a mission with half a wing gone, the tail in tatters, and the fuselage full of holes. That thing still brought its pilot home.
That’s not luck. That’s design.
Things that refuse to die
You know the Camel Trophy stories? Land Rover Defenders chased through Indonesian jungle. Chassis bent, parts snapped off, everything caked in mud. And they still arrived. Not because nothing went wrong on the way, because things went wrong the whole time. They arrived because they were built to break and keep driving anyway.
Now compare that to a modern car. One sensor dies and you’re parked on the hard shoulder waiting for a tow truck.
Somewhere along the way we started treating “robust” and “complex” as the same thing. More systems, more redundancy, more failovers. But complexity is a source of failure all by itself. Every extra component is one more thing that can break, and one more thing you have to understand when it does.
Why I do this for work
I build platforms for a living. Kubernetes clusters, GitOps pipelines, that kind of thing. And I catch myself asking the same question over and over: “what happens when this falls over?”
That’s a different question from “how do we stop this from falling over?”. Prevention matters, but it’s a separate problem. I want to know what happens when the thing goes down anyway, because it will. Servers crash. Networks do weird things. Code has bugs. I’m not being pessimistic here, that’s just the job.
Most of the systems I run into are built for the happy path. Everything works, requests come in, responses go out, the logs are green. Lovely. Then something goes sideways, which it always does, and it turns out nobody ever decided what should happen at that moment. There’s no plan, just a stack trace and a Slack channel filling up.
Kubernetes and the illusion of self-healing
Kubernetes gets sold as “self-healing”. Pod crashed? No problem, we’ll start a fresh one. Node fell off the cluster? The workloads get rescheduled somewhere else.
And yes, all of that is true. Up to a point.
The catch is that a Kubernetes cluster is itself a system that can fail. What happens when etcd corrupts? When the control plane goes down? When a network partition splits your nodes into two halves that both think they’re in charge?
Self-healing inside a cluster is great. But the cluster itself is still a single point of failure. That’s exactly what I’ve been experimenting with lately: clusters that are unbreakable together, where the whole thing keeps serving even when an entire cluster drops out.
That’s a story for another post.
The question I ask myself
Every time I build something, I try to ask: is this unbreakable?
Not perfect. Not indestructible. Can this thing keep doing its job while half of it is on fire?
Most of the time the honest answer is no. So the next question becomes: what would have to change to make the answer yes?
Sometimes the fix is simple. Sometimes the answer is “this whole design has to be different”. And sometimes the answer is “that’s more effort than this particular problem is worth”, which is fine too, as long as you made that call on purpose instead of finding out at 3 AM.
Not everything has to be unbreakable. The point is that you think about what happens when it breaks, before it breaks, while you still have the luxury of changing your mind.
That’s also why I’m a fan of chaos engineering: breaking things deliberately to find out what happens. I would much rather discover that in a controlled experiment on a Tuesday afternoon than at 3 AM with a phone buzzing on the nightstand.