Architecture

I write a lot about Kubernetes. I run it daily. I genuinely like it. So it might surprise you that I spend a fair amount of time talking people out of it. Here is the reality I keep walking into. A small team, a single product, a roadmap full of real features to build, and someone has decided the first milestone is a Kubernetes cluster. Three nodes minimum, etcd, a CNI, an ingress controller, cert-manager, a monitoring stack. Weeks of work before a single customer sees anything. Everyone nods along, because this is just how serious infrastructure looks in 2026. ...

My homelab started with one ArgoCD Application. Then a handful. The day I caught myself running kubectl apply -f for the fifteenth time to register yet another Application, I knew I’d built the exact thing I was trying to avoid: manual steps I had to remember, in an order I had to remember, with no record of what should exist. The App-of-Apps pattern fixes that with one idea. You create a single root Application by hand, and it creates everything else. After a cluster wipe I can rebuild the whole thing with one kubectl apply. That property is the entire reason I run it, and it’s the same reason I self-host in the first place: I want the repository to be the truth, not my memory. ...

What happens when your Kubernetes cluster can’t reach the internet? I don’t mean a slow connection. I mean no connection at all. Ships at sea. Remote mining sites. Factory floors with air-gapped networks. Military deployments. For a lot of people that sounds exotic, like a problem someone else has. I treat it as a baseline design requirement, and I’ll explain why it makes my homelab better even though I almost never actually pull the cable. ...

Everyone repeats the line that Kubernetes is self-healing. Pods die, they come back. Nodes drop, workloads reschedule. The system reconciles itself toward the state you declared, and most days you never have to think about it. Then one day the thing doing the healing is the thing that broke. The API server is down. etcd won’t respond. The scheduler is wedged. Now what? This is the question I actually care about, because “self-healing” is only useful if I understand its edges. I want to know what degrades gracefully and what takes the whole cluster with it. So I’ve run my clusters through a lot of failures on purpose: planned, unplanned, and a few “hold my beer” experiments on hardware I didn’t mind losing. Here is what actually happens when each piece breaks, and why most of it matters less than people fear. ...

When something goes wrong in Kubernetes, the trail usually leads back to etcd. API server timing out? Check etcd. Pods stuck in pending? Might be etcd. Cluster feels sluggish? Probably etcd. For a long time I treated etcd the way most operators do: as a black box that hums along next to the control plane. “The database.” You back it up and otherwise leave it alone. But black boxes feel like splinters to me, and the first time an etcd cluster fell over at 2am I realised I had no idea what I was actually looking at. So I learned. And it turns out the whole thing is built on a handful of ideas that, once they click, make most etcd problems diagnosable instead of terrifying. ...