For a couple of years my homelab cluster ran without much drama. Nodes came and went, workloads shifted around, and the worst I ever had to deal with was the occasional pod stuck in CrashLoopBackOff. The kind of stable where you stop thinking about what happens if it all goes away. Then one evening I ran a terraform destroy against the wrong workspace. By the time I noticed, the control plane was gone. Not degraded. Gone. ...
“Should I use ArgoCD or Flux?” I get this question over coffee, in Slack, in PR comments, probably once a week. People want me to declare a winner so they can stop thinking about it and go build something. Here is my bias, up front, so you can weigh everything that follows: I run ArgoCD on my own clusters. I have a few years of it under my belt and I like it. That preference is going to leak through no matter how careful I am, so I am telling you now instead of pretending I am neutral. ...
Here is the thing I never want to do again: ask a cluster what it is running and not trust the answer. For years that was normal. Someone ran kubectl apply, maybe from a laptop, maybe from a CI job nobody could find anymore, and the live state slowly drifted away from anything written down. When I switched to GitOps, that whole category of uncertainty disappeared. I push to Git, and the cluster converges to match. If I want to know what is deployed, I read a file. ...
What happens when your Kubernetes cluster can’t reach the internet? I don’t mean a slow connection. I mean no connection at all. Ships at sea. Remote mining sites. Factory floors with air-gapped networks. Military deployments. For a lot of people that sounds exotic, like a problem someone else has. I treat it as a baseline design requirement, and I’ll explain why it makes my homelab better even though I almost never actually pull the cable. ...
Everyone repeats the line that Kubernetes is self-healing. Pods die, they come back. Nodes drop, workloads reschedule. The system reconciles itself toward the state you declared, and most days you never have to think about it. Then one day the thing doing the healing is the thing that broke. The API server is down. etcd won’t respond. The scheduler is wedged. Now what? This is the question I actually care about, because “self-healing” is only useful if I understand its edges. I want to know what degrades gracefully and what takes the whole cluster with it. So I’ve run my clusters through a lot of failures on purpose: planned, unplanned, and a few “hold my beer” experiments on hardware I didn’t mind losing. Here is what actually happens when each piece breaks, and why most of it matters less than people fear. ...