ArgoCD

A standard Kubernetes Deployment had served me well for a long time. Push a new image tag, watch the pods roll, done. It was simple, it was declarative, and most of the time nothing went wrong. The rolling update even gave me a warm feeling of safety: old pods only get torn down once new ones are ready. That feeling is a lie. A rolling update protects you from pods that fail to start. It does nothing to protect you from pods that start perfectly and then serve broken responses. The container is healthy, the readiness probe is green, and your new code is quietly returning 500s to every single user. Within seconds, 100% of your traffic is hitting code that nobody validated under real load. ...

The whole pitch of GitOps is that Git is the source of truth. That promise holds right up until someone runs kubectl edit on a deployment at 2am to stop an incident, a mutating webhook quietly rewrites a resource, or a half-finished sync leaves your cluster somewhere between what Git wanted and what it got. Now Git says one thing and the cluster does another, and nobody told you. That gap is configuration drift, and it is the part of GitOps people forget to defend. The good news: ArgoCD already watches for it. The catch is that the defaults don’t do what you probably assume, and a few of them will bite you. This post walks from the simplest possible drift check up to the setup I actually run, one layer at a time. Stop wherever you have enough. ...

My homelab started with one ArgoCD Application. Then a handful. The day I caught myself running kubectl apply -f for the fifteenth time to register yet another Application, I knew I’d built the exact thing I was trying to avoid: manual steps I had to remember, in an order I had to remember, with no record of what should exist. The App-of-Apps pattern fixes that with one idea. You create a single root Application by hand, and it creates everything else. After a cluster wipe I can rebuild the whole thing with one kubectl apply. That property is the entire reason I run it, and it’s the same reason I self-host in the first place: I want the repository to be the truth, not my memory. ...

For a couple of years my homelab cluster ran without much drama. Nodes came and went, workloads shifted around, and the worst I ever had to deal with was the occasional pod stuck in CrashLoopBackOff. The kind of stable where you stop thinking about what happens if it all goes away. Then one evening I ran a terraform destroy against the wrong workspace. By the time I noticed, the control plane was gone. Not degraded. Gone. ...

“Should I use ArgoCD or Flux?” I get this question over coffee, in Slack, in PR comments, probably once a week. People want me to declare a winner so they can stop thinking about it and go build something. Here is my bias, up front, so you can weigh everything that follows: I run ArgoCD on my own clusters. I have a few years of it under my belt and I like it. That preference is going to leak through no matter how careful I am, so I am telling you now instead of pretending I am neutral. ...