Monitoring

You have Grafana. You have Prometheus metrics. You have logs in Loki and traces in Tempo. The data is all there. You also have 47 dashboards that nobody opens. I have done this to myself more than once. Something breaks at 2 AM, I bolt together a dashboard to see what’s going on, and then it just sits there forever. Multiply that by a year of incidents and a few “let me just add a panel for that” moments, and you end up with a Grafana that’s mostly archaeology. Nobody remembers what half the panels mean. The honest move is to delete most of them, but first it helps to understand what makes the survivors worth keeping. ...

The first time someone asked me “was this slower last month than it is now?”, I had no answer. My Prometheus only remembered two weeks. The data I needed had already aged out of local disk and been deleted. That gap is the whole reason this post exists. Prometheus is the default for Kubernetes metrics, and for good reason. It works beautifully right up until you need long-term storage, or a view across multiple clusters, or genuine high availability. Then you meet the wall. ...

The whole pitch of GitOps is that Git is the source of truth. That promise holds right up until someone runs kubectl edit on a deployment at 2am to stop an incident, a mutating webhook quietly rewrites a resource, or a half-finished sync leaves your cluster somewhere between what Git wanted and what it got. Now Git says one thing and the cluster does another, and nobody told you. That gap is configuration drift, and it is the part of GitOps people forget to defend. The good news: ArgoCD already watches for it. The catch is that the defaults don’t do what you probably assume, and a few of them will bite you. This post walks from the simplest possible drift check up to the setup I actually run, one layer at a time. Stop wherever you have enough. ...