Operations

My dashboard is a wall of green. Pods running, replicas matched, CPU comfortable, no alerts firing. I look at it and feel that small dopamine hit of “everything is fine.” And for the most part, it is fine. The cluster has been up for weeks. Nothing has fallen over. That green wall is also the most dangerous thing in my homelab, because it tells me nothing about what happens when something goes wrong. It only tells me that, right now, nothing has. ...

For a while my alerting worked fine. A handful of rules, pages were rare, and when one came in it meant something. Then the cluster grew, I bolted on the Prometheus Operator defaults, and “fine” quietly turned into noise. The tipping point was a 3 AM page. My phone buzzed, I groggily checked it: “High CPU usage on node-worker-3.” I looked at the graph, saw it had been sitting at 75% for ten minutes, and went back to sleep. Next night, same alert. A week later I’d stopped checking at all. ...