Resilience

Picture your homelab disk dying tonight. Not corrupting, not throwing SMART warnings for a week first, just gone. The GitLab that holds every repo you care about, the password vault, fifteen years of family photos, the home automation that runs your house. All of it on a drive that has decided it’s done. Can you answer what happens next without your stomach dropping? If not, you don’t have backups. You have hope, and hope is not a strategy you want to discover the limits of at 2am. ...

My dashboard is a wall of green. Pods running, replicas matched, CPU comfortable, no alerts firing. I look at it and feel that small dopamine hit of “everything is fine.” And for the most part, it is fine. The cluster has been up for weeks. Nothing has fallen over. That green wall is also the most dangerous thing in my homelab, because it tells me nothing about what happens when something goes wrong. It only tells me that, right now, nothing has. ...

In my previous post on Prometheus and Thanos, I set up the sidecar architecture. Thanos Sidecar runs next to Prometheus, uploads TSDB blocks to object storage, and exposes data to the Querier over gRPC. For clusters sitting in the same datacenter with a fat, stable link to your central infrastructure, it’s lovely. Everything pulls. Everything talks to everything. Life is good. Then I started putting Prometheus on clusters at the edge, and life got less good. ...

The first cluster I ever ran in anger had exactly one permission model: everything was cluster-admin. My CI pipeline, my monitoring stack, the little webhook receiver I threw together one afternoon. All of it could read every secret, delete every deployment, and touch every namespace. It worked great right up until I started thinking about what happens when one of those pods gets popped. Kubernetes RBAC (Role-Based Access Control) answers a single question: who can do what to which resources? The default answer on most clusters is “everyone can do everything,” and that answer quietly becomes your biggest liability. ...

A standard Kubernetes Deployment had served me well for a long time. Push a new image tag, watch the pods roll, done. It was simple, it was declarative, and most of the time nothing went wrong. The rolling update even gave me a warm feeling of safety: old pods only get torn down once new ones are ready. That feeling is a lie. A rolling update protects you from pods that fail to start. It does nothing to protect you from pods that start perfectly and then serve broken responses. The container is healthy, the readiness probe is green, and your new code is quietly returning 500s to every single user. Within seconds, 100% of your traffic is hitting code that nobody validated under real load. ...