Advanced

In my previous post on Prometheus and Thanos, I set up the sidecar architecture. Thanos Sidecar runs next to Prometheus, uploads TSDB blocks to object storage, and exposes data to the Querier over gRPC. For clusters sitting in the same datacenter with a fat, stable link to your central infrastructure, it’s lovely. Everything pulls. Everything talks to everything. Life is good. Then I started putting Prometheus on clusters at the edge, and life got less good. ...

Here’s how an ISO 27001 audit usually goes. Weeks before the auditor shows up, someone starts collecting screenshots. Configuration panels, firewall rules, a dashboard showing patches applied. Then come the Word documents describing what the systems are supposed to do. Then the change tickets, dug out of a ticketing system, each one referencing a vague “server maintenance” that nobody can fully reconstruct six months later. Everyone treats this as the cost of doing business. I did too, for years. ...

I’ve written about Talos Linux as the immutable Kubernetes OS, and I’ve compared Arch vs NixOS for workstations. One question keeps landing in my inbox after both: what about NixOS for the Kubernetes nodes themselves? It’s a fair question, because on paper these two look like siblings. NixOS and Talos are both declarative. Both can be immutable. Both put your configuration under version control. So why pick one over the other to run a cluster? ...

Here is how most of us run Kubernetes nodes. You install a general-purpose Linux distro, harden it with a CIS benchmark script, add an SSH key, set up a config management tool to keep drift in check, and then spend the next two years SSH-ing in to fix the things the config management tool didn’t catch. Every node is a little snowflake with its own history. We’ve accepted this as normal. ...

“We want to migrate to Kubernetes by November.” It was September. The client was an e-commerce company, and their biggest sales event of the year was Black Friday, in late November. I said no. They asked if I knew someone who might take it on anyway. I did. A fellow platform engineer, someone I respect and rate highly. I made the introduction but warned him about the timeline. He took the engagement, documented the same concerns I had, got them signed off. The client proceeded anyway. ...