You’ve got Prometheus for metrics, so you can already see what’s happening across your clusters. Metrics tell you a request latency spiked at 14:32. They don’t tell you the payment service threw a null pointer because someone shipped a config change with a typo. For that you need logs. The default answer for years was Elasticsearch. It’s powerful and flexible, and it indexes every single token in every log line. That full-text index is great until you look at the bill. You pay for it in CPU at ingest, in RAM to keep the index hot, and in storage that grows faster than your actual log volume. I ran an ELK stack in a previous job and spent more time tuning JVM heap sizes than reading logs. ...

In my previous post on Prometheus and Thanos, I set up the sidecar architecture. Thanos Sidecar runs next to Prometheus, uploads TSDB blocks to object storage, and exposes data to the Querier over gRPC. For clusters sitting in the same datacenter with a fat, stable link to your central infrastructure, it’s lovely. Everything pulls. Everything talks to everything. Life is good. Then I started putting Prometheus on clusters at the edge, and life got less good. ...

Here’s how an ISO 27001 audit usually goes. Weeks before the auditor shows up, someone starts collecting screenshots. Configuration panels, firewall rules, a dashboard showing patches applied. Then come the Word documents describing what the systems are supposed to do. Then the change tickets, dug out of a ticketing system, each one referencing a vague “server maintenance” that nobody can fully reconstruct six months later. Everyone treats this as the cost of doing business. I did too, for years. ...

Ask most people how to run a “real” hypervisor at home and you get the same shortlist: VMware, Hyper-V, or at minimum Proxmox. Something with a web UI, a clustering tab, a marketing page full of enterprise features. That mental model is so common that running virtual machines without one of those products feels like cutting corners. We’ve quietly accepted that serious virtualization comes with a vendor attached. Now flip it. The thing doing the actual work in all of those products is a Linux kernel module that has been production-grade for over a decade. KVM with libvirt gives you live migration, memory ballooning, CPU pinning, GPU passthrough, SR-IOV, nested virtualization. The features the glossy hypervisors advertise are kernel features. The web UI is a wrapper around them. ...

I’ve written about Talos Linux as the immutable Kubernetes OS, and I’ve compared Arch vs NixOS for workstations. One question keeps landing in my inbox after both: what about NixOS for the Kubernetes nodes themselves? It’s a fair question, because on paper these two look like siblings. NixOS and Talos are both declarative. Both can be immutable. Both put your configuration under version control. So why pick one over the other to run a cluster? ...