Advanced

In my previous post on Prometheus and Thanos, I covered the sidecar architecture — Thanos Sidecar runs alongside Prometheus, uploads TSDB blocks to object storage, and exposes data to the Querier. It works beautifully for clusters with stable connectivity to your central infrastructure. But what happens when your clusters are at the edge? When they might lose connectivity for hours or days? When you’re running dozens or hundreds of small clusters and don’t want sidecar complexity on each one? ...

Compliance audits are painful. Anyone who’s been through ISO 27001 certification knows the drill: weeks of documentation gathering, screenshots of configurations, evidence of change management processes, proof that what you say you do is what you actually do. But here’s something I’ve realized after running declarative infrastructure for years: systems like Talos and NixOS don’t just make infrastructure better — they make compliance dramatically easier. The same properties that make these systems reliable (immutability, reproducibility, auditability) are exactly what auditors want to see. ...

I’ve written about Talos Linux as the immutable Kubernetes OS, and I’ve compared Arch vs NixOS for workstations. But there’s a question I get asked often: what about NixOS for Kubernetes nodes? Both NixOS and Talos are declarative. Both can be immutable. Both version their configuration. So why would you choose one over the other for running Kubernetes? I’ve run both in production. Here’s what I’ve learned. The Philosophical Difference Before diving into specifics, understand the core difference: ...

The first time I tried to SSH into a Talos node, I got nothing. No shell, no connection, no familiar Linux prompt. My immediate reaction was confusion, then mild panic. How am I supposed to debug this thing? That was three years ago. Today, I can’t imagine running Kubernetes on anything else. What is Talos Linux? Talos Linux is a Linux distribution designed specifically for Kubernetes. But calling it a “Linux distribution” undersells how different it is. Talos strips away everything that makes a traditional Linux system… traditional. ...

“We want to migrate to Kubernetes by November.” It was September. The client was an e-commerce company. Their biggest sales event of the year — Black Friday — was in late November. I said no. They asked if I knew someone who might take it on anyway. I did. A fellow platform engineer — someone I respect, highly capable. I made the introduction, but warned him about the timeline. He took the engagement, documented the same concerns I had, got them signed off. The client proceeded anyway. ...