Posts

The first time it bit me I was halfway through a database migration over SSH. Train hit a tunnel, WiFi blinked, connection gone. The process running on the other end went with it. I sat there staring at a dead prompt wondering how much of the migration had actually committed. A terminal multiplexer would have saved me. The process keeps running on the server whether I’m attached or not, so a dropped connection becomes a non-event. I reconnect, reattach, and the work is exactly where I left it. That property alone is worth the setup cost, and it’s why I treat a multiplexer as part of the base layer on every machine I touch. ...

My homelab cluster runs in a closet at home. I do not. I work from coffee shops, from a client office, sometimes from a hotel with WiFi that feels actively hostile. And I still want to reach my own machines while I’m out there. For years the standard answer to that was to poke holes in your own front door. Forward a port on the router, wire up dynamic DNS so the changing home IP doesn’t break everything, write firewall rules, and then sit with the quiet hope that nobody scanning the internet stumbles onto the SSH daemon you just exposed. It works, in the sense that you can reach your stuff. It also means a part of your private network is now answering questions from strangers, all day, forever. ...

Every platform team eventually asks the same question: should we build an Internal Developer Platform? The honest answer is usually yes. The part that wrecks teams is the how. I’ve watched platforms that cost a small fortune get shipped and then quietly abandoned because nobody wanted to use them. I’ve also seen a couple of Helm charts and a Kyverno policy change how a whole team ships software. The gap between those two outcomes has almost nothing to do with budget or which fashionable tool you picked. It comes down to whether you started by solving a real problem or by building the platform you imagined developers should want. ...

You have Grafana. You have Prometheus metrics. You have logs in Loki and traces in Tempo. The data is all there. You also have 47 dashboards that nobody opens. I have done this to myself more than once. Something breaks at 2 AM, I bolt together a dashboard to see what’s going on, and then it just sits there forever. Multiply that by a year of incidents and a few “let me just add a panel for that” moments, and you end up with a Grafana that’s mostly archaeology. Nobody remembers what half the panels mean. The honest move is to delete most of them, but first it helps to understand what makes the survivors worth keeping. ...

My dashboard is a wall of green. Pods running, replicas matched, CPU comfortable, no alerts firing. I look at it and feel that small dopamine hit of “everything is fine.” And for the most part, it is fine. The cluster has been up for weeks. Nothing has fallen over. That green wall is also the most dangerous thing in my homelab, because it tells me nothing about what happens when something goes wrong. It only tells me that, right now, nothing has. ...