Advanced

Every platform team eventually asks the same question: should we build an Internal Developer Platform? The honest answer is usually yes. The part that wrecks teams is the how. I’ve watched platforms that cost a small fortune get shipped and then quietly abandoned because nobody wanted to use them. I’ve also seen a couple of Helm charts and a Kyverno policy change how a whole team ships software. The gap between those two outcomes has almost nothing to do with budget or which fashionable tool you picked. It comes down to whether you started by solving a real problem or by building the platform you imagined developers should want. ...

My dashboard is a wall of green. Pods running, replicas matched, CPU comfortable, no alerts firing. I look at it and feel that small dopamine hit of “everything is fine.” And for the most part, it is fine. The cluster has been up for weeks. Nothing has fallen over. That green wall is also the most dangerous thing in my homelab, because it tells me nothing about what happens when something goes wrong. It only tells me that, right now, nothing has. ...

The first time you run a stateful workload on a self-hosted cluster, you hit a wall. No cloud provider storage class to lean on. Just your nodes, their disks, and a Postgres pod that refuses to schedule because nothing can give it a PersistentVolume. So you start reading, and within an hour you’ve narrowed it down to two names that keep coming up: Longhorn and Rook-Ceph. I’ve run both in production. So let me get my bias out of the way before anything else: I default to Longhorn on small clusters, and I’ll explain exactly why later. Keep that in mind as you read, because it colours how I weigh things. Both are CNCF projects, both give you replicated block storage that survives a node dying, and both are good software. They just disagree about how much complexity you should be signing up for. ...

For a while my alerting worked fine. A handful of rules, pages were rare, and when one came in it meant something. Then the cluster grew, I bolted on the Prometheus Operator defaults, and “fine” quietly turned into noise. The tipping point was a 3 AM page. My phone buzzed, I groggily checked it: “High CPU usage on node-worker-3.” I looked at the graph, saw it had been sitting at 75% for ten minutes, and went back to sleep. Next night, same alert. A week later I’d stopped checking at all. ...

The first time a service stopped resolving in one of my clusters, I spent an evening reading iptables chains. Hundreds of rules, generated by kube-proxy, evaluated top to bottom. I never found the actual problem. I restarted a node and it went away. That bothered me more than the outage did. I was running something I couldn’t read. That feeling is why I moved to Cilium. It uses eBPF to push networking logic down into the Linux kernel and skips iptables entirely. You get better performance, you can actually see what your traffic is doing, and network policies stop being a guessing game. ...