Kubernetes

I write a lot about Kubernetes. I run it daily. I genuinely like it. So it might surprise you that I spend a fair amount of time talking people out of it. Here is the reality I keep walking into. A small team, a single product, a roadmap full of real features to build, and someone has decided the first milestone is a Kubernetes cluster. Three nodes minimum, etcd, a CNI, an ingress controller, cert-manager, a monitoring stack. Weeks of work before a single customer sees anything. Everyone nods along, because this is just how serious infrastructure looks in 2026. ...

As a kid I had a word for the things that fascinated me: unbreakable. Indestructible was never quite right, because indestructible means something that never breaks. Unbreakable is the better word. It means something that keeps working even after it breaks. I remember exactly when the fascination started. A photo of an A-10 Thunderbolt II that came back from a mission with half a wing gone, the tail in tatters, and the fuselage full of holes. That thing still brought its pilot home. ...

The first time someone asked me “was this slower last month than it is now?”, I had no answer. My Prometheus only remembered two weeks. The data I needed had already aged out of local disk and been deleted. That gap is the whole reason this post exists. Prometheus is the default for Kubernetes metrics, and for good reason. It works beautifully right up until you need long-term storage, or a view across multiple clusters, or genuine high availability. Then you meet the wall. ...

The first cluster I ever ran in anger had exactly one permission model: everything was cluster-admin. My CI pipeline, my monitoring stack, the little webhook receiver I threw together one afternoon. All of it could read every secret, delete every deployment, and touch every namespace. It worked great right up until I started thinking about what happens when one of those pods gets popped. Kubernetes RBAC (Role-Based Access Control) answers a single question: who can do what to which resources? The default answer on most clusters is “everyone can do everything,” and that answer quietly becomes your biggest liability. ...

I scanned my images with Trivy. I enforced policies with Kyverno. My workloads got cryptographic identity through SPIFFE. Three layers of prevention, all green, and for a while that felt like enough. Then I started asking the uncomfortable question. What happens after a pod is running? My scanners checked the image that went in. My admission controller checked the spec at deploy time. Neither of them is watching once the process is actually executing. If a container gets popped by a zero-day at 3am, every one of those controls has already done its job and gone home. ...