Advanced

Every vendor pitch deck right now has the same slide. “Autonomous cluster management.” An AI watches your Kubernetes cluster, spots problems, diagnoses them, and fixes them while you sleep. Platform engineers get to stop firefighting and the cluster heals itself. I run a homelab specifically because I want to understand what’s actually happening, not trust a black box. So when I see a claim like that, my first instinct is to test it myself rather than believe the slide. ...

I scanned my images with Trivy. I enforced policies with Kyverno. My workloads got cryptographic identity through SPIFFE. Three layers of prevention, all green, and for a while that felt like enough. Then I started asking the uncomfortable question. What happens after a pod is running? My scanners checked the image that went in. My admission controller checked the spec at deploy time. Neither of them is watching once the process is actually executing. If a container gets popped by a zero-day at 3am, every one of those controls has already done its job and gone home. ...

How does Service A know that Service B is actually Service B? I keep coming back to that question because the usual answer is uncomfortable. For years we trusted network location. Traffic from the right IP was legitimate, end of story. Zero trust took that assumption out back and shot it. Now every service has to prove who it is, every single request, no matter where it sits on the network. ...

I used to keep a wiki page titled “Cluster conventions”. Resource limits on everything. No :latest tags. No deploys in the default namespace. It was a good page. Nobody read it. Six months in, half the cluster broke those rules and I only found out when something fell over. A rule that lives in a doc is a suggestion. A rule the API server refuses to accept is governance. That gap is the whole reason this post exists. ...

A standard Kubernetes Deployment had served me well for a long time. Push a new image tag, watch the pods roll, done. It was simple, it was declarative, and most of the time nothing went wrong. The rolling update even gave me a warm feeling of safety: old pods only get torn down once new ones are ready. That feeling is a lie. A rolling update protects you from pods that fail to start. It does nothing to protect you from pods that start perfectly and then serve broken responses. The container is healthy, the readiness probe is green, and your new code is quietly returning 500s to every single user. Within seconds, 100% of your traffic is hitting code that nobody validated under real load. ...