Posts

The first cluster I ever ran in anger had exactly one permission model: everything was cluster-admin. My CI pipeline, my monitoring stack, the little webhook receiver I threw together one afternoon. All of it could read every secret, delete every deployment, and touch every namespace. It worked great right up until I started thinking about what happens when one of those pods gets popped. Kubernetes RBAC (Role-Based Access Control) answers a single question: who can do what to which resources? The default answer on most clusters is “everyone can do everything,” and that answer quietly becomes your biggest liability. ...

I scanned my images with Trivy. I enforced policies with Kyverno. My workloads got cryptographic identity through SPIFFE. Three layers of prevention, all green, and for a while that felt like enough. Then I started asking the uncomfortable question. What happens after a pod is running? My scanners checked the image that went in. My admission controller checked the spec at deploy time. Neither of them is watching once the process is actually executing. If a container gets popped by a zero-day at 3am, every one of those controls has already done its job and gone home. ...

How does Service A know that Service B is actually Service B? I keep coming back to that question because the usual answer is uncomfortable. For years we trusted network location. Traffic from the right IP was legitimate, end of story. Zero trust took that assumption out back and shot it. Now every service has to prove who it is, every single request, no matter where it sits on the network. ...

I used to keep a wiki page titled “Cluster conventions”. Resource limits on everything. No :latest tags. No deploys in the default namespace. It was a good page. Nobody read it. Six months in, half the cluster broke those rules and I only found out when something fell over. A rule that lives in a doc is a suggestion. A rule the API server refuses to accept is governance. That gap is the whole reason this post exists. ...

The first time I ran kubectl get secret myapp -o yaml and base64-decoded the value, I felt my stomach drop. There was my database password, sitting in etcd, readable by anyone who could reach the API with the right RBAC. Kubernetes Secrets are not secrets. They’re base64-encoded plain text with a fancy name. That’s the default, and it’s the thing nobody warns you about on day one. Every cloud provider has a fix for sale. AWS has Secrets Manager, Google has Secret Manager, Azure has Key Vault. They work. The catch shows up later: the day you need to migrate, the day you want to know exactly what happens to a secret after you write it, the day you realise your most sensitive data lives in a system you can’t inspect. ...