Posts

Kubernetes is designed to be self-healing, but what does that actually mean? More importantly: what happens when the components doing the healing themselves fail? I’ve run Kubernetes clusters through all kinds of failures — planned, unplanned, and “hold my beer” experiments. Here’s what actually happens when things break. The Components That Can Fail Before diving into failure scenarios, let’s map out what we’re working with: Control Plane: kube-apiserver: The API that everything talks to etcd: The database storing all cluster state kube-scheduler: Decides where pods run kube-controller-manager: Runs controllers (ReplicaSet, Deployment, etc.) cloud-controller-manager: Cloud provider integrations (if applicable) Node Components: ...

Kubernetes Network Policies are one of those features that everyone knows they should use but few actually understand. The YAML looks intimidating, the behavior is non-intuitive, and the mental model takes time to develop. I’ve spent hours debugging policies that “should work” but didn’t. Let me save you that pain with a visual approach to understanding Network Policies. The Default: Everything Talks to Everything By default, Kubernetes allows all pod-to-pod communication. Any pod can reach any other pod across any namespace. This is convenient for getting started but terrible for security. ...

When something goes wrong in Kubernetes, it’s often etcd. API server timing out? Check etcd. Pods stuck in pending? Might be etcd. Cluster feels slow? Probably etcd. Yet most Kubernetes operators treat etcd as a black box. It’s just “the database” that runs alongside the control plane. But understanding etcd makes you dramatically better at operating Kubernetes. Let me take you inside. What is etcd? etcd is a distributed key-value store. Think of it as a highly reliable dictionary that multiple servers agree on. Kubernetes uses it to store all cluster state: every pod, deployment, secret, configmap, and custom resource lives in etcd. ...

When I first set up a “production” Kubernetes cluster, I had one control plane node. It worked fine until it didn’t — a failed disk took down the entire cluster. Every pod, every service, everything. That day taught me what “single point of failure” really means. Kubernetes High Availability isn’t optional for production. But there’s a choice that trips up many people: do you run etcd on your control plane nodes (stacked), or on separate dedicated nodes (external)? Let me break down both approaches. ...