Advanced

Kubernetes is designed to be self-healing, but what does that actually mean? More importantly: what happens when the components doing the healing themselves fail? I’ve run Kubernetes clusters through all kinds of failures — planned, unplanned, and “hold my beer” experiments. Here’s what actually happens when things break. The Components That Can Fail Before diving into failure scenarios, let’s map out what we’re working with: Control Plane: kube-apiserver: The API that everything talks to etcd: The database storing all cluster state kube-scheduler: Decides where pods run kube-controller-manager: Runs controllers (ReplicaSet, Deployment, etc.) cloud-controller-manager: Cloud provider integrations (if applicable) Node Components: ...

When something goes wrong in Kubernetes, it’s often etcd. API server timing out? Check etcd. Pods stuck in pending? Might be etcd. Cluster feels slow? Probably etcd. Yet most Kubernetes operators treat etcd as a black box. It’s just “the database” that runs alongside the control plane. But understanding etcd makes you dramatically better at operating Kubernetes. Let me take you inside. What is etcd? etcd is a distributed key-value store. Think of it as a highly reliable dictionary that multiple servers agree on. Kubernetes uses it to store all cluster state: every pod, deployment, secret, configmap, and custom resource lives in etcd. ...

When I first set up a “production” Kubernetes cluster, I had one control plane node. It worked fine until it didn’t — a failed disk took down the entire cluster. Every pod, every service, everything. That day taught me what “single point of failure” really means. Kubernetes High Availability isn’t optional for production. But there’s a choice that trips up many people: do you run etcd on your control plane nodes (stacked), or on separate dedicated nodes (external)? Let me break down both approaches. ...