Kubernetes High Availability: Stacked vs External etcd Explained

The first “production” Kubernetes cluster I ran had a single control plane node. It hummed along happily for weeks, right up until a disk failed and took the whole thing with it. Every pod, every service, gone. That outage taught me what “single point of failure” actually feels like, and it pushed me toward a question that I keep seeing trip people up: when you build a cluster that survives node loss, do you run etcd on your control plane nodes, or on dedicated nodes of its own?

That choice has a name. Stacked etcd versus external etcd. Both are valid, both ship in real production clusters, and the right answer depends entirely on your context. I want to walk through them the way I wish someone had walked me through it, with the trade-offs laid bare instead of a vendor diagram telling me which is “best practice.”

A quick bias check before we start: I self-host everything, my clusters live on refurbished mini PCs in a homelab, and I value understanding what I run over chasing the most scalable architecture on paper. Keep that in mind. Your constraints will tilt the balance differently.

What etcd Actually Does

etcd is the part of Kubernetes that remembers things. It’s a distributed key-value store holding all cluster state: pod definitions, secrets, configmaps, service accounts, the lot. Every time you kubectl apply something, the desired state lands in etcd. The control plane components then read from it and work to make reality match.

So when etcd is down, your cluster is effectively brain-dead. The API server has nothing to serve, controllers can’t reconcile anything, and your running workloads drift with no one steering. etcd availability and cluster availability are the same number.

etcd stays consistent using the Raft consensus algorithm, which needs a quorum, a strict majority of members, to accept writes. Three members tolerate one failure. Five members tolerate two. Drop below quorum and etcd goes read-only until enough members come back. That quorum math is the thing every HA decision below revolves around, so it’s worth holding onto.

The Two Topologies

The whole comparison comes down to where etcd lives relative to the rest of the control plane.

Stacked means etcd runs on each control plane node, sitting right next to the API server, controller manager, and scheduler.

flowchart TD
    subgraph stacked["Stacked etcd Topology"]
        subgraph CP1["Control Plane Node 1"]
            API1["API Server"]
            Ctrl1["Controller"]
            Sched1["Scheduler"]
            etcd1["etcd"]
        end
        subgraph CP2["Control Plane Node 2"]
            API2["API Server"]
            Ctrl2["Controller"]
            Sched2["Scheduler"]
            etcd2["etcd"]
        end
        subgraph CP3["Control Plane Node 3"]
            API3["API Server"]
            Ctrl3["Controller"]
            Sched3["Scheduler"]
            etcd3["etcd"]
        end
    end

External means etcd runs on its own dedicated nodes, fully separated from the control plane.

flowchart TD
    subgraph external["External etcd Topology"]
        subgraph etcd_cluster["etcd Cluster"]
            E1["etcd Node 1"]
            E2["etcd Node 2"]
            E3["etcd Node 3"]
        end
        etcd_cluster --> control_plane
        subgraph control_plane["Control Plane Nodes"]
            subgraph CP1["Control Plane 1"]
                API1["API Server<br/>Controller<br/>Scheduler"]
            end
            subgraph CP2["Control Plane 2"]
                API2["API Server<br/>Controller<br/>Scheduler"]
            end
            subgraph CP3["Control Plane 3"]
                API3["API Server<br/>Controller<br/>Scheduler"]
            end
        end
    end

Same components, different blast radius. That difference is the whole story. Let me compare them on the criteria that actually decide the question: operational complexity, failure isolation, scaling flexibility, and resource behaviour.

Operational Complexity

Stacked wins here without much of a fight. Three nodes give you a fully HA control plane with etcd quorum baked in, and the lifecycle is coupled in a helpful way: add a control plane node and you automatically get an etcd member to go with it. kubeadm, K3s, and most installers default to stacked because it’s the path with the fewest moving parts. You run kubeadm init with --control-plane-endpoint, join two more nodes, and you’re done.

External etcd asks more of you up front. You bootstrap and certificate an etcd cluster yourself before kubeadm will even initialise the control plane, and from then on you’re maintaining two tiers with separate upgrade and backup procedures. More nodes, more network paths to secure, more things to reason about at 2am.

If your team hasn’t operated etcd before, that complexity is a real cost, not an abstract one. I’d rather understand three nodes deeply than half-understand six.

Failure Isolation

This is where external earns its keep. With stacked etcd, a dead node costs you two things at once: a control plane component and an etcd member. Lose a node and your quorum margin shrinks at the same moment your API capacity does. With external etcd those failures are independent. A control plane node dying leaves etcd quorum untouched, and an etcd node dying leaves the API servers serving.

For most clusters that coupling is fine, because three nodes still tolerate one failure either way. The isolation matters when you’re running tight SLAs and can’t afford a single node loss taking out two systems at once.

Scaling Flexibility

Stacked forces your etcd member count and your API server count to move together. That’s usually what you want, but not always. etcd actually performs worse past a certain member count because every write has to reach a majority, so you rarely want more than five members. Your API server tier, meanwhile, might genuinely benefit from scaling out further under heavy read load.

External decouples the two. Run three etcd members for stable consensus and five API servers for throughput, sized independently. On large clusters, past roughly a hundred nodes where API and etcd load climbs, that independence stops being theoretical.

Resource Behaviour

etcd is unusually sensitive to disk latency. It writes every committed entry to disk and expects that write to be fast, so when something else on the box hammers the disk, etcd starts missing heartbeats and can trigger an unnecessary leader election. On a stacked node, that “something else” is your own API server doing a large list operation. The two compete for the same disk.

External sidesteps this by giving etcd dedicated hardware. Fast NVMe for the etcd tier, more RAM for the API tier, each component tuned for what it actually does. You also get to back up, restore, and upgrade etcd without touching the control plane at all.

The Trade-offs Side by Side

The table is the quick reference. The honest answer lives in the next section.

So Which Should You Pick

Reach for stacked when you’re running a homelab or development cluster, when you’re under roughly a hundred nodes with sane workloads, when your team is newer to etcd operations, or when you’re on managed Kubernetes (EKS, GKE, AKS already handle this and the choice isn’t yours to make). The lower node count means less power draw, less cost, and fewer things to understand, which for a self-hoster is the whole point.

Reach for external when you’re past a hundred nodes and the API and etcd load justify dedicated hardware, when your SLAs genuinely can’t tolerate a node loss hitting two systems, when you need to scale the control plane and etcd independently, or when your team already operates etcd clusters comfortably and the added complexity is cheap for you.

There’s a middle path worth knowing about too. K3s defaults to embedded etcd (stacked, with a much lighter setup) but also supports an external SQL database (MySQL, PostgreSQL) in place of etcd entirely, which is genuinely handy if you already run and back up one of those.

Setting Up Stacked HA with kubeadm

# On first control plane node
kubeadm init \
  --control-plane-endpoint "loadbalancer.example.com:6443" \
  --upload-certs

# Save the join commands for other control planes
# Join additional control plane nodes
kubeadm join loadbalancer.example.com:6443 \
  --token <token> \
  --discovery-token-ca-cert-hash sha256:<hash> \
  --control-plane \
  --certificate-key <cert-key>

The load-bearing flag is --control-plane-endpoint. It should point at a load balancer in front of your API servers, never a single node IP, otherwise you’ve quietly reintroduced the single point of failure you were trying to remove.

Setting Up External etcd

For external etcd you bring up the etcd cluster first:

# On each etcd node, configure etcd
cat > /etc/etcd/etcd.conf.yaml << EOF
name: etcd-1
data-dir: /var/lib/etcd
initial-cluster-state: new
initial-cluster: etcd-1=https://10.0.0.1:2380,etcd-2=https://10.0.0.2:2380,etcd-3=https://10.0.0.3:2380
listen-peer-urls: https://10.0.0.1:2380
listen-client-urls: https://10.0.0.1:2379,https://127.0.0.1:2379
advertise-client-urls: https://10.0.0.1:2379
initial-advertise-peer-urls: https://10.0.0.1:2380
client-transport-security:
  cert-file: /etc/etcd/pki/server.crt
  key-file: /etc/etcd/pki/server.key
  trusted-ca-file: /etc/etcd/pki/ca.crt
peer-transport-security:
  cert-file: /etc/etcd/pki/peer.crt
  key-file: /etc/etcd/pki/peer.key
  trusted-ca-file: /etc/etcd/pki/ca.crt
EOF

Then point kubeadm at that external cluster:

# kubeadm-config.yaml
apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
etcd:
  external:
    endpoints:
      - https://10.0.0.1:2379
      - https://10.0.0.2:2379
      - https://10.0.0.3:2379
    caFile: /etc/kubernetes/pki/etcd/ca.crt
    certFile: /etc/kubernetes/pki/apiserver-etcd-client.crt
    keyFile: /etc/kubernetes/pki/apiserver-etcd-client.key

What I Actually Run

For me, stacked wins, and it isn’t close. I run stacked etcd in my homelab on three control plane nodes. It has survived disk failures, network blips, and the occasional “oops, I rebooted the wrong server” without losing the cluster. Stacked doesn’t mean fragile. It means coupled, and coupling three nodes I understand fully beats spreading state across six I half-understand.

But tie that back to my bias: I’m a self-hoster optimising for understanding and low node count, not a platform team running a thousand-node fleet under a five-nines SLA. If that’s your world, external etcd’s isolation is worth every extra node. Your context decides, not mine.

The thing that matters more than the topology is the node count underneath it. One node is not HA, it’s a single point of failure with extra steps. Two nodes is worse than one, because quorum needs a majority and 2/2 - 1 = 1, so you can’t actually tolerate any failure at all. Three is the floor for real HA, with either topology.

Kubernetes HA is about surviving failures, not preventing them. Disks die, networks flap, and someone reboots the wrong box eventually. The only question worth asking is whether the cluster shrugs and keeps going, and three control plane nodes with healthy etcd quorum is what lets it.