When something goes wrong in Kubernetes, the trail usually leads back to etcd. API server timing out? Check etcd. Pods stuck in pending? Might be etcd. Cluster feels sluggish? Probably etcd.

For a long time I treated etcd the way most operators do: as a black box that hums along next to the control plane. “The database.” You back it up and otherwise leave it alone. But black boxes feel like splinters to me, and the first time an etcd cluster fell over at 2am I realised I had no idea what I was actually looking at. So I learned. And it turns out the whole thing is built on a handful of ideas that, once they click, make most etcd problems diagnosable instead of terrifying.

Let me build it up from the bottom.

What is etcd?

etcd is a distributed key-value store. A highly reliable dictionary that multiple servers agree on. Kubernetes uses it to hold all cluster state: every pod, deployment, secret, configmap, and custom resource lives in there.

The name comes from “/etc distributed”. The Unix /etc directory holds system configuration, and etcd is that idea spread across nodes.

Four properties define it:

  • Strongly consistent: every node sees the same data at the same time
  • Highly available: it survives node failures, as long as you keep quorum
  • Watch-capable: clients can subscribe to changes
  • Ordered: operations happen in a strict sequence

Those four words explain almost everything etcd does. The rest of this post is really just unpacking how it pulls them off.

The Raft Consensus Algorithm

etcd uses Raft to stay consistent across nodes. If you understand Raft, you understand most of etcd’s behaviour, including the parts that look like bugs but aren’t.

Leader election

In any etcd cluster, one node is the leader. The leader handles all writes. Followers replicate the leader’s log.

flowchart LR
    L["Leader<br/>(etcd1)"] -->|replicates| F1["Follower<br/>(etcd2)"]
    L -->|replicates| F2["Follower<br/>(etcd3)"]
    L -.->|heartbeats| F1
    L -.->|heartbeats| F2

The leader sends heartbeats to followers. If a follower stops hearing from the leader for an election timeout (typically 1-2 seconds), it becomes a candidate and starts an election. This is why a flaky network shows up as constant leader churn: the followers keep thinking the leader died.

# See who's the leader
etcdctl endpoint status --cluster -w table

Log replication

When a write comes in, this is the path it takes:

  1. Client sends the write to the leader
  2. Leader appends it to its log
  3. Leader replicates to followers
  4. Once a majority confirms, the leader commits
  5. Leader responds to the client

Step 4 is the whole reason etcd needs a majority (quorum) to write. With 3 nodes you need 2. With 5 you need 3. Lose quorum and etcd goes read-only rather than risk diverging. That refusal to guess is exactly what you want from your source of truth.

Term numbers

Each election bumps the term number. Treat it as a logical clock. If a node sees a higher term, it knows a newer leader was elected and falls in line. A term number that keeps climbing fast is a tell-tale sign of an unstable cluster.

# Check cluster health and terms
etcdctl endpoint health --cluster -w table

Data Model

So far we have a consistent, replicated log. Now look at what actually gets stored in it.

etcd keeps data as key-value pairs in a flat namespace. Kubernetes layers a hierarchical convention on top:

/registry/pods/default/nginx-abc123
/registry/pods/kube-system/coredns-xyz789
/registry/deployments/default/my-app
/registry/secrets/default/my-secret

Everything sits under /registry/. The Kubernetes API server is mostly a translation layer between the Kubernetes API and etcd’s key-value model. When you kubectl get pods, you are reading a slice of that tree.

Revisions and MVCC

etcd uses Multi-Version Concurrency Control (MVCC). Every modification creates a new revision, and old revisions stick around for a while. That single design choice buys you three things:

  • Watch from revision: “tell me everything that changed since revision 12345”
  • Historical reads: “what was the value at revision 12345?”
  • Optimistic locking: “update only if still at revision 12345”
# Get current revision
etcdctl get / --prefix --limit=1 -w json | jq '.header.revision'

# Read a key with its revision
etcdctl get /registry/pods/default/nginx -w json | jq '.kvs[0].mod_revision'

That first capability, watch-from-revision, is the one Kubernetes leans on hardest. Hold that thought.

Why etcd Performance Matters

Kubernetes is chatty with etcd. Every API call, every controller reconciliation, every watch notification touches it. A slow etcd is a slow cluster, full stop. This is where the abstract Raft theory turns into very concrete operational pain.

Critical metrics

fsync latency: time to persist to disk. Aim for under 10ms. Past 20ms you start seeing problems, because remember step 4 of log replication waits on the disk.

# Check disk latency
etcdctl check perf --load="s"

Raft proposal latency: time to commit a write through Raft. Should be under 50ms.

Watch count: number of active watches. Kubernetes runs thousands of them.

Common performance problems

Slow disks: etcd writes to disk synchronously. Use SSDs, not spinning rust. NVMe if you can get it. This is the single most common cause of a sad cluster.

Network latency: Raft heartbeats and log replication want low latency between nodes. Keep etcd nodes on the same network segment.

Too many objects: a cluster with 100k pods leans on etcd far harder than one with 1k. Watch cardinality explodes.

Large objects: secrets stuffed with huge certificates, ConfigMaps holding megabytes of data, all of it flows through etcd.

Backing Up etcd

Here is the blunt version: if etcd data is gone, your cluster is gone. Everything else is derived state that can be rebuilt. This is the one thing you cannot afford to lose, so back it up on a schedule.

# Snapshot backup
etcdctl snapshot save /backup/etcd-$(date +%Y%m%d-%H%M%S).db

# Verify the backup
etcdctl snapshot status /backup/etcd-*.db -w table

For automated backups:

#!/bin/bash
# /etc/cron.daily/etcd-backup

BACKUP_DIR=/var/backups/etcd
RETENTION_DAYS=7

etcdctl snapshot save $BACKUP_DIR/etcd-$(date +%Y%m%d-%H%M%S).db

# Cleanup old backups
find $BACKUP_DIR -name "etcd-*.db" -mtime +$RETENTION_DAYS -delete

Restoring from Backup

Restoring etcd is serious business. You are resetting cluster state to a point in time, with all the consequences that implies.

# Stop etcd on all nodes first

# Restore on each node (different data-dir)
etcdctl snapshot restore /backup/etcd-snapshot.db \
  --name etcd1 \
  --initial-cluster etcd1=https://10.0.0.1:2380,etcd2=https://10.0.0.2:2380,etcd3=https://10.0.0.3:2380 \
  --initial-cluster-token etcd-cluster-restored \
  --initial-advertise-peer-urls https://10.0.0.1:2380 \
  --data-dir /var/lib/etcd-restored

# Start etcd with new data-dir

After restore, everything that happened after the snapshot is gone. Pods might reference deleted deployments. Services might point at endpoints that no longer exist. The cluster reconciles its way out of it, but expect some chaos on the way. Test this on a throwaway cluster before you ever need it for real.

Compaction and Defragmentation

Remember those old revisions MVCC keeps around? They do not vanish on their own. etcd holds them until compaction runs, and without it the database grows until it falls over.

# Check current database size
etcdctl endpoint status --cluster -w table

# Compact to revision
etcdctl compact 123456

# Defragment to reclaim space
etcdctl defrag --cluster

Kubernetes auto-compacts etcd via --etcd-compaction-interval, but defragmentation stays manual. Compaction drops the old revisions; defrag actually hands the freed disk space back to the filesystem.

Watching etcd

This is where that watch-from-revision feature pays off. Kubernetes controllers do their work by watching etcd. Something changes, etcd notifies the watchers immediately, no polling required.

# Watch all pod changes
etcdctl watch /registry/pods --prefix

# Watch specific namespace
etcdctl watch /registry/pods/default --prefix

That mechanism is how the scheduler learns about new pods the instant they appear, how the kubelet learns about pod assignments, how the whole control plane stays in sync without a central clock. The entire reconciliation model of Kubernetes rides on this one etcd primitive.

Debugging etcd Issues

Now that you know how the pieces fit, debugging stops being guesswork. Each symptom maps to a layer you already understand.

High latency

# Check if disk is the problem
etcdctl check perf --load="s"

# Check cluster health
etcdctl endpoint health --cluster -w table

# Check leader changes (too many = instability)
etcdctl endpoint status --cluster -w table

Disk space issues

# Check database size
etcdctl endpoint status --cluster -w table

# Force compaction and defrag
REV=$(etcdctl endpoint status -w json | jq '.[0].Status.header.revision')
etcdctl compact $REV
etcdctl defrag --cluster

Network issues

# Check if nodes can reach each other
for ep in https://10.0.0.1:2379 https://10.0.0.2:2379 https://10.0.0.3:2379; do
  etcdctl --endpoints=$ep endpoint health
done

etcd in Production

What this looks like on my own clusters:

  1. Dedicated SSDs: etcd wants fast, consistent disk I/O. Don’t share spindles with other workloads.

  2. Monitor relentlessly: etcd_disk_backend_commit_duration_seconds, etcd_network_peer_round_trip_time_seconds, etcd_server_leader_changes_seen_total. Those three tell you about disk, network, and stability respectively.

  3. Regular backups: daily at minimum, and test the restore now and then so you trust it.

  4. Separate network: where possible, put etcd peer traffic on its own network to dodge contention.

  5. Right-size the cluster: 3 nodes for most workloads, 5 for the critical ones. Beyond 5 you are mostly paying replication cost for very little extra resilience.

Wrapping Up

etcd is simple in concept and unforgiving in execution. It is the foundation everything else builds on, so when etcd is healthy Kubernetes is healthy, and when etcd struggles the whole cluster struggles with it.

The payoff for learning the internals is that the splinter stops being a splinter. Once you can see Raft consensus, quorum, and disk latency for what they are, etcd is a well-designed distributed system doing exactly what distributed systems do. No magic, just mechanism, and mechanism you can reason about at 2am.

Give it fast disks, back it up, watch the metrics, and it will quietly keep your cluster running.

etcd is the source of truth. Everything else in Kubernetes is derived state. Protect your source of truth.