Longhorn vs Rook-Ceph: Kubernetes Storage Compared

TL;DR Longhorn and Rook-Ceph both give you replicated block storage that survives a node dying. Longhorn is one Helm install and a UI that shows you what a volume is doing. Rook-Ceph is Ceph with a Kubernetes operator around it, so you get block, object and filesystem storage, and the operations that come with them. I run Longhorn on three nodes. Below is why, and when I’d switch.

Run your first stateful workload on a self-hosted cluster and you find out fast that nobody hands out PersistentVolumes for free. There’s no cloud provider storage class to lean on. You’ve got your nodes, their disks, and a Postgres pod that won’t schedule. An hour of reading later you’re left with two names that keep coming up. Longhorn and Rook-Ceph .

I’ve run both in production, and I’ll say up front that I default to Longhorn on small clusters. The why comes later, but you should know it while you read. Both are CNCF projects, both replicate block storage across nodes so a dead node doesn’t take your data with it, and both are good software. Where they disagree is how much complexity you should have to take on.

What you’re choosing between

Longhorn is distributed block storage written for Kubernetes from day one. Each volume gets replicated across your nodes with plain Linux storage primitives, and there’s no separate storage system underneath that you’d have to learn first.

Rook-Ceph is a Kubernetes operator wrapped around Ceph. Ceph is a distributed storage system that’s years older than Kubernetes, runs petabytes at places like CERN, and brings its whole feature set with it. Block, object, filesystem, erasure coding, all of it. Rook teaches Kubernetes how to drive it.

Most of what follows comes from that difference in age. Longhorn was born in your cluster. Ceph moved in from somewhere else and brought everything it owned. Some of that you’ll use. The rest you’ll still have to run.

Four things that decide it

A feature checklist won’t help you here, because both tools tick most of the boxes. What matters is what running it is like. I weigh four things. How much operational work I’m taking on. How it behaves when a node dies, because this is storage and that’s the whole job. How much of my RAM and CPU it wants. And how far it scales before it gets awkward. Performance matters too, but for most homelab and small-production workloads it isn’t what decides it, so I’ll get to it later.

Longhorn: the one that lives in your cluster

How Longhorn works

The model fits in your head, and that’s most of the appeal. Every PVC gets its own Longhorn engine, running as a pod. That engine writes your data to replicas on the local disks of several nodes, and it writes synchronously, so nothing is acknowledged until every replica has it. The workload talks to its volume over iSCSI, which the engine exposes.

When I want to know what a volume is doing, I open the Longhorn UI and look. Which replicas are healthy, where they live, whether a rebuild is running. No black box. That’s how I want to run infrastructure, as I wrote in Sovereign Infrastructure. I need to understand what I’m running, and Longhorn lets me.

Installing it

helm repo add longhorn https://charts.longhorn.io
helm repo update

helm install longhorn longhorn/longhorn \
  --namespace longhorn-system \
  --create-namespace

Basic configuration:

# longhorn-values.yaml
defaultSettings:
  defaultReplicaCount: 3
  defaultDataPath: /var/lib/longhorn
  storageMinimalAvailablePercentage: 15
  defaultLonghornStaticStorageClass: longhorn

persistence:
  defaultClass: true
  defaultClassReplicaCount: 3

What Longhorn does well

Helm install, and you have storage. No dedicated storage nodes, no pool topology to design first. The built-in web UI shows volumes, backup status and node health without you wiring anything up. Backups go straight to S3-compatible storage as incremental snapshots, which on my setup means I point it at my own MinIO and forget about it. And there’s no older system underneath being adapted, so nothing you meet is there for historical reasons.

Where Longhorn falls short

The limits are easy to state. Longhorn is fine for the workloads most of us run, but it isn’t built for extreme IOPS, because every volume’s traffic goes through its own engine pod and that pod is the ceiling. It works well up to roughly 100 nodes and gets awkward past that. And every replica is a full copy, so three replicas means three times the raw capacity. There’s no erasure coding to soften that.

Rook-Ceph: the one that moved in

How Rook-Ceph works

Ceph’s model is clever, and the cleverness is exactly why there’s more to learn. Each disk becomes an OSD, an Object Storage Daemon. Data spreads across those OSDs by the CRUSH algorithm and placement rules you write, so Ceph decides where each piece lives based on your failure domains instead of round-robin. On top of that you get three ways in. Block via RBD, S3-compatible object storage via RGW, and a real filesystem via CephFS. Holding it all together is a quorum of monitor daemons that track cluster state.

Every one of those parts is something you can inspect, which I like. It’s also something you’ll have to understand the first time it misbehaves.

Installing it

helm repo add rook-release https://charts.rook.io/release
helm repo update

# Install Rook operator
helm install rook-ceph rook-release/rook-ceph \
  --namespace rook-ceph \
  --create-namespace

# Create Ceph cluster
kubectl apply -f ceph-cluster.yaml

Cluster configuration:

# ceph-cluster.yaml
apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata:
  name: rook-ceph
  namespace: rook-ceph
spec:
  cephVersion:
    image: quay.io/ceph/ceph:v18.2.0

  mon:
    count: 3
    allowMultiplePerNode: false

  mgr:
    count: 2

  storage:
    useAllNodes: true
    useAllDevices: false
    deviceFilter: "^sd[b-z]"  # Use sdb, sdc, etc.

  resources:
    mon:
      requests:
        cpu: 500m
        memory: 1Gi
    osd:
      requests:
        cpu: 500m
        memory: 2Gi

What Ceph does well

Here the extra weight pays for itself. Ceph handles petabytes, the scale CERN and Bloomberg run it at, so you won’t outgrow it. Block, object and filesystem from one system, plus erasure coding, snapshots and cross-cluster mirroring. There’s a lot to tune, so a team that knows Ceph can shape it to a workload. And erasure coding cuts the overhead. Instead of paying 3x for replication you can land closer to 1.5x, and at large capacity that’s real money.

What Ceph costs you

The same power costs the same effort. Many more moving parts, and monitors, managers and OSDs all want resources and attention. Before you’ve stored a byte you’re running three monitors, two managers and your OSDs, and the memory they take is not small. Ceph carries decades of features and configuration, so the learning curve is steep and doesn’t flatten quickly. For real performance you often end up dedicating nodes to OSDs, hardware you’ve set aside for storage and nothing else. I don’t count any of that as a flaw. It’s what Ceph charges for what it gives you, and the price only makes sense if you’ll use what you bought.

Side by side

The table is fine for a quick look, but the decision lives in the rows you’ll feel. For me that’s resource overhead and complexity, because I pay for those every day the cluster runs, and for setup time only once.

When Longhorn is the right call

Reach for Longhorn when your situation looks like this:

1. Small to medium clusters (under 100 nodes)
2. Simplicity matters and you want storage that just works
3. Limited ops capacity, a small team that can’t spend its time babysitting storage
4. General workloads like databases and stateful apps with moderate I/O
5. Homelab or edge where resources are tight

# Typical Longhorn workload
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: postgres-data
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: longhorn
  resources:
    requests:
      storage: 100Gi

When Rook-Ceph is worth it

Reach for Rook-Ceph when:

1. Large clusters (100+ nodes)
2. Multiple storage types needed, block AND object AND filesystem from one system
3. Performance critical and you need to tune for specific workloads
4. Storage efficiency matters and erasure coding will save you real capacity
5. Dedicated storage team, people who can learn Ceph and operate it well

That last one is the point people skip. Ceph rewards a team that knows it and punishes one that doesn’t, and if nobody on the team owns the storage, the first person to learn Ceph properly will be doing it during an outage.

# Rook-Ceph with erasure coding
apiVersion: ceph.rook.io/v1
kind: CephBlockPool
metadata:
  name: replicated-pool
  namespace: rook-ceph
spec:
  failureDomain: host
  replicated:
    size: 3
---
apiVersion: ceph.rook.io/v1
kind: CephBlockPool
metadata:
  name: erasure-coded-pool
  namespace: rook-ceph
spec:
  failureDomain: host
  erasureCoded:
    dataChunks: 2
    codingChunks: 1

Performance, the criterion I parked

I parked this one earlier, and here’s why it rarely decides anything. For the workloads most people run, both are fast enough. The difference shows up at the edges.

Longhorn under load

# Tune replica count for performance vs durability
defaultSettings:
  defaultReplicaCount: 2  # Faster than 3, less durable

# Use dedicated disk path
defaultDataPath: /mnt/fast-ssd/longhorn

Longhorn is I/O bound by that per-volume engine pod. Push a high-IOPS workload through it and the engine is your bottleneck. That’s the trade you make for the simple architecture.

Rook-Ceph under load

# Dedicated OSD nodes
spec:
  placement:
    osd:
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
            - matchExpressions:
                - key: storage-node
                  operator: In
                  values:
                    - "true"

# NVMe optimization
storage:
  config:
    osdsPerDevice: "1"
    storeType: bluestore

Ceph can saturate modern NVMe drives when it’s configured properly. “When it’s configured properly” is doing a lot of work in that sentence, and getting there is the work you signed up for.

Backups

Backups matter more to me than raw throughput, because storage you can’t restore from will lose data at some point and you won’t know when.

Longhorn backups

Built in. Configure an S3 target:

defaultSettings:
  backupTarget: s3://longhorn-backups@us-east-1/
  backupTargetCredentialSecret: longhorn-s3-credentials

Schedule backups per volume:

apiVersion: longhorn.io/v1beta1
kind: RecurringJob
metadata:
  name: daily-backup
spec:
  cron: "0 2 * * *"
  task: backup
  groups:
    - default
  retain: 7

Rook-Ceph backups

There’s no equivalent built-in flow, so you reach for Velero with Ceph CSI snapshots:

velero install \
  --provider aws \
  --plugins velero/velero-plugin-for-csi \
  --features=EnableCSI

Or native Ceph mirroring for disaster recovery between clusters, which is nice once you’re at the scale where a second cluster exists.

Three nodes, so Longhorn

I said at the top that I default to Longhorn. I run Longhorn in my homelab, and this is the configuration:

# My Longhorn configuration
defaultSettings:
  defaultReplicaCount: 2  # 3 nodes, 2 replicas
  defaultDataPath: /mnt/storage/longhorn
  backupTarget: s3://backups@minio/longhorn/
  backupTargetCredentialSecret: minio-credentials
  storageMinimalAvailablePercentage: 20

persistence:
  defaultClass: true

Three nodes is the deciding fact. That’s too small to carry Ceph’s overhead, where the monitors and managers alone would eat capacity I can’t spare. When a volume misbehaves I open the Longhorn UI and read what’s wrong. With Ceph I’d be paging through internals working out which daemon does what, and I’ve done enough of that already. Backups land in my own MinIO over S3, and on small nodes every spare MB stays spare.

The day I’m running 50-plus nodes, or need object storage next to block, I’ll switch to Ceph and pay the complexity gladly, because then I’d be using what I paid for. That day isn’t here. Your cluster might put it a lot closer. If you’ve got the team and the scale, pick Ceph and don’t let my three nodes talk you out of it.

Migrating later if you outgrow Longhorn

Starting on Longhorn and worried you’re painting yourself into a corner? You aren’t. The path out is boring, which is the best thing you can say about a migration:

1. Back up the data from the Longhorn volume
2. Deploy Rook-Ceph alongside it
3. Restore into Ceph volumes
4. Update workloads to use the new StorageClass
5. Retire Longhorn once everything’s moved

Both speak CSI, so your workloads see the same interface either way. You change a StorageClass and you’re done.

Picking the complexity you can carry

Storage is the part of Kubernetes that punishes mistakes hardest. Get it wrong and you lose data, which is the one failure you can’t roll back. Over-build it and you spend your weeks feeding complexity you never needed.

Map it to scale and the answer usually falls out. Homelab and small clusters point at Longhorn. Medium production goes either way, depending on whether you’d use the extra Ceph features. Large scale points at Ceph. Both are solid software. What separates them is how much operational weight you want to carry, and only you know that for your own cluster.

Mine is three nodes and Longhorn, and it survives a node dying, which is the failure I built it for. When it outgrows that, the five steps above are the whole plan.

• Infrastructure
• Architecture