Backup Strategy for Your Homelab: The 3-2-1 Rule in Practice

Your homelab runs your GitLab, your passwords, your photos, your home automation. What happens when the disk fails?

If you can’t answer that question confidently, you don’t have backups. You have hope.

The 3-2-1 rule has been around for decades because it works. Three copies, two different media, one offsite. Here’s how to actually implement it.

The 3-2-1 Rule Explained

flowchart TD
    subgraph rule["3-2-1 Backup Rule"]
        Data["Original Data"]

        subgraph three["3 Copies"]
            C1["Copy 1<br/>(Original)"]
            C2["Copy 2<br/>(Local Backup)"]
            C3["Copy 3<br/>(Offsite)"]
        end

        subgraph two["2 Media Types"]
            M1["NVMe/SSD"]
            M2["HDD/NAS"]
        end

        subgraph one["1 Offsite"]
            Off["Cloud/Remote"]
        end
    end

    Data --> C1
    Data --> C2
    Data --> C3
    C1 --> M1
    C2 --> M2
    C3 --> Off

Why Three Copies?

Copy 1: Your live data (original)
Copy 2: Local backup (fast restore)
Copy 3: Offsite backup (disaster recovery)

One copy is not a backup. Two copies can both fail in the same disaster (fire, flood, ransomware). Three copies with separation gives you real resilience.

Why Two Media Types?

Different failure modes:

SSDs can fail silently (bit rot)
HDDs have mechanical failures
RAID is not a backup (protects against drive failure, not data corruption)

Different media means different failure scenarios don’t take out all copies.

Why One Offsite?

Your house can burn down. Your neighborhood can flood. Your entire city can lose power. Offsite means survival even when everything local is gone.

What to Back Up

Critical (Daily Backup)

Data	Why	Tool
Databases	Can’t recreate	pg_dump, Velero
Secrets/credentials	Security critical	Vault export, External Secrets
Configuration	System state	Git (already offsite)
Personal files	Irreplaceable	Restic

Important (Weekly Backup)

Data	Why	Tool
Container images	Rebuild takes time	Registry backup
Persistent volumes	Stateful workloads	Longhorn/Velero
Logs (compressed)	Forensics	Loki snapshots

Rebuildable (Don’t Backup)

Base OS (reinstall from ISO)
Downloaded packages (re-download)
Cached data (regenerates)
Temporary files

Don’t waste backup space on data you can recreate.

Backup Tools

Restic: File-Level Backups

Restic is my go-to for file backups. It’s fast, encrypted, deduplicated, and supports multiple backends.

# Initialize repository
restic init --repo /mnt/backup/restic

# Or with S3 backend
restic init --repo s3:s3.amazonaws.com/my-bucket

# Backup a directory
restic backup /home/user/documents

# Backup with exclusions
restic backup /data \
  --exclude="*.tmp" \
  --exclude=".cache" \
  --exclude="node_modules"

# List snapshots
restic snapshots

# Restore
restic restore latest --target /restore/location

Automated Restic Backups

#!/bin/bash
# /usr/local/bin/backup.sh

export RESTIC_REPOSITORY="s3:s3.eu-west-1.amazonaws.com/homelab-backups"
export RESTIC_PASSWORD_FILE="/etc/restic/password"
export AWS_ACCESS_KEY_ID="your-key"
export AWS_SECRET_ACCESS_KEY="your-secret"

# Backup
restic backup /data/important \
  --exclude-caches \
  --tag homelab \
  --tag daily

# Prune old snapshots (keep 7 daily, 4 weekly, 12 monthly)
restic forget \
  --keep-daily 7 \
  --keep-weekly 4 \
  --keep-monthly 12 \
  --prune

# Check repository integrity
restic check

Cron job:

# /etc/cron.d/restic-backup
0 3 * * * root /usr/local/bin/backup.sh >> /var/log/restic-backup.log 2>&1

Velero: Kubernetes Backups

Velero backs up Kubernetes resources and persistent volumes.

# Install Velero with S3 backend
velero install \
  --provider aws \
  --plugins velero/velero-plugin-for-aws:v1.8.0 \
  --bucket velero-backups \
  --backup-location-config region=eu-west-1 \
  --secret-file ./credentials-velero \
  --use-volume-snapshots=true \
  --snapshot-location-config region=eu-west-1

Scheduled Backups

apiVersion: velero.io/v1
kind: Schedule
metadata:
  name: daily-backup
  namespace: velero
spec:
  schedule: "0 3 * * *"
  template:
    includedNamespaces:
      - production
      - gitlab
      - monitoring
    excludedResources:
      - events
      - pods
    ttl: 720h  # Keep for 30 days
    storageLocation: default
    volumeSnapshotLocations:
      - default

Restore from Velero

# List backups
velero backup get

# Describe a backup
velero backup describe daily-backup-20260518030000

# Restore entire backup
velero restore create --from-backup daily-backup-20260518030000

# Restore specific namespace
velero restore create --from-backup daily-backup-20260518030000 \
  --include-namespaces gitlab

Longhorn Backups

Longhorn has built-in backup to S3:

# Configure backup target
apiVersion: longhorn.io/v1beta1
kind: Setting
metadata:
  name: backup-target
  namespace: longhorn-system
value: "s3://longhorn-backups@eu-west-1/"
---
apiVersion: longhorn.io/v1beta1
kind: Setting
metadata:
  name: backup-target-credential-secret
  namespace: longhorn-system
value: "longhorn-s3-secret"

Schedule recurring backups:

apiVersion: longhorn.io/v1beta1
kind: RecurringJob
metadata:
  name: daily-backup
  namespace: longhorn-system
spec:
  cron: "0 3 * * *"
  task: backup
  groups:
    - default
  retain: 7
  concurrency: 2

Offsite Options

Cloud Storage

Provider	Cost	Pros	Cons
Backblaze B2	$0.005/GB	Cheap, S3-compatible	US-based
Wasabi	$0.0059/GB	No egress fees	90-day minimum
AWS S3 Glacier	$0.004/GB	Very cheap	Slow retrieval
Hetzner Storage Box	€3.81/1TB	EU-based, cheap	SFTP/WebDAV only

Second Location

If you have a friend/family member with a homelab:

flowchart LR
    subgraph your["Your Home"]
        YourData["Your Data"]
        YourBackup["Their Backup<br/>(encrypted)"]
    end

    subgraph their["Their Home"]
        TheirData["Their Data"]
        TheirBackup["Your Backup<br/>(encrypted)"]
    end

    YourData -->|Encrypted| TheirBackup
    TheirData -->|Encrypted| YourBackup

Mutual offsite backup. Both encrypted so neither can read the other’s data.

Self-Hosted Cloud

Run your own S3-compatible storage at a second location:

# MinIO at remote location
apiVersion: apps/v1
kind: Deployment
metadata:
  name: minio
spec:
  template:
    spec:
      containers:
        - name: minio
          image: minio/minio
          args:
            - server
            - /data
          env:
            - name: MINIO_ROOT_USER
              valueFrom:
                secretKeyRef:
                  name: minio-credentials
                  key: user
            - name: MINIO_ROOT_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: minio-credentials
                  key: password

Access via Tailscale for security.

Database Backups

PostgreSQL

#!/bin/bash
# Kubernetes PostgreSQL backup

NAMESPACE="gitlab"
POD=$(kubectl get pod -n $NAMESPACE -l app=postgresql -o jsonpath='{.items[0].metadata.name}')
DATE=$(date +%Y%m%d_%H%M%S)

# Dump all databases
kubectl exec -n $NAMESPACE $POD -- \
  pg_dumpall -U postgres | \
  gzip > /backup/postgres_${DATE}.sql.gz

# Upload to S3
restic backup /backup/postgres_${DATE}.sql.gz --tag postgres --tag daily

Vault Backup

# Export Vault data (requires root token)
vault operator raft snapshot save /backup/vault_$(date +%Y%m%d).snap

# Encrypt and upload
gpg --encrypt --recipient backup@example.com /backup/vault_$(date +%Y%m%d).snap
restic backup /backup/vault_$(date +%Y%m%d).snap.gpg --tag vault

Testing Restores

A backup you haven’t tested is not a backup.

Monthly Restore Test

#!/bin/bash
# test-restore.sh

# Create test namespace
kubectl create namespace restore-test

# Restore from Velero
velero restore create test-restore \
  --from-backup $(velero backup get -o json | jq -r '.items[0].metadata.name') \
  --include-namespaces gitlab \
  --namespace-mappings gitlab:restore-test

# Wait for restore
velero restore wait test-restore

# Verify pods are running
kubectl get pods -n restore-test

# Test application (example: GitLab)
kubectl port-forward -n restore-test svc/gitlab 8080:80 &
curl -s http://localhost:8080/health | grep "ok"

# Cleanup
kubectl delete namespace restore-test

Document Recovery Procedures

For each critical system:

# GitLab Recovery Procedure

## Prerequisites
- Access to Velero backups
- Access to PostgreSQL backups
- GitLab Helm values

## Steps
1. Restore PostgreSQL from backup
2. Restore GitLab PVCs with Velero
3. Deploy GitLab with same Helm values
4. Verify user login works
5. Verify repositories are accessible

## Estimated Time: 45 minutes
## Last Tested: 2026-05-01

Monitoring Backups

Prometheus Alerts

groups:
  - name: backup-alerts
    rules:
      - alert: BackupFailed
        expr: restic_backup_last_successful_timestamp < (time() - 86400)
        for: 1h
        labels:
          severity: critical
        annotations:
          summary: "Backup hasn't succeeded in 24 hours"

      - alert: BackupStorageLow
        expr: restic_repository_size_bytes / restic_repository_max_bytes > 0.9
        for: 1h
        labels:
          severity: warning
        annotations:
          summary: "Backup storage over 90% full"

Backup Dashboard

Track in Grafana:

Last successful backup time
Backup duration trend
Storage usage
Restore test results

My Backup Setup

flowchart TD
    subgraph homelab["Homelab (K3s)"]
        PV["Persistent Volumes"]
        DB["Databases"]
        Config["Configs (Git)"]
    end

    subgraph local["Local Backup (NAS)"]
        Longhorn["Longhorn Snapshots"]
        Restic1["Restic Repository"]
    end

    subgraph offsite["Offsite (Backblaze B2)"]
        Velero["Velero Backups"]
        Restic2["Restic Offsite"]
    end

    PV --> Longhorn
    PV --> Velero
    DB --> Restic1
    Restic1 --> Restic2
    Config --> Git["GitLab (self-hosted)"]
    Git --> GitMirror["GitHub Mirror"]

Schedule

What	Frequency	Retention	Location
Longhorn snapshots	Hourly	24 hours	Local NVMe
Longhorn backups	Daily	7 days	NAS
Velero full backup	Daily	30 days	Backblaze B2
Database dumps	Daily	30 days	Backblaze B2
Git repos	Push	Forever	GitHub mirror

Costs

Backblaze B2: ~€5/month for 200GB
NAS storage: Already owned
Total: ~€5/month for peace of mind

Common Mistakes

“RAID is my backup”

RAID protects against drive failure. It doesn’t protect against:

Accidental deletion
Ransomware
Software bugs corrupting data
Fire/flood/theft

“I’ll restore when I need to”

If you’ve never restored, you don’t know if your backups work. Test quarterly at minimum.

“I backup everything”

Backing up 10TB of movies you can re-download wastes money and time. Prioritize irreplaceable data.

“My backup is in the same room”

A fire takes out your server AND your backup drive. Offsite is non-negotiable.

Why This Matters

Data loss is not a question of if, but when:

Drives fail (3-5% annual failure rate)
Humans make mistakes (rm -rf wrong directory)
Software has bugs (database corruption)
Bad things happen (fire, flood, theft)

The difference between “minor inconvenience” and “catastrophic loss” is a tested backup strategy.

Your homelab stores things that matter. Protect them accordingly.

The best time to set up backups was before you needed them. The second best time is now.

The 3-2-1 Rule Explained#

Why Three Copies?#

Why Two Media Types?#

Why One Offsite?#

What to Back Up#

Critical (Daily Backup)#

Important (Weekly Backup)#

Rebuildable (Don’t Backup)#

Backup Tools#

Restic: File-Level Backups#

Automated Restic Backups#

Velero: Kubernetes Backups#

Scheduled Backups#

Restore from Velero#

Longhorn Backups#

Offsite Options#

Cloud Storage#

Second Location#

Self-Hosted Cloud#

Database Backups#

PostgreSQL#

Vault Backup#

Testing Restores#

Monthly Restore Test#

Document Recovery Procedures#

Monitoring Backups#

Prometheus Alerts#

Backup Dashboard#

My Backup Setup#

Schedule#

Costs#

Common Mistakes#

“RAID is my backup”#

“I’ll restore when I need to”#

“I backup everything”#

“My backup is in the same room”#

Why This Matters#