GitOps Disaster Recovery: Rebuilding Your Cluster from Git

For a couple of years my homelab cluster ran without much drama. Nodes came and went, workloads shifted around, and the worst I ever had to deal with was the occasional pod stuck in CrashLoopBackOff. The kind of stable where you stop thinking about what happens if it all goes away.

Then one evening I ran a terraform destroy against the wrong workspace. By the time I noticed, the control plane was gone. Not degraded. Gone.

That is the moment disaster recovery stops being a planning slide and becomes a question you have to answer right now: the cluster is gone, the hardware is dead, the region is down, or you fat-fingered a destroy. What do you do next?

If you have been doing GitOps properly, the answer is almost boring. You provision a fresh cluster, point ArgoCD at Git, and wait. Your infrastructure rebuilds itself from the repository. That is the whole promise: Git is your backup.

I want to walk through why that works, what has to be true for it to work, and where it quietly fails you. Because the first time I tested this for real, I learned that “Git is your backup” is true for infrastructure and a dangerous lie for data.

Why GitOps Changes Disaster Recovery

Traditional DR is a second copy of a primary thing. You take the cluster, you snapshot etcd, you run Velero, you ship those backups somewhere safe, and you keep a restore procedure you hopefully tested. Recovery time gets measured in hours, and most of those hours go to copying state back into place.

GitOps flips which thing is primary. The cluster stops being the source of truth and becomes a derived artifact. The real state lives in Git. The cluster is just the current rendering of that state, and you can throw it away and render it again.

flowchart TD
    subgraph traditional["Traditional"]
        TC["Cluster<br/>(primary)"] -->|backup| TB["Backup<br/>(secondary)"]
    end

    subgraph gitops["GitOps"]
        GG["Git<br/>(primary)"] -->|sync| GC["Cluster<br/>(derived)"]
    end

Once you internalise that, recovery changes shape. You are not restoring a cluster. You are bootstrapping a new one and letting it reconcile. Recovery time drops to minutes, because there is no separate backup to copy back. This is resilience by design rather than resilience bolted on: the system that builds your cluster is the same system that rebuilds it.

It also fits how I want to understand my infrastructure. When the whole thing is declared in a repo I can read, there are no hidden bits of state that only exist because someone ran kubectl apply at 2am and never wrote it down.

What Has to Be in Git for This to Work

Here is the catch, and it is where most GitOps setups quietly fail their own DR test: this only works if everything is actually in Git. Anything that exists only in the running cluster is something you will lose and have to remember to recreate by hand, under pressure, while production is down.

So before walking through recovery, walk through what needs to live in the repo.

Application manifests

All your workload definitions. Deployments, Services, ConfigMaps, Helm charts with their values files, Kustomize overlays. This is the easy part, and most teams already have it. If you are doing GitOps at all, your apps are here.

Infrastructure configuration

The cluster-level resources that are easy to forget because nobody deploys them as part of a release:

• Namespaces
• RBAC (Roles, RoleBindings, ServiceAccounts)
• NetworkPolicies
• ResourceQuotas
• PodSecurityPolicies / Pod Security Standards

These define your security boundaries and isolation. A cluster you rebuilt without its NetworkPolicies is a cluster with the blast doors left open. Treat them as first-class GitOps content, not an afterthought.

Platform components

The tools your applications assume are already there:

• Ingress controllers
• Cert-manager
• Monitoring stack (Prometheus, Grafana)
• Logging (Loki, Fluentd)
• Service mesh, if you run one

If these are not in Git, your apps come back but nothing routes to them, nothing has a certificate, and you are flying blind because your observability stack never deployed.

Secrets strategy

Secrets are the one thing you genuinely cannot commit in plaintext, so they need a strategy that survives losing the cluster:

• Sealed Secrets: encrypted in Git, decrypted in-cluster by a controller
• External Secrets: fetched at runtime from Vault or a cloud secrets manager
• SOPS: files encrypted with keys you hold

The property that matters for DR is reproducibility. A secret that exists only inside the dead cluster is a secret you have lost. Whichever tool you pick, the encrypted or referenced form has to be in Git and the decryption key has to live somewhere you can still reach after the cluster is gone.

Persistent data

Here is the hard truth I learned the unpleasant way: GitOps does not back up your data.

Databases, object storage, anything stateful, none of it is in Git and none of it comes back when ArgoCD syncs. GitOps rebuilds the infrastructure that runs your data. The data itself needs a separate, tested backup strategy, and if you skip that step you will rebuild a perfect, empty cluster and feel sick about it.

The Recovery Process

With all of that in Git, recovery becomes a sequence you can actually run. Here is the path I follow.

Step 0: Have a recovery playbook

Before anything breaks, write down how to come back:

• How to provision a fresh cluster
• The bootstrap commands for ArgoCD
• Where Vault lives and where its backups are
• External dependencies (DNS, load balancers)

Then run it. A playbook you have never executed is fiction, and you do not want to discover the gaps while production is down. The first time I “tested” mine, I found three undocumented manual steps that would have cost me an hour at the worst possible moment.

Step 1: Provision a new cluster

Whatever your method, get a working Kubernetes cluster:

# Terraform
cd terraform/clusters/production
terraform apply

# Or managed Kubernetes
gcloud container clusters create production-recovery \
  --region europe-west1 \
  --num-nodes 3

# Or bare metal
talosctl apply-config --nodes 10.0.0.10 --file controlplane.yaml

How you get the cluster is deliberately separate from GitOps. GitOps takes over once you have a functioning API server to talk to.

Step 2: Install ArgoCD

kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

# Wait for it to be ready
kubectl wait --for=condition=Ready pods --all -n argocd --timeout=300s

Step 3: Restore secrets access

Wire External Secrets back to Vault so the rest of the sync has the credentials it needs:

# Configure Vault authentication for the new cluster
kubectl create secret generic vault-token \
  -n external-secrets \
  --from-literal=token=$VAULT_TOKEN

Step 4: Connect ArgoCD to Git

# Add your GitOps repository
argocd repo add https://github.com/yourorg/gitops-repo.git \
  --username git \
  --password $GIT_TOKEN

# Or via Secret
kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: gitops-repo
  namespace: argocd
  labels:
    argocd.argoproj.io/secret-type: repository
stringData:
  url: https://github.com/yourorg/gitops-repo.git
  username: git
  password: ${GIT_TOKEN}
EOF

Step 5: Apply the root application

If you use the App-of-Apps pattern (you should), one manifest pulls in everything else:

kubectl apply -f - <<EOF
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: root
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/yourorg/gitops-repo.git
    targetRevision: main
    path: clusters/production
  destination:
    server: https://kubernetes.default.svc
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
EOF

Step 6: Wait

This is the part that still feels slightly magic to me. ArgoCD reconciles everything on its own:

1. Discovers all applications under the root app
2. Creates namespaces and RBAC
3. Deploys platform components
4. Deploys your applications

Watch it work:

argocd app list
argocd app get root --refresh

Or in the UI: kubectl port-forward svc/argocd-server -n argocd 8080:443

Step 7: Restore data

This is where your separate data backup strategy earns its keep:

# Restore database
pg_restore -d myapp < /backups/myapp-latest.dump

# Or Velero restore
velero restore create --from-backup production-daily-latest

Step 8: Update DNS and load balancers

Point external traffic at the new cluster:

• Update DNS records
• Reconfigure CDN origins
• Update load balancer targets

How Fast Can You Actually Recover

The honest answer is “it depends, and the dependency is your data.” For the infrastructure half, here is what I typically see:

Infrastructure total: 20-60 minutes.

Data restore is the wildcard. It scales with your data volume and your backup design, and it is almost always the longest leg of the recovery. The clean, fast part is the GitOps part. The slow, anxious part is the data part, which is exactly the part GitOps does not touch.

Where It Bites You

I have hit most of these the hard way, so here is what actually goes wrong.

Missing secrets

If Vault is unreachable, External Secrets cannot fetch anything and your workloads never start. You rebuild the whole cluster and then watch every pod wait on a secret that will not arrive.

Mitigation: run Vault in HA on infrastructure separate from the cluster it serves, back it up regularly, and consider replicating it to a second location.

External dependencies

Your cluster probably leans on things that are not in Git at all:

• Cloud IAM roles
• DNS zones
• SSL certificates from Let’s Encrypt
• External databases

Mitigation: document every external dependency, and better still, manage them with Terraform or Crossplane in the same repo so they rebuild alongside everything else.

Circular dependencies

ArgoCD has to be running before it can deploy ArgoCD. The classic bootstrap problem, and it will trip you if your only copy of “how to install ArgoCD” was a managed-by-ArgoCD application.

Mitigation: keep a bootstrap script that works with plain kubectl, no ArgoCD required. Once ArgoCD is up, let it manage itself from there.

Data loss

GitOps rebuilds infrastructure, not data. A stale database backup means you recover the cluster and still lose hours or days of data.

Mitigation: a separate, tested data backup strategy with real restore tests, not just backup jobs you assume are working.

The Recovery Manifest

After my own near-miss, I started keeping a single file that documents everything recovery needs in one place. When things are on fire, you do not want to be grep-ing through a repo for the Vault address.

# recovery-manifest.yaml
recovery:
  cluster:
    provisioner: terraform
    path: terraform/clusters/production

  secrets:
    method: external-secrets
    vault-address: https://vault.internal.company.com
    vault-backup: s3://backups/vault-snapshots/

  gitops:
    repo: https://github.com/yourorg/gitops-repo.git
    path: clusters/production
    root-app: apps/root.yaml

  data:
    databases:
      - name: postgres-main
        backup-location: s3://backups/postgres/
        restore-command: |
          pg_restore -d main /backups/latest.dump
    volumes:
      - name: uploads
        backup-location: s3://backups/uploads/

  external-dependencies:
    - DNS zone: managed in Route53, terraform/dns/
    - Load balancer: created by terraform/clusters/
    - Vault: vault.internal.company.com (HA, separate infra)

  contacts:
    - oncall: ops-team@company.com
    - escalation: platform-lead@company.com

Commit it to Git. Review it quarterly. Test it at least once a year.

The Mindset This Requires

GitOps DR asks you to think about your clusters differently, and the shift is worth saying out loud:

1. Clusters are cattle, not pets. When recovery is cheap, you stop white-knuckling the protection of any single cluster.

2. State belongs in Git. Every kubectl apply you run without committing the change is DR debt that comes due at the worst time.

3. Data is its own problem. GitOps handles infrastructure cleanly and never pretends to handle your data. That boundary is the most important thing to remember.

4. Practice recovery. The first time you run a real recovery should not be during a real disaster.

My Recovery Checklist

## Pre-Disaster (do now)
- [ ] All manifests in Git
- [ ] Secrets strategy documented and tested
- [ ] Vault backed up and HA configured
- [ ] Recovery playbook written
- [ ] External dependencies documented
- [ ] Data backup strategy implemented
- [ ] Recovery tested in last 90 days

## During Recovery
- [ ] Provision new cluster
- [ ] Install ArgoCD
- [ ] Restore secrets decryption capability
- [ ] Connect ArgoCD to Git
- [ ] Apply root application
- [ ] Monitor sync progress
- [ ] Restore data from backups
- [ ] Update DNS/load balancers
- [ ] Verify application health
- [ ] Notify stakeholders

The best disaster recovery is the one you never need. The second best is the one you have rehearsed often enough that when it finally happens for real, it feels like just another Tuesday.