Drift Detection with ArgoCD: How to Know If Your Cluster Is Still in Sync

The whole pitch of GitOps is that Git is the source of truth. That promise holds right up until someone runs kubectl edit on a deployment at 2am to stop an incident, a mutating webhook quietly rewrites a resource, or a half-finished sync leaves your cluster somewhere between what Git wanted and what it got. Now Git says one thing and the cluster does another, and nobody told you.

That gap is configuration drift, and it is the part of GitOps people forget to defend. The good news: ArgoCD already watches for it. The catch is that the defaults don’t do what you probably assume, and a few of them will bite you. This post walks from the simplest possible drift check up to the setup I actually run, one layer at a time. Stop wherever you have enough.

What Drift Actually Is

Drift is the moment the real state of your cluster stops matching the desired state in Git.

flowchart LR
    subgraph git["Git says (Source of truth)"]
        G1["replicas: 3"]
        G2["image: v1.2.3"]
        G3["cpu: 100m"]
    end

    subgraph cluster["Cluster has (Actual state)"]
        C1["replicas: 5"]
        C2["image: v1.2.3"]
        C3["cpu: 200m"]
    end

    git -.->|"!="| cluster

Git says three replicas, the cluster runs five. How does that happen? A few usual suspects:

Manual changes: someone ran kubectl scale or kubectl edit and meant to fix it in Git later
Horizontal Pod Autoscaler: HPA bumped replicas on its own
Mutating webhooks: an admission controller rewrote the resource on the way in
Controller side effects: an operator changed a field it manages
Partial syncs: a sync died halfway through

Some of that drift is wanted (HPA doing its job). Most of it is not. The real problem is that from the outside you can’t tell the wanted kind from the dangerous kind, so you have to treat all of it as a signal.

Why I Care About This

Undetected drift quietly cancels the reason you adopted GitOps in the first place. Four things break:

Audit trail: “what’s deployed?” stops being “read the Git history” and becomes “go poke the live cluster”
Disaster recovery: rebuild from Git and you get a cluster that doesn’t match the one you lost
Security: an unauthorised change to a Deployment sits there unnoticed
Reproducibility: two clusters built from the same repo end up subtly different

This connects straight to the sovereignty thing I keep banging on about: I want to understand what I’m running, and a cluster that has silently wandered off from its declared state is a black box I built for myself. Drift detection is how I keep Git honest.

The Simplest Version: Sync Status

You don’t need any extra tooling to start. ArgoCD already compares Git to the live cluster on a loop and reports one of three states:

Synced: cluster matches Git exactly
OutOfSync: it found differences
Unknown: it can’t figure out the state

Application	Sync Status	Health
frontend	Synced	Healthy
backend	OutOfSync	Healthy
database	Synced	Healthy
cache	Synced	Degraded

OutOfSync is your drift signal. That’s the entire basic version: glance at the dashboard, anything not green is drift. It works, but staring at a dashboard is a terrible detection strategy, and the default sync behaviour might not be what you expect. So let’s add a layer.

Layer 1: Self-Heal, and Why It’s a Double-Edged Tool

ArgoCD can revert drift for you automatically:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: frontend
spec:
  syncPolicy:
    automated:
      selfHeal: true  # Revert manual changes
      prune: true     # Delete orphaned resources

With selfHeal: true, the moment someone runs kubectl scale deployment frontend --replicas=5, ArgoCD notices the gap and puts it back to whatever Git says, usually within seconds. This is the self-healing infrastructure ideal in action: the system corrects itself instead of waiting for a human to notice.

It also has sharp edges. That same automation will revert an intentional change, stomp on an HPA adjustment, and make a genuine emergency hotfix bounce right back. For most apps I leave selfHeal on anyway, because the cost of letting drift accumulate is higher than the cost of the occasional reverted change. Just go in knowing the trade-off rather than discovering it during an incident.

Layer 1b: Telling ArgoCD What to Ignore

Some fields are supposed to live outside Git, and self-heal will fight you over them unless you say so. The classic case is replica count when HPA owns it:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: autoscaled-app
spec:
  ignoreDifferences:
    - group: apps
      kind: Deployment
      jsonPointers:
        - /spec/replicas
    - group: ""
      kind: Service
      jsonPointers:
        - /spec/clusterIP

This tells ArgoCD to stop reporting drift on those specific fields. The ones I reach for most:

/spec/replicas when HPA is in charge
/spec/clusterIP because Kubernetes assigns it
/metadata/annotations that controllers add on their own
/status, which is always controller-managed

The goal here is to stop the noise so that when something does show as OutOfSync, it means something.

Layer 2: Detect Without Auto-Fixing

Self-heal is the right default, but not everywhere. For the systems where I want a human to look before anything moves, I turn the automation off and keep the detection on:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: critical-app
spec:
  syncPolicy:
    automated:
      selfHeal: false  # Don't auto-fix
      prune: false     # Don't auto-delete

Now ArgoCD still flags OutOfSync and shows you the diff, but it waits for you to decide. This earns its keep for critical production where you want eyes on every change, for debugging the source of recurring drift, and for apps that are deliberately managed partly outside GitOps.

Layer 2b: Get Notified Instead of Watching

Detection only helps if it reaches you. Wire up ArgoCD notifications so drift comes to you rather than you going to the dashboard:

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-notifications-cm
  namespace: argocd
data:
  trigger.on-sync-status-unknown: |
    - when: app.status.sync.status == 'OutOfSync'
      send: [app-out-of-sync]
  template.app-out-of-sync: |
    message: |
      Application {{.app.metadata.name}} is OutOfSync.
      Sync Status: {{.app.status.sync.status}}
      Health: {{.app.status.health.status}}
      Repository: {{.app.spec.source.repoURL}}

Point that at Slack, PagerDuty, or email. Drift should make a noise.

Reading the Diff

When something drifts, ArgoCD will tell you exactly what changed, which is half the battle:

argocd app diff frontend

Or click the OutOfSync app in the UI to see the same thing:

--- Git (desired)
+++ Cluster (actual)
@@ -1,4 +1,4 @@
 spec:
-  replicas: 3
+  replicas: 5
   template:
     spec:

This is where understanding beats guessing. You see the field, the old value, the new value, and you can start asking who or what did it.

One Thing Worth Getting Straight: Refresh vs Sync

These two operations sound similar and do very different things, and mixing them up causes accidents.

Refresh compares Git to the cluster and updates the status. It changes nothing.

argocd app get frontend --refresh

Sync applies the Git state to the cluster. It changes things.

argocd app sync frontend

Refresh is safe and runs on its own every three minutes by default. Sync alters live state, so do it deliberately (or let automation do it for you, knowing what that means).

The Full Picture: My Drift Strategy

Here’s how I actually split this across environments. None of it is exotic; it’s just the layers above, applied with a bit of judgement.

Development / Staging

selfHeal: true to revert all drift
prune: true to clean up orphaned resources
Fast feedback, no ceremony, pure GitOps

Production (most apps)

selfHeal: true and prune: true
alerts on every OutOfSync event
and crucially, investigate why it drifted rather than just letting the heal paper over it

Production (critical or special)

selfHeal: false, human review required
prune: false, manual deletion only
strict alerts and an explicit sync approval before anything changes

HPA-managed apps

ignoreDifferences:
  - group: apps
    kind: Deployment
    jsonPointers:
      - /spec/replicas
syncPolicy:
  automated:
    selfHeal: true  # For other fields

When Drift Shows Up, Chase the Source

Healing the drift fixes the symptom. Finding out why it happened fixes the problem:

Check audit logs: who ran kubectl?

kubectl get events --field-selector reason=Update

Check controller logs: did an operator do it?
Check admission webhooks: is something mutating resources on admission?
```
kubectl get mutatingwebhookconfigurations
```
Check the diff: what changed, exactly?
```
argocd app diff app-name
```

Better Than Detecting: Don’t Let It Drift

Detection is the safety net. The cheaper win is making most drift impossible in the first place:

RBAC restrictions: limit who can change resources at all

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: readonly
rules:
  - apiGroups: ["*"]
    resources: ["*"]
    verbs: ["get", "list", "watch"]  # No create/update/delete

Policy enforcement: use Kyverno to block manual changes outright

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-gitops
spec:
  rules:
    - name: block-manual-changes
      match:
        resources:
          kinds:
            - Deployment
      exclude:
        subjects:
          - kind: ServiceAccount
            name: argocd-application-controller
      validate:
        message: "Changes must go through GitOps"
        deny: {}

Training: get the team into the habit of changing Git, not the cluster. Tooling helps, but a shared norm helps more.

Track Drift as a Metric

If you run Prometheus, treat drift as something you measure over time, not just an instantaneous status:

# Prometheus query for out-of-sync apps
count(argocd_app_info{sync_status="OutOfSync"})

And alert when it stays non-zero for longer than you’d expect a sync to take:

- alert: GitOpsDriftDetected
  expr: count(argocd_app_info{sync_status="OutOfSync"}) > 0
  for: 10m
  labels:
    severity: warning
  annotations:
    summary: "GitOps drift detected"
    description: "One or more applications are OutOfSync with Git"

My Checklist for Drift-Free GitOps

[ ] selfHeal enabled for most applications
[ ] ignoreDifferences configured for HPA-managed replicas
[ ] Notifications set up for OutOfSync events
[ ] RBAC restricts direct cluster modifications
[ ] Policy enforcement prevents manual changes
[ ] Monitoring alerts on drift
[ ] Team trained on GitOps workflow

Look back at where we started: a dashboard with a green or red light next to each app. Everything since then is just making that signal trustworthy and acting on it before it costs you. Drift is the quiet erosion of the one guarantee GitOps gives you, which is that Git reflects reality. Detect it, heal it automatically where that’s safe, and dig into why it happened the rest of the time. That last part is what keeps the cluster from becoming a black box you built yourself.

What Drift Actually Is#

Why I Care About This#

The Simplest Version: Sync Status#

Layer 1: Self-Heal, and Why It’s a Double-Edged Tool#

Layer 1b: Telling ArgoCD What to Ignore#

Layer 2: Detect Without Auto-Fixing#

Layer 2b: Get Notified Instead of Watching#

Reading the Diff#

One Thing Worth Getting Straight: Refresh vs Sync#

The Full Picture: My Drift Strategy#

Development / Staging#

Production (most apps)#

Production (critical or special)#

HPA-managed apps#

When Drift Shows Up, Chase the Source#

Better Than Detecting: Don’t Let It Drift#

Track Drift as a Metric#

My Checklist for Drift-Free GitOps#