Progressive Delivery with Argo Rollouts: Canary and Blue-Green Deployments

A standard Kubernetes Deployment had served me well for a long time. Push a new image tag, watch the pods roll, done. It was simple, it was declarative, and most of the time nothing went wrong. The rolling update even gave me a warm feeling of safety: old pods only get torn down once new ones are ready.

That feeling is a lie. A rolling update protects you from pods that fail to start. It does nothing to protect you from pods that start perfectly and then serve broken responses. The container is healthy, the readiness probe is green, and your new code is quietly returning 500s to every single user. Within seconds, 100% of your traffic is hitting code that nobody validated under real load.

So the question I kept coming back to: how do you ship a new version without betting all of your users on it being correct? I want to understand what a release is actually doing before it touches everyone, and I want the system to pull the plug on its own when I’m not watching.

Argo Rollouts is my answer. It’s a drop-in replacement for a Deployment that shifts traffic to the new version in steps, checks metrics along the way, and rolls back automatically when the numbers go bad. Same declarative spirit, far smaller blast radius.

Why Progressive Delivery?

Look at what a standard Deployment costs you when a bug ships:

Time 0:00 - Deploy new version
Time 0:02 - All pods running new version
Time 0:05 - Errors start appearing
Time 0:08 - Alerts fire
Time 0:15 - Engineer investigates
Time 0:25 - Rollback initiated
Time 0:27 - All pods back to old version

Blast radius: 100% of users for ~25 minutes

With progressive delivery:

Time 0:00 - Deploy new version (5% traffic)
Time 0:05 - Automated analysis detects errors
Time 0:06 - Automatic rollback

Blast radius: 5% of users for ~6 minutes

That gap is the whole point. I’m not trying to prevent failures, because I can’t. Bugs ship no matter how good the tests are. What I can do is make sure that when one slips through, it hits five people for six minutes instead of everyone for half an hour. That’s resilience by design: assume things break, and limit how much they can hurt.

Two Strategies: Canary vs Blue-Green

Argo Rollouts supports a few strategies. Two cover almost everything I do.

Canary

Traffic shifts gradually from old to new version:

Step 1:  5% new, 95% old   (test the waters)
Step 2: 20% new, 80% old   (expand if healthy)
Step 3: 50% new, 50% old   (halfway point)
Step 4: 100% new, 0% old   (full rollout)

Best for stateless services and anything with enough traffic that a small slice gives you real signal. This is what I reach for by default.

Blue-Green

Two complete environments, instant switch:

Before:  100% blue (old)    0% green (new)
Deploy:  100% blue          green ready, not receiving traffic
Switch:    0% blue        100% green

Best for services that need an instant flip back, for testing the new version against production data before it goes live, and for cases where you want both versions up at the same time to compare them. The cost is double the resources during the switch, since you run two full copies.

Installing Argo Rollouts

kubectl create namespace argo-rollouts
kubectl apply -n argo-rollouts -f https://github.com/argoproj/argo-rollouts/releases/latest/download/install.yaml

That works, but applying a release manifest by hand is the kind of thing I forget to do consistently. For GitOps with ArgoCD, let the controller manage it as an Application instead:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: argo-rollouts
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://argoproj.github.io/argo-helm
    chart: argo-rollouts
    targetRevision: 2.35.1
    helm:
      values: |
        dashboard:
          enabled: true
  destination:
    server: https://kubernetes.default.svc
    namespace: argo-rollouts
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Canary Rollout Example

The nice part is how little changes. A Rollout looks almost identical to a Deployment. Same replicas, same selector, same pod template. You swap the kind and add a strategy block:

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: my-app
spec:
  replicas: 10
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
        - name: app
          image: my-app:v2.0.0
          ports:
            - containerPort: 8080
  strategy:
    canary:
      steps:
        - setWeight: 5
        - pause: { duration: 5m }
        - setWeight: 20
        - pause: { duration: 5m }
        - setWeight: 50
        - pause: { duration: 5m }
        - setWeight: 100

This creates a 20-minute gradual rollout:

Send 5% of traffic to new version, wait 5 minutes
If healthy, increase to 20%, wait 5 minutes
If healthy, increase to 50%, wait 5 minutes
Complete rollout to 100%

If you abort at any point, traffic snaps back to the old version. The old ReplicaSet never went away, so there’s nothing to rebuild.

Traffic Management

Here’s a detail that trips people up. By default, Argo Rollouts approximates the traffic weight using replica count. Ask for 5% with 10 replicas and it can’t give you 5%, it gives you the nearest it can manage with whole pods. For most apps that approximation is fine. When you need the weight to actually mean 5%, you wire the Rollout into your ingress so the proxy splits traffic precisely.

I run Traefik, so that’s the one I use in production. The Nginx and Istio configs are here too for completeness.

With Traefik

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: my-app
spec:
  strategy:
    canary:
      canaryService: my-app-canary
      stableService: my-app-stable
      trafficRouting:
        traefik:
          weightedTraefikServiceName: my-app-weighted
      steps:
        - setWeight: 10
        - pause: { duration: 2m }
        - setWeight: 50
        - pause: { duration: 2m }

With supporting TraefikService:

apiVersion: traefik.io/v1alpha1
kind: TraefikService
metadata:
  name: my-app-weighted
spec:
  weighted:
    services:
      - name: my-app-stable
        port: 80
        weight: 100  # Managed by Argo Rollouts
      - name: my-app-canary
        port: 80
        weight: 0    # Managed by Argo Rollouts

With Nginx Ingress

apiVersion: argoproj.io/v1alpha1
kind: Rollout
spec:
  strategy:
    canary:
      canaryService: my-app-canary
      stableService: my-app-stable
      trafficRouting:
        nginx:
          stableIngress: my-app-ingress

With Istio

apiVersion: argoproj.io/v1alpha1
kind: Rollout
spec:
  strategy:
    canary:
      trafficRouting:
        istio:
          virtualService:
            name: my-app-vsvc
            routes:
              - primary

Blue-Green Rollout

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: my-app
spec:
  replicas: 5
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
        - name: app
          image: my-app:v2.0.0
  strategy:
    blueGreen:
      activeService: my-app-active
      previewService: my-app-preview
      autoPromotionEnabled: false
      prePromotionAnalysis:
        templates:
          - templateName: smoke-tests
      postPromotionAnalysis:
        templates:
          - templateName: load-test

This creates:

my-app-active: Points to current production version
my-app-preview: Points to new version for testing

The new version runs in full but sees no production traffic until you promote it. With autoPromotionEnabled: false, that promotion is a deliberate decision you make after poking at the preview service yourself.

Automated Analysis

Everything so far still leans on a human watching a dashboard and deciding when to continue. That’s where I lose trust in the process, because I’m not always watching, and at 2am I’m definitely not. The part of Argo Rollouts I actually care about is automated analysis: the controller queries your metrics during the rollout and aborts on its own when they go bad.

Analysis Template

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: success-rate
spec:
  args:
    - name: service-name
  metrics:
    - name: success-rate
      interval: 1m
      successCondition: result[0] >= 0.95
      failureLimit: 3
      provider:
        prometheus:
          address: http://prometheus.monitoring:9090
          query: |
            sum(rate(http_requests_total{service="{{args.service-name}}",status=~"2.."}[5m])) /
            sum(rate(http_requests_total{service="{{args.service-name}}"}[5m]))

This template queries Prometheus every minute for the HTTP success rate, treats anything at or above 95% as healthy, and fails the rollout after 3 consecutive bad readings. The failureLimit matters: a single scrape blip shouldn’t kill a deploy, but three in a row is a real signal.

Using Analysis in Rollout

apiVersion: argoproj.io/v1alpha1
kind: Rollout
spec:
  strategy:
    canary:
      steps:
        - setWeight: 5
        - pause: { duration: 2m }
        - analysis:
            templates:
              - templateName: success-rate
            args:
              - name: service-name
                value: my-app
        - setWeight: 50
        - pause: { duration: 5m }
        - analysis:
            templates:
              - templateName: success-rate
            args:
              - name: service-name
                value: my-app

Now the rollout shifts 5% of traffic, waits two minutes, then runs the analysis. If success rate is below 95% for three readings, it rolls back without me touching anything. If it passes, traffic goes to 50% and the same check runs again before the full rollout completes. The deploy now defends itself.

Multiple Analysis Metrics

Success rate alone misses things. A version can return 200s while taking three seconds to do it, which is its own kind of broken. So I check several signals at once, and any one of them failing aborts the rollout:

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: comprehensive-check
spec:
  args:
    - name: service-name
  metrics:
    # HTTP success rate
    - name: success-rate
      interval: 1m
      successCondition: result[0] >= 0.95
      failureLimit: 3
      provider:
        prometheus:
          address: http://prometheus.monitoring:9090
          query: |
            sum(rate(http_requests_total{service="{{args.service-name}}",status=~"2.."}[5m])) /
            sum(rate(http_requests_total{service="{{args.service-name}}"}[5m]))

    # P99 latency
    - name: latency-p99
      interval: 1m
      successCondition: result[0] < 0.5
      failureLimit: 3
      provider:
        prometheus:
          address: http://prometheus.monitoring:9090
          query: |
            histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket{service="{{args.service-name}}"}[5m])) by (le))

    # Error rate
    - name: error-rate
      interval: 1m
      successCondition: result[0] < 0.01
      failureLimit: 2
      provider:
        prometheus:
          address: http://prometheus.monitoring:9090
          query: |
            sum(rate(http_requests_total{service="{{args.service-name}}",status=~"5.."}[5m])) /
            sum(rate(http_requests_total{service="{{args.service-name}}"}[5m]))

Web-Based Job Analysis

Not every check fits a Prometheus query. Sometimes you just want to hit a couple of endpoints and confirm they answer. Argo Rollouts can run a Kubernetes Job as an analysis step, so a failed curl aborts the rollout the same way a bad metric does:

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: smoke-test
spec:
  metrics:
    - name: smoke-test
      provider:
        job:
          spec:
            backoffLimit: 1
            template:
              spec:
                containers:
                  - name: smoke
                    image: curlimages/curl
                    command:
                      - /bin/sh
                      - -c
                      - |
                        curl -f http://my-app-canary/health || exit 1
                        curl -f http://my-app-canary/api/status || exit 1
                restartPolicy: Never

Dashboard and CLI

Monitor rollouts with the Argo Rollouts kubectl plugin:

# Install plugin
curl -LO https://github.com/argoproj/argo-rollouts/releases/latest/download/kubectl-argo-rollouts-darwin-amd64
chmod +x kubectl-argo-rollouts-darwin-amd64
mv kubectl-argo-rollouts-darwin-amd64 /usr/local/bin/kubectl-argo-rollouts

# Watch rollout progress
kubectl argo rollouts get rollout my-app -w

# Manually promote (if autoPromotion is disabled)
kubectl argo rollouts promote my-app

# Abort and rollback
kubectl argo rollouts abort my-app

# View dashboard
kubectl argo rollouts dashboard

The -w watch view is what I keep open during a release. It shows the current weight, which analysis runs are passing, and how far along each step is, all in the terminal where I already live. The web dashboard does the same thing with graphs if you prefer that.

Integration with GitOps

This is where it clicks into the rest of my workflow. I don’t run any of those kubectl commands during a normal deploy. The image tag lives in Git, ArgoCD syncs it, and the progressive rollout starts on its own.

# In your GitOps repo
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: my-app
spec:
  template:
    spec:
      containers:
        - name: app
          image: my-app:v2.1.0  # Update this line

The semantic versioning pipeline creates the tag, that bumps the image in the GitOps repo, ArgoCD syncs the change, and the Rollout takes it from there. Git is the only thing I touch, and the controller does the careful part.

Notifications

Since I’m deliberately not watching every rollout, I want a ping when one finishes or aborts:

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: my-app
  annotations:
    notifications.argoproj.io/subscribe.on-rollout-completed.slack: my-channel
    notifications.argoproj.io/subscribe.on-rollout-aborted.slack: my-channel

You configure the notification controller separately, and it shares its config with ArgoCD notifications if you already run those.

My Production Setup

Here’s the actual Rollout I run for my API, pulling together everything above:

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: api
spec:
  replicas: 5
  revisionHistoryLimit: 3
  selector:
    matchLabels:
      app: api
  template:
    metadata:
      labels:
        app: api
    spec:
      containers:
        - name: api
          image: registry.example.com/api:v1.0.0
          ports:
            - containerPort: 8080
          readinessProbe:
            httpGet:
              path: /health
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 5
          resources:
            requests:
              cpu: 100m
              memory: 128Mi
            limits:
              cpu: 500m
              memory: 512Mi
  strategy:
    canary:
      canaryService: api-canary
      stableService: api-stable
      trafficRouting:
        traefik:
          weightedTraefikServiceName: api-weighted
      steps:
        # Phase 1: Smoke test
        - setWeight: 5
        - pause: { duration: 2m }
        - analysis:
            templates:
              - templateName: smoke-test

        # Phase 2: Limited exposure
        - setWeight: 25
        - pause: { duration: 5m }
        - analysis:
            templates:
              - templateName: success-rate
            args:
              - name: service-name
                value: api

        # Phase 3: Majority traffic
        - setWeight: 75
        - pause: { duration: 10m }
        - analysis:
            templates:
              - templateName: success-rate
            args:
              - name: service-name
                value: api

        # Phase 4: Full rollout
        - setWeight: 100
      rollbackWindow:
        revisions: 2

A few choices in there are deliberate. The smoke test runs first at 5%, so an obviously broken build dies in two minutes before it reaches real users. The pauses get longer as the weight climbs, because more traffic means I want more signal before trusting the next jump. And rollbackWindow keeps the last two revisions warm so a revert is instant, not a rebuild.

When Not to Use Progressive Delivery

I should be honest about the cost. Progressive delivery adds moving parts: extra services, traffic routing config, analysis templates, metrics that have to actually exist. That complexity buys you nothing in a few cases, and forcing it in anyway just adds friction:

Breaking database schema changes. If the new code can’t coexist with the old schema, running both versions at once is a bug, not a safety feature. You need the whole app on one version.
Single-user or internal tools. There’s no meaningful traffic to split, so a canary tells you nothing. A plain Deployment is the honest choice.
Tightly coupled services that must upgrade in lockstep. Shifting one of them gradually breaks the contract with the others.

For a production service handling real users over HTTP, though, the complexity earns its keep on the first bad deploy it catches.

Why This Matters

Every deploy is a bet that your new code behaves in production. Tests narrow the odds, but they don’t settle the bet, because the staging environment is never quite the real one.

What progressive delivery changes is the size of the wager. A bad release reaches a slice of users for a couple of minutes, the controller spots it in the metrics, and traffic flows back to the version you know works, all without waking anyone up. I get to understand what a release is doing while it’s happening, and the system protects itself when I’m not looking. For me that’s the whole reason to run my own platform: not chasing deploys that never fail, but building one that fails small and recovers on its own.

Why Progressive Delivery?#

Two Strategies: Canary vs Blue-Green#

Canary#

Blue-Green#

Installing Argo Rollouts#

Canary Rollout Example#

Traffic Management#

With Traefik#

With Nginx Ingress#

With Istio#

Blue-Green Rollout#

Automated Analysis#

Analysis Template#

Using Analysis in Rollout#

Multiple Analysis Metrics#

Web-Based Job Analysis#

Dashboard and CLI#

Integration with GitOps#

Notifications#

My Production Setup#

When Not to Use Progressive Delivery#

Why This Matters#