Resilience

In my previous post on Prometheus and Thanos, I covered the sidecar architecture — Thanos Sidecar runs alongside Prometheus, uploads TSDB blocks to object storage, and exposes data to the Querier. It works beautifully for clusters with stable connectivity to your central infrastructure. But what happens when your clusters are at the edge? When they might lose connectivity for hours or days? When you’re running dozens or hundreds of small clusters and don’t want sidecar complexity on each one? ...

When everything has cluster-admin, nothing is secure. Kubernetes RBAC (Role-Based Access Control) exists to answer one question: who can do what to which resources? Most clusters answer incorrectly: “everyone can do everything.” This isn’t just a security problem — it’s a resilience problem. When a service account gets compromised, how much damage can it do? When someone runs the wrong command, what’s the blast radius? Least privilege limits that radius. ...

Every deployment is a risk. The question isn’t whether something will go wrong — it’s how much damage it will cause when it does. Traditional Kubernetes deployments are all-or-nothing. You push a new version, and within seconds, 100% of your traffic hits the new code. If there’s a bug, everyone sees it. If the service crashes, all users are affected. Progressive delivery changes this equation. Instead of deploying to everyone at once, you gradually shift traffic to the new version, validating at each step. If something goes wrong, only a fraction of users are affected. ...

Your cluster is gone. Complete failure. The cloud region is down, the hardware died, or someone ran the wrong terraform destroy. Everything is gone. Now what? If you’ve been doing GitOps right, the answer is: spin up a new cluster, point ArgoCD at Git, wait. Your entire infrastructure recreates itself. This is the ultimate promise of GitOps: Git is your backup. Why GitOps Changes Disaster Recovery Traditional DR involves: Regular backups of cluster state Backup storage (etcd snapshots, Velero backups) Tested restore procedures Recovery time measured in hours GitOps DR is different: ...