Resilience

Your homelab runs your GitLab, your passwords, your photos, your home automation. What happens when the disk fails? If you can’t answer that question confidently, you don’t have backups. You have hope. The 3-2-1 rule has been around for decades because it works. Three copies, two different media, one offsite. Here’s how to actually implement it. The 3-2-1 Rule Explained flowchart TD subgraph rule["3-2-1 Backup Rule"] Data["Original Data"] subgraph three["3 Copies"] C1["Copy 1<br/>(Original)"] C2["Copy 2<br/>(Local Backup)"] C3["Copy 3<br/>(Offsite)"] end subgraph two["2 Media Types"] M1["NVMe/SSD"] M2["HDD/NAS"] end subgraph one["1 Offsite"] Off["Cloud/Remote"] end end Data --> C1 Data --> C2 Data --> C3 C1 --> M1 C2 --> M2 C3 --> Off Why Three Copies? Copy 1: Your live data (original) Copy 2: Local backup (fast restore) Copy 3: Offsite backup (disaster recovery) One copy is not a backup. Two copies can both fail in the same disaster (fire, flood, ransomware). Three copies with separation gives you real resilience. ...

Your cluster looks healthy. Pods are running. Metrics are green. Everything works. Until a node fails during peak traffic. Or the database connection pool exhausts. Or that one service nobody remembers deploying starts consuming all available memory. You can wait for these things to happen in production at 3 AM. Or you can break things intentionally, on your terms, and fix the weaknesses before they become outages. This is chaos engineering. ...

In my previous post on Prometheus and Thanos, I covered the sidecar architecture — Thanos Sidecar runs alongside Prometheus, uploads TSDB blocks to object storage, and exposes data to the Querier. It works beautifully for clusters with stable connectivity to your central infrastructure. But what happens when your clusters are at the edge? When they might lose connectivity for hours or days? When you’re running dozens or hundreds of small clusters and don’t want sidecar complexity on each one? ...

When everything has cluster-admin, nothing is secure. Kubernetes RBAC (Role-Based Access Control) exists to answer one question: who can do what to which resources? Most clusters answer incorrectly: “everyone can do everything.” This isn’t just a security problem — it’s a resilience problem. When a service account gets compromised, how much damage can it do? When someone runs the wrong command, what’s the blast radius? Least privilege limits that radius. ...

Every deployment is a risk. The question isn’t whether something will go wrong — it’s how much damage it will cause when it does. Traditional Kubernetes deployments are all-or-nothing. You push a new version, and within seconds, 100% of your traffic hits the new code. If there’s a bug, everyone sees it. If the service crashes, all users are affected. Progressive delivery changes this equation. Instead of deploying to everyone at once, you gradually shift traffic to the new version, validating at each step. If something goes wrong, only a fraction of users are affected. ...