Resilience on Tom Meurs

Backup Strategy for Your Homelab: The 3-2-1 Rule in Practice

Mon, 18 May 2026 19:30:00 +0100

TL;DR Three copies, two kinds of media, one of them offsite. Restic for files, Velero for the cluster, a Backblaze bucket for the offsite copy, and a restore test on a schedule. Mine costs about €5 a month, and the restore test is the part I still have to make myself do.

Picture the disk in your homelab dying tonight. Not corrupting, not throwing SMART warnings for a week first, just gone. The git server with every repo you care about, the password vault, fifteen years of family photos, the home automation that runs the house. All of it on a drive that’s decided it’s done.

Chaos Engineering: Breaking Your Cluster to Make It Stronger

Tue, 28 Apr 2026 19:45:00 +0100

TL;DR Chaos engineering is breaking your own cluster on purpose, with the hypothesis written down first and a rollback ready. I run it on the homelab with Litmus: a weekly pod-delete, a monthly node drain, a game day every quarter. So far it has found four things the dashboard would never have shown me.

My dashboard is a wall of green. Pods running, replicas matched, CPU comfortable, no alerts firing. I look at it and get that small hit of “everything’s fine.” And mostly it is. The cluster’s been up for weeks and nothing has fallen over.

Alerting That Works: From Alert Fatigue to Actionable Notifications

Thu, 16 Apr 2026 20:15:00 +0100

TL;DR Most default alert rules page on a number with no user behind it, and after a week of that you stop reading pages. I alert on error rate, P99 latency, a disk that fills within 24 hours and a certificate under seven days, about ten rules in total, each with a runbook. Alertmanager sends critical to the pager, warning to Slack during office hours, info nowhere. Everything that doesn’t need a human gets deleted.

Thanos Remote Write: Push-Based Metrics for Edge and Multi-Cluster

Fri, 27 Mar 2026 20:38:00 +0100

TL;DR Thanos Receive turns the metrics flow around. Each edge Prometheus pushes samples over remote_write to one central endpoint, buffers in its WAL when the link drops, and never sees an object storage credential. Below is the Receive StatefulSet, the hashring, the edge config, two ways to lock the endpoint down, and the hybrid I run with sidecars on the core clusters.

In my previous post on Prometheus and Thanos, I set up the sidecar architecture. Thanos Sidecar runs next to Prometheus, uploads TSDB blocks to object storage, and exposes the data to the Querier over gRPC. For clusters in the same datacenter with a fat, stable link to the central infrastructure, I’ve got no complaints. Everything pulls, and everything can reach everything else.

Unbreakable - My Fascination.

Tue, 23 Dec 2025 20:00:00 +0100

As a kid I called the things that fascinated me unbreakable. Indestructible never fit, because indestructible means it never breaks. Unbreakable means it keeps working after it breaks.

I know exactly when it started. A photo of an A-10 Thunderbolt II back from a mission with half a wing gone, the tail in tatters and the fuselage full of holes. It still brought its pilot home.

Luck had nothing to do with it. Somebody sat at a drawing board years before that mission and decided the plane had to come home looking like that.

Kubernetes RBAC: Least Privilege in Practice

Tue, 19 Aug 2025 10:00:00 +0200

TL;DR Kubernetes ships with a permission model that hands every pod a token and lets the default service account collect rights nobody remembers granting. Least privilege means a service account per workload, no token unless the app talks to the API, and roles that name the namespace, the resource and the verb. Below are the four RBAC resources, the five settings that cap the blast radius, the bindings I copy most often, and how I check and enforce all of it.

Progressive Delivery with Argo Rollouts: Canary and Blue-Green Deployments

Fri, 20 Jun 2025 10:00:00 +0200

A standard Kubernetes Deployment had served me well for a long time. Push a new image tag, watch the pods roll, done. It was simple, it was declarative, and most of the time nothing went wrong. The rolling update even gave me a warm feeling of safety: old pods only get torn down once new ones are ready.

That feeling is a lie. A rolling update protects you from pods that fail to start. It does nothing to protect you from pods that start perfectly and then serve broken responses. The container is healthy, the readiness probe is green, and your new code is quietly returning 500s to every single user. Within seconds, 100% of your traffic is hitting code that nobody validated under real load.

GitOps Disaster Recovery: Rebuilding Your Cluster from Git

Wed, 09 Apr 2025 19:52:00 +0200

For a couple of years my homelab cluster ran without much drama. Nodes came and went, workloads shifted around, and the worst I ever had to deal with was the occasional pod stuck in CrashLoopBackOff. The kind of stable where you stop thinking about what happens if it all goes away.

Then one evening I ran a terraform destroy against the wrong workspace. By the time I noticed, the control plane was gone. Not degraded. Gone.

Graceful Degradation in Kubernetes: What Happens When Components Fail

Thu, 20 Feb 2025 14:22:00 +0100

Everyone repeats the line that Kubernetes is self-healing. Pods die, they come back. Nodes drop, workloads reschedule. The system reconciles itself toward the state you declared, and most days you never have to think about it.

Then one day the thing doing the healing is the thing that broke. The API server is down. etcd won’t respond. The scheduler is wedged. Now what?

This is the question I actually care about, because “self-healing” is only useful if I understand its edges. I want to know what degrades gracefully and what takes the whole cluster with it. So I’ve run my clusters through a lot of failures on purpose: planned, unplanned, and a few “hold my beer” experiments on hardware I didn’t mind losing. Here is what actually happens when each piece breaks, and why most of it matters less than people fear.