Intermediate

Your homelab runs your GitLab, your passwords, your photos, your home automation. What happens when the disk fails? If you can’t answer that question confidently, you don’t have backups. You have hope. The 3-2-1 rule has been around for decades because it works. Three copies, two different media, one offsite. Here’s how to actually implement it. The 3-2-1 Rule Explained flowchart TD subgraph rule["3-2-1 Backup Rule"] Data["Original Data"] subgraph three["3 Copies"] C1["Copy 1<br/>(Original)"] C2["Copy 2<br/>(Local Backup)"] C3["Copy 3<br/>(Offsite)"] end subgraph two["2 Media Types"] M1["NVMe/SSD"] M2["HDD/NAS"] end subgraph one["1 Offsite"] Off["Cloud/Remote"] end end Data --> C1 Data --> C2 Data --> C3 C1 --> M1 C2 --> M2 C3 --> Off Why Three Copies? Copy 1: Your live data (original) Copy 2: Local backup (fast restore) Copy 3: Offsite backup (disaster recovery) One copy is not a backup. Two copies can both fail in the same disaster (fire, flood, ransomware). Three copies with separation gives you real resilience. ...

You have Grafana. You have Prometheus metrics. You have logs in Loki and traces in Tempo. You also have 47 dashboards that nobody looks at. Dashboard rot is real. Teams create dashboards for every possible metric, every service, every potential issue. Six months later, nobody remembers what half of them show or why they exist. Good dashboards are different. They get opened daily. They answer questions before you ask. They help you understand your system, not just display numbers. ...

You don’t need a cloud provider to run Kubernetes. You don’t need expensive servers. You need three mini-PCs and an afternoon. This is how I built my homelab cluster — the same cluster that runs my GitLab, monitoring, home automation, and everything else I refuse to trust to someone else’s computer. Why K3s? K3s is Kubernetes, simplified: Single binary — ~70MB, includes everything Low resource — Runs on Raspberry Pi, runs great on mini-PCs Production ready — Same API, same workloads, less overhead Batteries included — Built-in ingress, load balancer, storage It’s not “Kubernetes lite.” It’s Kubernetes without the enterprise cruft. ...

Manual certificate management is a recipe for outages. Certificates expire at 3 AM on a holiday weekend. Renewal processes live in tribal knowledge. Teams deploy services without HTTPS because “it’s too complicated.” cert-manager automates everything. Define what certificates you need, and cert-manager handles issuance, renewal, and Kubernetes Secret management. Forever. This is one of the first things I install in every cluster. How cert-manager Works flowchart TD subgraph cluster["Kubernetes Cluster"] CM["cert-manager"] CERT["Certificate<br/>Resource"] SECRET["TLS Secret"] INGRESS["Ingress"] end subgraph external["External"] LE["Let's Encrypt<br/>ACME Server"] DNS["DNS Provider"] end CERT -->|"watches"| CM CM -->|"creates"| SECRET CM <-->|"ACME protocol"| LE CM <-->|"DNS challenge"| DNS SECRET -->|"mounts"| INGRESS You create a Certificate resource cert-manager requests a certificate from the issuer (Let’s Encrypt, Vault, etc.) cert-manager completes the challenge (HTTP-01 or DNS-01) cert-manager stores the certificate in a Kubernetes Secret Your Ingress/Gateway uses the Secret for TLS Renewal happens automatically 30 days before expiration. ...

You have metrics telling you something is slow. You have logs telling you errors happened. But which request failed? Where did the latency come from? Which service in the chain caused the timeout? This is where distributed tracing comes in. It follows individual requests as they flow through your microservices, showing you exactly what happened and where. The Observability Triangle flowchart TD subgraph observability["Complete Observability"] M["Metrics<br/>(Prometheus/Thanos)<br/>WHAT is happening"] L["Logs<br/>(Loki)<br/>WHY it happened"] T["Traces<br/>(Tempo)<br/>WHERE it happened"] end M <--> L L <--> T T <--> M G["Grafana"] --> M G --> L G --> T Metrics answer: “What is the error rate? What is the latency?” Logs answer: “What error message? What was the context?” Traces answer: “Which service? Which call? What was the path?” Together, they give you complete understanding. ...