Observability on Tom Meurs

Grafana Dashboards That Actually Get Used

Sat, 02 May 2026 20:15:00 +0100

TL;DR Each dashboard answers one question, and within five seconds I know whether I need to care. Four kinds of dashboard cover everything, variables and links tie them together, and the JSON lives in Git. My homelab runs five, and anything that goes a month unopened gets deleted.

You have Grafana. You have Prometheus metrics. You have logs in Loki and traces in Tempo. The data’s all there.

You also have 47 dashboards nobody opens.

Alerting That Works: From Alert Fatigue to Actionable Notifications

Thu, 16 Apr 2026 20:15:00 +0100

TL;DR Most default alert rules page on a number with no user behind it, and after a week of that you stop reading pages. I alert on error rate, P99 latency, a disk that fills within 24 hours and a certificate under seven days, about ten rules in total, each with a runbook. Alertmanager sends critical to the pager, warning to Slack during office hours, info nowhere. Everything that doesn’t need a human gets deleted.

Distributed Tracing with Tempo and OpenTelemetry

Sat, 04 Apr 2026 19:47:00 +0100

TL;DR Tempo is where my traces go. The OpenTelemetry Collector receives spans from every service, tail sampling keeps every error and every slow request, and Tempo writes the rest to MinIO with three days of retention. From a span in Grafana I click through to the Loki lines with the same trace ID. My production values are at the bottom.

Your metrics say something is slow. Your logs say errors happened. Fine. Which request failed, where did the latency come from, and which service in the chain ate the timeout? Neither of them can tell you.

Loki for Kubernetes Logging: The Prometheus-Like Approach

Tue, 31 Mar 2026 21:15:00 +0100

TL;DR Loki indexes labels and stores the log text as compressed chunks, so a month of logs fits on commodity hardware. The trade is full-text search. Below is the single-binary install I’d hand anyone first, the same chart on MinIO with three replicas, and the knobs I didn’t touch until the volume was real.

You’ve got Prometheus for metrics, so you can already see what’s happening across your clusters. Metrics tell you request latency spiked at 14:32. They don’t tell you the payment service threw a null pointer because someone shipped a config change with a typo. For that you need the log line.

Thanos Remote Write: Push-Based Metrics for Edge and Multi-Cluster

Fri, 27 Mar 2026 20:38:00 +0100

TL;DR Thanos Receive turns the metrics flow around. Each edge Prometheus pushes samples over remote_write to one central endpoint, buffers in its WAL when the link drops, and never sees an object storage credential. Below is the Receive StatefulSet, the hashring, the edge config, two ways to lock the endpoint down, and the hybrid I run with sidecars on the core clusters.

In my previous post on Prometheus and Thanos, I set up the sidecar architecture. Thanos Sidecar runs next to Prometheus, uploads TSDB blocks to object storage, and exposes the data to the Querier over gRPC. For clusters in the same datacenter with a fat, stable link to the central infrastructure, I’ve got no complaints. Everything pulls, and everything can reach everything else.

Prometheus and Thanos: Metrics at Scale

Sun, 31 Aug 2025 10:00:00 +0200

TL;DR Prometheus on its own keeps a couple of weeks of metrics on local disk and can’t answer a question across clusters. Thanos puts a sidecar next to each Prometheus, ships the blocks to object storage, and puts a querier in front that reads all of it back through the same PromQL. Below is the install, the four components, the HA and multi-cluster wiring, and the numbers I run in production.