Kubernetes

For a long time my certificates renewed the way most people’s do: a calendar reminder, a manual certbot run, and a quiet hope that I’d remember before the thing actually expired. It worked. It worked right up until the morning a service threw cert errors at me and I had no idea why, because the renewal cron had been silently failing for weeks. That’s the part nobody tells you about manual TLS. The failure doesn’t announce itself. The cert just expires, usually at the worst possible moment, and you find out because a browser is yelling at someone. Renewal knowledge ends up living in one person’s head. Teams skip HTTPS on internal services because wiring it up by hand is annoying enough to put off. ...

The first time a service stopped resolving in one of my clusters, I spent an evening reading iptables chains. Hundreds of rules, generated by kube-proxy, evaluated top to bottom. I never found the actual problem. I restarted a node and it went away. That bothered me more than the outage did. I was running something I couldn’t read. That feeling is why I moved to Cilium. It uses eBPF to push networking logic down into the Linux kernel and skips iptables entirely. You get better performance, you can actually see what your traffic is doing, and network policies stop being a guessing game. ...

Your metrics say something is slow. Your logs say errors happened. Great. Now answer me this: which request actually failed, where did the latency come from, and which service in the chain ate the timeout? Metrics and logs both shrug at that. I hit this wall the first time a checkout flow started timing out under load. Ten services in the path, every one of them green on its own dashboard, and no way to follow a single doomed request from front door to failure. That gap is exactly what distributed tracing fills. It follows one request as it moves through your services and shows you precisely what happened and where it stalled. ...

You’ve got Prometheus for metrics, so you can already see what’s happening across your clusters. Metrics tell you a request latency spiked at 14:32. They don’t tell you the payment service threw a null pointer because someone shipped a config change with a typo. For that you need logs. The default answer for years was Elasticsearch. It’s powerful and flexible, and it indexes every single token in every log line. That full-text index is great until you look at the bill. You pay for it in CPU at ingest, in RAM to keep the index hot, and in storage that grows faster than your actual log volume. I ran an ELK stack in a previous job and spent more time tuning JVM heap sizes than reading logs. ...

In my previous post on Prometheus and Thanos, I set up the sidecar architecture. Thanos Sidecar runs next to Prometheus, uploads TSDB blocks to object storage, and exposes data to the Querier over gRPC. For clusters sitting in the same datacenter with a fat, stable link to your central infrastructure, it’s lovely. Everything pulls. Everything talks to everything. Life is good. Then I started putting Prometheus on clusters at the edge, and life got less good. ...