TL;DR Prometheus on its own keeps a couple of weeks of metrics on local disk and can’t answer a question across clusters. Thanos puts a sidecar next to each Prometheus, ships the blocks to object storage, and puts a querier in front that reads all of it back through the same PromQL. Below is the install, the four components, the HA and multi-cluster wiring, and the numbers I run in production.
The first time someone asked me “was this slower last month than it is now?”, I had no answer. My Prometheus remembered two weeks. The data I needed had aged out of local disk and was gone. That gap is why this post exists.
Prometheus is the default for Kubernetes metrics, and I’d pick it again. It works right up to the point where you need long-term storage, or a view across more than one cluster, or high availability that survives a node. Then it stops, and it stops hard.
Thanos extends Prometheus instead of replacing it. You keep the setup you already understand, bolt a few components onto it, and your retention is limited by your bucket instead of your disk, with one query across every cluster thrown in. That “keep what you know” part is what sold me. I’d rather grow a system I can already reason about than swap it for something that promises to do everything and explains none of it.
Where a single Prometheus stops
A single Prometheus has four hard limits. It’s one node, with no clustering or HA built in. It writes to local disk, so retention is whatever fits. It sees one cluster, and there’s no way to query across two. And it never downsamples, so a metric from last year costs the same space as one from this morning.
For one small cluster with two weeks of retention, none of that hurts. For a multi-cluster production setup with compliance requirements, every one of the four is a blocker. You’ll hit them. The only open question is which one first.
What Thanos adds
Thanos wraps a handful of components around the Prometheus you already have:
The sidecar runs next to Prometheus and uploads its blocks to object storage. The store gateway serves that history back out of the bucket. The querier answers PromQL by asking the sidecars and the store gateway and merging what comes back. The compactor downsamples and deduplicates whatever sits in the bucket.
Four moving parts, each with one job I can hold in my head. I trust a design like that a lot more than one binary that does everything.
Install it with the Bitnami chart
Start with the Bitnami Helm chart and get something running:
helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update
helm install thanos bitnami/thanos \
--namespace monitoring \
--create-namespace \
--set objstoreConfig="$(cat thanos-objstore.yaml)"
The object store config in thanos-objstore.yaml points Thanos at your bucket:
type: s3
config:
bucket: thanos-metrics
endpoint: minio.storage:9000
access_key: ${MINIO_ACCESS_KEY}
secret_key: ${MINIO_SECRET_KEY}
insecure: true # For MinIO without TLS
That’s the skeleton. Nothing is uploading into it yet, and that’s the next step.
Prometheus with the sidecar
Tell Prometheus to run the Thanos sidecar next to it:
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
name: prometheus
namespace: monitoring
spec:
replicas: 2 # HA pair
retention: 2h # Short local retention, Thanos handles long-term
# Thanos sidecar configuration
thanos:
baseImage: quay.io/thanos/thanos
version: v0.34.0
objectStorageConfig:
key: thanos.yaml
name: thanos-objstore-secret
# External labels for deduplication
externalLabels:
cluster: production
replica: $(POD_NAME)
# Let Thanos sidecar access Prometheus data
storage:
volumeClaimTemplate:
spec:
storageClassName: longhorn
resources:
requests:
storage: 50Gi
The sidecar does three things:
- Exposes Prometheus data to the Thanos querier over gRPC
- Uploads completed TSDB blocks to object storage
- Answers Store API queries for recent data
Look at that retention: 2h. Local disk now holds only the most recent slice. The history lives in object storage, and the sidecar keeps feeding it.
The three components that read blocks back
With the sidecar shipping blocks, you need the pieces that read them again.
Querier
apiVersion: apps/v1
kind: Deployment
metadata:
name: thanos-querier
namespace: monitoring
spec:
replicas: 2
template:
spec:
containers:
- name: thanos-query
image: quay.io/thanos/thanos:v0.34.0
args:
- query
- --http-address=0.0.0.0:9090
- --grpc-address=0.0.0.0:10901
# Connect to sidecars
- --store=dnssrv+_grpc._tcp.prometheus-operated.monitoring.svc
# Connect to store gateway
- --store=dnssrv+_grpc._tcp.thanos-store.monitoring.svc
# Deduplication
- --query.replica-label=replica
ports:
- name: http
containerPort: 9090
- name: grpc
containerPort: 10901
Store gateway
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: thanos-store
namespace: monitoring
spec:
replicas: 2
template:
spec:
containers:
- name: thanos-store
image: quay.io/thanos/thanos:v0.34.0
args:
- store
- --http-address=0.0.0.0:10902
- --grpc-address=0.0.0.0:10901
- --data-dir=/var/thanos/store
- --objstore.config-file=/etc/thanos/objstore.yaml
volumeMounts:
- name: objstore-config
mountPath: /etc/thanos
- name: data
mountPath: /var/thanos/store
volumes:
- name: objstore-config
secret:
secretName: thanos-objstore-secret
volumeClaimTemplates:
- metadata:
name: data
spec:
storageClassName: longhorn
resources:
requests:
storage: 10Gi
Compactor
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: thanos-compactor
namespace: monitoring
spec:
replicas: 1 # Only one compactor!
template:
spec:
containers:
- name: thanos-compact
image: quay.io/thanos/thanos:v0.34.0
args:
- compact
- --http-address=0.0.0.0:10902
- --data-dir=/var/thanos/compact
- --objstore.config-file=/etc/thanos/objstore.yaml
- --retention.resolution-raw=30d
- --retention.resolution-5m=90d
- --retention.resolution-1h=1y
- --wait
volumeMounts:
- name: objstore-config
mountPath: /etc/thanos
- name: data
mountPath: /var/thanos/compact
That replicas: 1 on the compactor isn’t a typo. Run two and they fight over the same blocks and corrupt your data. I put a loud comment on that line so future me doesn’t get clever with it.
The retention flags are where the cost comes down. Raw data stays 30 days at full resolution, the 5m downsampled series stay 90 days, and the 1h series stay 1 year. Older data takes less space because there’s less of it per hour. You trade resolution for history, and for anything older than a couple of weeks that’s the trade I want anyway.
GitOps deployment
I don’t click any of this together by hand. It goes through ArgoCD like everything else, so the cluster state lives in Git where I can read it, review it and roll it back:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: thanos
namespace: argocd
spec:
project: default
source:
repoURL: https://charts.bitnami.com/bitnami
chart: thanos
targetRevision: 12.20.0
helm:
values: |
objstoreConfig: |-
type: s3
config:
bucket: thanos-metrics
endpoint: minio.storage:9000
insecure: true
query:
enabled: true
replicaCount: 2
stores:
- dnssrv+_grpc._tcp.prometheus-operated.monitoring.svc
storegateway:
enabled: true
replicaCount: 2
persistence:
size: 20Gi
compactor:
enabled: true
retentionResolutionRaw: 30d
retentionResolution5m: 90d
retentionResolution1h: 1y
persistence:
size: 50Gi
ruler:
enabled: false # Use Prometheus rules instead
receive:
enabled: false # Using sidecar mode
destination:
server: https://kubernetes.default.svc
namespace: monitoring
Two Prometheus replicas, one set of graphs
Two Prometheus replicas scraping the same targets only help if something stitches their results back together. Thanos is that something.
# Run two Prometheus instances
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
spec:
replicas: 2
externalLabels:
replica: $(POD_NAME) # Different for each replica
Both instances scrape everything. The querier deduplicates on the replica label so you don’t get every line twice:
# Querier configuration
args:
- --query.replica-label=replica
- --query.replica-label=prometheus_replica
Lose a node, or just one Prometheus pod, and the dashboards keep drawing from the replica that’s left. That’s what the second one is for.
One query across every cluster
Point several clusters at the same Thanos deployment and give each one its own cluster label. The first one:
externalLabels:
cluster: production-eu
replica: $(POD_NAME)
And the second:
externalLabels:
cluster: production-us
replica: $(POD_NAME)
Now the querier aggregates across both:
# Total requests across all clusters
sum(rate(http_requests_total[5m]))
# Requests by cluster
sum by (cluster) (rate(http_requests_total[5m]))
One PromQL query, every cluster, one answer. No more three Grafana tabs open, one per cluster, adding the numbers up in your head.
Grafana doesn’t need to know
Grafana doesn’t need to know Thanos exists. Point it at the querier and it thinks it’s talking to a plain Prometheus:
apiVersion: v1
kind: ConfigMap
metadata:
name: grafana-datasources
data:
thanos.yaml: |
apiVersion: 1
datasources:
- name: Thanos
type: prometheus
url: http://thanos-query.monitoring:9090
access: proxy
isDefault: true
jsonData:
timeInterval: "15s"
Every dashboard you already built keeps working. On the Grafana side, the change is that one URL.
Recording rules for the expensive queries
Some queries are expensive enough that you don’t want to run them on every dashboard refresh. Compute them once a minute and read the result:
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: recording-rules
spec:
groups:
- name: aggregations
interval: 1m
rules:
# Pre-aggregate request rate by service
- record: service:http_requests:rate5m
expr: sum by (service) (rate(http_requests_total[5m]))
# Pre-aggregate error rate
- record: service:http_errors:rate5m
expr: sum by (service) (rate(http_requests_total{status=~"5.."}[5m]))
# Pre-compute availability
- record: service:availability:ratio
expr: |
1 - (
service:http_errors:rate5m /
service:http_requests:rate5m
)
Dashboards then query the cheap service:* series instead of grinding through raw data on every load.
Alerting stays with Prometheus
Keep alerting close to the data. Run Alertmanager against Prometheus rather than against Thanos:
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
spec:
alerting:
alertmanagers:
- namespace: monitoring
name: alertmanager
port: web
ruleSelector:
matchLabels:
role: alert-rules
Thanos Ruler exists and it works. It’s also one more component to understand and one more thing that can be broken while you’re already debugging something else. For most setups Prometheus alerting is plenty, and I’d rather have fewer parts that can surprise me.
Watching Thanos itself
Thanos exposes its own Prometheus metrics, which is just as well, because the thing watching your systems is itself a system that breaks. These are the ones I watch:
# Sidecar upload success
thanos_shipper_uploads_total
thanos_shipper_upload_failures_total
# Store gateway performance
thanos_bucket_store_series_fetch_duration_seconds
thanos_bucket_store_block_loads_total
# Compactor health
thanos_compact_group_compactions_total
thanos_compact_group_compaction_failures_total
# Querier performance
thanos_query_gate_duration_seconds
And alert when uploads start failing, because a silent sidecar means you’re losing history without anyone noticing:
- alert: ThanosSidecarUploadFailing
expr: increase(thanos_shipper_upload_failures_total[1h]) > 0
for: 15m
labels:
severity: warning
annotations:
summary: "Thanos sidecar failing to upload blocks"
What the storage costs
This is where downsampling pays for itself:
| Resolution | Data per day | 1 year cost (S3) |
|---|---|---|
| Raw (15s) | ~100MB/target | ~$4/target |
| 5m downsample | ~3MB/target | ~$0.12/target |
| 1h downsample | ~0.5MB/target | ~$0.02/target |
Keep raw data forever and the storage bill grows a little every month until somebody asks about it. Downsample, and the old data costs close to nothing.
For self-hosted object storage, MinIO does the job and keeps the data on hardware I own:
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: minio
spec:
template:
spec:
containers:
- name: minio
image: minio/minio:latest
args:
- server
- /data
- --console-address
- ":9001"
env:
- name: MINIO_ROOT_USER
valueFrom:
secretKeyRef:
name: minio-credentials
key: root-user
- name: MINIO_ROOT_PASSWORD
valueFrom:
secretKeyRef:
name: minio-credentials
key: root-password
What I run, all of it together
Put the pieces together and this is roughly what runs in production:
# Prometheus with sidecar
prometheus:
replicas: 2
retention: 6h # Very short, Thanos handles long-term
thanos:
objectStorageConfig:
name: thanos-objstore
externalLabels:
cluster: production
environment: prod
# Thanos components
thanos:
query:
replicaCount: 2
stores:
- dnssrv+_grpc._tcp.prometheus-operated.monitoring.svc
- dnssrv+_grpc._tcp.thanos-store.monitoring.svc
storegateway:
replicaCount: 2
persistence:
size: 50Gi
compactor:
retentionResolutionRaw: 14d
retentionResolution5m: 60d
retentionResolution1h: 365d
persistence:
size: 100Gi
# Object storage
minio:
replicas: 4
persistence:
size: 500Gi
Each of those numbers has a reason. Local retention is 6h because the sidecar uploads often, so there’s no point hoarding on local disk. Raw retention is 14d, which covers any debugging session I’m likely to have. The 1h series stay 1 year, enough trend history for capacity planning. And the bucket is self-hosted MinIO , so the metrics stay on my hardware, with no cloud bill and no third party holding them.
Same chart as the install at the top. The values are what changed.
What the metrics are for
Metrics are how I understand my systems instead of guessing about them. The questions are the same in every incident and every planning session:
- Is this service healthy?
- What changed right before it broke?
- Are we meeting our SLOs ?
- Where is optimization worth the effort?
Lose long-term metrics and “compared to when?” has no answer. Lose cross-cluster queries and you only ever see one piece at a time. Prometheus plus Thanos gives you retention measured in years, one view across clusters and an HA pair, behind the same Prometheus interface you already know how to read.
That question from the start of this post, “was this slower last month?”, has an answer now. One query against the querier, and the data is there because the sidecar has been uploading it all along.