The first time someone asked me “was this slower last month than it is now?”, I had no answer. My Prometheus only remembered two weeks. The data I needed had already aged out of local disk and been deleted. That gap is the whole reason this post exists.
Prometheus is the default for Kubernetes metrics, and for good reason. It works beautifully right up until you need long-term storage, or a view across multiple clusters, or genuine high availability. Then you meet the wall.
Thanos extends Prometheus instead of replacing it. You keep the setup you already understand, bolt on a few components, and get unlimited retention and global querying. That “keep what you know” part matters to me. I would rather grow a system I can already reason about than swap it for a black box that promises to do everything.
Where Standalone Prometheus Stops
A single Prometheus has four hard limits:
- Single node - No native clustering or HA
- Local storage - Retention is capped by disk size
- Single cluster view - You can’t query across clusters
- No downsampling - Old data eats as much space as new data
For one small cluster with two weeks of retention, none of this hurts. For a production multi-cluster setup with compliance requirements, every one of these is a blocker. The question is not whether you hit them, but when.
The Shape of Thanos
Thanos wraps a handful of components around your existing Prometheus:
flowchart TD
subgraph clusterA["Cluster A"]
PA["Prometheus + Sidecar"]
end
subgraph clusterB["Cluster B"]
PB["Prometheus + Sidecar"]
end
PA --> OS["Object Storage<br/>(S3/MinIO/GCS)"]
PB --> OS
OS --> Q["Querier"]
OS --> SG["Store Gateway"]
OS --> C["Compactor"]
SG --> Q
Q --> G["Grafana"]
Sidecar - Runs next to Prometheus, uploads blocks to object storage Store Gateway - Serves historical data back from object storage Querier - Aggregates data from sidecars and the store gateway Compactor - Downsamples and deduplicates data in object storage
Four moving parts, and each one does a single job you can hold in your head. That’s the kind of design I trust.
The Simplest Version: Just Install It
Start with the Bitnami Helm chart and get something running:
helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update
helm install thanos bitnami/thanos \
--namespace monitoring \
--create-namespace \
--set objstoreConfig="$(cat thanos-objstore.yaml)"
The object store config (thanos-objstore.yaml) points Thanos at your bucket:
type: s3
config:
bucket: thanos-metrics
endpoint: minio.storage:9000
access_key: ${MINIO_ACCESS_KEY}
secret_key: ${MINIO_SECRET_KEY}
insecure: true # For MinIO without TLS
That’s the whole skeleton. Everything after this is layering real-world detail on top.
Wiring Prometheus to the Sidecar
Next, tell Prometheus to run the Thanos sidecar alongside it:
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
name: prometheus
namespace: monitoring
spec:
replicas: 2 # HA pair
retention: 2h # Short local retention, Thanos handles long-term
# Thanos sidecar configuration
thanos:
baseImage: quay.io/thanos/thanos
version: v0.34.0
objectStorageConfig:
key: thanos.yaml
name: thanos-objstore-secret
# External labels for deduplication
externalLabels:
cluster: production
replica: $(POD_NAME)
# Let Thanos sidecar access Prometheus data
storage:
volumeClaimTemplate:
spec:
storageClassName: longhorn
resources:
requests:
storage: 50Gi
The sidecar does three things:
- Exposes Prometheus data to the Thanos Querier over gRPC
- Uploads completed TSDB blocks to object storage
- Answers Store API queries for recent data
Notice the retention: 2h. Local disk now only holds the most recent slice. The long history lives in object storage, where it belongs.
Going Further: The Three Server Components
With the sidecar shipping blocks, you need the components that read them back.
Querier
apiVersion: apps/v1
kind: Deployment
metadata:
name: thanos-querier
namespace: monitoring
spec:
replicas: 2
template:
spec:
containers:
- name: thanos-query
image: quay.io/thanos/thanos:v0.34.0
args:
- query
- --http-address=0.0.0.0:9090
- --grpc-address=0.0.0.0:10901
# Connect to sidecars
- --store=dnssrv+_grpc._tcp.prometheus-operated.monitoring.svc
# Connect to store gateway
- --store=dnssrv+_grpc._tcp.thanos-store.monitoring.svc
# Deduplication
- --query.replica-label=replica
ports:
- name: http
containerPort: 9090
- name: grpc
containerPort: 10901
Store Gateway
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: thanos-store
namespace: monitoring
spec:
replicas: 2
template:
spec:
containers:
- name: thanos-store
image: quay.io/thanos/thanos:v0.34.0
args:
- store
- --http-address=0.0.0.0:10902
- --grpc-address=0.0.0.0:10901
- --data-dir=/var/thanos/store
- --objstore.config-file=/etc/thanos/objstore.yaml
volumeMounts:
- name: objstore-config
mountPath: /etc/thanos
- name: data
mountPath: /var/thanos/store
volumes:
- name: objstore-config
secret:
secretName: thanos-objstore-secret
volumeClaimTemplates:
- metadata:
name: data
spec:
storageClassName: longhorn
resources:
requests:
storage: 10Gi
Compactor
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: thanos-compactor
namespace: monitoring
spec:
replicas: 1 # Only one compactor!
template:
spec:
containers:
- name: thanos-compact
image: quay.io/thanos/thanos:v0.34.0
args:
- compact
- --http-address=0.0.0.0:10902
- --data-dir=/var/thanos/compact
- --objstore.config-file=/etc/thanos/objstore.yaml
- --retention.resolution-raw=30d
- --retention.resolution-5m=90d
- --retention.resolution-1h=1y
- --wait
volumeMounts:
- name: objstore-config
mountPath: /etc/thanos
- name: data
mountPath: /var/thanos/compact
That replicas: 1 on the compactor is not a typo. Run two and they will fight over the same blocks and corrupt your data. I learned to put a big comment there so future-me does not get clever.
The retention tiers do the heavy lifting on cost:
- Raw data: 30 days at full resolution
- 5m downsampled: 90 days
- 1h downsampled: 1 year
Older data takes less space because it has been downsampled. You trade resolution for history, which is exactly the trade you want for anything older than a couple of weeks.
GitOps Deployment
I do not click any of this together by hand. It goes through ArgoCD like everything else, so the cluster state lives in Git where I can read it, review it, and roll it back:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: thanos
namespace: argocd
spec:
project: default
source:
repoURL: https://charts.bitnami.com/bitnami
chart: thanos
targetRevision: 12.20.0
helm:
values: |
objstoreConfig: |-
type: s3
config:
bucket: thanos-metrics
endpoint: minio.storage:9000
insecure: true
query:
enabled: true
replicaCount: 2
stores:
- dnssrv+_grpc._tcp.prometheus-operated.monitoring.svc
storegateway:
enabled: true
replicaCount: 2
persistence:
size: 20Gi
compactor:
enabled: true
retentionResolutionRaw: 30d
retentionResolution5m: 90d
retentionResolution1h: 1y
persistence:
size: 50Gi
ruler:
enabled: false # Use Prometheus rules instead
receive:
enabled: false # Using sidecar mode
destination:
server: https://kubernetes.default.svc
namespace: monitoring
Advanced Patterns: HA That Actually Survives a Node
Two Prometheus replicas scraping the same targets is only useful if something stitches their results back together. Thanos is that something.
# Run two Prometheus instances
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
spec:
replicas: 2
externalLabels:
replica: $(POD_NAME) # Different for each replica
Both instances scrape everything. The Querier deduplicates on the replica label so you don’t see doubled graphs:
# Querier configuration
args:
- --query.replica-label=replica
- --query.replica-label=prometheus_replica
Queries come back deduplicated automatically. Lose a node, lose a Prometheus pod, and your dashboards keep working from the surviving replica. That is resilience you can actually feel during an incident, instead of a checkbox in a runbook.
One Global View Across Clusters
Point multiple clusters at the same Thanos deployment and give each one a distinct cluster label.
Cluster A Prometheus:
externalLabels:
cluster: production-eu
replica: $(POD_NAME)
Cluster B Prometheus:
externalLabels:
cluster: production-us
replica: $(POD_NAME)
Now the Querier aggregates across both:
# Total requests across all clusters
sum(rate(http_requests_total[5m]))
# Requests by cluster
sum by (cluster) (rate(http_requests_total[5m]))
One PromQL query, every cluster, one answer. No more tab-juggling between three different Grafana instances to reconstruct what happened.
Grafana Integration
Grafana does not need to know Thanos exists. Point it at the Querier and it thinks it is talking to plain Prometheus:
apiVersion: v1
kind: ConfigMap
metadata:
name: grafana-datasources
data:
thanos.yaml: |
apiVersion: 1
datasources:
- name: Thanos
type: prometheus
url: http://thanos-query.monitoring:9090
access: proxy
isDefault: true
jsonData:
timeInterval: "15s"
Every dashboard you already built keeps working. Point it at Thanos and move on.
Recording Rules for Performance
Some queries are expensive enough that you do not want to run them on every dashboard refresh. Pre-compute them once and read the result:
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: recording-rules
spec:
groups:
- name: aggregations
interval: 1m
rules:
# Pre-aggregate request rate by service
- record: service:http_requests:rate5m
expr: sum by (service) (rate(http_requests_total[5m]))
# Pre-aggregate error rate
- record: service:http_errors:rate5m
expr: sum by (service) (rate(http_requests_total{status=~"5.."}[5m]))
# Pre-compute availability
- record: service:availability:ratio
expr: |
1 - (
service:http_errors:rate5m /
service:http_requests:rate5m
)
Dashboards then query the cheap service:* metrics instead of grinding through raw data on every load.
Alerting Architecture
Keep alerting close to the data. Run Alertmanager with Prometheus rather than with Thanos:
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
spec:
alerting:
alertmanagers:
- namespace: monitoring
name: alertmanager
port: web
ruleSelector:
matchLabels:
role: alert-rules
Thanos Ruler exists and it works, but it adds another component to reason about and another failure mode to debug at 3am. For most setups Prometheus alerting is plenty, and fewer moving parts means fewer things that surprise you.
Monitoring Thanos Itself
Thanos exposes its own Prometheus metrics, which is convenient because the thing watching your systems is itself a system that can break. Watch these:
# Sidecar upload success
thanos_shipper_uploads_total
thanos_shipper_upload_failures_total
# Store gateway performance
thanos_bucket_store_series_fetch_duration_seconds
thanos_bucket_store_block_loads_total
# Compactor health
thanos_compact_group_compactions_total
thanos_compact_group_compaction_failures_total
# Querier performance
thanos_query_gate_duration_seconds
And alert when uploads start failing, because a silent sidecar means you are quietly losing history:
- alert: ThanosSidecarUploadFailing
expr: increase(thanos_shipper_upload_failures_total[1h]) > 0
for: 15m
labels:
severity: warning
annotations:
summary: "Thanos sidecar failing to upload blocks"
Storage Considerations
Here is where downsampling pays for itself:
| Resolution | Data per day | 1 year cost (S3) |
|---|---|---|
| Raw (15s) | ~100MB/target | ~$4/target |
| 5m downsample | ~3MB/target | ~$0.12/target |
| 1h downsample | ~0.5MB/target | ~$0.02/target |
Keeping raw data forever is how you end up with a storage bill that quietly grows into a problem. Downsample aggressively and old data costs almost nothing.
For self-hosted object storage, MinIO does the job and keeps the data on hardware I own:
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: minio
spec:
template:
spec:
containers:
- name: minio
image: minio/minio:latest
args:
- server
- /data
- --console-address
- ":9001"
env:
- name: MINIO_ROOT_USER
valueFrom:
secretKeyRef:
name: minio-credentials
key: root-user
- name: MINIO_ROOT_PASSWORD
valueFrom:
secretKeyRef:
name: minio-credentials
key: root-password
The Full Picture: My Production Setup
Pulling every layer together, this is roughly what runs in production:
# Prometheus with sidecar
prometheus:
replicas: 2
retention: 6h # Very short, Thanos handles long-term
thanos:
objectStorageConfig:
name: thanos-objstore
externalLabels:
cluster: production
environment: prod
# Thanos components
thanos:
query:
replicaCount: 2
stores:
- dnssrv+_grpc._tcp.prometheus-operated.monitoring.svc
- dnssrv+_grpc._tcp.thanos-store.monitoring.svc
storegateway:
replicaCount: 2
persistence:
size: 50Gi
compactor:
retentionResolutionRaw: 14d
retentionResolution5m: 60d
retentionResolution1h: 365d
persistence:
size: 100Gi
# Object storage
minio:
replicas: 4
persistence:
size: 500Gi
The decisions behind those numbers:
- 6h local retention - The sidecar uploads often, so there is no reason to hoard data on local disk
- 14d raw retention - Full resolution covers any recent debugging session
- 1 year 1h retention - Enough trend history for capacity planning
- Self-hosted MinIO - Data sovereignty, no cloud bill, no third party holding my metrics
Compare that to the three-line Helm install at the top. Same building blocks, just tuned for a system I have to live with.
Why This Matters
Metrics are how I understand my systems instead of guessing about them. They answer the questions that come up during every incident and every planning session:
- Is this service healthy?
- What changed right before the incident?
- Are we meeting our SLOs?
- Where is optimization actually worth the effort?
Lose long-term metrics and you lose the ability to answer “compared to when?”. Lose cross-cluster queries and you only ever see one piece of the picture at a time. Prometheus plus Thanos gives you unlimited retention, a global view, and high availability, all behind the same Prometheus interface you already know how to read.
That question from the start of this post, “was this slower last month?”, now has an answer I can pull up in a single query. Going from “what’s happening right now” to “what has been happening” is the difference between firefighting and actually steering the system.