Thanos Remote Write: Push-Based Metrics for Edge and Multi-Cluster

TL;DR Thanos Receive turns the metrics flow around. Each edge Prometheus pushes samples over remote_write to one central endpoint, buffers in its WAL when the link drops, and never sees an object storage credential. Below is the Receive StatefulSet, the hashring, the edge config, two ways to lock the endpoint down, and the hybrid I run with sidecars on the core clusters.

In my previous post on Prometheus and Thanos, I set up the sidecar architecture. Thanos Sidecar runs next to Prometheus, uploads TSDB blocks to object storage, and exposes the data to the Querier over gRPC. For clusters in the same datacenter with a fat, stable link to the central infrastructure, I’ve got no complaints. Everything pulls, and everything can reach everything else.

Then I started putting Prometheus on clusters at the edge.

Those clusters drop off the network for hours when a site loses its uplink. Some sit behind firewalls that allow outbound HTTPS and nothing else. And I went from a handful of clusters to a few dozen, with more on the way. The sidecar model was fighting me at every one of them.

The fix is to stop pulling and start pushing. Thanos has a component for that, called Receive, and this post is about running it.

Where the sidecar model runs out of road

The sidecar setup makes four assumptions. Every one of them held in the datacenter.

1. Thanos Sidecar runs on every Prometheus instance
2. Every cluster gets direct object storage access to upload its own blocks
3. The network stays up long enough for those block uploads to finish
4. gRPC reaches back from the central Querier to each cluster for live queries

They stop holding at the edges of the topology. An edge site where the internet link comes and goes. An air-gapped site that can only talk outbound, never inbound. Hundreds of small clusters where one sidecar each adds up to real overhead. A multi-tenant platform where you don’t even own all the clusters pushing data at you.

Assumption 2 bit me first. Handing object storage credentials to every edge site, including sites I don’t fully control, is a blast radius I don’t want. Assumption 3 bit me second. A factory site that loses its uplink for an afternoon stops uploading blocks, and there’s no story for catching up afterwards. The blocks sit there until someone notices.

So I wanted metrics from clusters I can’t reliably reach, and I wanted my storage keys on none of them. Does Thanos have a component for that? It does.

Thanos Receive flips the direction

Receive turns the arrows around. Instead of sidecars pushing blocks out to object storage and the Querier reaching in, each Prometheus pushes its samples to one central Receive, and Receive owns all the storage plumbing.

The part that makes this cheap to adopt is that Prometheus already has remote_write built in. You don’t install anything new at the edge. It’s a standard Prometheus feature that streams samples to any compatible endpoint, and Receive implements that endpoint. The edge side of this is a few lines of config.

What push buys you at the edge

1. The edge deployment shrinks to almost nothing

No sidecar, no storage credentials at the edge, no gRPC ports to expose inbound. The edge cluster runs Prometheus with a remote_write URL.

# The ENTIRE edge configuration
global:
  external_labels:
    cluster: factory-site-a
    region: europe-west

remote_write:
  - url: https://thanos-receive.central.example.com/api/v1/receive
    bearer_token_file: /etc/prometheus/token

That’s the whole edge config. The edge cluster has no idea object storage exists, never mind which bucket or which provider. It scrapes locally and ships samples upstream. This is the sovereignty angle too. A site I don’t fully control gets a write-only token to one endpoint instead of a key to my storage.

2. It survives losing the network

Drop the link and Prometheus keeps scraping, buffering the unsent samples locally. Bring the link back and it drains the backlog. The Write-Ahead Log (WAL) does this without you wiring up anything.

remote_write:
  - url: https://thanos-receive.central.example.com/api/v1/receive
    queue_config:
      capacity: 100000        # Buffer size
      max_shards: 50          # Parallel send workers
      min_shards: 1
      max_samples_per_send: 5000
      batch_send_deadline: 30s
      min_backoff: 1s
      max_backoff: 5m         # Exponential backoff on failure

I’ve watched a factory cluster sit dark for four hours and then quietly backfill the whole gap once its uplink came back. Nobody touched anything. The WAL had held the samples the entire time.

3. Outbound-only works with the firewall

Edge sites tend to have paranoid firewalls, and honestly they should. Inbound connections get blocked. VPNs are a maintenance tax I’d rather not pay across dozens of sites. Outbound HTTPS to one known hostname, though, is almost always fine. Push fits these networks as they are, so I don’t have to argue with anyone about opening a port.

4. Less running at the edge

Dropping the sidecar means less memory on the edge nodes, one fewer container to patch and watch, no gRPC connections to keep alive, and fewer ways for things to break. On a refurbished mini PC at a remote site, every bit of that counts.

Setting it up

The Receive deployment on the central cluster

Receive is stateful. It writes incoming samples to local disk first, then uploads blocks to object storage on its own schedule, so it needs persistent volumes. Run it as a StatefulSet:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: thanos-receive
  namespace: monitoring
spec:
  serviceName: thanos-receive
  replicas: 3
  selector:
    matchLabels:
      app: thanos-receive
  template:
    metadata:
      labels:
        app: thanos-receive
    spec:
      containers:
        - name: thanos-receive
          image: quay.io/thanos/thanos:v0.34.0
          args:
            - receive
            - --http-address=0.0.0.0:10902
            - --grpc-address=0.0.0.0:10901
            - --remote-write.address=0.0.0.0:19291
            - --tsdb.path=/var/thanos/receive
            - --objstore.config-file=/etc/thanos/objstore.yaml
            - --label=receive_replica="$(POD_NAME)"
            - --tsdb.retention=6h
            # Hashring for HA distribution
            - --receive.hashrings-file=/etc/thanos/hashring.json
            - --receive.local-endpoint=$(POD_NAME).thanos-receive.monitoring.svc:10901
          ports:
            - name: http
              containerPort: 10902
            - name: grpc
              containerPort: 10901
            - name: remote-write
              containerPort: 19291
          volumeMounts:
            - name: data
              mountPath: /var/thanos/receive
            - name: objstore-config
              mountPath: /etc/thanos
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
      volumes:
        - name: objstore-config
          secret:
            secretName: thanos-objstore-secret
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        storageClassName: longhorn
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 50Gi

The hashring keeps replicas honest

Run more than one Receive replica and you need a hashring to decide which replica owns which incoming series:

[
  {
    "endpoints": [
      "thanos-receive-0.thanos-receive.monitoring.svc:10901",
      "thanos-receive-1.thanos-receive.monitoring.svc:10901",
      "thanos-receive-2.thanos-receive.monitoring.svc:10901"
    ]
  }
]

Store this in a ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: thanos-receive-hashring
  namespace: monitoring
data:
  hashring.json: |
    [
      {
        "endpoints": [
          "thanos-receive-0.thanos-receive.monitoring.svc:10901",
          "thanos-receive-1.thanos-receive.monitoring.svc:10901",
          "thanos-receive-2.thanos-receive.monitoring.svc:10901"
        ]
      }
    ]

The hashring pins every time series to a consistent Receive instance, so samples for one series always land on the same replica. Without it, several replicas taking writes at once would give you duplicated or split data.

Exposing Receive to the outside

Edge clusters need to reach Receive from off-cluster, so put it behind a load balancer and an Ingress:

apiVersion: v1
kind: Service
metadata:
  name: thanos-receive-lb
  namespace: monitoring
spec:
  type: LoadBalancer
  ports:
    - name: remote-write
      port: 443
      targetPort: 19291
  selector:
    app: thanos-receive
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: thanos-receive
  namespace: monitoring
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt
    nginx.ingress.kubernetes.io/backend-protocol: "HTTP"
spec:
  tls:
    - hosts:
        - thanos-receive.example.com
      secretName: thanos-receive-tls
  rules:
    - host: thanos-receive.example.com
      http:
        paths:
          - path: /api/v1/receive
            pathType: Prefix
            backend:
              service:
                name: thanos-receive
                port:
                  number: 19291

Pointing the edge Prometheus at Receive

On each edge cluster, this is all the Prometheus operator config you need to start pushing:

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: prometheus
  namespace: monitoring
spec:
  replicas: 1
  retention: 24h  # Local retention for edge queries

  externalLabels:
    cluster: factory-site-a
    region: europe-west
    environment: production

  remoteWrite:
    - url: https://thanos-receive.example.com/api/v1/receive
      bearerTokenSecret:
        name: thanos-remote-write-token
        key: token
      queueConfig:
        capacity: 50000
        maxShards: 20
        minShards: 1
        maxSamplesPerSend: 2000
        batchSendDeadline: 30s
        minBackoff: 1s
        maxBackoff: 5m
      writeRelabelConfigs:
        # Drop high-cardinality metrics you don't need centrally
        - sourceLabels: [__name__]
          regex: 'go_.*'
          action: drop

One thing I learned the annoying way. Get your externalLabels right from the start, because they’re how you tell which site a metric came from once everything is pooled centrally. Forget them and you’ll be staring at a Grafana panel unable to say whether a spike is the factory or the warehouse. I have done that staring.

Locking it down

A Receive endpoint open to the internet with no auth is a free metrics-injection service for anyone who finds it. Don’t ship that. I’ve used two options.

Bearer token, when simple is enough

Generate a token, hand it to the edge clusters, and have Receive check it:

# Generate token
openssl rand -hex 32 > receive-token.txt

# Create secret in central cluster
kubectl create secret generic thanos-receive-auth \
  --from-file=token=receive-token.txt \
  -n monitoring

# Create secret in edge cluster
kubectl create secret generic thanos-remote-write-token \
  --from-file=token=receive-token.txt \
  -n monitoring

Configure Receive to validate tokens:

# Add to Receive container args
- --receive.tenant-header=THANOS-TENANT
- --receive.default-tenant-id=anonymous

mTLS, when a shared token isn’t good enough

For sites where a single shared secret feels too fragile, mutual TLS gives each cluster its own identity:

# Receive args
- --grpc-server-tls-cert=/etc/tls/server.crt
- --grpc-server-tls-key=/etc/tls/server.key
- --grpc-server-tls-client-ca=/etc/tls/ca.crt

# Edge Prometheus remoteWrite
remoteWrite:
  - url: https://thanos-receive.example.com/api/v1/receive
    tlsConfig:
      certFile: /etc/prometheus/tls/client.crt
      keyFile: /etc/prometheus/tls/client.key
      caFile: /etc/prometheus/tls/ca.crt

Keeping tenants apart

If different teams or customers push into the same Receive, a tenant header keeps their data separated:

# Edge cluster A
remoteWrite:
  - url: https://thanos-receive.example.com/api/v1/receive
    headers:
      THANOS-TENANT: tenant-a

# Edge cluster B
remoteWrite:
  - url: https://thanos-receive.example.com/api/v1/receive
    headers:
      THANOS-TENANT: tenant-b

Receive stores the data with the tenant label, so a query can select one tenant:

# Query only tenant-a data
http_requests_total{tenant_id="tenant-a"}

Every sample carries its tenant label, so one Receive cluster can serve several tenants without their data bleeding into each other’s queries.

Surviving the flaky link

The whole reason I went down this road was the edge connectivity problem, so this is the part I care about most. How graceful is the degradation? Graceful enough that I stopped worrying about the edge links.

Tuning the WAL buffer

Prometheus parks unsent samples in its Write-Ahead Log and replays them when the endpoint comes back. The buffer size is yours to tune for how bad your worst outage gets:

remoteWrite:
  - url: https://thanos-receive.example.com/api/v1/receive
    queueConfig:
      capacity: 100000       # Samples to buffer
      maxShards: 30          # Parallel senders when catching up
      maxSamplesPerSend: 5000
      batchSendDeadline: 60s
      minBackoff: 1s
      maxBackoff: 10m        # Don't hammer when down

For a typical edge cluster scraping every 30s, a 100k sample buffer covers roughly 3-4 hours of metrics. Your mileage depends on scrape interval and how many targets each site has, so size it against your own worst-case outage rather than mine.

Watch the remote write from the edge

On the edge cluster, alert when remote write starts failing so you know the buffer is filling:

- alert: PrometheusRemoteWriteFailing
  expr: |
    rate(prometheus_remote_storage_failed_samples_total[5m]) > 0
  for: 15m
  labels:
    severity: warning
  annotations:
    summary: "Prometheus remote write failing"
    description: "Edge cluster {{ $labels.cluster }} failing to send metrics"

Watch for silent edges from the centre

The edge-side alert only fires if the edge can still reach you. The case that scares me is a site that goes completely dark, so catch that one centrally:

- alert: EdgeClusterMissing
  expr: |
    absent_over_time(up{job="prometheus", cluster=~"edge-.*"}[1h])
  for: 30m
  labels:
    severity: critical
  annotations:
    summary: "Edge cluster not reporting"
    description: "No metrics from {{ $labels.cluster }} for 1 hour"

Sidecar or Receive, per cluster

Receive and the sidecar solve different networks, and the table shows where they differ:

Sidecar fits when all your clusters have stable connectivity, you want minimal central infrastructure, and real-time queries across every cluster matter. Receive fits when edge clusters have intermittent connectivity, you’re scaling to many (50+) clusters, edge resources are constrained, or the firewall rules are outbound-only.

You don’t have to choose globally. The healthiest setup I’ve run is a hybrid, with sidecars on the core clusters that have stable links, Receive for everything at the edge, and both feeding the same Querier and the same object storage. The Querier doesn’t care how the data arrived.

How I run this

That hybrid is what’s in production for me:

# Central infrastructure cluster: Sidecar
# (stable connectivity, needs real-time queries)
prometheus:
  thanos:
    enabled: true
    objectStorageConfig:
      name: thanos-objstore

# Edge manufacturing sites: Remote Write
# (intermittent connectivity, resource constrained)
prometheus:
  remoteWrite:
    - url: https://thanos-receive.central:443/api/v1/receive
      bearerTokenSecret:
        name: edge-token
      queueConfig:
        capacity: 150000  # 6+ hours buffer
        maxBackoff: 15m

# All feed the same Thanos Query
thanos:
  query:
    stores:
      - dnssrv+_grpc._tcp.prometheus-operated.monitoring.svc  # Sidecars
      - dnssrv+_grpc._tcp.thanos-receive.monitoring.svc        # Receive
      - dnssrv+_grpc._tcp.thanos-store.monitoring.svc          # Historical

Edge sites still lose connectivity for hours. They reconnect, the metrics backfill on their own, and the central Grafana dashboards show the full picture no matter which path the data took to get there.

What this solved

The original problem was that I couldn’t see clusters I couldn’t reliably reach, and I didn’t want to hand storage credentials to sites I don’t fully trust. Receive answers both. Edge sites ride out connectivity loss on their local WAL, they hold nothing but a write-only token to a single endpoint, and all the storage complexity stays in one place I do control.

The deeper reason I like it lines up with how I think about infrastructure generally. Push the simple thing to the edge and keep the complicated thing central, and each edge site becomes something I can reason about end to end. The firewall rules match what the security team would have asked for anyway. When a site goes dark now, the WAL fills, the central alert fires after an hour, and the gap backfills when the link returns. Nobody opens an incident for it.

Pull works when the network stays up, and push works when it doesn’t, so I run both. The four-hour factory outage from earlier in this post is what convinced me. Nobody touched anything, and the graph has no hole in it.

• Prometheus
• Architecture
• Resilience