Thanos Remote Write: Push-Based Metrics for Edge and Multi-Cluster

In my previous post on Prometheus and Thanos, I set up the sidecar architecture. Thanos Sidecar runs next to Prometheus, uploads TSDB blocks to object storage, and exposes data to the Querier over gRPC. For clusters sitting in the same datacenter with a fat, stable link to your central infrastructure, it’s lovely. Everything pulls. Everything talks to everything. Life is good.

Then I started putting Prometheus on clusters at the edge, and life got less good.

These clusters drop off the network for hours when a site loses its uplink. Some of them sit behind firewalls that only allow outbound HTTPS, full stop. And I went from a handful of clusters to a few dozen, with more coming. Suddenly the sidecar model was fighting me at every site.

The fix is to stop pulling and start pushing. Thanos has a component for exactly that: Receive.

Where the sidecar model runs out of road

The sidecar setup makes a few assumptions, and each one is reasonable until it isn’t:

Thanos Sidecar runs on every Prometheus instance
Every cluster gets direct object storage access to upload its own blocks
The network stays up long enough for those block uploads to finish
gRPC reaches back from the central Querier to each cluster for live queries

Inside one datacenter, none of this is a problem. The trouble shows up at the edges of your topology:

Edge locations where the internet link comes and goes
Air-gapped sites that can only talk outbound, never inbound
Hundreds of small clusters where one sidecar each turns into real overhead
Multi-tenant platforms where you don’t even own all the clusters pushing data at you

Assumption 2 is the one that bit me first. Handing object storage credentials to every edge site, including ones I don’t fully control, is a blast radius I don’t want. Assumption 3 is the one that bit me second: a factory site that loses its uplink for an afternoon just stops uploading blocks, and there’s no graceful story for catching up.

So how do you collect metrics from clusters you can’t reliably reach, without shipping storage credentials to every one of them?

Thanos Receive flips the direction

Receive turns the arrows around. Instead of sidecars pushing blocks out to object storage and the Querier reaching in, each Prometheus pushes its samples to one central Receive component, and that component owns all the storage plumbing.

flowchart TD
    subgraph edge["Edge Clusters"]
        subgraph E1["Site A (Factory)"]
            P1["Prometheus"]
        end
        subgraph E2["Site B (Warehouse)"]
            P2["Prometheus"]
        end
        subgraph E3["Site C (Retail)"]
            P3["Prometheus"]
        end
    end

    subgraph central["Central Cluster"]
        R["Thanos Receive<br/>(HA cluster)"]
        Q["Thanos Querier"]
        SG["Store Gateway"]
        C["Compactor"]
        OS["Object Storage"]
    end

    P1 -->|"remote_write"| R
    P2 -->|"remote_write"| R
    P3 -->|"remote_write"| R

    R --> OS
    OS --> SG
    SG --> Q
    R --> Q
    OS --> C

    Q --> G["Grafana"]

Here’s the part that makes this cheap to adopt: Prometheus already has remote_write built in. You don’t install anything new at the edge. It’s a standard Prometheus feature that streams samples to any compatible endpoint, and Receive implements that endpoint. The edge side of this is config, not software.

What push buys you at the edge

1. The edge deployment shrinks to almost nothing

No sidecar. No storage credentials at the edge. No gRPC ports to expose inbound. The edge cluster runs Prometheus with a remote_write URL and that’s the whole story.

# The ENTIRE edge configuration
global:
  external_labels:
    cluster: factory-site-a
    region: europe-west

remote_write:
  - url: https://thanos-receive.central.example.com/api/v1/receive
    bearer_token_file: /etc/prometheus/token

That’s the lot. The edge cluster has no idea object storage exists, never mind which bucket or which provider. It scrapes locally and ships samples upstream. This is the sovereignty angle too: the site I don’t fully control gets a write-only token to one endpoint, not a key to my storage.

2. It survives losing the network

Drop the link and Prometheus keeps scraping, buffering the unsent samples locally. Bring the link back and it drains the backlog. The Write-Ahead Log (WAL) does this without you wiring up anything.

remote_write:
  - url: https://thanos-receive.central.example.com/api/v1/receive
    queue_config:
      capacity: 100000        # Buffer size
      max_shards: 50          # Parallel send workers
      min_shards: 1
      max_samples_per_send: 5000
      batch_send_deadline: 30s
      min_backoff: 1s
      max_backoff: 5m         # Exponential backoff on failure

I’ve watched a factory cluster sit dark for four hours and then quietly backfill the whole gap once its uplink came back. Nobody touched anything. The WAL had held the samples the entire time.

3. Outbound-only works with the firewall, not against it

Edge sites tend to have paranoid firewalls, and honestly they should. Inbound connections get blocked. VPNs are a maintenance tax I’d rather not pay across dozens of sites. Outbound HTTPS to one known hostname, though, is almost always fine. Push fits the shape of these networks instead of demanding they bend for me.

4. Less running at the edge

Dropping the sidecar means less memory on edge nodes, one fewer container to patch and watch, no gRPC connections to keep alive, and fewer ways for things to break. On a refurbished mini PC at a remote site, every bit of that counts.

Setting it up

Central cluster: the Receive deployment

Receive is stateful. It writes incoming samples to local disk first, then uploads blocks to object storage on its own schedule, so it needs persistent volumes. Run it as a StatefulSet:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: thanos-receive
  namespace: monitoring
spec:
  serviceName: thanos-receive
  replicas: 3
  selector:
    matchLabels:
      app: thanos-receive
  template:
    metadata:
      labels:
        app: thanos-receive
    spec:
      containers:
        - name: thanos-receive
          image: quay.io/thanos/thanos:v0.34.0
          args:
            - receive
            - --http-address=0.0.0.0:10902
            - --grpc-address=0.0.0.0:10901
            - --remote-write.address=0.0.0.0:19291
            - --tsdb.path=/var/thanos/receive
            - --objstore.config-file=/etc/thanos/objstore.yaml
            - --label=receive_replica="$(POD_NAME)"
            - --tsdb.retention=6h
            # Hashring for HA distribution
            - --receive.hashrings-file=/etc/thanos/hashring.json
            - --receive.local-endpoint=$(POD_NAME).thanos-receive.monitoring.svc:10901
          ports:
            - name: http
              containerPort: 10902
            - name: grpc
              containerPort: 10901
            - name: remote-write
              containerPort: 19291
          volumeMounts:
            - name: data
              mountPath: /var/thanos/receive
            - name: objstore-config
              mountPath: /etc/thanos
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
      volumes:
        - name: objstore-config
          secret:
            secretName: thanos-objstore-secret
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        storageClassName: longhorn
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 50Gi

The hashring keeps replicas honest

Run more than one Receive replica and you need a hashring to decide which replica owns which incoming series:

[
  {
    "endpoints": [
      "thanos-receive-0.thanos-receive.monitoring.svc:10901",
      "thanos-receive-1.thanos-receive.monitoring.svc:10901",
      "thanos-receive-2.thanos-receive.monitoring.svc:10901"
    ]
  }
]

Store this in a ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: thanos-receive-hashring
  namespace: monitoring
data:
  hashring.json: |
    [
      {
        "endpoints": [
          "thanos-receive-0.thanos-receive.monitoring.svc:10901",
          "thanos-receive-1.thanos-receive.monitoring.svc:10901",
          "thanos-receive-2.thanos-receive.monitoring.svc:10901"
        ]
      }
    ]

The hashring pins every time series to a consistent Receive instance, so samples for one series always land on the same replica. That’s what stops you from getting duplicated or split data when several replicas are taking writes at once.

Exposing Receive to the outside

Edge clusters need to reach Receive from off-cluster, so put it behind a load balancer and an Ingress:

apiVersion: v1
kind: Service
metadata:
  name: thanos-receive-lb
  namespace: monitoring
spec:
  type: LoadBalancer
  ports:
    - name: remote-write
      port: 443
      targetPort: 19291
  selector:
    app: thanos-receive
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: thanos-receive
  namespace: monitoring
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt
    nginx.ingress.kubernetes.io/backend-protocol: "HTTP"
spec:
  tls:
    - hosts:
        - thanos-receive.example.com
      secretName: thanos-receive-tls
  rules:
    - host: thanos-receive.example.com
      http:
        paths:
          - path: /api/v1/receive
            pathType: Prefix
            backend:
              service:
                name: thanos-receive
                port:
                  number: 19291

Edge cluster: pointing Prometheus at Receive

On each edge cluster, this is all the Prometheus operator config you need to start pushing:

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: prometheus
  namespace: monitoring
spec:
  replicas: 1
  retention: 24h  # Local retention for edge queries

  externalLabels:
    cluster: factory-site-a
    region: europe-west
    environment: production

  remoteWrite:
    - url: https://thanos-receive.example.com/api/v1/receive
      bearerTokenSecret:
        name: thanos-remote-write-token
        key: token
      queueConfig:
        capacity: 50000
        maxShards: 20
        minShards: 1
        maxSamplesPerSend: 2000
        batchSendDeadline: 30s
        minBackoff: 1s
        maxBackoff: 5m
      writeRelabelConfigs:
        # Drop high-cardinality metrics you don't need centrally
        - sourceLabels: [__name__]
          regex: 'go_.*'
          action: drop

One thing I learned the annoying way: get your externalLabels right from the start. They’re how you tell which site a metric came from once everything is pooled centrally. Forget them and you’ll be staring at a Grafana panel unable to say whether a spike is the factory or the warehouse.

Locking it down

A Receive endpoint open to the internet with no auth is a free metrics-injection service for anyone who finds it. Don’t ship that. Two options I’ve used:

Bearer token, when simple is enough

Generate a token, hand it to the edge clusters, and have Receive check it:

# Generate token
openssl rand -hex 32 > receive-token.txt

# Create secret in central cluster
kubectl create secret generic thanos-receive-auth \
  --from-file=token=receive-token.txt \
  -n monitoring

# Create secret in edge cluster
kubectl create secret generic thanos-remote-write-token \
  --from-file=token=receive-token.txt \
  -n monitoring

Configure Receive to validate tokens:

# Add to Receive container args
- --receive.tenant-header=THANOS-TENANT
- --receive.default-tenant-id=anonymous

mTLS, when a shared token isn’t good enough

For sites where a single shared secret feels too fragile, mutual TLS gives each cluster its own identity:

# Receive args
- --grpc-server-tls-cert=/etc/tls/server.crt
- --grpc-server-tls-key=/etc/tls/server.key
- --grpc-server-tls-client-ca=/etc/tls/ca.crt

# Edge Prometheus remoteWrite
remoteWrite:
  - url: https://thanos-receive.example.com/api/v1/receive
    tlsConfig:
      certFile: /etc/prometheus/tls/client.crt
      keyFile: /etc/prometheus/tls/client.key
      caFile: /etc/prometheus/tls/ca.crt

Keeping tenants apart

If different teams or customers push into the same Receive, you can keep their data separated with a tenant header:

# Edge cluster A
remoteWrite:
  - url: https://thanos-receive.example.com/api/v1/receive
    headers:
      THANOS-TENANT: tenant-a

# Edge cluster B
remoteWrite:
  - url: https://thanos-receive.example.com/api/v1/receive
    headers:
      THANOS-TENANT: tenant-b

Data is stored with the tenant label, enabling isolation in queries:

# Query only tenant-a data
http_requests_total{tenant_id="tenant-a"}

Receive tags every sample with its tenant label, so one Receive cluster can serve several isolated tenants without their data bleeding into each other’s queries.

Surviving the flaky link

The whole reason I went down this road was the edge connectivity problem, so it’s worth spending a moment on how graceful the degradation actually is.

Tuning the WAL buffer

Prometheus parks unsent samples in its Write-Ahead Log and replays them when the endpoint comes back. The buffer size is yours to tune for how bad your worst outage gets:

remoteWrite:
  - url: https://thanos-receive.example.com/api/v1/receive
    queueConfig:
      capacity: 100000       # Samples to buffer
      maxShards: 30          # Parallel senders when catching up
      maxSamplesPerSend: 5000
      batchSendDeadline: 60s
      minBackoff: 1s
      maxBackoff: 10m        # Don't hammer when down

For a typical edge cluster scraping every 30s, a 100k sample buffer covers roughly 3-4 hours of metrics. Your mileage depends on scrape interval and how many targets each site has, so size it against your actual worst-case outage rather than mine.

Watch the remote write from the edge

On the edge cluster, alert when remote write starts failing so you know the buffer is filling:

- alert: PrometheusRemoteWriteFailing
  expr: |
    rate(prometheus_remote_storage_failed_samples_total[5m]) > 0
  for: 15m
  labels:
    severity: warning
  annotations:
    summary: "Prometheus remote write failing"
    description: "Edge cluster {{ $labels.cluster }} failing to send metrics"

Watch for silent edges from the centre

The edge-side alert only fires if the edge can still reach you. The case that actually scares me is a site that goes completely dark, so catch that one centrally:

- alert: EdgeClusterMissing
  expr: |
    absent_over_time(up{job="prometheus", cluster=~"edge-.*"}[1h])
  for: 30m
  labels:
    severity: critical
  annotations:
    summary: "Edge cluster not reporting"
    description: "No metrics from {{ $labels.cluster }} for 1 hour"

Sidecar or Receive: picking per cluster

Receive isn’t a replacement for the sidecar, it’s the right tool for a different network. Here’s how the two stack up:

Aspect	Sidecar	Receive
Edge/offline support	Poor	Excellent
Real-time queries	Yes (via gRPC)	Yes (via Receive gRPC)
Resource at edge	Higher	Lower
Object storage access	Required at edge	Central only
Network direction	Outbound + inbound	Outbound only
Complexity at edge	Higher	Lower
Block upload latency	2h (block size)	Real-time

Use Sidecar when:

All clusters have stable connectivity
You want minimal central infrastructure
Real-time queries across all clusters matter

Use Receive when:

Edge clusters have intermittent connectivity
You’re scaling to many (50+) clusters
Edge resources are constrained
Outbound-only firewall rules

You don’t have to choose globally. The healthiest setup I’ve run is a hybrid: sidecar on the core, stable clusters, Receive for everything at the edge, both feeding the same Querier and the same object storage. The Querier doesn’t care how the data arrived.

How I actually run this

That hybrid is what’s in production for me:

# Central infrastructure cluster: Sidecar
# (stable connectivity, needs real-time queries)
prometheus:
  thanos:
    enabled: true
    objectStorageConfig:
      name: thanos-objstore

# Edge manufacturing sites: Remote Write
# (intermittent connectivity, resource constrained)
prometheus:
  remoteWrite:
    - url: https://thanos-receive.central:443/api/v1/receive
      bearerTokenSecret:
        name: edge-token
      queueConfig:
        capacity: 150000  # 6+ hours buffer
        maxBackoff: 15m

# All feed the same Thanos Query
thanos:
  query:
    stores:
      - dnssrv+_grpc._tcp.prometheus-operated.monitoring.svc  # Sidecars
      - dnssrv+_grpc._tcp.thanos-receive.monitoring.svc        # Receive
      - dnssrv+_grpc._tcp.thanos-store.monitoring.svc          # Historical

Edge sites still lose connectivity for hours. They reconnect, the metrics backfill on their own, and the central Grafana dashboards show the full picture no matter which path the data took to get there.

What this actually solved

The original problem was straightforward: I couldn’t see clusters I couldn’t reliably reach, and I didn’t want to hand storage credentials to sites I don’t fully trust. Receive answers both. Edge sites ride out connectivity loss on their local WAL, they hold nothing but a write-only token to a single endpoint, and all the storage complexity stays in one place I do control.

The deeper reason I like it lines up with how I think about infrastructure generally. Pushing the simple thing to the edge and keeping the complicated thing central means each edge site is something I can reason about end to end, and the firewall rules match what the security team would have wanted anyway. When a site goes dark, that’s expected behaviour with a defined recovery, not an incident.

Pull is the right answer when everything stays connected. Push is the right answer when the network has other plans. Run both, and your monitoring stops depending on perfect conditions you were never going to get.

Where the sidecar model runs out of road#

Thanos Receive flips the direction#

What push buys you at the edge#

1. The edge deployment shrinks to almost nothing#

2. It survives losing the network#

3. Outbound-only works with the firewall, not against it#

4. Less running at the edge#

Setting it up#

Central cluster: the Receive deployment#

The hashring keeps replicas honest#

Exposing Receive to the outside#

Edge cluster: pointing Prometheus at Receive#

Locking it down#

Bearer token, when simple is enough#

mTLS, when a shared token isn’t good enough#

Keeping tenants apart#

Surviving the flaky link#

Tuning the WAL buffer#

Watch the remote write from the edge#

Watch for silent edges from the centre#

Sidecar or Receive: picking per cluster#

How I actually run this#

What this actually solved#