Distributed Tracing with Tempo and OpenTelemetry

TL;DR Tempo is where my traces go. The OpenTelemetry Collector receives spans from every service, tail sampling keeps every error and every slow request, and Tempo writes the rest to MinIO with three days of retention. From a span in Grafana I click through to the Loki lines with the same trace ID. My production values are at the bottom.

Your metrics say something is slow. Your logs say errors happened. Fine. Which request failed, where did the latency come from, and which service in the chain ate the timeout? Neither of them can tell you.

The first time this bit me, a checkout flow had started timing out under load. Ten services in the path, every one of them green on its own dashboard, and no way to follow one request from the front door to the place it died. I suspected the payment service because it felt like the payment service.

Distributed tracing closes that gap. It follows one request through every service it touches and shows you where it stalled. I want to understand what I run instead of guessing at it, and tracing is what turns “I think it’s the payment service” into “here’s the span, here’s the slow query”. So here’s the setup, one Helm chart at a time, ending with the values I run.

Metrics, logs and traces

Metrics tell you the error rate and the latency. Logs tell you the error message and what was going on around it. Traces tell you which service, which call, and which path the request took.

Any one of them on its own leaves you half blind. Put all three in the same Grafana and you can follow a request from the alert to the query that caused it.

A trace is a tree of spans

A trace is a tree of spans, one tree per request:

Each box is a span. A span has a name, like HTTP GET /orders, a duration, a parent span, a status of success or error, and whatever attributes you attached to it, user_id and order_id being the usual ones.

The trace ID is what holds the tree together. Every span from the same request carries it, however many services that request crossed.

Why I run Tempo

I run Grafana Tempo because it fits how the rest of my stack already works. It writes to object storage and keeps no index beyond the trace ID, so it stays cheap at high volume and there’s no cluster to babysit. And Grafana treats it as a native data source, which is what makes the click from a span to a log line work later on.

That’s the decision to understand before you install anything. Tempo indexes trace IDs and nothing else, the same trick Loki pulls for logs. No index on spans, no index on attributes. Storage stays cheap, and the price is that you can’t ask it for “every trace with user_id=123” out of the box.

So how do you find a trace ID? Through the other two. Metrics point you at the bad five minutes, a log line carries the trace ID, and you jump into Tempo for the whole path. Sounds like a chore. Once the data sources are wired together it’s one click in Grafana, and that wiring is further down.

How the pieces fit

Your applications carry an OpenTelemetry SDK that emits spans. The OTel Collector sits in the middle, receives the telemetry, processes it and fans it out to the right backend. Tempo parks the traces in object storage. Grafana is where you look at them, next to the metrics and logs from the same request.

Two Helm charts, then the instrumentation.

Installing Tempo

The smallest Tempo that receives traces, via Helm :

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

helm install tempo grafana/tempo \
  --namespace monitoring \
  --values tempo-values.yaml

A single binary with traces on a local volume:

# tempo-values.yaml
tempo:
  storage:
    trace:
      backend: local
      local:
        path: /var/tempo/traces

  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317
        http:
          endpoint: 0.0.0.0:4318

persistence:
  enabled: true
  size: 50Gi

That’s enough to start receiving traces, and enough to lose all of them when the pod’s volume dies. Use it for a first look. Before you rely on it during an incident, the next values file swaps the local disk for object storage and splits Tempo into its components.

# tempo-values.yaml
tempo:
  storage:
    trace:
      backend: s3
      s3:
        bucket: tempo-traces
        endpoint: minio.storage:9000
        access_key: ${MINIO_ACCESS_KEY}
        secret_key: ${MINIO_SECRET_KEY}
        insecure: true

  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317
        http:
          endpoint: 0.0.0.0:4318

  # Retention
  compactor:
    compaction:
      block_retention: 48h

# Distributed mode for scale
distributor:
  replicas: 2
ingester:
  replicas: 3
querier:
  replicas: 2
compactor:
  replicas: 1

I point it at my own MinIO rather than a hosted bucket, so the trace data stays on hardware I own. Same reasoning as everything else in the stack.

Installing the OpenTelemetry Collector

Every span, metric and log line goes through the OTel Collector before it lands anywhere:

helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts

helm install otel-collector open-telemetry/opentelemetry-collector \
  --namespace monitoring \
  --values otel-collector-values.yaml

The collector’s config:

# otel-collector-values.yaml
mode: deployment
replicaCount: 2

config:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317
        http:
          endpoint: 0.0.0.0:4318

  processors:
    batch:
      timeout: 1s
      send_batch_size: 1024

    # Add Kubernetes metadata
    k8sattributes:
      auth_type: serviceAccount
      extract:
        metadata:
          - k8s.namespace.name
          - k8s.pod.name
          - k8s.deployment.name

    # Sample to reduce volume (adjust rate as needed)
    probabilistic_sampler:
      sampling_percentage: 10

  exporters:
    otlp/tempo:
      endpoint: tempo.monitoring:4317
      tls:
        insecure: true

    prometheus:
      endpoint: 0.0.0.0:8889
      namespace: otel

  service:
    pipelines:
      traces:
        receivers: [otlp]
        processors: [k8sattributes, batch]
        exporters: [otlp/tempo]

      metrics:
        receivers: [otlp]
        processors: [batch]
        exporters: [prometheus]

Instrumenting your applications

Tempo and the collector don’t do anything until your apps emit spans. There are two ways in, and you’ll probably use both.

Auto-instrumentation

For a lot of languages OpenTelemetry hooks into the framework for you and you write zero tracing code. This is where I start with any service I didn’t write myself.

Java:

apiVersion: apps/v1
kind: Deployment
spec:
  template:
    spec:
      containers:
        - name: app
          image: my-java-app:latest
          env:
            - name: JAVA_TOOL_OPTIONS
              value: "-javaagent:/otel/opentelemetry-javaagent.jar"
            - name: OTEL_SERVICE_NAME
              value: "order-service"
            - name: OTEL_EXPORTER_OTLP_ENDPOINT
              value: "http://otel-collector.monitoring:4317"
          volumeMounts:
            - name: otel-agent
              mountPath: /otel
      initContainers:
        - name: otel-agent
          image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:latest
          command: [cp, /javaagent.jar, /otel/opentelemetry-javaagent.jar]
          volumeMounts:
            - name: otel-agent
              mountPath: /otel
      volumes:
        - name: otel-agent
          emptyDir: {}

Python:

FROM python:3.11
RUN pip install opentelemetry-distro opentelemetry-exporter-otlp
RUN opentelemetry-bootstrap -a install
CMD ["opentelemetry-instrument", "python", "app.py"]

Node.js:

// tracing.js - require this first
const { NodeSDK } = require('@opentelemetry/sdk-node');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-grpc');
const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node');

const sdk = new NodeSDK({
  traceExporter: new OTLPTraceExporter({
    url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT || 'http://otel-collector:4317',
  }),
  instrumentations: [getNodeAutoInstrumentations()],
  serviceName: process.env.OTEL_SERVICE_NAME || 'my-service',
});

sdk.start();

Auto-instrumentation gets you HTTP, gRPC and database calls for free. It can’t see your business logic. When you want a span around “process this order” with the order ID attached, you write that span yourself.

Manual instrumentation

For your own spans and the attributes you’ll search on:

// Go example
import (
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/attribute"
)

func ProcessOrder(ctx context.Context, orderID string) error {
    tracer := otel.Tracer("order-service")
    ctx, span := tracer.Start(ctx, "process-order")
    defer span.End()

    // Add attributes
    span.SetAttributes(
        attribute.String("order.id", orderID),
        attribute.String("order.type", "standard"),
    )

    // Create child span for sub-operation
    ctx, childSpan := tracer.Start(ctx, "validate-inventory")
    err := validateInventory(ctx, orderID)
    childSpan.End()

    if err != nil {
        span.RecordError(err)
        return err
    }

    return nil
}

Context propagation

Here’s the part that breaks most first attempts. A trace stays one trace only if the trace ID travels with the request to the next service. Drop it on one hop and you get two traces that don’t know about each other.

HTTP headers (automatic with instrumentation):

traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01
tracestate: vendor=value

gRPC metadata (also automatic with instrumentation)

Auto-instrumentation handles this for you. If you build HTTP calls by hand, you inject and extract the context yourself:

// Inject context into outgoing request
req, _ := http.NewRequestWithContext(ctx, "GET", url, nil)
otel.GetTextMapPropagator().Inject(ctx, propagation.HeaderCarrier(req.Header))

// Extract context from incoming request
ctx := otel.GetTextMapPropagator().Extract(r.Context(), propagation.HeaderCarrier(r.Header))

Grafana integration

This is the part that makes tracing worth having. With the right data source config, Grafana jumps from a trace to the matching logs and metrics without you copying a trace ID anywhere.

Add Tempo as a data source:

apiVersion: v1
kind: ConfigMap
metadata:
  name: grafana-datasources
data:
  tempo.yaml: |
    apiVersion: 1
    datasources:
      - name: Tempo
        type: tempo
        url: http://tempo.monitoring:3100
        access: proxy
        jsonData:
          tracesToLogs:
            datasourceUid: loki
            tags: ['app', 'namespace']
          tracesToMetrics:
            datasourceUid: prometheus
            tags: ['service.name']
          serviceMap:
            datasourceUid: prometheus
          nodeGraph:
            enabled: true
          search:
            hide: false
          lokiSearch:
            datasourceUid: loki

Finding a trace

In Grafana Explore:

1. Select Tempo data source
2. Choose “Search” tab
3. Filter by service name, duration, status
4. Click a trace to see the waterfall

From a trace to its logs

With tracesToLogs configured, a span links to the log lines it produced:

1. Open a trace
2. Click a span
3. Click “Logs for this span”
4. See Loki logs with the same trace ID

That click is what the data source wiring buys you. No grep across pods, no guessing which one logged the error.

From a trace to its metrics

The reverse works too. From a slow trace you pivot to the latency histogram and error rate of that service, and you can tell whether you’re looking at one unlucky request or a trend.

Service graph

Tempo can build a service dependency graph from the traces themselves:

# Enable metrics generator in Tempo
tempo:
  metricsGenerator:
    enabled: true
    remoteWriteUrl: http://prometheus.monitoring:9090/api/v1/write

That produces metrics like:

• traces_service_graph_request_total
• traces_service_graph_request_failed_total
• traces_service_graph_request_server_seconds

Grafana renders those as a map of traffic and error rates between services, built from real requests rather than from an architecture diagram nobody updated.

Sampling

Once real traffic hits this, the next problem is volume. Storing every trace at production scale gets expensive fast, so you sample. There are two ways to do it, and the difference matters.

Head sampling

# OTel Collector
processors:
  probabilistic_sampler:
    sampling_percentage: 10  # Keep 10% of traces

Dead simple, with one flaw. The decision is made before the request finishes, so a coin flip can throw away the exact trace that errored. The traces you’d want during an incident are the ones this drops.

Tail sampling

processors:
  tail_sampling:
    decision_wait: 10s
    policies:
      # Always keep errors
      - name: errors
        type: status_code
        status_code:
          status_codes: [ERROR]

      # Always keep slow traces
      - name: slow
        type: latency
        latency:
          threshold_ms: 1000

      # Sample 5% of everything else
      - name: probabilistic
        type: probabilistic
        probabilistic:
          sampling_percentage: 5

Tail sampling waits until the request is done and decides then. Every error kept, everything slow kept, a 5% net over the boring successes. It’s what I run, because the trace I go looking for during an incident is exactly the kind head sampling drops.

What I run in production

Here’s roughly what I run. Object storage on MinIO, tail sampling that keeps errors and slow requests, and the metrics generator on so the service graph comes with it:

# Tempo with object storage
tempo:
  storage:
    trace:
      backend: s3
      s3:
        bucket: tempo-traces
        endpoint: minio.storage:9000
  compactor:
    compaction:
      block_retention: 72h  # 3 days of traces
  metricsGenerator:
    enabled: true
    remoteWriteUrl: http://prometheus:9090/api/v1/write

# OTel Collector with tail sampling
otel-collector:
  config:
    processors:
      tail_sampling:
        policies:
          - name: errors
            type: status_code
            status_codes: [ERROR]
          - name: slow
            type: latency
            threshold_ms: 500
          - name: sample-rest
            type: probabilistic
            sampling_percentage: 5

The 5% keeps the volume manageable, and the metrics generator gives me the dependency map without extra work. Retention is 72 hours, and that sounds short until you notice that nearly every trace you go looking for is from the last hour. Anything that lasts longer than three days shows up in the metrics, and metrics are cheap to keep around for ages.

Debugging with a trace

Back to the checkout that was timing out. With tracing in place, the same problem reads like this:

1. Alert fires: High latency on checkout service
2. Check metrics: P99 latency spiked at 14:32
3. Find traces: Search Tempo for checkout-service, duration > 1s, time range 14:30-14:35
4. Analyze trace: See that payment-service call took 4.2s
5. Drill into span: See db.statement attribute showing slow query
6. Check logs: Jump to Loki logs for that span, see connection pool exhaustion
7. Fix: Increase connection pool size

Steps 3 to 6 used to be guesswork. Now they’re clicks in Grafana.

Why this matters

Microservices split work across teams nicely and make debugging miserable. One request touches ten services, and when it breaks, the logs show you the error text and the metrics show you the symptom. The path the request took, the thing that tells you which service to open, lives in the traces.

Put Prometheus and Thanos for metrics, Loki for logs and Tempo for traces in one Grafana, correlated, on hardware you own. That last part is what I care about most. No vendor sees my request paths, no SaaS bill grows with my trace volume, and when a request fails I can open the box and read what happened to it.

The next time something times out, I search Tempo for the service, sort by duration, open the longest trace and read which span ate the time. Then I click through to its logs.

• Monitoring
• Understanding