Distributed Tracing with Tempo and OpenTelemetry

Your metrics say something is slow. Your logs say errors happened. Great. Now answer me this: which request actually failed, where did the latency come from, and which service in the chain ate the timeout? Metrics and logs both shrug at that.

I hit this wall the first time a checkout flow started timing out under load. Ten services in the path, every one of them green on its own dashboard, and no way to follow a single doomed request from front door to failure. That gap is exactly what distributed tracing fills. It follows one request as it moves through your services and shows you precisely what happened and where it stalled.

The whole point of this blog is understanding what you run instead of guessing. Tracing is the piece that turns “I think it’s the payment service” into “here is the span, here is the slow query.” Let me build it up from nothing.

The Observability Triangle

flowchart TD
    subgraph observability["Complete Observability"]
        M["Metrics<br/>(Prometheus/Thanos)<br/>WHAT is happening"]
        L["Logs<br/>(Loki)<br/>WHY it happened"]
        T["Traces<br/>(Tempo)<br/>WHERE it happened"]
    end

    M <--> L
    L <--> T
    T <--> M

    G["Grafana"] --> M
    G --> L
    G --> T

• Metrics answer: “What is the error rate? What is the latency?”
• Logs answer: “What error message? What was the context?”
• Traces answer: “Which service? Which call? What was the path?”

Each one on its own leaves you half-blind. Wire all three into the same Grafana and you can actually reason about what your cluster did, not what you hope it did.

What is a Trace?

A trace is a tree of spans representing the work done for a single request:

flowchart LR
    subgraph trace["Trace: order-12345"]
        A["API Gateway<br/>250ms"] --> B["Order Service<br/>180ms"]
        B --> C["Inventory Check<br/>45ms"]
        B --> D["Payment Service<br/>120ms"]
        D --> E["Bank API<br/>95ms"]
        B --> F["Notification<br/>15ms"]
    end

Each box is a span. Spans have:

• Name: What operation (e.g., “HTTP GET /orders”)
• Duration: How long it took
• Parent: Which span initiated this one
• Attributes: Key-value metadata (user_id, order_id, etc.)
• Status: Success/error

The trace ID is the thread that stitches every span from the same request together, no matter how many services it crossed.

Why Tempo?

I run Grafana Tempo for a few reasons that line up with how the rest of my stack already works:

1. Cost-effective - Object storage backend, no indexing
2. Simple - No complex cluster management
3. Scalable - Handles massive trace volumes
4. Integrated - Native Grafana support, links to metrics/logs

The big design choice is that Tempo only indexes trace IDs, the same trick Loki pulls for logs. It does not index spans or attributes. That keeps storage cheap, but there is a real cost: you query by trace ID, so you cannot search for “all traces with user_id=123” out of the box.

How do you find a trace ID in the first place, then? You use the other two corners of the triangle. Metrics point you at the bad time window, logs hand you a trace ID, and you jump into Tempo to see the whole path. It sounds like a chore, but in practice the jump is one click in Grafana once you wire it up.

Architecture

flowchart TD
    subgraph apps["Applications"]
        A1["Service A<br/>(instrumented)"]
        A2["Service B<br/>(instrumented)"]
        A3["Service C<br/>(instrumented)"]
    end

    subgraph collector["OpenTelemetry"]
        OC["OTel Collector"]
    end

    A1 -->|"OTLP"| OC
    A2 -->|"OTLP"| OC
    A3 -->|"OTLP"| OC

    OC -->|"traces"| T["Tempo"]
    OC -->|"metrics"| P["Prometheus"]
    OC -->|"logs"| L["Loki"]

    T --> OS["Object Storage"]
    T --> G["Grafana"]
    P --> G
    L --> G

Your applications carry OpenTelemetry SDKs that emit spans. The OTel Collector sits in the middle, receiving telemetry, processing it, and fanning it out to the right backends. Tempo parks the traces in object storage. Grafana is where you actually look at them and correlate against metrics and logs.

Now let me build that up piece by piece, starting with the smallest thing that works.

Installing Tempo

Smallest viable Tempo, via Helm:

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

helm install tempo grafana/tempo \
  --namespace monitoring \
  --values tempo-values.yaml

Basic single-binary deployment, traces on a local volume:

# tempo-values.yaml
tempo:
  storage:
    trace:
      backend: local
      local:
        path: /var/tempo/traces

  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317
        http:
          endpoint: 0.0.0.0:4318

persistence:
  enabled: true
  size: 50Gi

That is enough to start receiving traces. It is also enough to lose them all when the pod’s volume dies, which is fine for a first look and unacceptable for anything you care about. So the next layer swaps local storage for object storage and scales the components out.

# tempo-values.yaml
tempo:
  storage:
    trace:
      backend: s3
      s3:
        bucket: tempo-traces
        endpoint: minio.storage:9000
        access_key: ${MINIO_ACCESS_KEY}
        secret_key: ${MINIO_SECRET_KEY}
        insecure: true

  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317
        http:
          endpoint: 0.0.0.0:4318

  # Retention
  compactor:
    compaction:
      block_retention: 48h

# Distributed mode for scale
distributor:
  replicas: 2
ingester:
  replicas: 3
querier:
  replicas: 2
compactor:
  replicas: 1

I point Tempo at my own MinIO rather than a hosted bucket, so the trace data stays on hardware I own. Same reasoning as the rest of the stack.

Installing OpenTelemetry Collector

The OTel Collector is the pipeline every bit of telemetry flows through before it lands anywhere:

helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts

helm install otel-collector open-telemetry/opentelemetry-collector \
  --namespace monitoring \
  --values otel-collector-values.yaml

Collector configuration:

# otel-collector-values.yaml
mode: deployment
replicaCount: 2

config:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317
        http:
          endpoint: 0.0.0.0:4318

  processors:
    batch:
      timeout: 1s
      send_batch_size: 1024

    # Add Kubernetes metadata
    k8sattributes:
      auth_type: serviceAccount
      extract:
        metadata:
          - k8s.namespace.name
          - k8s.pod.name
          - k8s.deployment.name

    # Sample to reduce volume (adjust rate as needed)
    probabilistic_sampler:
      sampling_percentage: 10

  exporters:
    otlp/tempo:
      endpoint: tempo.monitoring:4317
      tls:
        insecure: true

    prometheus:
      endpoint: 0.0.0.0:8889
      namespace: otel

  service:
    pipelines:
      traces:
        receivers: [otlp]
        processors: [k8sattributes, batch]
        exporters: [otlp/tempo]

      metrics:
        receivers: [otlp]
        processors: [batch]
        exporters: [prometheus]

Instrumenting Applications

Tempo and the collector are useless until your apps actually emit spans. There are two ways in, and you will probably use both.

Auto-Instrumentation (Easy Mode)

For a lot of languages, OpenTelemetry hooks the framework for you and you write zero tracing code. This is where I start with any service I did not write myself.

Java:

apiVersion: apps/v1
kind: Deployment
spec:
  template:
    spec:
      containers:
        - name: app
          image: my-java-app:latest
          env:
            - name: JAVA_TOOL_OPTIONS
              value: "-javaagent:/otel/opentelemetry-javaagent.jar"
            - name: OTEL_SERVICE_NAME
              value: "order-service"
            - name: OTEL_EXPORTER_OTLP_ENDPOINT
              value: "http://otel-collector.monitoring:4317"
          volumeMounts:
            - name: otel-agent
              mountPath: /otel
      initContainers:
        - name: otel-agent
          image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:latest
          command: [cp, /javaagent.jar, /otel/opentelemetry-javaagent.jar]
          volumeMounts:
            - name: otel-agent
              mountPath: /otel
      volumes:
        - name: otel-agent
          emptyDir: {}

Python:

FROM python:3.11
RUN pip install opentelemetry-distro opentelemetry-exporter-otlp
RUN opentelemetry-bootstrap -a install
CMD ["opentelemetry-instrument", "python", "app.py"]

Node.js:

// tracing.js - require this first
const { NodeSDK } = require('@opentelemetry/sdk-node');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-grpc');
const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node');

const sdk = new NodeSDK({
  traceExporter: new OTLPTraceExporter({
    url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT || 'http://otel-collector:4317',
  }),
  instrumentations: [getNodeAutoInstrumentations()],
  serviceName: process.env.OTEL_SERVICE_NAME || 'my-service',
});

sdk.start();

Auto-instrumentation gets you HTTP, gRPC, and DB calls for free. What it cannot see is your business logic, so when you want a span around “process this order” with the order ID attached, you reach for the manual API.

Manual Instrumentation (More Control)

For custom spans and the attributes you actually care about:

// Go example
import (
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/attribute"
)

func ProcessOrder(ctx context.Context, orderID string) error {
    tracer := otel.Tracer("order-service")
    ctx, span := tracer.Start(ctx, "process-order")
    defer span.End()

    // Add attributes
    span.SetAttributes(
        attribute.String("order.id", orderID),
        attribute.String("order.type", "standard"),
    )

    // Create child span for sub-operation
    ctx, childSpan := tracer.Start(ctx, "validate-inventory")
    err := validateInventory(ctx, orderID)
    childSpan.End()

    if err != nil {
        span.RecordError(err)
        return err
    }

    return nil
}

Context Propagation

Here is the gotcha that breaks most first attempts. A trace only stays a single trace if the trace ID travels with the request from one service to the next. Drop that context on a hop and you get two disconnected traces instead of one, with no idea they belonged together.

HTTP headers (automatic with instrumentation):

traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01
tracestate: vendor=value

gRPC metadata (also automatic with instrumentation)

The auto-instrumentation handles propagation for you. If you are hand-rolling HTTP calls, you inject and extract the context yourself:

// Inject context into outgoing request
req, _ := http.NewRequestWithContext(ctx, "GET", url, nil)
otel.GetTextMapPropagator().Inject(ctx, propagation.HeaderCarrier(req.Header))

// Extract context from incoming request
ctx := otel.GetTextMapPropagator().Extract(r.Context(), propagation.HeaderCarrier(r.Header))

Grafana Integration

This is the part that makes the whole triangle pay off. With the right data source config, Grafana lets you jump from a trace straight to the matching logs and metrics without copy-pasting a trace ID anywhere.

Add Tempo as a data source:

apiVersion: v1
kind: ConfigMap
metadata:
  name: grafana-datasources
data:
  tempo.yaml: |
    apiVersion: 1
    datasources:
      - name: Tempo
        type: tempo
        url: http://tempo.monitoring:3100
        access: proxy
        jsonData:
          tracesToLogs:
            datasourceUid: loki
            tags: ['app', 'namespace']
          tracesToMetrics:
            datasourceUid: prometheus
            tags: ['service.name']
          serviceMap:
            datasourceUid: prometheus
          nodeGraph:
            enabled: true
          search:
            hide: false
          lokiSearch:
            datasourceUid: loki

Finding Traces

In Grafana Explore:

1. Select Tempo data source
2. Choose “Search” tab
3. Filter by service name, duration, status
4. Click a trace to see the waterfall

Trace to Logs

With tracesToLogs set up, a span links straight to the logs that came from it:

1. Open a trace
2. Click a span
3. Click “Logs for this span”
4. See Loki logs with the same trace ID

That single click is the payoff for wiring the data sources together. No grepping, no guessing which pod logged what.

Trace to Metrics

The reverse direction works too. From a slow trace you can pivot to the latency histograms and error rates for that service, so you can tell whether you are staring at one unlucky request or a real trend.

Service Graph

Tempo can build a service dependency graph straight from your traces:

# Enable metrics generator in Tempo
tempo:
  metricsGenerator:
    enabled: true
    remoteWriteUrl: http://prometheus.monitoring:9090/api/v1/write

This creates metrics like:

• traces_service_graph_request_total
• traces_service_graph_request_failed_total
• traces_service_graph_request_server_seconds

Grafana renders that as an interactive map of traffic flow and error rates between services, derived from real requests instead of an architecture diagram nobody updated.

Sampling Strategies

Once real traffic hits this, the next problem is volume. Storing every single trace at production scale gets expensive fast, so you sample. There are two flavours, and the difference matters.

Head Sampling (At Collection)

# OTel Collector
processors:
  probabilistic_sampler:
    sampling_percentage: 10  # Keep 10% of traces

Dead simple, and it has one nasty flaw: the decision happens before the request finishes, so a coin flip can throw away the exact trace that errored out. The interesting traces are the ones you most want to keep.

Tail Sampling (After Collection)

processors:
  tail_sampling:
    decision_wait: 10s
    policies:
      # Always keep errors
      - name: errors
        type: status_code
        status_code:
          status_codes: [ERROR]

      # Always keep slow traces
      - name: slow
        type: latency
        latency:
          threshold_ms: 1000

      # Sample 5% of everything else
      - name: probabilistic
        type: probabilistic
        probabilistic:
          sampling_percentage: 5

Tail sampling waits until the request is done, then decides. Keep every error, keep everything slow, throw a 5% net over the boring successes. That is the version I actually run, because the traces you want during an incident are exactly the ones head sampling tends to drop.

My Production Setup

Pulling the layers together, here is roughly what I run. Object storage on MinIO, tail sampling that protects errors and slow requests, and the metrics generator on so I get the service graph for free:

# Tempo with object storage
tempo:
  storage:
    trace:
      backend: s3
      s3:
        bucket: tempo-traces
        endpoint: minio.storage:9000
  compactor:
    compaction:
      block_retention: 72h  # 3 days of traces
  metricsGenerator:
    enabled: true
    remoteWriteUrl: http://prometheus:9090/api/v1/write

# OTel Collector with tail sampling
otel-collector:
  config:
    processors:
      tail_sampling:
        policies:
          - name: errors
            type: status_code
            status_codes: [ERROR]
          - name: slow
            type: latency
            threshold_ms: 500
          - name: sample-rest
            type: probabilistic
            sampling_percentage: 5

Key decisions:

• 72h retention - Enough to debug recent issues
• Tail sampling - Keep all errors and slow traces
• 5% general sampling - Manageable volume
• Service graph - Visual dependency map

Three days of retention sounds short until you notice that almost every trace you go looking for is from the last hour. If an issue survives longer than that, it shows up in metrics, and metrics are cheap to keep around for ages.

Debugging with Traces

Back to that timing-out checkout from the top. Here is how the whole thing reads with tracing in place:

1. Alert fires: High latency on checkout service
2. Check metrics: P99 latency spiked at 14:32
3. Find traces: Search Tempo for checkout-service, duration > 1s, time range 14:30-14:35
4. Analyze trace: See that payment-service call took 4.2s
5. Drill into span: See db.statement attribute showing slow query
6. Check logs: Jump to Loki logs for that span, see connection pool exhaustion
7. Fix: Increase connection pool size

Every one of those steps was guesswork before. Now it is a chain of clicks, and the guessing is gone.

Why This Matters

Microservices are great for splitting work across teams and miserable for debugging. One user request can touch ten services, and when it breaks, logs only show you the error text while metrics only show you the symptom. The causation, the actual path the request took, lives in the traces.

Stack Prometheus and Thanos for metrics, Loki for logs, and Tempo for traces, and you can see all three in one Grafana, correlated, running on hardware you own. That last part is the bit I care about most: no vendor sees your request paths, no SaaS bill scales with your trace volume, and when it breaks at 2am you can actually open the box and understand why.

That is the difference between “works on my machine” and a span that points at a slow query. One is a shrug, the other is an answer.