Loki for Kubernetes Logging: The Prometheus-Like Approach

TL;DR Loki indexes labels and stores the log text as compressed chunks, so a month of logs fits on commodity hardware. The trade is full-text search. Below is the single-binary install I’d hand anyone first, the same chart on MinIO with three replicas, and the knobs I didn’t touch until the volume was real.

You’ve got Prometheus for metrics, so you can already see what’s happening across your clusters. Metrics tell you request latency spiked at 14:32. They don’t tell you the payment service threw a null pointer because someone shipped a config change with a typo. For that you need the log line.

The default answer for years was Elasticsearch. It’s powerful and flexible, and it indexes every token in every log line. That full-text index is great until you look at the bill. You pay for it in CPU at ingest, in RAM to keep the index hot, and in storage that grows faster than the logs themselves. I ran an ELK stack in a previous job and spent more time tuning JVM heap sizes than reading logs.

Loki indexes a small set of labels and stores the log text itself as compressed chunks. Same trick that makes Prometheus cheap for metrics, now pointed at logs. You give up arbitrary full-text search and get back a system you can afford to run on commodity hardware.

Why Loki, and what it costs you

Grafana Labs made four choices that explain the rest of this post. Only the labels get indexed, and the log lines are compressed and written to object storage. The labels come straight from Kubernetes metadata, so there’s no mapping config to maintain. The dashboards, alerting and Explore view are the same Grafana you already use for metrics. And there’s no JVM to babysit and no shard rebalancing to wake up for.

The cost is that you can’t grep across all your logs without first narrowing by label. You have to know which namespace, app or pod you care about before you start filtering content. Do I miss that? Only when I don’t know which app logged the thing, and that’s rarer than I expected. Kubernetes debugging starts from “the checkout pods are unhappy” and drills in, and that’s a label selector. If your job is open-ended log analytics where you don’t know what you’re hunting for, that constraint will bite, and I come back to it at the end.

Architecture

Three moving parts.

Promtail runs on each node as a DaemonSet, tails the log files under /var/log/pods, attaches labels, and ships the lines to Loki.

Loki takes those lines, indexes them by label, compresses the bodies into chunks, and writes the chunks to object storage.

Grafana queries Loki with LogQL and shows the results in Explore or on a dashboard.

If you’ve run Prometheus you’ve seen this shape before, collectors feeding a server feeding Grafana, and Grafana Labs copied it on purpose.

One process on one disk

Start with the Grafana Helm charts.

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

# Install Loki with simple scalable deployment
helm install loki grafana/loki \
  --namespace monitoring \
  --create-namespace \
  --values loki-values.yaml

For a homelab or a small cluster, run Loki as a single binary backed by the local filesystem. No object storage, no replication, one process writing chunks to a PVC. This is the version I’d start anyone on, because you see logs in Grafana in about five minutes and can decide whether you like the label model before you wire up MinIO .

# loki-values.yaml
loki:
  auth_enabled: false

  commonConfig:
    replication_factor: 1

  storage:
    type: filesystem

  schemaConfig:
    configs:
      - from: 2024-01-01
        store: tsdb
        object_store: filesystem
        schema: v13
        index:
          prefix: index_
          period: 24h

singleBinary:
  replicas: 1
  persistence:
    size: 50Gi

# Disable components not needed for single binary
backend:
  replicas: 0
read:
  replicas: 0
write:
  replicas: 0

The single binary is enough to learn LogQL on. What it doesn’t survive is the node dying, because the chunks live on that node’s PVC. Once that matters, move the chunks into object storage. Here’s the same chart pointed at S3-compatible storage with a replication factor of three:

# loki-values.yaml
loki:
  auth_enabled: false

  commonConfig:
    replication_factor: 3

  storage:
    type: s3
    s3:
      endpoint: minio.storage:9000
      bucketnames: loki-chunks
      access_key_id: ${MINIO_ACCESS_KEY}
      secret_access_key: ${MINIO_SECRET_KEY}
      insecure: true

  schemaConfig:
    configs:
      - from: 2024-01-01
        store: tsdb
        object_store: s3
        schema: v13
        index:
          prefix: index_
          period: 24h

# Scalable deployment
backend:
  replicas: 3
read:
  replicas: 3
write:
  replicas: 3

Getting logs in with Promtail

Loki doesn’t read anything by itself. Promtail runs on every node and forwards whatever lands in the pod log directory.

helm install promtail grafana/promtail \
  --namespace monitoring \
  --set config.clients[0].url=http://loki:3100/loki/api/v1/push

A more realistic Promtail config does some work on the way in. It parses the CRI log format, drops the noisy filename label, and pulls a few fields out of nginx access logs into labels you can query on.

# promtail-values.yaml
config:
  clients:
    - url: http://loki:3100/loki/api/v1/push

  snippets:
    # Add Kubernetes metadata as labels
    pipelineStages:
      - cri: {}
      - labeldrop:
          - filename
      - match:
          selector: '{app="nginx"}'
          stages:
            - regex:
                expression: '^(?P<remote_addr>[\d\.]+) - (?P<remote_user>\S+) \[(?P<time_local>[^\]]+)\] "(?P<request>[^"]+)" (?P<status>\d+)'
            - labels:
                status:

# DaemonSet tolerations for all nodes
tolerations:
  - operator: Exists

Labels are the index

If you understand one thing about Loki, make it this. Labels are the entire index. They decide how logs get stored and how fast a query returns. A bad label set is how a system that’s supposed to be light on RAM ends up falling over.

Promtail hands you these Kubernetes labels for free:

Now the mistake I made too. Every unique combination of label values creates a separate stream, and Loki keeps an index entry per stream. Add a label with thousands of distinct values and you’ve created thousands of tiny streams. This is high cardinality, and it’s the fastest way to make Loki miserable. A request_id label generates a new stream for every request. That’s millions of streams a day, an index that no longer fits in memory, and queries that crawl.

So a label should be a dimension you’d group or filter by, with a handful of possible values. Anything unbounded stays in the log body, where you filter it with LogQL at query time instead of indexing it.

Good labels (bounded, you filter on them):

• namespace, app, environment, team

Bad labels (unbounded, they belong in the log line):

• request_id, user_id, trace_id, timestamp

Querying with LogQL

LogQL is how you read your logs back. If you know PromQL, you already know most of it. Pick a stream with label selectors, then pipe it through filters and parsers.

Basic queries

# All logs from a namespace
{namespace="production"}

# Specific app
{app="frontend", namespace="production"}

# Multiple containers
{container=~"nginx|envoy"}

# Exclude a namespace
{namespace!="kube-system"}

Filtering content

# Lines containing "error"
{app="frontend"} |= "error"

# Lines NOT containing "health"
{app="frontend"} != "health"

# Regex match
{app="frontend"} |~ "status=(4|5)[0-9]{2}"

# Case insensitive
{app="frontend"} |~ "(?i)error"

Parsing and extracting

# Parse JSON logs
{app="api"} | json

# Extract specific field
{app="api"} | json | status_code >= 500

# Parse with pattern
{app="nginx"} | pattern `<ip> - - [<_>] "<method> <path> <_>" <status>`

# Use extracted fields
{app="nginx"} | pattern `<_> - - [<_>] "<method> <path> <_>" <status>` | status >= 400

This is where keeping request_id out of the labels stops costing you anything. The {...} selector hits the index and narrows you to a few streams. Everything after the pipe runs over just those lines, not your whole log volume, and | json pulls the request id back out at query time.

Aggregations, or metrics from logs

# Count errors per app
sum by (app) (count_over_time({namespace="production"} |= "error" [5m]))

# Rate of requests
sum(rate({app="nginx"} | pattern `<_> "<method> <path> <_>" <status>` [1m])) by (status)

# Bytes per namespace
sum by (namespace) (bytes_over_time({job="kubernetes-pods"}[1h]))

Wiring it into Grafana

The reason I picked Loki over running a separate log stack is that the logs land in the same Grafana I already use for metrics. One data source to add, and the rest of the workflow is identical.

apiVersion: v1
kind: ConfigMap
metadata:
  name: grafana-datasources
  namespace: monitoring
data:
  loki.yaml: |
    apiVersion: 1
    datasources:
      - name: Loki
        type: loki
        url: http://loki:3100
        access: proxy
        jsonData:
          maxLines: 1000

Explore

Explore is where I spend most of my Loki time when something’s on fire. Pick the Loki data source, build the selector in the label browser, add a content filter, and click a line to see the lines around it.

Dashboard panels

Add logs to your dashboards:

{
  "type": "logs",
  "datasource": "Loki",
  "targets": [
    {
      "expr": "{namespace=\"production\", app=\"frontend\"} |= \"error\"",
      "refId": "A"
    }
  ],
  "options": {
    "showTime": true,
    "showLabels": false,
    "wrapLogMessage": true
  }
}

Metrics and logs in one dashboard

# Prometheus panel showing error rate
sum(rate(http_requests_total{status=~"5.."}[5m])) by (app)

# Loki panel showing error logs
{app="$app"} |= "error"

The $app template variable feeds both panels. You spot a 5xx spike in the metric, glance down, and the matching error lines are already there. No tab-switching, no copy-pasting timestamps into a different tool. That view is what I wanted from a log system when I stopped tuning heap sizes.

Alerting on a log line

Once you’re querying logs, you can alert on them too. Sometimes the signal you care about only exists in a log line, not in a metric, and you don’t want to wait for someone to notice. Loki gives you two ways to do this, Grafana alert rules or Loki’s own ruler component.

# Grafana alert rule
apiVersion: 1
groups:
  - name: LogAlerts
    rules:
      - alert: HighErrorRate
        expr: |
          sum(count_over_time({namespace="production"} |= "error" [5m])) > 100
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High error rate in production logs"

The ruler runs the same LogQL inside Loki itself, so the alert fires even if Grafana is down. Handy for the things you can’t miss, like a payment service logging CRITICAL:

# loki-rules.yaml
groups:
  - name: errors
    rules:
      - alert: CriticalError
        expr: |
          count_over_time({app="payment-service"} |= "CRITICAL" [1m]) > 0
        for: 0m
        labels:
          severity: critical
        annotations:
          summary: "Critical error in payment service"

Retention and storage

Every chunk you keep is disk you pay for. Loki’s compactor handles retention and deletes chunks past your cutoff. Thirty days covers almost every “what happened last week” question without the bucket growing without bound.

loki:
  limits_config:
    retention_period: 30d

  compactor:
    working_directory: /var/loki/compactor
    retention_enabled: true
    retention_delete_delay: 2h
    retention_delete_worker_count: 150

If you run multi-tenancy, you can set retention per tenant. Keep production around longer, let dev logs expire fast:

loki:
  limits_config:
    retention_period: 30d  # Default

  overrides:
    production:
      retention_period: 90d  # Keep production logs longer
    development:
      retention_period: 7d   # Dev logs expire faster

Performance tuning

You can run Loki for a long time without touching any of this. These are the knobs you reach for once you’re at real volume and something feels slow.

Chunk size

Bigger chunks mean fewer index entries and better compression, at the cost of higher latency on small queries. It’s a throughput-versus-latency dial:

loki:
  ingester:
    chunk_target_size: 1572864  # 1.5MB
    chunk_idle_period: 30m
    max_chunk_age: 2h

Query limits

Someone will eventually run an unbounded query over six months of logs and wonder why the cluster got slow. Limits stop that before it starts:

loki:
  limits_config:
    max_query_length: 721h           # Max time range
    max_query_parallelism: 32        # Concurrent sub-queries
    max_entries_limit_per_query: 5000

Caching

Repeated queries over the same time range get a lot faster with a memcached in front of the chunk and results paths:

loki:
  memcached:
    chunk_cache:
      enabled: true
      host: memcached.monitoring
    results_cache:
      enabled: true
      host: memcached.monitoring

Structured logging pays off here

Loki rewards you for logging in JSON. A structured line means | json pulls out every field as something you can filter on, no regex archaeology required.

{
  "level": "error",
  "message": "Failed to process order",
  "order_id": "12345",
  "error": "payment declined",
  "duration_ms": 234
}

Now a query like this works, pulling level and duration_ms out of the line at query time:

{app="order-service"} | json | level="error" | duration_ms > 1000

Most runtimes have a one-liner for JSON output:

# Spring Boot
logging.pattern.console: '{"timestamp":"%d","level":"%p","logger":"%c","message":"%m"}%n'

# Node.js with winston
const logger = winston.createLogger({
  format: winston.format.json(),
});

# Go with zap
logger, _ := zap.NewProduction()

The values I run

Here’s everything from above in one config. Object storage on MinIO, three replicas, thirty-day retention, ingest rate limits, and Promtail on every node including the control plane.

# Loki with object storage
loki:
  auth_enabled: false
  commonConfig:
    replication_factor: 3
  storage:
    type: s3
    s3:
      endpoint: minio.storage:9000
      bucketnames: loki-data
  limits_config:
    retention_period: 30d
    ingestion_rate_mb: 10
    ingestion_burst_size_mb: 20

# Three-replica deployment
backend:
  replicas: 3
  persistence:
    size: 50Gi
read:
  replicas: 3
write:
  replicas: 3
  persistence:
    size: 50Gi

# Promtail on all nodes
promtail:
  tolerations:
    - operator: Exists
  config:
    clients:
      - url: http://loki:3100/loki/api/v1/push

The bucket is on MinIO rather than a cloud provider for the reason in Sovereign Infrastructure, which is that my logs shouldn’t depend on a provider I can’t inspect or walk away from. Retention is 30 days because that answers the questions I ask without paying to hoard a year of noise. Three replicas so a node falling over doesn’t take my logs with it. And Promtail on the control plane too, because the logs you need most during an incident are often from the components you forgot to collect.

Loki vs Elasticsearch

Elasticsearch does one thing Loki can’t, which is search you didn’t plan for. The two systems optimise for different questions, so the question to ask is which one you ask more often.

Loki fits when you query by known dimensions like namespace, app and pod, when cost matters, when you want fewer moving parts to operate, and when you’re already in Grafana. Elasticsearch fits when you need full-text search across all logs, when you don’t know what you’re looking for, and when log analytics is a primary use case rather than a debugging aid.

Why I run this one

I can afford to keep a month of logs on hardware I own, the labels match how I already think about my cluster, and it lives in the same Grafana I open every day. When something breaks, the answer is one LogQL query in the panel under the metric that alerted.

Pair it with Prometheus and Thanos for metrics and traces on top, and you’ve got all three signals on infrastructure you can read the config of. The spike at 14:32 from the top of this post gets a |= "error" panel under its graph, and the typo’d config key is in one of those lines.

• Monitoring
• Architecture