Loki for Kubernetes Logging: The Prometheus-Like Approach

You’ve got Prometheus for metrics, so you can already see what’s happening across your clusters. Metrics tell you a request latency spiked at 14:32. They don’t tell you the payment service threw a null pointer because someone shipped a config change with a typo. For that you need logs.

The default answer for years was Elasticsearch. It’s powerful and flexible, and it indexes every single token in every log line. That full-text index is great until you look at the bill. You pay for it in CPU at ingest, in RAM to keep the index hot, and in storage that grows faster than your actual log volume. I ran an ELK stack in a previous job and spent more time tuning JVM heap sizes than reading logs.

Loki flips the model. Instead of indexing the content of every line, it indexes a small set of labels and stores the log text itself as compressed chunks. Same trick that makes Prometheus cheap for metrics, now pointed at logs. You give up arbitrary full-text search and you get back a system you can actually afford to run on commodity hardware.

This post builds Loki up in layers: the smallest thing that works, then a production setup, then the tuning knobs you reach for once you’re running it for real. Stop reading whenever you have enough.

Why Loki, and what it costs you

Grafana Labs built Loki around a few deliberate choices:

Cost efficient - Only the labels get indexed. Log lines are compressed and dropped into object storage.
Kubernetes native - Labels come straight from Kubernetes metadata. No mapping config to maintain.
Grafana integration - Same dashboards, same alerting, same Explore view you already use for metrics.
Operationally simple - No JVM to babysit, no shard rebalancing at 3am.

The honest trade-off: you can’t grep across all your logs without first narrowing by label. You have to know which namespace, app, or pod you care about before you start filtering content. For Kubernetes debugging that’s almost always how you work anyway. You start from “the checkout pods are unhappy” and drill in. If your actual job is open-ended log analytics where you don’t know what you’re hunting for, that constraint will bite, and I’ll come back to it at the end.

Architecture

flowchart TD
    subgraph cluster["Kubernetes Cluster"]
        subgraph nodes["Nodes"]
            P1["Promtail<br/>(DaemonSet)"]
            P2["Promtail"]
            P3["Promtail"]
        end
        PODS["Pod Logs<br/>(/var/log/pods)"]
    end

    P1 --> PODS
    P2 --> PODS
    P3 --> PODS

    P1 --> L["Loki"]
    P2 --> L
    P3 --> L

    L --> OS["Object Storage<br/>(chunks)"]
    L --> G["Grafana"]

Three moving parts, and that’s the whole thing:

Promtail runs on each node as a DaemonSet, tails the log files under /var/log/pods, attaches labels, and ships the lines to Loki.

Loki takes those lines, indexes them by label, compresses the bodies into chunks, and writes the chunks to object storage.

Grafana queries Loki with LogQL and shows the results in Explore or on a dashboard.

If you’ve run Prometheus, this shape will feel familiar. That’s the point.

The simplest version that works

Start with the Grafana Helm charts.

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

# Install Loki with simple scalable deployment
helm install loki grafana/loki \
  --namespace monitoring \
  --create-namespace \
  --values loki-values.yaml

For a homelab or a small cluster, run Loki as a single binary backed by the local filesystem. No object storage, no replication, just one process writing chunks to a PVC. This is the version I’d start anyone on, because it lets you see logs in Grafana in about five minutes and decide whether you like the model before you wire up MinIO.

# loki-values.yaml
loki:
  auth_enabled: false

  commonConfig:
    replication_factor: 1

  storage:
    type: filesystem

  schemaConfig:
    configs:
      - from: 2024-01-01
        store: tsdb
        object_store: filesystem
        schema: v13
        index:
          prefix: index_
          period: 24h

singleBinary:
  replicas: 1
  persistence:
    size: 50Gi

# Disable components not needed for single binary
backend:
  replicas: 0
read:
  replicas: 0
write:
  replicas: 0

That single binary is a fine place to stop if you’re just learning. Once you want logs to survive a node dying, move the chunks off the local disk and into object storage. Here’s the same chart pointed at S3-compatible storage with a replication factor of three:

# loki-values.yaml
loki:
  auth_enabled: false

  commonConfig:
    replication_factor: 3

  storage:
    type: s3
    s3:
      endpoint: minio.storage:9000
      bucketnames: loki-chunks
      access_key_id: ${MINIO_ACCESS_KEY}
      secret_access_key: ${MINIO_SECRET_KEY}
      insecure: true

  schemaConfig:
    configs:
      - from: 2024-01-01
        store: tsdb
        object_store: s3
        schema: v13
        index:
          prefix: index_
          period: 24h

# Scalable deployment
backend:
  replicas: 3
read:
  replicas: 3
write:
  replicas: 3

Getting logs in with Promtail

Loki on its own is an empty bucket. Promtail is what fills it. It runs on every node and forwards whatever lands in the pod log directory.

helm install promtail grafana/promtail \
  --namespace monitoring \
  --set config.clients[0].url=http://loki:3100/loki/api/v1/push

A more realistic Promtail config does a bit of work on the way in: parse the CRI log format, drop the noisy filename label, and pull a few fields out of nginx access logs into labels you can query on.

# promtail-values.yaml
config:
  clients:
    - url: http://loki:3100/loki/api/v1/push

  snippets:
    # Add Kubernetes metadata as labels
    pipelineStages:
      - cri: {}
      - labeldrop:
          - filename
      - match:
          selector: '{app="nginx"}'
          stages:
            - regex:
                expression: '^(?P<remote_addr>[\d\.]+) - (?P<remote_user>\S+) \[(?P<time_local>[^\]]+)\] "(?P<request>[^"]+)" (?P<status>\d+)'
            - labels:
                status:

# DaemonSet tolerations for all nodes
tolerations:
  - operator: Exists

The one thing to get right: labels

If you understand one thing about Loki, make it this. Labels are the entire index. They decide how logs get stored and how fast your queries are. Get them right and Loki is fast and cheap. Get them wrong and you’ll wonder why a system that’s supposed to be lightweight is falling over.

Promtail hands you these Kubernetes labels for free:

Label	Source	Example
`namespace`	Pod namespace	`default`, `monitoring`
`pod`	Pod name	`nginx-abc123`
`container`	Container name	`nginx`, `sidecar`
`node_name`	Node	`worker-1`
`app`	Pod label	`nginx`
`job`	Scrape config	`kubernetes-pods`

Now the gotcha that catches everyone, me included. Every unique combination of label values creates a separate stream, and Loki keeps an index entry per stream. Add a label with thousands of distinct values and you’ve just created thousands of tiny streams. This is high cardinality, and it’s the fastest way to make Loki miserable. A request_id label will generate a new stream for every request. That’s millions of streams a day, an index that no longer fits in memory, and queries that crawl.

The rule of thumb: a label should be a dimension you’d group or filter by, with a handful of possible values. Anything unbounded stays in the log body, where you filter it with LogQL at query time instead of indexing it.

Good labels (bounded, you filter on them):

namespace, app, environment, team

Bad labels (unbounded, they belong in the log line):

request_id, user_id, trace_id, timestamp

Querying with LogQL

LogQL is how you actually read your logs back. If you know PromQL, you already know most of it: pick a stream with label selectors, then pipe it through filters and parsers.

Basic Queries

# All logs from a namespace
{namespace="production"}

# Specific app
{app="frontend", namespace="production"}

# Multiple containers
{container=~"nginx|envoy"}

# Exclude a namespace
{namespace!="kube-system"}

Filtering Content

# Lines containing "error"
{app="frontend"} |= "error"

# Lines NOT containing "health"
{app="frontend"} != "health"

# Regex match
{app="frontend"} |~ "status=(4|5)[0-9]{2}"

# Case insensitive
{app="frontend"} |~ "(?i)error"

Parsing and Extracting

# Parse JSON logs
{app="api"} | json

# Extract specific field
{app="api"} | json | status_code >= 500

# Parse with pattern
{app="nginx"} | pattern `<ip> - - [<_>] "<method> <path> <_>" <status>`

# Use extracted fields
{app="nginx"} | pattern `<_> - - [<_>] "<method> <path> <_>" <status>` | status >= 400

This is where the labels-plus-filter split pays off. The {...} selector hits the index and narrows you to a few streams instantly. Everything after the pipe runs over just those lines, not your whole log volume. That’s why keeping request_id out of the labels doesn’t actually cost you anything: you parse it back out at query time with | json.

Aggregations (Log Metrics)

# Count errors per app
sum by (app) (count_over_time({namespace="production"} |= "error" [5m]))

# Rate of requests
sum(rate({app="nginx"} | pattern `<_> "<method> <path> <_>" <status>` [1m])) by (status)

# Bytes per namespace
sum by (namespace) (bytes_over_time({job="kubernetes-pods"}[1h]))

Wiring it into Grafana

The reason I picked Loki over running a separate log stack is right here: the logs land in the same Grafana I already use for metrics. One data source to add, and the rest of the workflow is identical.

apiVersion: v1
kind: ConfigMap
metadata:
  name: grafana-datasources
  namespace: monitoring
data:
  loki.yaml: |
    apiVersion: 1
    datasources:
      - name: Loki
        type: loki
        url: http://loki:3100
        access: proxy
        jsonData:
          maxLines: 1000

Explore View

Explore is where I spend most of my Loki time when something’s on fire:

Select Loki data source
Build query with label browser
Filter with content matches
Click on log lines for context

Dashboard Panels

Add logs to your dashboards:

{
  "type": "logs",
  "datasource": "Loki",
  "targets": [
    {
      "expr": "{namespace=\"production\", app=\"frontend\"} |= \"error\"",
      "refId": "A"
    }
  ],
  "options": {
    "showTime": true,
    "showLabels": false,
    "wrapLogMessage": true
  }
}

Correlating Metrics and Logs

This is the payoff. One dashboard, metrics and logs side by side.

# Prometheus panel showing error rate
sum(rate(http_requests_total{status=~"5.."}[5m])) by (app)

# Loki panel showing error logs
{app="$app"} |= "error"

The $app template variable feeds both panels. You spot a 5xx spike in the metric, glance down, and the matching error lines are already there. No tab-switching, no copy-pasting timestamps into a different tool.

Advanced patterns: alerting on logs

Once you’re querying logs, you can alert on them too. Sometimes the signal you care about only exists in a log line, not in a metric, and you don’t want to wait for someone to notice. Loki gives you two ways to do this: Grafana alert rules, or Loki’s own ruler component.

# Grafana alert rule
apiVersion: 1
groups:
  - name: LogAlerts
    rules:
      - alert: HighErrorRate
        expr: |
          sum(count_over_time({namespace="production"} |= "error" [5m])) > 100
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High error rate in production logs"

The ruler runs the same LogQL inside Loki itself, so the alert fires even if Grafana is down. Handy for the things you really can’t miss, like a payment service logging CRITICAL:

# loki-rules.yaml
groups:
  - name: errors
    rules:
      - alert: CriticalError
        expr: |
          count_over_time({app="payment-service"} |= "CRITICAL" [1m]) > 0
        for: 0m
        labels:
          severity: critical
        annotations:
          summary: "Critical error in payment service"

Retention and storage

Logs that live forever are logs you’re paying to store forever. Loki’s compactor handles retention, deleting chunks past your cutoff. Thirty days covers almost every “what happened last week” question without your storage growing without bound.

loki:
  limits_config:
    retention_period: 30d

  compactor:
    working_directory: /var/loki/compactor
    retention_enabled: true
    retention_delete_delay: 2h
    retention_delete_worker_count: 150

If you run multi-tenancy, you can set retention per tenant. Keep production around longer, let dev logs expire fast:

loki:
  limits_config:
    retention_period: 30d  # Default

  overrides:
    production:
      retention_period: 90d  # Keep production logs longer
    development:
      retention_period: 7d   # Dev logs expire faster

Performance tuning

You can run Loki for a long time without touching any of this. These are the knobs you reach for once you’re at real volume and something feels slow.

Chunk Size

Bigger chunks mean fewer index entries and better compression, at the cost of higher latency on small queries. It’s a throughput-versus-latency dial:

loki:
  ingester:
    chunk_target_size: 1572864  # 1.5MB
    chunk_idle_period: 30m
    max_chunk_age: 2h

Query Limits

Someone will eventually run an unbounded query over six months of logs and wonder why the cluster got slow. Limits stop that before it starts:

loki:
  limits_config:
    max_query_length: 721h           # Max time range
    max_query_parallelism: 32        # Concurrent sub-queries
    max_entries_limit_per_query: 5000

Caching

Repeated queries over the same time range get a lot faster with a memcached in front of the chunk and results paths:

loki:
  memcached:
    chunk_cache:
      enabled: true
      host: memcached.monitoring
    results_cache:
      enabled: true
      host: memcached.monitoring

Structured logging pays off here

Loki rewards you for logging in JSON. A structured line means | json pulls out every field as something you can filter on, no regex archaeology required.

{
  "level": "error",
  "message": "Failed to process order",
  "order_id": "12345",
  "error": "payment declined",
  "duration_ms": 234
}

Now a query like this just works, pulling level and duration_ms out of the line at query time:

{app="order-service"} | json | level="error" | duration_ms > 1000

Most runtimes have a one-liner for JSON output:

# Spring Boot
logging.pattern.console: '{"timestamp":"%d","level":"%p","logger":"%c","message":"%m"}%n'

# Node.js with winston
const logger = winston.createLogger({
  format: winston.format.json(),
});

# Go with zap
logger, _ := zap.NewProduction()

The full picture: my production setup

Here’s everything from the layers above pulled into one config. Object storage on MinIO, three replicas, thirty-day retention, ingest rate limits, and Promtail on every node including the control plane.

# Loki with object storage
loki:
  auth_enabled: false
  commonConfig:
    replication_factor: 3
  storage:
    type: s3
    s3:
      endpoint: minio.storage:9000
      bucketnames: loki-data
  limits_config:
    retention_period: 30d
    ingestion_rate_mb: 10
    ingestion_burst_size_mb: 20

# Three-replica deployment
backend:
  replicas: 3
  persistence:
    size: 50Gi
read:
  replicas: 3
write:
  replicas: 3
  persistence:
    size: 50Gi

# Promtail on all nodes
promtail:
  tolerations:
    - operator: Exists
  config:
    clients:
      - url: http://loki:3100/loki/api/v1/push

Why each of those:

Object storage on MinIO, not a cloud bucket. Same reason I self-host everything else, covered in Sovereign Infrastructure: my logs shouldn’t depend on a provider I can’t inspect or walk away from.
30-day retention, because that answers the questions I actually ask without paying to hoard a year of noise.
Three replicas, so a node falling over doesn’t take my logs with it. Resilience is a setting here, not a project.
Promtail on all nodes, control plane included, because the logs you need most during an incident are often from the components you forgot to collect.

Loki vs Elasticsearch

Aspect	Loki	Elasticsearch
Indexing	Labels only	Full-text
Storage cost	Lower	Higher
Query flexibility	Label-first	Full-text search
Operations	Simpler	Complex
Memory usage	Lower	Higher (JVM)
Grafana integration	Native	Good

I’m not here to tell you Elasticsearch is bad. It does something Loki genuinely can’t, which is search you didn’t plan for. The two systems optimise for different questions, so the honest answer is “which question do you ask more often?”

Choose Loki when:

You query by known dimensions (namespace, app, pod)
Cost matters
You want operational simplicity
You’re already in the Grafana ecosystem

Choose Elasticsearch when:

You need full-text search across all logs
You don’t know what you’re looking for
Log analytics is a primary use case

Why this matters

Remember where this started: metrics told you something broke, and you needed the logs to find out why. The reason I run Loki instead of a heavier stack comes down to what it asks of me. I can afford to keep a month of logs on commodity hardware, the labels match how I already think about my cluster, and it lives in the same Grafana I open every day. When I need to understand a failure, the answer is one query away, not one JVM heap dump away.

Pair it with Prometheus and Thanos for metrics and traces on top, and you’ve got the full observability picture on infrastructure you own and can reason about. That last part is the whole point. A logging system you can’t afford to run, or can’t understand when it misbehaves, is just another black box bolted onto the ones you were trying to debug.

Why Loki, and what it costs you#

Architecture#

The simplest version that works#

Getting logs in with Promtail#

The one thing to get right: labels#

Querying with LogQL#

Basic Queries#

Filtering Content#

Parsing and Extracting#

Aggregations (Log Metrics)#

Wiring it into Grafana#

Explore View#

Dashboard Panels#

Correlating Metrics and Logs#

Advanced patterns: alerting on logs#

Retention and storage#

Performance tuning#

Chunk Size#

Query Limits#

Caching#

Structured logging pays off here#

The full picture: my production setup#

Loki vs Elasticsearch#

Why this matters#