Alerting That Works: From Alert Fatigue to Actionable Notifications

TL;DR Most default alert rules page on a number with no user behind it, and after a week of that you stop reading pages. I alert on error rate, P99 latency, a disk that fills within 24 hours and a certificate under seven days, about ten rules in total, each with a runbook. Alertmanager sends critical to the pager, warning to Slack during office hours, info nowhere. Everything that doesn’t need a human gets deleted.

For a while my alerting was fine. A handful of rules, pages were rare, and when one came in it meant something. Then the cluster grew, I bolted on the Prometheus Operator defaults, and “fine” turned into noise without me noticing.

The page that did it came in at 3 AM. “High CPU usage on node-worker-3.” I looked at the graph, saw the node had been sitting at 75% for ten minutes, and went back to sleep. Next night, same alert. A week later I’d stopped opening them at all.

That’s alert fatigue, and it’s the dangerous kind of broken because it looks like it’s working. The pages keep coming and you keep ignoring them. When everything pages, a real incident lands on the same pile, and the first person to notice is a user.

So which alerts deserve to wake someone up? That’s the only question in this post, and the tooling has little to do with the answer. I’ve run systems on both ends, drowning in pages and systems where a page is rare enough that I trust it the second it fires. The difference between the two is how you decide what gets to interrupt a person.

It’s also the value I keep coming back to on this blog, understanding over trust. An alert whose meaning you can’t explain is a black box wired to your pager. You can’t reason about it, so you learn to ignore it.

Why default alerts make it worse

Most monitoring setups arrive with alerting rules pre-loaded. The Prometheus Operator ships hundreds of them, and they fire on everything:

• CPU > 80%
• Memory > 80%
• Disk > 80%
• Pod restarts > 0
• Any 5xx error

I understand why they ship. Four things still go wrong:

1. No context. Is 80% CPU bad? Depends on the workload. A batch job pinning a core is doing its job.
2. No impact. Is a user affected right now? The alert has no idea, and neither do you, half awake.
3. No action. What am I supposed to do about it? Usually nothing.
4. Too sensitive. A short spike fires an alert that resolves itself before you’ve unlocked your phone.

You end up with pages that don’t need a human, and humans who’ve learned not to trust pages. The two feed each other.

How I decide what gets to page me

A better tool wouldn’t have fixed this. What fixed it was a few rules about what an alert is allowed to be.

1. Alert on symptoms, not causes

Bad: “Pod restarted” Good: “Service error rate > 1%”

Nobody using your service cares whether a pod restarted. They care whether the thing works. So alert on what the user experiences, and go chase the cause after you’ve been woken up for a real reason. A pod that crash-loops while the service stays healthy, because the others picked up the load, is the self-healing I set the cluster up for. I don’t want a page for it.

2. Alert only on what needs a hand

If nobody has to do anything, it has no business paging anyone. Informational stuff goes on a dashboard or into a daily review. Not to your phone at night.

The test I use is one question. “If I get this alert, what do I do?”

• Answer is “investigate and probably nothing” → not an alert
• Answer is “this specific remediation step” → fine, that’s an alert

3. Every alert needs a runbook

When the thing fires, what do you do about it? If you can’t write that down ahead of time, the alert isn’t finished. This is the low-friction principle applied to the version of me that’s half asleep. That version shouldn’t have to reverse-engineer what the daytime version meant.

annotations:
  runbook_url: https://wiki.example.com/runbooks/high-error-rate
  summary: "Error rate exceeds SLO"
  description: "Service {{ $labels.service }} error rate is {{ $value }}%"

4. Severity has to mean something

The first time a critical fires that didn’t need anyone out of bed, you’ve taught yourself that critical doesn’t mean critical. A few of those and the whole scale is worthless. So I guard that label.

Tie alerts to an SLO instead of a threshold

The alerts I trust most map onto a Service Level Objective. The SLO already says how much badness is acceptable, so the alert inherits that meaning instead of guessing at a threshold.

# SLO: 99.9% availability
# Error budget: 0.1% = 43 minutes/month

groups:
  - name: slo-alerts
    rules:
      # Fast burn: consuming error budget quickly
      - alert: HighErrorRateFastBurn
        expr: |
          (
            sum(rate(http_requests_total{status=~"5.."}[5m])) /
            sum(rate(http_requests_total[5m]))
          ) > 0.01
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Error rate exceeds 1% (fast burn)"
          description: "At this rate, monthly error budget exhausts in {{ $value | humanizeDuration }}"

      # Slow burn: steadily consuming error budget
      - alert: HighErrorRateSlowBurn
        expr: |
          (
            sum(rate(http_requests_total{status=~"5.."}[1h])) /
            sum(rate(http_requests_total[1h]))
          ) > 0.001
        for: 1h
        labels:
          severity: warning
        annotations:
          summary: "Error rate elevated (slow burn)"

The two burn rates do different jobs:

• Fast burn means you’re torching the error budget right now. Wake someone up.
• Slow burn means you’re drifting toward an SLO breach over hours. Deal with it during work time.

The alerts I run

Here’s the concrete stuff. Not the whole catalogue, just the categories that have earned a place on my pager.

Availability

# Service is down
- alert: ServiceDown
  expr: up{job="my-service"} == 0
  for: 1m
  labels:
    severity: critical
  annotations:
    summary: "Service {{ $labels.job }} is down"
    runbook_url: https://wiki/runbooks/service-down

# High error rate (user impact)
- alert: HighErrorRate
  expr: |
    sum by (service) (rate(http_requests_total{status=~"5.."}[5m])) /
    sum by (service) (rate(http_requests_total[5m])) > 0.05
  for: 5m
  labels:
    severity: critical
  annotations:
    summary: "{{ $labels.service }} error rate above 5%"

Latency

# P99 latency too high
- alert: HighLatency
  expr: |
    histogram_quantile(0.99,
      sum by (le, service) (rate(http_request_duration_seconds_bucket[5m]))
    ) > 2
  for: 10m
  labels:
    severity: warning
  annotations:
    summary: "{{ $labels.service }} P99 latency above 2s"

Capacity

These two are predictive, which is the point. I’d rather get a warning during office hours that a disk fills tomorrow than a critical page at midnight because it already did.

# Disk filling (predictive)
- alert: DiskFillingUp
  expr: |
    predict_linear(node_filesystem_avail_bytes{fstype!="tmpfs"}[6h], 24*60*60) < 0
  for: 1h
  labels:
    severity: warning
  annotations:
    summary: "Disk {{ $labels.mountpoint }} will fill within 24h"

# Certificate expiring
- alert: CertificateExpiringSoon
  expr: |
    certmanager_certificate_expiration_timestamp_seconds - time() < 7*24*60*60
  for: 1h
  labels:
    severity: warning
  annotations:
    summary: "Certificate {{ $labels.name }} expires in less than 7 days"

What I deliberately don’t alert on

This list matters as much as the one above. Every rule in it is something a default config will happily set up for you, and every one of them is noise.

# DON'T: Generic resource usage
- alert: HighCPU
  expr: node_cpu_seconds_total > 0.8  # Bad: no context

# DON'T: Expected behavior
- alert: PodRestart
  expr: increase(kube_pod_container_status_restarts_total[1h]) > 0  # Bad: restarts are normal

# DON'T: Things that auto-resolve
- alert: PodPending
  expr: kube_pod_status_phase{phase="Pending"} == 1
  for: 1m  # Bad: too short, scheduler will handle it

Routing it all through Alertmanager

Deciding what’s worth a page is half the work. The other half is getting the page to the right place at the right time. Critical goes to the pager, warning goes to Slack and only during work hours, info goes to a log and nowhere else.

# alertmanager.yaml
global:
  resolve_timeout: 5m

route:
  receiver: 'default'
  group_by: ['alertname', 'severity']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h

  routes:
    # Critical: page immediately
    - match:
        severity: critical
      receiver: 'pagerduty'
      continue: true

    # Warning: Slack during work hours
    - match:
        severity: warning
      receiver: 'slack-ops'
      active_time_intervals:
        - work_hours

    # Info: just log
    - match:
        severity: info
      receiver: 'null'

receivers:
  - name: 'pagerduty'
    pagerduty_configs:
      - service_key: '<key>'
        severity: '{{ .CommonLabels.severity }}'

  - name: 'slack-ops'
    slack_configs:
      - api_url: '<webhook>'
        channel: '#ops-alerts'
        title: '{{ .CommonAnnotations.summary }}'
        text: '{{ .CommonAnnotations.description }}'

  - name: 'null'

time_intervals:
  - name: work_hours
    time_intervals:
      - weekdays: ['monday:friday']
        times:
          - start_time: '09:00'
            end_time: '18:00'

Silencing and inhibition

Two more ways to keep the volume down. Silence what you already know about, and suppress the alerts that are downstream symptoms of a bigger one.

Silences for maintenance

# Create silence during maintenance
amtool silence add alertname=~".+" --duration=2h --comment="Planned maintenance"

# Silence specific service
amtool silence add service="api" --duration=1h --comment="Deploying v2.3.0"

Inhibition rules

When one alert implies a pile of others, suppress the pile. If the whole cluster is down, I don’t need forty separate pages telling me each service on it is down too.

inhibit_rules:
  # If cluster is down, don't alert on individual services
  - source_match:
      alertname: 'ClusterDown'
    target_match_re:
      alertname: '.+'
    equal: ['cluster']

  # If node is down, don't alert on pods on that node
  - source_match:
      alertname: 'NodeDown'
    target_match_re:
      alertname: 'Pod.+'
    equal: ['node']

Surfacing alert state in Grafana

Not every alert should page, but you still want to see them. The ones that don’t wake you up live on a dashboard. That’s where info and recently resolved alerts go to get reviewed instead of ignored.

# Prometheus datasource query
ALERTS{alertstate="firing"}

Create an alerts overview panel showing:

• Currently firing alerts
• Recent alert history
• Alert trends over time

Link from alerts to the relevant dashboard:

annotations:
  dashboard_url: https://grafana/d/abc123?var-service={{ $labels.service }}

The human side of on-call

All the config in the world doesn’t help if the rotation around it is a mess. A few things I’ve found worth writing down.

Rotation structure

Primary    → First responder, handles all pages
Secondary  → Backup if primary doesn't respond in 10 min
Escalation → Management, for major incidents

Response time SLAs

Post-incident review

Every critical alert is worth a quick look afterwards, even when the fix was easy:

1. Was the alert necessary?
2. Did it give enough context?
3. Was the runbook helpful?
4. Could the underlying thing be prevented?

If the same alert keeps firing and the answer is always “do nothing”, the alert is broken. Fix it or delete it.

What my setup looks like

# Core SLO alerts only
groups:
  - name: slo
    rules:
      # Availability: error rate
      - alert: HighErrorRate
        expr: error_rate > 0.01
        for: 5m
        severity: critical

      # Latency: P99
      - alert: HighLatency
        expr: p99_latency > 2
        for: 10m
        severity: warning

      # Capacity: predictive
      - alert: DiskFillingUp
        expr: disk_will_fill_24h
        severity: warning

      # Security: cert expiry
      - alert: CertExpiring
        expr: cert_expires_7d
        severity: warning

# That's it. ~10 alert rules total.

About ten rules, and every one of them earns its keep:

• SLO-based, so each alert is tied to something a user would feel.
• Minimal, because a rule I can’t justify is a rule I delete.
• Clear severity. Critical means get out of bed, warning means it waits until morning.
• Runbook on everything, so half-asleep me has a next step that was written down in advance.

Digging out if you’re already buried

If you’re reading this from inside a pile of noisy alerts, you don’t fix it by adding more. You fix it by deleting. Roughly the order I’d go in:

1. Audit what you have. How many alerts fired last month, and how many of those led to anyone doing anything?
2. Delete the dead ones. If a rule hasn’t fired in 90 days, you probably don’t need it, and if it has fired but nobody acted, you definitely don’t.
3. Raise the thresholds that always auto-resolve. If an alert clears itself before you respond, it was set too tight.
4. Add a for duration so brief spikes can’t page you.
5. Merge the near-duplicates. One “high error rate” beats forty per-endpoint variations of the same thing.

Every page needs a human. If it doesn’t, take it off the pager.

Why I bother with all this

A noisy alerting setup looks healthy while it rots your reliability. The engineer on call learns the pages are mostly junk, starts skimming them, and one night skims past the one that mattered. That’s the failure I’m designing against, and no tool fixes it, because the tool did exactly what it was told.

So I keep the volume low enough that every page still gets read, and I only page on things a user would feel.

The observability stack with Loki and Tempo produces far more data than a person can watch. Alerting decides which slice of it gets to interrupt someone. Let too much through and you’re back where I started, ignoring your own phone at 3 AM.

The CPU rule that woke me up was the first one I deleted. What’s left is the ten rules above, each with a runbook I wrote while awake.

• Monitoring
• Prometheus
• Resilience