TL;DR Each dashboard answers one question, and within five seconds I know whether I need to care. Four kinds of dashboard cover everything, variables and links tie them together, and the JSON lives in Git. My homelab runs five, and anything that goes a month unopened gets deleted.
You have Grafana. You have Prometheus metrics. You have logs in Loki and traces in Tempo. The data’s all there.
You also have 47 dashboards nobody opens.
I’ve done this to myself more than once. Something breaks, I bolt a dashboard together to see what’s going on, and it sits there forever. A year of incidents and a few “let me add a panel for that” moments later, most of my Grafana was archaeology. Nobody remembered what half the panels meant, me included. Most of it should go. But before you delete, it helps to know what the survivors have that the rest don’t.
The dashboards I still open have one thing in common. I open one and know within five seconds whether I need to care. That’s the whole bar, and this post is how I get there, from the smallest dashboard worth keeping up to the five I run on my homelab cluster.
There’s a reason I keep coming back to this. A pile of metrics you can’t read is a black box, and this one you built yourself. Observability is supposed to buy understanding. A dashboard nobody can parse buys nothing.
Where dashboards go wrong
The failure modes first. I’ve shipped every one of these.
Too many panels. Thirty panels, thirty metrics, no answers, because nobody reads thirty graphs at a glance. They compete for attention and none of them wins.
Wrong abstraction level. CPU, memory and disk for every individual pod is what you want when you’re elbow-deep in a problem. It’s useless when you want to know if the service is up. Those are different questions, and they want different dashboards.
No context. “500 requests/second.” Good? Bad? Normal? Without something to compare it against, the number is decoration.
Built and forgotten. The incident dashboard from two years ago, still querying services that got renamed in a refactor nobody told it about. Flat lines, and everyone’s learned to look past it.
One dashboard, one question
The rule that fixes most of this is boring. Each dashboard answers exactly one question.
| Question | Dashboard |
|---|---|
| “Is the system healthy?” | Service overview |
| “Why is this service slow?” | Service deep dive |
| “What happened at 3 AM?” | Incident investigation |
| “Are we meeting SLOs ?” | SLO dashboard |
Keep those apart. The moment you bolt a debugging panel onto the health-check dashboard, you’ve started the slide back to 47.
The simplest useful dashboard is “is everything OK?”. A handful of stat panels, big and colour-coded, red when something’s wrong. If that’s the only dashboard you ever build, you’re ahead of most teams I’ve worked with.
Link down instead of cramming it all in
Once the one-glance dashboard works, the next question is “OK, which service?”. Don’t answer that on the same screen. Link to a deeper one. The model I use:
Level 1: Is everything OK? (green/yellow/red)
↓
Level 2: Which service has problems?
↓
Level 3: What specific metric is wrong?
↓
Level 4: Raw data for debugging
Each level is its own dashboard or section, wired together with links. You stop at whichever level answers your question. Most days that’s level 1.
Give every number something to compare against
A raw number tells you almost nothing. The same value is fine on a Tuesday afternoon and an incident in the middle of the night. So I add context wherever I can.
- Thresholds. The panel changes colour when a metric crosses an SLO boundary, so you don’t do the arithmetic in your head.
- Comparisons. Today against last week, this deploy against the previous one.
- Annotations. Deployments, incidents and config changes drawn straight onto the graph.
The annotations have saved me more debugging time than anything else on that list. Latency spikes, there’s a deployment marker right under the spike, done.
Readable in five seconds
If I can’t read a dashboard in five seconds it’s too complex, and I’ll stop opening it, which defeats the point. So I stick to three panel types.
- Stat panels for current state, big numbers, colour-coded
- Time series for trends, so you can see when something started
- Heatmaps for distributions like latency percentiles
And I skip the things that look informative and aren’t. Tables with 50 rows, graphs with 20 overlapping lines, pie charts. Especially pie charts.
The dashboards you need
With the rules in place, these are the dashboards that get a permanent spot. Four kinds. I’ve never needed a fifth kind, only more copies of these four.
Service overview
The “is everything OK?” dashboard, one per service or service group. This is the one you’ll open most.
# Panels (top to bottom, left to right)
Row 1: Status (stat panels)
- Availability (% uptime last 24h)
- Error Rate (current, color-coded)
- P99 Latency (current, color-coded)
- Active Instances (count)
Row 2: Trends (time series)
- Request Rate (stacked by status code)
- Latency (P50, P95, P99 lines)
- Error Rate (percentage over time)
Row 3: Resources (gauges)
- CPU Usage (% of limit)
- Memory Usage (% of limit)
- Pod Count (current vs desired)
The queries behind it:
# Availability (percentage of successful requests)
sum(rate(http_requests_total{status!~"5.."}[5m])) /
sum(rate(http_requests_total[5m])) * 100
# Error rate
sum(rate(http_requests_total{status=~"5.."}[5m])) /
sum(rate(http_requests_total[5m])) * 100
# P99 latency
histogram_quantile(0.99,
sum(rate(http_request_duration_seconds_bucket[5m])) by (le)
)
SLO dashboard
This one tracks Service Level Objectives and tells you whether you’re burning your error budget faster than you can afford. It’s the backbone of meaningful alerting, because you want to page someone when the budget is burning, and you don’t want to page anyone for a single request that hiccups.
Row 1: SLO Status (stat panels)
- Availability SLO (99.9% target)
- Latency SLO (P99 < 500ms target)
- Error Budget Remaining (%)
Row 2: Burn Rate (time series)
- Error budget consumption rate
- Projected budget exhaustion
Row 3: SLI Breakdown (time series)
- Availability over time
- Latency percentiles over time
The error budget query:
# Error budget remaining (for 99.9% SLO over 30 days)
1 - (
(1 - (sum(rate(http_requests_total{status!~"5.."}[30d])) /
sum(rate(http_requests_total[30d])))) /
0.001 # 0.1% error budget
)
Kubernetes cluster dashboard
A bird’s-eye view of cluster health. Resist the urge to put per-pod detail here. That belongs in the debugging dashboard, and mixing the two is how you end up back at thirty panels.
Row 1: Cluster Status
- Nodes Ready / Total
- Pods Running / Scheduled
- PVCs Bound / Total
Row 2: Resource Utilization
- CPU: Used / Requested / Allocatable
- Memory: Used / Requested / Allocatable
Row 3: Workload Health
- Deployments Available / Total
- StatefulSets Ready / Total
- DaemonSets Ready / Total
Row 4: Recent Events (table)
- Warning events from last hour
Debugging dashboard
The one place where complexity is allowed. When you’re investigating an incident you want detail, variables, and the ability to slice by namespace and pod. The five-second rule doesn’t apply when you’re already neck-deep.
Row 1: Service Selection (variables)
- Namespace dropdown
- Service dropdown
- Pod dropdown
Row 2: Request Flow
- Incoming requests by endpoint
- Outgoing requests by destination
- Error breakdown by type
Row 3: Resource Details
- CPU per container (time series)
- Memory per container (time series)
- Network I/O (time series)
Row 4: Logs Integration
- Link to Loki with pre-filtered query
- Recent error log count
Panels worth stealing
The individual panels I reach for again and again. Copy them, point them at your own metrics, done.
Request rate with status codes
sum by (status_code) (
rate(http_requests_total{service="$service"}[5m])
)
Stack the series and colour them by status. 2xx green, 3xx blue, 4xx yellow, 5xx red. When the graph goes red you don’t need the legend.
Latency heatmap
sum(rate(http_request_duration_seconds_bucket{service="$service"}[5m])) by (le)
Render it as a heatmap. You get the full latency distribution over time, and the hot spots show up where an average would have hidden them. A p99 spike that a line graph smooths away is a bright band here.
Resource usage vs limits
# CPU usage percentage of limit
sum(rate(container_cpu_usage_seconds_total{pod=~"$pod"}[5m])) /
sum(kube_pod_container_resource_limits{resource="cpu", pod=~"$pod"}) * 100
A gauge with thresholds at 0-70 green, 70-90 yellow, 90-100 red. Usage as a percentage of the limit rather than raw cores, so the same panel works for every service without retuning.
Pod restart indicator
sum(increase(kube_pod_container_status_restarts_total{namespace="$namespace"}[1h])) by (pod)
A stat panel that only appears when the count is above zero, linked straight to the pod logs. A pod quietly restarting every few minutes is the kind of slow bleed I want to see before it turns into an outage.
Variables, one dashboard for every service
Variables are what stop you cloning the same dashboard fifty times. Define them once and the dashboard re-targets itself from a dropdown. These three cover most of what I need:
# Namespace variable
name: namespace
query: label_values(kube_pod_info, namespace)
multi: true
include_all: true
# Service variable (filtered by namespace)
name: service
query: label_values(kube_pod_info{namespace=~"$namespace"}, pod)
multi: false
# Time comparison
name: comparison
options:
- 1h ago
- 1d ago
- 1w ago
Then reference them in the queries:
# Current value
sum(rate(http_requests_total{namespace="$namespace"}[5m]))
# Comparison value (offset by selected time)
sum(rate(http_requests_total{namespace="$namespace"}[5m] offset $comparison))
That offset $comparison trick gives you a “vs last week” overlay almost for free. It’s the context rule from earlier, in one query.
Annotations, so you stop guessing what changed
Annotations mark events directly on the graphs. Two sources are worth setting up:
# Deployment annotations
name: Deployments
datasource: Prometheus
query: changes(kube_deployment_status_observed_generation{deployment="$service"}[5m]) > 0
tags: deployment
# Alert annotations
name: Alerts
datasource: Alertmanager
filter: service="$service"
tags: alert
Now a latency spike with a deployment marker under it answers “what changed?” before you’ve opened a terminal. Half my incidents turn out to be “we deployed something”, and the marker tells me that straight away.
Wiring dashboards together
Links between dashboards are what make the levels work. The one-glance view hands you to the detail view, and you never hunt through folders.
Data links
On a service panel, link straight to that service’s detail dashboard:
URL: /d/service-detail?var-service=${__field.labels.service}
Title: View ${__field.labels.service} details
Panel links
On the cluster overview, link down into the namespace views:
URL: /d/namespace-view?var-namespace=${namespace}
Title: Drill down to namespace
Explore links
And link to Loki , so going from a metric to the matching logs is one click:
URL: /explore?left=["now-1h","now","Loki",{"expr":"{namespace=\"${namespace}\",pod=~\"${pod}\"}"}]
Title: View logs in Explore
Keeping it organised
Past a handful of dashboards, structure stops being optional. A little discipline here is the difference between finding the right dashboard in two seconds and giving up and querying Prometheus by hand.
Folder structure
├── Overview
│ ├── Platform Health
│ └── SLO Summary
├── Services
│ ├── API Gateway
│ ├── Auth Service
│ └── ...
├── Infrastructure
│ ├── Kubernetes Cluster
│ ├── Nodes
│ └── Storage
├── Debugging
│ ├── Service Debug
│ └── Incident Investigation
└── Archive
└── (old dashboards, hidden)
Naming convention
[Category] - [Subject] - [Type]
Examples:
- Services - API Gateway - Overview
- Services - API Gateway - Debug
- Infrastructure - Kubernetes - Nodes
- SLO - Platform - 30d
Starring and the home dashboard
Set the “is everything OK?” dashboard as the Grafana home, so it’s the first thing you see after logging in, and star the few you open daily. The faster the important ones are to reach, the less you fall back on old habits.
Dashboards as code
Clicking dashboards together in the UI is fine until someone fat-fingers a panel, or the Grafana pod gets rescheduled and you find out the dashboard only lived in its memory. The fix is the same one I apply to everything else. Put it in Git and let ArgoCD deploy it. If I can’t rebuild a dashboard from source, I don’t own it.
Grafana dashboard ConfigMap
apiVersion: v1
kind: ConfigMap
metadata:
name: service-overview-dashboard
labels:
grafana_dashboard: "1"
data:
service-overview.json: |
{
"title": "Service Overview",
"uid": "service-overview",
"panels": [
...
]
}
Drop in a ConfigMap with the grafana_dashboard: "1" label and the sidecar picks it up. The dashboard JSON now lives next to the service it monitors, and it gets reviewed in the same merge request that ships the service.
Grafonnet (Jsonnet library)
Hand-editing dashboard JSON gets old fast. Grafonnet generates it from code, so a shared panel is a function you call instead of JSON you copy and then forget to keep in sync:
local grafana = import 'grafonnet/grafana.libsonnet';
local dashboard = grafana.dashboard;
local prometheus = grafana.prometheus;
dashboard.new(
'Service Overview',
uid='service-overview',
tags=['service', 'overview'],
)
.addPanel(
grafana.statPanel.new(
'Error Rate',
datasource='Prometheus',
).addTarget(
prometheus.target(
'sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) * 100',
)
),
gridPos={ x: 0, y: 0, w: 6, h: 4 },
)
What I run at home
Here’s the whole thing in practice. On my homelab cluster I keep exactly five dashboards:
- Home. Everything OK? One glance.
- SLOs. Am I meeting my objectives, and how’s the error budget?
- Cluster. Kubernetes health from the top down.
- Service Debug. The variable-driven one, for when something’s wrong.
- Incident. The messy one that pulls metrics, logs and traces together.
That’s the lot. I resist a sixth, actively. Every one of these gets opened weekly or more, and that’s the only test I apply. A dashboard that goes a month without being opened gets deleted, and so far I haven’t missed one.
Set that next to the 47 from the top of this post. I’m not collecting less data, my observability stack hoovers up plenty. I’m reading five dashboards I trust instead of ignoring fifty.
Fighting dashboard rot
Five stays five because I keep pruning. Left alone, any Grafana drifts back towards archaeology. A little process keeps it at five.
Review on a schedule
Once a month I go through everything and ask three things. Which dashboards got opened? Do the queries still point at services that exist? Do the thresholds still match how the system behaves? Metrics get renamed, SLOs get tightened, and a threshold that made sense a year ago can quietly stop meaning anything.
Sunset what nobody uses
Not opened in 30 days, it moves to the Archive folder. Sixty days in Archive, it’s gone. Git has the history if I’m wrong.
Write down what each one is for
Every dashboard gets a short description, so the next person knows why it exists. The next person is usually me, six months on, having forgotten everything.
Purpose: Quick health check for production services
Audience: On-call engineers
When to use: Daily check, incident first response
Links to: Service Debug dashboard for details
Why I bother
A dashboard is there to tell you something true about your system, fast enough that you act on it. A bad one is noise, and noise teaches people to ignore the monitoring altogether. That’s worse than no monitoring, because you think you’re covered.
Collecting metrics is easy, and Grafana makes drawing graphs easy, so data was never the bottleneck. Reading it is. The five on my homelab are in Git, each one gets opened every week, and when one stops being opened it goes.