Prometheus Troubleshooting

Establish whether the problem is collection, storage, query, or alerting — they look identical from a blank dashboard.

Purpose

Use this when:

• A target is DOWN or missing from /targets.
• A metric that used to exist returns no data.
• A PromQL query returns nothing, or something implausible.
• Prometheus is OOMing, restarting, or using far more memory than expected.
• An alert did not fire during an incident, or fires constantly.

Do not use this when:

• Prometheus is fine and you are debugging the application it monitors — use observability-debugging.
• You are designing alerts and SLOs rather than fixing them — use slo-design.

What this skill assumes: access to the Prometheus UI or API, and ideally the scrape configuration. Many of these checks are answered by Prometheus's own metrics about itself.

Operating Procedure

1. Localise the layer. Is the metric being scraped, stored, queried, or alerted on? Check in that order — each layer depends on the one before.
2. Check the target first. /targets and up answer most "missing metrics" reports in seconds.
3. Query the raw series before the aggregation. A broken rate() on a healthy counter is a query problem, not a data problem.
4. Use Prometheus's own metrics to diagnose Prometheus. prometheus_tsdb_* and scrape_series_added are the real evidence.
5. Fix at the right layer. A missing label is fixed in relabelling; a missing metric is fixed in the application.
6. Verify with the same query that failed.

An empty query result is Observed but ambiguous: it means "no matching series", never "the value is zero". Never report an empty result as a zero.

Investigation

Step 1 - is it being scraped

up{job="<job>"}
up == 0
scrape_samples_scraped{job="<job>"}
scrape_duration_seconds{job="<job>"} > 10

up is synthesised by Prometheus for every scrape: 1 for success, 0 for failure. If up is 0, no metric from that target can exist and nothing else in this skill matters yet.

If up returns no series at all, the target is not in the configuration — a service discovery or relabelling problem, not a scrape failure. These are different faults that produce the same empty dashboard.

Then read /targets in the UI for the exact error:

A /metrics endpoint that takes longer than scrape_timeout produces intermittent gaps rather than a clean failure — sawtooth graphs with no obvious error.

Step 2 - service discovery and relabelling

count by (job) (up)
count by (job, namespace) (up)

Compare against what you expect to exist. In Kubernetes, when a ServiceMonitor or PodMonitor exists but no target appears, work through this order — it is almost always one of them:

1. The selector does not match the Service's labels.
2. The Service port is not named, or the name does not match port: in the monitor.
3. namespaceSelector excludes the namespace.
4. The Prometheus resource's own serviceMonitorSelector does not select your monitor. This one is invisible from the monitor's side and catches people repeatedly.

Check the generated configuration rather than reasoning about the CRDs:

kubectl get secret -n <ns> prometheus-<name> -o jsonpath='{.data.prometheus\.yaml\.gz}' \
  | base64 -d | gunzip | grep -A10 '<job-name>'

This is the configuration Prometheus actually loaded. If your job is not in it, the problem is entirely in the operator's selectors.

Step 3 - the metric itself

{__name__=~"<partial_name>.*"}
count({__name__=~".+", job="<job>"}) by (__name__)

Confirm the metric name, including the suffix. _total on counters and _bucket/_sum/_count on histograms are real parts of the name, and querying http_requests when the metric is http_requests_total returns nothing with no error.

rate(http_requests_total[5m])

rate() returning empty on a series that clearly exists has three standard causes:

1. Fewer than two samples in the window. With a 60s scrape interval, [1m] may contain one point. Rule of thumb: the range must be at least 4x the scrape interval.
2. Applied to a gauge. rate() is only defined for counters; on a gauge it produces nonsense or nothing.
3. The series is stale. Prometheus marks a series stale 5 minutes after the last sample, and it disappears from instant queries.

Step 4 - cardinality and memory

topk(10, count by (__name__) ({__name__=~".+"}))
topk(10, count by (job) ({__name__=~".+"}))
sum(scrape_series_added) by (job)
prometheus_tsdb_head_series

prometheus_tsdb_head_series is the total number of active series and the primary driver of Prometheus memory. Rough guidance: budget a few KB of RAM per active series, so a million series needs several GB before query overhead.

High cardinality comes from putting unbounded values in labels. The usual offenders, in order:

A metric with 20 labels of 10 values each is 10^20 possible series. Cardinality is multiplicative, which is why it goes from fine to fatal with one added label.

sum(rate(prometheus_target_scrapes_exceeded_sample_limit_total[5m]))

Non-zero means a target is being rejected for exposing too many series — a protection working correctly, and a signal to fix the application's labels.

Step 5 - queries and alerts

prometheus_engine_query_duration_seconds{quantile="0.99"}
ALERTS{alertstate="firing"}
ALERTS_FOR_STATE

For an alert that did not fire, check in this order:

1. Does the expression return data now? Run it in the UI.
2. Was it returning data at the time? An alert on rate(errors[5m]) > 0.05 never fires if the errors metric disappears entirely when the service dies — an absent series is not greater than anything. This is the most dangerous alerting bug there is, and the fix is an absent() or up == 0 alert alongside it.
3. Is for: longer than the incident lasted?
4. Did Alertmanager receive it and suppress it — inhibition, silence, or grouping?

curl -sS http://<alertmanager>:9093/api/v2/silences | jq -r '.[] | select(.status.state=="active") | .comment'

A forgotten silence from a previous incident is a common reason an alert "did not fire".

Decision Tree

Metric or alert problem
├── up == 0            → scrape failing → read the error on /targets
├── up has no series   → target not discovered → selectors and relabelling
├── up == 1 but metric missing
│   ├── Wrong name or suffix       → check with __name__ regex
│   ├── Not exposed by the app     → curl the /metrics endpoint directly
│   └── Dropped by metric_relabel_configs → read the generated config
├── Series exists, rate() empty
│   ├── Range < 4x scrape interval → widen the range
│   ├── Metric is a gauge          → use delta() / deriv() or no function
│   └── Series stale (>5 min old)  → the target stopped exposing it
├── Prometheus OOM / slow
│   ├── head_series growing        → cardinality → find the metric and the label
│   └── Query duration high        → unbounded queries or too-long ranges
└── Alert did not fire
    ├── Expression returns nothing when the service is down → needs absent()/up==0
    ├── for: longer than the event
    └── Alertmanager silence, inhibition, or grouping

Common Causes

Safety

Read-only

All PromQL queries, /targets, /api/v1/status/config, and reading the generated scrape configuration. Note that a very expensive query — an unbounded regex over a long range — can itself put a strained Prometheus under enough load to affect alert evaluation. Bound your queries during an incident.

Potentially disruptive

• Reloading Prometheus configuration (POST /-/reload or SIGHUP) — brief, but a bad configuration fails to load and leaves the previous one running, which is confusing under pressure. Validate with promtool check config first.
• Restarting Prometheus — on a large TSDB, replaying the WAL can take minutes, during which there is no alerting at all. Never restart Prometheus during an incident without telling the incident channel that alerting is offline.
• Adding a metric_relabel_configs drop rule — stops collection of that metric immediately and permanently. The historical data remains; nothing new is added. Confirm nothing alerts on it.
• Creating an Alertmanager silence — suppresses notifications. Always set an explicit expiry; an indefinite silence is how the next incident goes unnoticed.
• Changing retention or storage settings — restarts the server and can delete data outside the new retention window.

Destructive

• Deleting time-series data via the admin API (/api/v1/admin/tsdb/delete_series, followed by clean_tombstones) — permanently removes history. It is disabled unless --web.enable-admin-api is set, which is a deliberate choice. Required first: confirm the series selector matches only what you intend (run it as a normal query and check the count), confirm nobody needs the history for an ongoing investigation, and get explicit approval.
• Deleting the Prometheus PersistentVolumeClaim to "reset" a broken server — destroys all historical metrics. Required first: confirm remote-write or a long-term store holds the data, or accept the permanent loss in writing. A Prometheus that will not start is nearly always fixable by other means; check disk space and WAL corruption messages before considering this.

Verification

Before: 0/12 targets up for job=api; dashboards empty; no alerts firing
After:  12/12 targets up; rate(http_requests_total[5m]) returns 12 series; test alert fired

count(up{job="<job>"} == 1)
rate(<metric>[5m])
prometheus_tsdb_head_series

Verify all four layers, because fixing one commonly reveals the next:

1. Scrape — up == 1 and scrape_samples_scraped non-zero.
2. Storage — the series is queryable over a range, not just instantaneously.
3. Query — the exact expression from the dashboard or alert returns what you expect.
4. Alert — force the condition in a non-production environment and confirm the notification actually arrives. An alert rule that evaluates correctly but routes nowhere is not working.

Anti-Patterns

Bad: The graph is empty, so the service is down. Better: Check up first. An empty graph much more often means a scrape or query problem than a dead service — and treating a monitoring failure as an outage wastes the first ten minutes of an incident.

Bad: Add a label so we can break the metric down by user. Better: Cardinality is multiplicative and unbounded labels take down the Prometheus server for everyone. Per-user breakdowns belong in logs or traces.

Bad: Alert on rate(http_errors_total[5m]) > 0.05. Better: If the process dies, that series disappears and the alert silently stops firing. Pair every such alert with absent() or up == 0, or express it as a ratio with a denominator that also disappears.

Bad: Restart Prometheus when data looks wrong. Better: WAL replay can take minutes with no alerting. Diagnose first; a restart rarely fixes a scrape configuration problem anyway.

Bad: Increase scrape interval to reduce load. Better: That reduces resolution everywhere and breaks rate() over short ranges. Reduce cardinality instead — it is nearly always where the cost is.

Bad: Silence the noisy alert. Better: A noisy alert is either mistuned or reporting something real that nobody is fixing. Silencing it indefinitely removes the signal permanently. Fix the threshold, or delete the alert on purpose.

Composition

• observability-debugging — the metrics work and the question is what they mean.
• slo-design — designing alerts that are worth firing.
• kubernetes-troubleshooting — Prometheus itself is a pod and can fail like any other.
• kubernetes-resource-debugging — Prometheus OOMing because of its memory limit.

References

• Prometheus: Querying basics
• Prometheus: Instrumentation best practices — including what not to put in a label.
• Prometheus Operator: troubleshooting ServiceMonitors