Prometheus troubleshooting

Find out why Prometheus does not scrape a target, does not load a rule, or does not alert, and propose a fix. Read first, ask before changing. Run the investigation commands freely. Run a command under an "Ask first" line only after the user approves that exact command; if your credentials do not allow it, give it to the user to run. Never run a "Proposed fix"; hand it to the user.

With the Prometheus Operator, a Prometheus custom resource selects ServiceMonitors, PodMonitors, Probes, ScrapeConfigs and PrometheusRules by label and namespace, and the operator turns them into Prometheus' configuration. A ServiceMonitor selects Services by label and names one of their ports. Something can go wrong at each step: the selection, the Service, the port, the scrape itself.

The commands use placeholders: monitoring (where Prometheus runs), my-prometheus (the Prometheus resource), my-namespace, my-app (the ServiceMonitor and its Service) and my-rules. Replace them with the real names. Prometheus' HTTP API is reached through the API server's service proxy on the governing Service prometheus-operated (port name web); this needs get on services/proxy, which the built-in view role does not have. If the proxy is forbidden, say so and ask the user for the Targets page of the Prometheus UI.

Never read Secrets. The operator stores the generated Prometheus and Alertmanager configuration in Secrets (prometheus-<name>, alertmanager-<name>-generated), and the upstream troubleshooting guide decodes them. Do not: use the selection rules, the operator's events and logs, and Prometheus' own API instead. For the same reason never call Prometheus' /api/v1/status/config or Alertmanager's /api/v2/status: they return that generated configuration (credentials masked, but webhook URLs and routes not). Access to the service proxy covers the whole Service, so this rule is yours to keep.

When to use

• A new ServiceMonitor or PodMonitor shows no targets, or the job is missing.
• Targets are down.
• A PrometheusRule's alerts do not appear, do not fire, or do not reach Alertmanager.
• The Prometheus or Alertmanager resource is not Available, or its pods crash.

Hand-offs, by skill name. When the evidence points elsewhere, continue with that skill; if it is not installed, keep going with the steps here and tell the user which skill would help:

• The target's pods are not running or not Ready: k8s-workload-troubleshooting.
• The target is up but a NetworkPolicy or DNS blocks the scrape: k8s-network-troubleshooting.
• Prometheus' volume is full or will not bind: k8s-storage-troubleshooting.

Investigate

Work through the steps in order. Stop when you have the cause and its evidence.

1. Read the Prometheus resource: its status, and which monitors and rules it selects.

kubectl get prometheuses -A
kubectl get prometheus my-prometheus -n monitoring -o jsonpath='{.spec.serviceMonitorSelector}{"\n"}{.spec.serviceMonitorNamespaceSelector}{"\n"}{.spec.podMonitorSelector}{"\n"}{.spec.ruleSelector}{"\n"}{.spec.ruleNamespaceSelector}{"\n"}{.status.conditions}{"\n"}'

2. Read the ServiceMonitor and compare it with the selection and with the Service.

kubectl get servicemonitors,podmonitors -A
kubectl get servicemonitor my-app -n my-namespace -o yaml
kubectl get service my-app -n my-namespace -o jsonpath='{.metadata.labels}{"\n"}{.spec.ports}{"\n"}'
kubectl get endpointslices -n my-namespace -l kubernetes.io/service-name=my-app
kubectl get namespace my-namespace --show-labels

3. Ask Prometheus what it scrapes: the jobs it has, the active targets with their health and last error, and the targets it dropped.

kubectl get --raw '/api/v1/namespaces/monitoring/services/prometheus-operated:web/proxy/api/v1/scrape_pools'
kubectl get --raw '/api/v1/namespaces/monitoring/services/prometheus-operated:web/proxy/api/v1/targets?state=active' | jq -r '.data.activeTargets[] | [.labels.job, .health, .scrapeUrl, .lastError] | @tsv'
kubectl get --raw '/api/v1/namespaces/monitoring/services/prometheus-operated:web/proxy/api/v1/targets?state=dropped' | jq -r '.data.droppedTargets[].discoveredLabels | [.__meta_kubernetes_service_name, .__meta_kubernetes_endpoint_port_name, .__address__] | @tsv'

A ServiceMonitor appears as the job serviceMonitor/<namespace>/<name>/<endpoint index>.

4. Read the operator's events and logs. The operator rejects invalid objects with an event and logs objects it skips.

kubectl events -n my-namespace --for servicemonitor/my-app
kubectl get deployments -A -l app.kubernetes.io/name=prometheus-operator
kubectl logs -n monitoring deployment/prometheus-operator --tail=200 | grep -i -E "skipping|rejected|my-app"

5. Open the matching reference and follow its steps:

Read the results

What the common findings mean (Prometheus Operator 0.94, Prometheus 3.14):

• No job serviceMonitor/<ns>/<name>/... in scrape_pools: the Prometheus does not select the ServiceMonitor. Its labels do not match serviceMonitorSelector, or its namespace does not match serviceMonitorNamespaceSelector. An empty selector ({}) matches everything; an unset (null) selector matches nothing for monitors and rules, and only Prometheus' own namespace for the namespace selectors.
• The job exists, no active targets, a dropped target: the ServiceMonitor matches a Service, but its endpoints[].port is not the name of one of the Service's ports (it must be the port's name, not its number), or the Service has no ready endpoints. The dropped target's __meta_kubernetes_endpoint_port_name shows the real port name.
• The job exists, no targets at all, nothing dropped: spec.selector matches no Service, or namespaceSelector leaves the Service's namespace out.
• A target down with server returned HTTP status 404 Not Found: the path is wrong (endpoints[].path, default /metrics). Other lastError values are in references/targets-down.md.
• An event InvalidConfiguration, "<name>" was rejected due to invalid configuration: ...: the operator refused the object; the message says why.

State the cause in one sentence, then the evidence you saw (the selector and labels, the port names, the target's health and error), then the proposed fix.

Proposed fix

Show each fix to the user before running it: what it changes and how to undo it. Patching a ServiceMonitor, PodMonitor or PrometheusRule (labels, port, path) is reversible, so it goes under "Ask first" and runs only after the user says yes. Changing the Prometheus or Alertmanager resource (selectors, resources, retention), Alertmanager's configuration and anything with Secrets are never run by you: hand them to the user as proposed fixes. Monitors usually come from the app's Helm chart or manifests; fix them there too. Write each undo with the values you found.

Give a ServiceMonitor the label the Prometheus selects:

Ask first: the agent runs this only after the user approves it.

kubectl label servicemonitor my-app -n my-namespace release=my-prometheus
# Undo: kubectl label servicemonitor my-app -n my-namespace release-

Point a ServiceMonitor at the Service's real port name:

Ask first: the agent runs this only after the user approves it.

kubectl patch servicemonitor my-app -n my-namespace --type=json -p '[{"op": "replace", "path": "/spec/endpoints/0/port", "value": "http"}]'
# Undo: kubectl patch servicemonitor my-app -n my-namespace --type=json -p '[{"op": "replace", "path": "/spec/endpoints/0/port", "value": "metrics"}]'

Prometheus picks up the change by itself within a minute or two; check scrape_pools and the targets again.

Sources

Checked against the Prometheus Operator 0.94 and Prometheus 3.14 documentation and source:

• Prometheus Operator troubleshooting: selection, port names, events; its step that decodes the configuration Secret is left out here.
• API reference: selector semantics (empty versus null), ServiceMonitor endpoints.
• RBAC for the CRDs and storage.
• Prometheus HTTP API (targets, scrape pools, rules, alerts, TSDB status), storage and the 3.0 migration guide.
• Manually constructing API server proxy URLs.
• Messages from the source: resource_selector.go, operator.go, scrape.go. The behaviour and messages quoted here were also seen on a Kubernetes 1.37 kind cluster with Prometheus Operator 0.94.2 and Prometheus 3.14.0.