agentleFS
k8s-servantk8s-servant/k8s-skills/skills/k8s-grafana-troubleshooting/SKILL.md

k8s-grafana-troubleshooting

Skill38 starsChanged 17 days agoApache-2.0View on GitHub

Use when Grafana on Kubernetes fails: the Grafana pod crash-loops after a change to its provisioning ("Datasource provisioning error", "Only one datasource per organization can be marked as default", "failed to parse"), a provisioned dashboard or datasource does not appear ("failed to load dashboard from", sidecar ConfigMap labels), panels show "No data" or the datasource returns "Bad Gateway" or "connection refused", "database is locked" with several replicas, permission errors on the data volume, admin login or sidecar "401 Unauthorized", or grafana-operator resources with "NoMatchingInstance". Investigates with kubectl get, describe, events and logs, never reading Grafana's Secrets, and proposes fixes. Reads first and asks before changing anything.

The file

We split this skill into the rules it gives an agent

3 rules. About 0 are common, 1 is distinctive, and 2 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈1

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 2

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Never run a "Proposed fix".

    line 26
  2. Very specific:

    Never read Secrets: not the admin password, not datasource credentials, not grafana.ini values kept in Secrets.

    line 39
  3. Very specific:

    Avoid kubectl describe pod here: it prints the containers' environment values, and Grafana pods often carry passwords there.

    line 76

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Grafana troubleshooting

Find out why Grafana does not start, does not show a datasource or dashboard, or shows no data, and propose a fix. Read first, ask before changing. Run the investigation commands freely. Run a command under an "Ask first" line only after the user approves that exact command; if your credentials do not allow it, give it to the user to run. Never run a "Proposed fix"; hand it to the user.

On Kubernetes, Grafana is usually provisioned: datasources and dashboard providers are YAML files under /etc/grafana/provisioning/, and dashboards are JSON files, all mounted from ConfigMaps. With the Grafana Helm chart (also inside kube-prometheus-stack) a sidecar container copies ConfigMaps labelled grafana_dashboard or grafana_datasource into the pod. With the grafana-operator, Grafana, GrafanaDashboard and GrafanaDatasource resources do the same.

The commands use placeholders: monitoring (where Grafana runs), my-grafana (its Deployment and Service), my-grafana-datasources and my-grafana-dashboards (its ConfigMaps) and my-app (a dashboard). Replace them with the real names.

Never read Secrets: not the admin password, not datasource credentials, not grafana.ini values kept in Secrets. The sidecar can also read Secrets with the same labels; look only at ConfigMaps. Do not log in to Grafana's API with the admin password; its /api/health needs no login.

When to use

  • The Grafana pod is in CrashLoopBackOff or not Ready, often right after a change.
  • A provisioned datasource or dashboard is missing in the UI.
  • Panels show "No data", or a datasource test fails.
  • Logins fail, or the sidecar logs 401 Unauthorized.
  • grafana-operator resources do not sync.

Hand-offs, by skill name. When the evidence points elsewhere, continue with that skill; if it is not installed, keep going with the steps here and tell the user which skill would help:

  • Grafana runs and reaches Prometheus, but Prometheus has no data for the query: k8s-prometheus-troubleshooting.
  • The datasource's Service exists but traffic does not reach it (NetworkPolicy, DNS): k8s-network-troubleshooting.
  • The pod is Pending or its volume does not mount: k8s-workload-troubleshooting, k8s-storage-troubleshooting.

Investigate

Work through the steps in order. Stop when you have the cause and its evidence.

1. See the pods. During a rollout the old pod can keep running while the new one crashes; look at each pod, not only the Deployment.

kubectl get deployments,pods -n monitoring -l app=my-grafana -o wide
kubectl get pods -n monitoring -l app=my-grafana -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.status.phase}{"\n"}{range .status.containerStatuses[*]}{"  "}{.name}{" ready="}{.ready}{" restarts="}{.restartCount}{" last="}{.lastState.terminated.reason}{"\n"}{end}{end}'
kubectl events -n monitoring

Avoid kubectl describe pod here: it prints the containers' environment values, and Grafana pods often carry passwords there. If you see a credential anyway, do not repeat it.

The chart labels its pods app.kubernetes.io/name=grafana; use that selector there.

2. Read the logs of the failing pod, the previous run if it restarted. Each pod has the grafana container, and with the chart one container per sidecar (grafana-sc-dashboard, grafana-sc-datasources).

kubectl logs -n monitoring my-grafana-0 -c grafana --previous --tail=50
kubectl logs -n monitoring deployment/my-grafana -c grafana --tail=200 | grep -E "level=error|level=warn|Error:"
kubectl logs -n monitoring deployment/my-grafana -c grafana --tail=500 | grep logger=provisioning

Replace my-grafana-0 with the failing pod's name from step 1.

3. Read the provisioning ConfigMaps the pod mounts.

kubectl get deployment my-grafana -n monitoring -o jsonpath='{range .spec.template.spec.volumes[*]}{.name}{" "}{.configMap.name}{"\n"}{end}'
kubectl get configmap my-grafana-datasources -n monitoring -o yaml
kubectl get configmaps -n monitoring -l grafana_dashboard
kubectl get configmaps -n monitoring -l grafana_datasource

4. For "No data" or a failing datasource, check the address it uses against the Services that exist.

kubectl get services -A | grep -i -E "prometheus|loki|tempo"
kubectl get endpointslices -n monitoring
kubectl get --raw '/api/v1/namespaces/monitoring/services/my-grafana:http/proxy/api/health'

The last command needs get on services/proxy; if it is forbidden, skip it.

5. Open the matching reference and follow its steps:

What you seeReference
CrashLoopBackOff, Datasource provisioning error, failed to parse, a datasource or dashboard missing, sidecar not picking up ConfigMapsreferences/provisioning.md
"No data", Bad Gateway, connection refused, wrong datasource UID in a dashboardreferences/datasources-no-data.md
database is locked, permission errors, plugins, login and 401 Unauthorized, grafana-operator NoMatchingInstancereferences/runtime-and-operator.md

Read the results

What the common findings mean (Grafana 13.2):

  • The new pod exits with Error: ✗ invalid service state: Failed ... Datasource provisioning error: <reason>: Grafana refuses a provisioned datasource file and stops; the pod crash-loops. datasource.yaml config is invalid. Only one datasource per organization can be marked as default means two datasources have isDefault: true. A YAML error or an unknown field shows up the same way.
  • logger=provisioning.dashboard ... level=error msg="failed to load dashboard from " file=<path> error=<JSON error>: that dashboard file is not valid JSON. Grafana skips it and keeps running, so only that dashboard is missing.
  • the same UID is used more than once or dashboard title is not unique in folder: two files define the same dashboard; only one wins.
  • A datasource whose URL names a Service that does not exist (or the wrong port): Grafana starts normally; every query fails, shown in the UI as "Bad Gateway" or a connection error, and in the log only when a query runs.
  • /api/health returns "database": "ok": Grafana itself is up; failing (HTTP 503) means its database is not reachable.

State the cause in one sentence, then the evidence you saw (the log line, the ConfigMap entry, the Services), then the proposed fix.

Proposed fix

Show each fix to the user before running it: what it changes and how to undo it. Provisioning lives in ConfigMaps that come from a Helm chart, kube-prometheus-stack values or a GitOps repository: fix it there. Patching a ConfigMap that only holds provisioning or dashboards, and restarting Grafana to reload it, are reversible, so they go under "Ask first" and run only after the user says yes. Anything with Secrets (admin password, datasource credentials), Grafana's database, its volume and the replica count are never run by you: hand them to the user as proposed fixes. Write each undo with the values you found.

Restart Grafana after its provisioning ConfigMap is fixed (Grafana reads datasource files only at start):

Ask first: the agent runs this only after the user approves it.

kubectl rollout restart deployment/my-grafana -n monitoring
# Undo: kubectl rollout undo deployment/my-grafana -n monitoring

Sources

Checked against the Grafana 13.2 documentation and source (v13.2.4), the Grafana Helm chart 14.0 and grafana-operator 5.25:

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.