k8s-grafana-troubleshooting
Skill38 starsChanged 17 days agoApache-2.0View on GitHub
Use when Grafana on Kubernetes fails: the Grafana pod crash-loops after a change to its provisioning ("Datasource provisioning error", "Only one datasource per organization can be marked as default", "failed to parse"), a provisioned dashboard or datasource does not appear ("failed to load dashboard from", sidecar ConfigMap labels), panels show "No data" or the datasource returns "Bad Gateway" or "connection refused", "database is locked" with several replicas, permission errors on the data volume, admin login or sidecar "401 Unauthorized", or grafana-operator resources with "NoMatchingInstance". Investigates with kubectl get, describe, events and logs, never reading Grafana's Secrets, and proposes fixes. Reads first and asks before changing anything.
In this repository32 other files it gives its agents.
- AGENTS.mdAGENTS.md
- CLAUDE.mdCLAUDE.md
- k8s-aks-troubleshootingSkill
- k8s-api-troubleshootingSkill
- k8s-argocd-troubleshootingSkill
- k8s-autoscaling-troubleshootingSkill
- k8s-cert-manager-troubleshootingSkill
- k8s-cilium-troubleshootingSkill
- k8s-cloudnative-pg-troubleshootingSkill
- k8s-cluster-reviewSkill
- k8s-coredns-troubleshootingSkill
- k8s-envoy-gateway-troubleshootingSkill
- k8s-envoy-troubleshootingSkill
- k8s-external-secrets-troubleshootingSkill
- k8s-fluent-bit-troubleshootingSkill
- k8s-flux-troubleshootingSkill
- k8s-gatekeeper-troubleshootingSkill
- k8s-gateway-api-troubleshootingSkill
- k8s-helm-troubleshootingSkill
- k8s-istio-troubleshootingSkill
- k8s-kyverno-troubleshootingSkill
- k8s-loki-troubleshootingSkill
- k8s-network-troubleshootingSkill
- k8s-node-troubleshootingSkill
- k8s-opentelemetry-collector-troubleshootingSkill
- k8s-prometheus-troubleshootingSkill
- k8s-rbac-troubleshootingSkill
- k8s-storage-troubleshootingSkill
- k8s-triageSkill
- k8s-upgrade-readinessSkill
- k8s-velero-troubleshootingSkill
- k8s-workload-troubleshootingSkill
The file
We split this skill into the rules it gives an agent
3 rules. About 0 are common, 1 is distinctive, and 2 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈1
Distinctive · estimate
This author's own choices. Worth a look.
2
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 26
Never run a "Proposed fix".
- Very specific: line 39
Never read Secrets: not the admin password, not datasource credentials, not grafana.ini values kept in Secrets.
- Very specific: line 76
Avoid kubectl describe pod here: it prints the containers' environment values, and Grafana pods often carry passwords there.
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Grafana troubleshooting
Find out why Grafana does not start, does not show a datasource or dashboard, or shows no data, and propose a fix. Read first, ask before changing. Run the investigation commands freely. Run a command under an "Ask first" line only after the user approves that exact command; if your credentials do not allow it, give it to the user to run. Never run a "Proposed fix"; hand it to the user.
On Kubernetes, Grafana is usually provisioned: datasources and dashboard providers
are YAML files under /etc/grafana/provisioning/, and dashboards are JSON files, all
mounted from ConfigMaps. With the Grafana Helm chart (also inside kube-prometheus-stack)
a sidecar container copies ConfigMaps labelled grafana_dashboard or
grafana_datasource into the pod. With the grafana-operator, Grafana,
GrafanaDashboard and GrafanaDatasource resources do the same.
The commands use placeholders: monitoring (where Grafana runs), my-grafana (its
Deployment and Service), my-grafana-datasources and my-grafana-dashboards (its
ConfigMaps) and my-app (a dashboard). Replace them with the real names.
Never read Secrets: not the admin password, not datasource credentials, not
grafana.ini values kept in Secrets. The sidecar can also read Secrets with the same
labels; look only at ConfigMaps. Do not log in to Grafana's API with the admin
password; its /api/health needs no login.
When to use
- The Grafana pod is in CrashLoopBackOff or not Ready, often right after a change.
- A provisioned datasource or dashboard is missing in the UI.
- Panels show "No data", or a datasource test fails.
- Logins fail, or the sidecar logs
401 Unauthorized. - grafana-operator resources do not sync.
Hand-offs, by skill name. When the evidence points elsewhere, continue with that skill; if it is not installed, keep going with the steps here and tell the user which skill would help:
- Grafana runs and reaches Prometheus, but Prometheus has no data for the query:
k8s-prometheus-troubleshooting. - The datasource's Service exists but traffic does not reach it (NetworkPolicy, DNS):
k8s-network-troubleshooting. - The pod is Pending or its volume does not mount:
k8s-workload-troubleshooting,k8s-storage-troubleshooting.
Investigate
Work through the steps in order. Stop when you have the cause and its evidence.
1. See the pods. During a rollout the old pod can keep running while the new one crashes; look at each pod, not only the Deployment.
kubectl get deployments,pods -n monitoring -l app=my-grafana -o wide
kubectl get pods -n monitoring -l app=my-grafana -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.status.phase}{"\n"}{range .status.containerStatuses[*]}{" "}{.name}{" ready="}{.ready}{" restarts="}{.restartCount}{" last="}{.lastState.terminated.reason}{"\n"}{end}{end}'
kubectl events -n monitoringAvoid kubectl describe pod here: it prints the containers' environment values, and
Grafana pods often carry passwords there. If you see a credential anyway, do not repeat
it.
The chart labels its pods app.kubernetes.io/name=grafana; use that selector there.
2. Read the logs of the failing pod, the previous run if it restarted. Each pod has
the grafana container, and with the chart one container per sidecar
(grafana-sc-dashboard, grafana-sc-datasources).
kubectl logs -n monitoring my-grafana-0 -c grafana --previous --tail=50
kubectl logs -n monitoring deployment/my-grafana -c grafana --tail=200 | grep -E "level=error|level=warn|Error:"
kubectl logs -n monitoring deployment/my-grafana -c grafana --tail=500 | grep logger=provisioningReplace my-grafana-0 with the failing pod's name from step 1.
3. Read the provisioning ConfigMaps the pod mounts.
kubectl get deployment my-grafana -n monitoring -o jsonpath='{range .spec.template.spec.volumes[*]}{.name}{" "}{.configMap.name}{"\n"}{end}'
kubectl get configmap my-grafana-datasources -n monitoring -o yaml
kubectl get configmaps -n monitoring -l grafana_dashboard
kubectl get configmaps -n monitoring -l grafana_datasource4. For "No data" or a failing datasource, check the address it uses against the Services that exist.
kubectl get services -A | grep -i -E "prometheus|loki|tempo"
kubectl get endpointslices -n monitoring
kubectl get --raw '/api/v1/namespaces/monitoring/services/my-grafana:http/proxy/api/health'The last command needs get on services/proxy; if it is forbidden, skip it.
5. Open the matching reference and follow its steps:
| What you see | Reference |
CrashLoopBackOff, Datasource provisioning error, failed to parse, a datasource or dashboard missing, sidecar not picking up ConfigMaps | references/provisioning.md |
"No data", Bad Gateway, connection refused, wrong datasource UID in a dashboard | references/datasources-no-data.md |
database is locked, permission errors, plugins, login and 401 Unauthorized, grafana-operator NoMatchingInstance | references/runtime-and-operator.md |
Read the results
What the common findings mean (Grafana 13.2):
- The new pod exits with
Error: ✗ invalid service state: Failed ... Datasource provisioning error: <reason>: Grafana refuses a provisioned datasource file and stops; the pod crash-loops.datasource.yaml config is invalid. Only one datasource per organization can be marked as defaultmeans two datasources haveisDefault: true. A YAML error or an unknown field shows up the same way. logger=provisioning.dashboard ... level=error msg="failed to load dashboard from " file=<path> error=<JSON error>: that dashboard file is not valid JSON. Grafana skips it and keeps running, so only that dashboard is missing.the same UID is used more than onceordashboard title is not unique in folder: two files define the same dashboard; only one wins.- A datasource whose URL names a Service that does not exist (or the wrong port): Grafana starts normally; every query fails, shown in the UI as "Bad Gateway" or a connection error, and in the log only when a query runs.
/api/healthreturns"database": "ok": Grafana itself is up;failing(HTTP 503) means its database is not reachable.
State the cause in one sentence, then the evidence you saw (the log line, the ConfigMap entry, the Services), then the proposed fix.
Proposed fix
Show each fix to the user before running it: what it changes and how to undo it. Provisioning lives in ConfigMaps that come from a Helm chart, kube-prometheus-stack values or a GitOps repository: fix it there. Patching a ConfigMap that only holds provisioning or dashboards, and restarting Grafana to reload it, are reversible, so they go under "Ask first" and run only after the user says yes. Anything with Secrets (admin password, datasource credentials), Grafana's database, its volume and the replica count are never run by you: hand them to the user as proposed fixes. Write each undo with the values you found.
Restart Grafana after its provisioning ConfigMap is fixed (Grafana reads datasource files only at start):
Ask first: the agent runs this only after the user approves it.
kubectl rollout restart deployment/my-grafana -n monitoring
# Undo: kubectl rollout undo deployment/my-grafana -n monitoringSources
Checked against the Grafana 13.2 documentation and source (v13.2.4), the Grafana Helm chart 14.0 and grafana-operator 5.25:
- Troubleshooting and Grafana on Kubernetes.
- Provisioning:
datasource and dashboard files,
isDefault, providers. - Prometheus datasource troubleshooting.
- High availability: a shared database for several replicas.
- Upgrade to Grafana 13.
- The Grafana Helm chart (sidecar labels, image, admin password) and the grafana-operator documentation.
- Messages from the source: provisioning.go, datasources config_reader.go, dashboards file_reader.go. The messages quoted here were seen with Grafana 13.2.4 on a Kubernetes 1.37 kind cluster; the exact wording of the start-up error differs from older releases.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

