k8s-opentelemetry-collector-troubleshooting
Skill38 starsChanged 17 days agoApache-2.0View on GitHub
Use when telemetry (traces, metrics, logs) does not get through the OpenTelemetry Collector on Kubernetes: the collector crashes at start ("invalid configuration", "references receiver ... which is not configured", "unknown type"), applications get "connection refused" on 4317 or 4318 (a receiver listening on localhost, or not in a pipeline), exporters fail ("Exporting failed. Will retry the request after interval", "Exporting failed. Dropping data.", "sending queue is full", "no children to pick from"), the memory_limiter refuses data, k8s_attributes lacks RBAC, or the OpenTelemetry Operator does not create a collector or inject auto-instrumentation. Investigates with kubectl get, logs and the collector's own metrics, and proposes fixes. Reads first and asks before changing anything.
In this repository32 other files it gives its agents.
- AGENTS.mdAGENTS.md
- CLAUDE.mdCLAUDE.md
- k8s-aks-troubleshootingSkill
- k8s-api-troubleshootingSkill
- k8s-argocd-troubleshootingSkill
- k8s-autoscaling-troubleshootingSkill
- k8s-cert-manager-troubleshootingSkill
- k8s-cilium-troubleshootingSkill
- k8s-cloudnative-pg-troubleshootingSkill
- k8s-cluster-reviewSkill
- k8s-coredns-troubleshootingSkill
- k8s-envoy-gateway-troubleshootingSkill
- k8s-envoy-troubleshootingSkill
- k8s-external-secrets-troubleshootingSkill
- k8s-fluent-bit-troubleshootingSkill
- k8s-flux-troubleshootingSkill
- k8s-gatekeeper-troubleshootingSkill
- k8s-gateway-api-troubleshootingSkill
- k8s-grafana-troubleshootingSkill
- k8s-helm-troubleshootingSkill
- k8s-istio-troubleshootingSkill
- k8s-kyverno-troubleshootingSkill
- k8s-loki-troubleshootingSkill
- k8s-network-troubleshootingSkill
- k8s-node-troubleshootingSkill
- k8s-prometheus-troubleshootingSkill
- k8s-rbac-troubleshootingSkill
- k8s-storage-troubleshootingSkill
- k8s-triageSkill
- k8s-upgrade-readinessSkill
- k8s-velero-troubleshootingSkill
- k8s-workload-troubleshootingSkill
The file
Files
OpenTelemetry Collector troubleshooting
Find out why telemetry does not pass through the OpenTelemetry Collector, and propose a fix. Read first, ask before changing. Run the investigation commands freely. Run a command under an "Ask first" line only after the user approves that exact command; if your credentials do not allow it, give it to the user to run. Never run a "Proposed fix"; hand it to the user.
A collector's config defines receivers (OTLP on 4317 for gRPC and 4318 for HTTP,
Prometheus, ...), processors (memory_limiter, batch, k8s_attributes, ...),
exporters (otlp_grpc, otlp_http, debug, ...) and extensions
(health_check, ...), and then the service.pipelines that connect them. A component
that no pipeline uses is not started at all, without a warning. The collector checks the
config at start and exits with Error: ... when it is invalid. At run time it logs
failed exports with the exporter's name and the error. Collectors often run as an agent
(DaemonSet, sidecar) that forwards to a gateway (Deployment), so follow the data hop by
hop.
The commands use placeholders: my-namespace, my-collector (the collector workload,
its ConfigMap and Service), my-backend (where it exports to) and my-app (an
application that sends telemetry). Replace them with the real names. Never read Secrets.
Exporters and auth extensions can carry API keys and tokens in headers: read the
receivers, the pipelines and each exporter's endpoint, never dump whole exporters or
extensions blocks, and never repeat a header value. Do not use print-config: it
resolves ${env:...} values, which may come from Secrets.
When to use
- Traces, metrics or logs are missing at the backend.
- The collector pods crash, restart, or are OOMKilled.
- Applications log export errors to the collector (
connection refused, timeouts). - The OpenTelemetry Operator's collector or auto-instrumentation does not appear.
Hand-offs, by skill name. When the evidence points elsewhere, continue with that skill; if it is not installed, keep going with the steps here and tell the user which skill would help:
- The collector's Service has no endpoints, or a NetworkPolicy blocks it:
k8s-network-troubleshooting. - The backend refuses the data (Loki 400 or 429, Prometheus remote write errors):
k8s-loki-troubleshooting,k8s-prometheus-troubleshooting. k8s_attributesor the operator is refused by RBAC:k8s-rbac-troubleshooting.- The operator's webhook certificate comes from cert-manager and is not issued:
k8s-cert-manager-troubleshooting.
Investigate
Work through the steps in order. Stop when you have the cause and its evidence.
1. See the collector and the application. A crash points to the config; an application error names what it could not reach.
kubectl get pods -n my-namespace -o wide
kubectl logs -n my-namespace deployment/my-app --tail=20
kubectl get services,endpointslices -n my-namespace2. Read the collector's log, current and before the last restart: config errors,
the endpoints its receivers listen on (Starting GRPC server, Starting HTTP server),
and export failures.
kubectl logs -n my-namespace deployment/my-collector --tail=100
kubectl logs -n my-namespace deployment/my-collector --previous --tail=30
kubectl logs -n my-namespace deployment/my-collector --tail=500 | grep -E 'error|warn|Starting (GRPC|HTTP) server|Exporting failed'3. Read the config's pipelines, receivers and exporter endpoints, not the whole config.
kubectl get deployment my-collector -n my-namespace -o jsonpath='{.spec.template.spec.containers[0].args}{"\n"}{range .spec.template.spec.volumes[*]}{.name}{" "}{.configMap.name}{.secret.secretName}{"\n"}{end}'
kubectl get configmap my-collector -n my-namespace -o jsonpath='{.data.config\.yaml}' | grep -E '^ *(receivers|exporters|processors|pipelines|protocols|grpc|http|endpoint|traces|metrics|logs):'4. If the OpenTelemetry Operator manages the collector, read its resource and its status.
kubectl get opentelemetrycollectors,instrumentations -A
kubectl get opentelemetrycollector my-collector -n my-namespace -o jsonpath='{.spec.mode}{" "}{.spec.image}{"\n"}{.spec.config.service}{"\n"}{range .status.conditions[*]}{.type}={.status} {.reason}: {.message}{"\n"}{end}'
kubectl get events -n my-namespace --field-selector involvedObject.kind=OpenTelemetryCollector5. Open the matching reference and follow its steps:
| What you see | Reference |
| A crash at start, config errors, components not in a pipeline, localhost receivers, ports and protocols | references/config-and-receivers.md |
| Export failures, retries, queues, memory_limiter, k8s_attributes, internal metrics, the operator and auto-instrumentation | references/export-and-operator.md |
Read the results
What the common findings mean (collector 0.163):
Error: invalid configuration: service::pipelines::logs: references receiver "otlp/app" which is not configured: a pipeline names a component that has no definition (a typo, or a missing/namesuffix). The collector exits; CrashLoopBackOff.unknown type: "..." for id: "..." (valid values: [...]): a component type this distribution does not include, or a misspelled one. The core image has fewer components thancontribandk8s.- Applications get
connection refusedfrom the collector's Service, and the log showsStarting GRPC server {"endpoint": "127.0.0.1:4317"}(or no such line for that receiver): the receiver listens on localhost only;0.0.0.0shows as[::]:4317. Since v0.104.0 the OTLP receiver defaults tolocalhost; in Kubernetes it needsendpoint: 0.0.0.0:4317(or the pod IP). A receiver missing from every pipeline gives the same symptom and logs noStartingline at all. Exporting failed. Will retry the request after interval.with"error": "rpc error: code = Unavailable desc = no children to pick from": the gRPC exporter'sendpointresolves to no address (a misspelled or missing Service); gRPC does not say "no such host".connection refusedorcontext deadline exceededinstead means the name resolves but nothing answers (the backend is down, a wrong port, a NetworkPolicy). Logged at info level; data waits in the queue until retries run out (Dropping data.,no more retries left).Exporting failed. Rejecting data.withsending queue is full: the backend is too slow or down for longer than the queue holds.Memory usage is above soft limit. Refusing data.: thememory_limiterprotects the collector; senders getdata refused due to high memory usageand retry.
State the cause in one sentence, then the evidence you saw (the log line, the config entry, the application's error), then the proposed fix.
Proposed fix
Show each fix to the user before running it: what it changes and how to undo it. A collector's config is shared by every application that sends to it: never change it yourself; hand it to the user as a proposed fix. Prefer fixing the source (the Helm values, the OpenTelemetryCollector resource, the manifest in Git). The collector reads its config at start, so a changed ConfigMap needs a restart (the operator rolls its collectors itself). Write each undo with the values you found.
Fix the config in its source, apply it and restart the collector:
Proposed fix: review before running. The agent does not run this.
kubectl apply -f my-collector-config.yaml
kubectl rollout restart deployment/my-collector -n my-namespace
# Undo: kubectl apply -f my-collector-config-before.yaml, then restart my-collector againOnce the collector works, restart an application whose client gave up, so it retries:
Ask first: the agent runs this only after the user approves it.
kubectl rollout restart deployment/my-app -n my-namespace
# Undo: none needed; the restart only recreates the pods with the same templateSources
Checked against the OpenTelemetry documentation and the collector source (core and contrib v0.163.0, distribution v0.163.2) and the operator v0.162.0:
- Collector configuration and troubleshooting.
- Internal telemetry: metrics on 8888 and their names.
- The
localhostdefault: core CHANGELOG v0.104.0 and otlpreceiver factory.go. - Messages from the source: otelcol/config.go and exporterhelper. "Dropping data because sending_queue is full" no longer exists in 0.163.
- Operator, its compatibility table and auto-instrumentation troubleshooting.
- The log lines quoted here were seen on a Kubernetes 1.37 kind cluster with the contrib collector 0.163.2.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

