agentleFS
k8s-servantk8s-servant/k8s-skills/skills/k8s-opentelemetry-collector-troubleshooting/SKILL.md

k8s-opentelemetry-collector-troubleshooting

Skill38 starsChanged 17 days agoApache-2.0View on GitHub

Use when telemetry (traces, metrics, logs) does not get through the OpenTelemetry Collector on Kubernetes: the collector crashes at start ("invalid configuration", "references receiver ... which is not configured", "unknown type"), applications get "connection refused" on 4317 or 4318 (a receiver listening on localhost, or not in a pipeline), exporters fail ("Exporting failed. Will retry the request after interval", "Exporting failed. Dropping data.", "sending queue is full", "no children to pick from"), the memory_limiter refuses data, k8s_attributes lacks RBAC, or the OpenTelemetry Operator does not create a collector or inject auto-instrumentation. Investigates with kubectl get, logs and the collector's own metrics, and proposes fixes. Reads first and asks before changing anything.

The file

Files
SKILL.md

OpenTelemetry Collector troubleshooting

Find out why telemetry does not pass through the OpenTelemetry Collector, and propose a fix. Read first, ask before changing. Run the investigation commands freely. Run a command under an "Ask first" line only after the user approves that exact command; if your credentials do not allow it, give it to the user to run. Never run a "Proposed fix"; hand it to the user.

A collector's config defines receivers (OTLP on 4317 for gRPC and 4318 for HTTP, Prometheus, ...), processors (memory_limiter, batch, k8s_attributes, ...), exporters (otlp_grpc, otlp_http, debug, ...) and extensions (health_check, ...), and then the service.pipelines that connect them. A component that no pipeline uses is not started at all, without a warning. The collector checks the config at start and exits with Error: ... when it is invalid. At run time it logs failed exports with the exporter's name and the error. Collectors often run as an agent (DaemonSet, sidecar) that forwards to a gateway (Deployment), so follow the data hop by hop.

The commands use placeholders: my-namespace, my-collector (the collector workload, its ConfigMap and Service), my-backend (where it exports to) and my-app (an application that sends telemetry). Replace them with the real names. Never read Secrets. Exporters and auth extensions can carry API keys and tokens in headers: read the receivers, the pipelines and each exporter's endpoint, never dump whole exporters or extensions blocks, and never repeat a header value. Do not use print-config: it resolves ${env:...} values, which may come from Secrets.

When to use

  • Traces, metrics or logs are missing at the backend.
  • The collector pods crash, restart, or are OOMKilled.
  • Applications log export errors to the collector (connection refused, timeouts).
  • The OpenTelemetry Operator's collector or auto-instrumentation does not appear.

Hand-offs, by skill name. When the evidence points elsewhere, continue with that skill; if it is not installed, keep going with the steps here and tell the user which skill would help:

  • The collector's Service has no endpoints, or a NetworkPolicy blocks it: k8s-network-troubleshooting.
  • The backend refuses the data (Loki 400 or 429, Prometheus remote write errors): k8s-loki-troubleshooting, k8s-prometheus-troubleshooting.
  • k8s_attributes or the operator is refused by RBAC: k8s-rbac-troubleshooting.
  • The operator's webhook certificate comes from cert-manager and is not issued: k8s-cert-manager-troubleshooting.

Investigate

Work through the steps in order. Stop when you have the cause and its evidence.

1. See the collector and the application. A crash points to the config; an application error names what it could not reach.

kubectl get pods -n my-namespace -o wide
kubectl logs -n my-namespace deployment/my-app --tail=20
kubectl get services,endpointslices -n my-namespace

2. Read the collector's log, current and before the last restart: config errors, the endpoints its receivers listen on (Starting GRPC server, Starting HTTP server), and export failures.

kubectl logs -n my-namespace deployment/my-collector --tail=100
kubectl logs -n my-namespace deployment/my-collector --previous --tail=30
kubectl logs -n my-namespace deployment/my-collector --tail=500 | grep -E 'error|warn|Starting (GRPC|HTTP) server|Exporting failed'

3. Read the config's pipelines, receivers and exporter endpoints, not the whole config.

kubectl get deployment my-collector -n my-namespace -o jsonpath='{.spec.template.spec.containers[0].args}{"\n"}{range .spec.template.spec.volumes[*]}{.name}{" "}{.configMap.name}{.secret.secretName}{"\n"}{end}'
kubectl get configmap my-collector -n my-namespace -o jsonpath='{.data.config\.yaml}' | grep -E '^ *(receivers|exporters|processors|pipelines|protocols|grpc|http|endpoint|traces|metrics|logs):'

4. If the OpenTelemetry Operator manages the collector, read its resource and its status.

kubectl get opentelemetrycollectors,instrumentations -A
kubectl get opentelemetrycollector my-collector -n my-namespace -o jsonpath='{.spec.mode}{" "}{.spec.image}{"\n"}{.spec.config.service}{"\n"}{range .status.conditions[*]}{.type}={.status} {.reason}: {.message}{"\n"}{end}'
kubectl get events -n my-namespace --field-selector involvedObject.kind=OpenTelemetryCollector

5. Open the matching reference and follow its steps:

What you seeReference
A crash at start, config errors, components not in a pipeline, localhost receivers, ports and protocolsreferences/config-and-receivers.md
Export failures, retries, queues, memory_limiter, k8s_attributes, internal metrics, the operator and auto-instrumentationreferences/export-and-operator.md

Read the results

What the common findings mean (collector 0.163):

  • Error: invalid configuration: service::pipelines::logs: references receiver "otlp/app" which is not configured: a pipeline names a component that has no definition (a typo, or a missing /name suffix). The collector exits; CrashLoopBackOff.
  • unknown type: "..." for id: "..." (valid values: [...]): a component type this distribution does not include, or a misspelled one. The core image has fewer components than contrib and k8s.
  • Applications get connection refused from the collector's Service, and the log shows Starting GRPC server {"endpoint": "127.0.0.1:4317"} (or no such line for that receiver): the receiver listens on localhost only; 0.0.0.0 shows as [::]:4317. Since v0.104.0 the OTLP receiver defaults to localhost; in Kubernetes it needs endpoint: 0.0.0.0:4317 (or the pod IP). A receiver missing from every pipeline gives the same symptom and logs no Starting line at all.
  • Exporting failed. Will retry the request after interval. with "error": "rpc error: code = Unavailable desc = no children to pick from": the gRPC exporter's endpoint resolves to no address (a misspelled or missing Service); gRPC does not say "no such host". connection refused or context deadline exceeded instead means the name resolves but nothing answers (the backend is down, a wrong port, a NetworkPolicy). Logged at info level; data waits in the queue until retries run out (Dropping data., no more retries left).
  • Exporting failed. Rejecting data. with sending queue is full: the backend is too slow or down for longer than the queue holds.
  • Memory usage is above soft limit. Refusing data.: the memory_limiter protects the collector; senders get data refused due to high memory usage and retry.

State the cause in one sentence, then the evidence you saw (the log line, the config entry, the application's error), then the proposed fix.

Proposed fix

Show each fix to the user before running it: what it changes and how to undo it. A collector's config is shared by every application that sends to it: never change it yourself; hand it to the user as a proposed fix. Prefer fixing the source (the Helm values, the OpenTelemetryCollector resource, the manifest in Git). The collector reads its config at start, so a changed ConfigMap needs a restart (the operator rolls its collectors itself). Write each undo with the values you found.

Fix the config in its source, apply it and restart the collector:

Proposed fix: review before running. The agent does not run this.

kubectl apply -f my-collector-config.yaml
kubectl rollout restart deployment/my-collector -n my-namespace
# Undo: kubectl apply -f my-collector-config-before.yaml, then restart my-collector again

Once the collector works, restart an application whose client gave up, so it retries:

Ask first: the agent runs this only after the user approves it.

kubectl rollout restart deployment/my-app -n my-namespace
# Undo: none needed; the restart only recreates the pods with the same template

Sources

Checked against the OpenTelemetry documentation and the collector source (core and contrib v0.163.0, distribution v0.163.2) and the operator v0.162.0:

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.