k8s-fluent-bit-troubleshooting
Skill38 starsChanged 16 days agoApache-2.0View on GitHub
Use when Fluent Bit on Kubernetes does not deliver logs: the DaemonSet crashes on a config error ("tried to instance a plugin name that doesn't exist", "unknown configuration property", "configuration file contains errors"), an output fails ("failed to flush chunk", "cannot be retried", "no upstream connections available", "getaddrinfo", "Connection refused"), records arrive without Kubernetes metadata ("could not get meta for POD"), lines arrive unparsed or not at all (tail path, parser, CRI versus Docker format), or inputs pause ("paused (mem buf overlimit"). Covers the Helm chart and the Fluent Operator. Investigates with kubectl get, logs and Fluent Bit's read-only HTTP endpoints, and proposes fixes. Reads first and asks before changing anything.
In this repository32 other files it gives its agents.
- AGENTS.mdAGENTS.md
- CLAUDE.mdCLAUDE.md
- k8s-aks-troubleshootingSkill
- k8s-api-troubleshootingSkill
- k8s-argocd-troubleshootingSkill
- k8s-autoscaling-troubleshootingSkill
- k8s-cert-manager-troubleshootingSkill
- k8s-cilium-troubleshootingSkill
- k8s-cloudnative-pg-troubleshootingSkill
- k8s-cluster-reviewSkill
- k8s-coredns-troubleshootingSkill
- k8s-envoy-gateway-troubleshootingSkill
- k8s-envoy-troubleshootingSkill
- k8s-external-secrets-troubleshootingSkill
- k8s-flux-troubleshootingSkill
- k8s-gatekeeper-troubleshootingSkill
- k8s-gateway-api-troubleshootingSkill
- k8s-grafana-troubleshootingSkill
- k8s-helm-troubleshootingSkill
- k8s-istio-troubleshootingSkill
- k8s-kyverno-troubleshootingSkill
- k8s-loki-troubleshootingSkill
- k8s-network-troubleshootingSkill
- k8s-node-troubleshootingSkill
- k8s-opentelemetry-collector-troubleshootingSkill
- k8s-prometheus-troubleshootingSkill
- k8s-rbac-troubleshootingSkill
- k8s-storage-troubleshootingSkill
- k8s-triageSkill
- k8s-upgrade-readinessSkill
- k8s-velero-troubleshootingSkill
- k8s-workload-troubleshootingSkill
The file
Files
Fluent Bit troubleshooting
Find out why Fluent Bit does not ship logs, and propose a fix. Read first, ask before changing. Run the investigation commands freely. Run a command under an "Ask first" line only after the user approves that exact command; if your credentials do not allow it, give it to the user to run. Never run a "Proposed fix"; hand it to the user.
Fluent Bit runs as a DaemonSet: on every node a tail input reads the container log
files under /var/log/containers/, the kubernetes filter adds pod metadata from the
API server, and outputs send the records to a backend (Elasticsearch, Loki, an HTTP
endpoint, a forwarder). Its own log names each part as [input:tail:tail.0],
[filter:kubernetes:kubernetes.0], [output:http:http.0], so the log usually names the
cause. The Helm chart keeps the config in a ConfigMap (fluent-bit.conf); the Fluent
Operator builds it from custom resources into a Secret.
The commands use placeholders: my-namespace, my-fluent-bit (the DaemonSet, its
ConfigMap and its ServiceAccount), my-fluent-bit-pod (one of its pods) and
my-backend (the Service an output sends to). Replace them with the real names. Never
read Secrets: not the Fluent Operator's rendered config, not Secrets that outputs take
credentials from. The ConfigMap may hold credentials inline (HTTP_Passwd,
http_api_key, cloud_auth, bearer_token, AWS keys): read it, but never repeat those
values. Avoid kubectl describe pod on Fluent Bit: it prints literal environment
values. Never request /api/v1/trace/... (a GET turns tracing on) and never POST to
/api/v2/reload.
When to use
- Logs are missing, late, or incomplete at the log backend.
- Fluent Bit pods crash, restart, or are not Ready.
- Records arrive without
kubernetesfields, or with the raw<time> stdout F <line>text inlog.
Hand-offs, by skill name. When the evidence points elsewhere, continue with that skill; if it is not installed, keep going with the steps here and tell the user which skill would help:
- The backend refuses what Fluent Bit sends (Loki 400, 401, 429):
k8s-loki-troubleshooting. - The backend's Service has no endpoints or a NetworkPolicy blocks it:
k8s-network-troubleshooting. - The kubernetes filter is refused by RBAC:
k8s-rbac-troubleshooting. - Pods are Pending, OOMKilled or evicted:
k8s-workload-troubleshooting.
Investigate
Work through the steps in order. Stop when you have the cause and its evidence.
1. See the DaemonSet and its pods: ready on every node, restarts.
kubectl get daemonsets -A -l app.kubernetes.io/name=fluent-bit
kubectl get pods -n my-namespace -l app.kubernetes.io/name=fluent-bit -o wide2. Read Fluent Bit's log, current and before the last restart. Errors and warnings name the input, filter or output.
kubectl logs -n my-namespace daemonset/my-fluent-bit --tail=100
kubectl logs -n my-namespace daemonset/my-fluent-bit --previous --tail=50
kubectl logs -n my-namespace daemonset/my-fluent-bit --tail=500 | grep -E '\[ ?(error|warn)\]'3. Read the config: inputs (Path, parsers), filters, outputs (Name, Host,
Port, Match). Do not repeat credentials you see.
kubectl get daemonset my-fluent-bit -n my-namespace -o jsonpath='{.spec.template.spec.containers[0].args}{"\n"}{range .spec.template.spec.volumes[*]}{.name}{" "}{.configMap.name}{.secret.secretName}{.hostPath.path}{"\n"}{end}'
kubectl get configmap my-fluent-bit -n my-namespace -o yaml4. Check what the outputs and the kubernetes filter depend on: the backend's Service, and the ServiceAccount's access to pods and namespaces.
kubectl get services,endpointslices -A | grep -i my-backend
kubectl get clusterrolebindings -o wide | grep my-fluent-bit
kubectl auth can-i list pods --as=system:serviceaccount:my-namespace:my-fluent-bit5. Ask Fluent Bit itself (port 2020, if HTTP_Server is on), through the API
server's pod proxy, if your credentials allow get on pods/proxy (the built-in view
role does not). Each pod reports for its own node.
kubectl get --raw /api/v1/namespaces/my-namespace/pods/my-fluent-bit-pod:2020/proxy/api/v2/health
kubectl get --raw /api/v1/namespaces/my-namespace/pods/my-fluent-bit-pod:2020/proxy/api/v1/metrics6. Open the matching reference and follow its steps:
| What you see | Reference |
| A crash at start, config errors, the Fluent Operator's resources, hot reload | references/config-and-operator.md |
| Outputs failing, retries, dropped chunks, backpressure, missing metadata, tail and parsers, the HTTP endpoints | references/pipeline.md |
Read the results
What the common findings mean (Fluent Bit 5.1):
[config] section 'htttp' tried to instance a plugin name that doesn't existthenconfiguration file contains errors, aborting.: a misspelled pluginName; the pod exits and goes to CrashLoopBackOff.[net] getaddrinfo(host='...', err=4): Domain name not foundwith[output:http:http.0] no upstream connections available to ...: the output'sHostdoes not resolve. Then[engine] failed to flush chunk '...', retry in N secondsand, once retries run out,chunk '...' cannot be retried: those logs are dropped. The defaultRetry_Limitis 1; the Helm chart setsFalse(retry forever).[net] TCP connection failed: host:port (Connection refused): the name resolves, but nothing listens there (the backend is down, or the port is wrong).[filter:kubernetes:kubernetes.0] could not get meta for POD ...: the filter cannot read pod metadata (RBAC,Kube_URL, the API server unreachable). Fluent Bit keeps running and ships records withoutkubernetesfields; the HTTP status is only logged at debug level.- Records with
logholding2026-... stdout F <line>: the parser does not match the runtime's log format; containerd and CRI-O write CRI lines, so usemultiline.parser docker, cri. [input] tail.0 paused (mem buf overlimit: the output cannot keep up, and the tail input stops reading until memory frees; lines can be lost on rotation.
State the cause in one sentence, then the evidence you saw (the log lines with their plugin prefix, the config entries), then the proposed fix.
Proposed fix
Show each fix to the user before running it: what it changes and how to undo it. Fluent Bit's config and RBAC serve every node and every workload's logs: never change them yourself; hand them to the user as proposed fixes. Prefer fixing the source (the Helm values or the Fluent Operator's resources). Fluent Bit reads its config only at start unless hot reload is on, so a changed ConfigMap needs a restart. Write each undo with the values you found.
Fix the config in its source, apply it, and restart the DaemonSet so it reads it:
Proposed fix: review before running. The agent does not run this.
kubectl apply -f my-fluent-bit-configmap.yaml
kubectl rollout restart daemonset/my-fluent-bit -n my-namespace
# Undo: kubectl apply -f my-fluent-bit-configmap-before.yaml, then restart the DaemonSet againBind the role the kubernetes filter needs (read on pods and namespaces) to Fluent
Bit's ServiceAccount:
Proposed fix: review before running. The agent does not run this.
kubectl create clusterrolebinding my-fluent-bit --clusterrole=my-fluent-bit --serviceaccount=my-namespace:my-fluent-bit
# Undo: kubectl delete clusterrolebinding my-fluent-bitOnce the config or the access is fixed, restart Fluent Bit so it picks it up:
Ask first: the agent runs this only after the user approves it.
kubectl rollout restart daemonset/my-fluent-bit -n my-namespace
# Undo: none needed; the restart only recreates the pods with the same templateSources
Checked against the Fluent Bit 5.1 documentation and source (v5.1.5), the Helm chart fluent-bit 0.58.3 and the Fluent Operator v3.10.1:
- Kubernetes installation and the Kubernetes filter, including its troubleshooting section.
- Tail input, HTTP output, scheduling and retries, backpressure, monitoring and hot reload.
- The Helm chart and the Fluent Operator.
- Messages from the source: flb_config.c, flb_engine.c and kube_meta.c. The log lines quoted here were seen on a Kubernetes 1.37 kind cluster with Fluent Bit 5.1.5.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

