Loki troubleshooting

Find out why Loki does not accept, keep or return logs, and propose a fix. Read first, ask before changing. Run the investigation commands freely. Run a command under an "Ask first" line only after the user approves that exact command; if your credentials do not allow it, give it to the user to run. Never run a "Proposed fix"; hand it to the user.

Clients (Promtail, Grafana Alloy, Fluent Bit, the OpenTelemetry Collector) push log lines to /loki/api/v1/push. The distributor validates each line (labels, size, age) and applies the tenant's ingestion rate; the ingester applies per-stream limits and writes chunks to storage; queriers answer LogQL queries. In monolithic mode one pod (-target=all) runs every part; in microservices mode each part is its own workload. Loki answers a refused push with 400 (invalid, not retried), 429 (rate limit, retried) or 401 (no tenant), and logs refused pushes as msg="write operation failed" details="<reason>", except 401s, which it does not log.

The commands use placeholders: my-namespace, my-loki (the Loki workload, its Service and its ConfigMap) and my-app-logger (a client). Replace them with the real names. Never read Secrets. Loki's config may hold storage keys: if it is in a Secret, do not read it; if it is in a ConfigMap, read only the parts you need and never repeat a key, password or token you see. Prefer the /config endpoint, which masks typed secret fields. Avoid kubectl describe pod on Loki: it prints literal environment values.

Never call /flush or /ingester/shutdown, not even with GET: Loki flushes all chunks or shuts the ingester down. Never POST to a ring page (forget= removes an instance). Use only the read endpoints listed here.

When to use

• A log agent reports 4xx errors from Loki, or logs are missing in Grafana.
• Loki pods crash, restart, or are not Ready.
• Queries fail, time out, or return nothing for data that was sent.

Hand-offs, by skill name. When the evidence points elsewhere, continue with that skill; if it is not installed, keep going with the steps here and tell the user which skill would help:

• The client cannot reach Loki at all (DNS, Service, NetworkPolicy): k8s-network-troubleshooting.
• The log agent itself fails (Fluent Bit, the OpenTelemetry Collector): k8s-fluent-bit-troubleshooting, k8s-opentelemetry-collector-troubleshooting.
• Grafana's Loki datasource fails while Loki answers: k8s-grafana-troubleshooting.
• Loki pods are Pending or OOMKilled: k8s-workload-troubleshooting; a full volume: k8s-storage-troubleshooting.

Investigate

Work through the steps in order. Stop when you have the cause and its evidence.

1. See Loki's pods and the client's result. The HTTP status the client got points to the part that refused it.

kubectl get pods -n my-namespace -o wide
kubectl logs -n my-namespace deployment/my-app-logger --tail=20

2. Read Loki's log: refused pushes, start-up errors, storage and ring errors.

kubectl logs -n my-namespace deployment/my-loki --tail=200 | grep -E 'level=(error|warn)'
kubectl logs -n my-namespace deployment/my-loki --previous --tail=50
kubectl get events -n my-namespace --sort-by=.lastTimestamp

3. Check where the config lives and what it sets: auth_enabled, the limits and the schema. Check the kind of object first; a Secret is not read. Match only the keys you need, so storage credentials near them are not printed.

kubectl get deployment my-loki -n my-namespace -o jsonpath='{.spec.template.spec.containers[0].args}{"\n"}{range .spec.template.spec.volumes[*]}{.name}{" "}{.configMap.name}{.secret.secretName}{"\n"}{end}'
kubectl get configmap my-loki -n my-namespace -o jsonpath='{.data.config\.yaml}' | grep -E 'auth_enabled|store:|schema:|period:|reject_old_samples|ingestion_rate|ingestion_burst|per_stream_rate|max_global_streams|max_line_size'

4. Ask Loki itself, through the API server's service proxy, if your credentials allow get on services/proxy (the built-in view role does not). With auth_enabled: true the query endpoints need a tenant header and answer no org id here; the status endpoints below do not.

kubectl get --raw /api/v1/namespaces/my-namespace/services/my-loki:http-metrics/proxy/ready
kubectl get --raw /api/v1/namespaces/my-namespace/services/my-loki:http-metrics/proxy/services
kubectl get --raw "/api/v1/namespaces/my-namespace/services/my-loki:http-metrics/proxy/config?mode=diff"
kubectl get --raw /api/v1/namespaces/my-namespace/services/my-loki:http-metrics/proxy/metrics | grep -E 'loki_discarded_samples_total|status_code="(4|5)'

5. Open the matching reference and follow its steps:

Read the results

What the common findings mean (Loki 3.7):

• Client 400 and msg="write operation failed" details="entry for stream '{app=\"my-app\", service_name=\"my-app\"}' has timestamp too old: 2020-09-13T12:26:40Z, oldest acceptable timestamp is: ...": the lines are older than reject_old_samples_max_age (7 days by default). Loki adds the service_name label itself. The client's clock or its timestamps are wrong, or it replays old logs.
• Client 401 and nothing in Loki's log: auth_enabled: true (multi-tenant) and the client sends no X-Scope-OrgID header; Loki answers no org id. Every client and the Grafana datasource need the tenant ID.
• Client 429 with Per stream rate limit exceeded or ingestion rate limit exceeded for user ...: a stream or the tenant sends more than per_stream_rate_limit (3MB/s) or ingestion_rate_mb (4MB/s) allow. Clients retry 429s, so logs arrive late or not at all.
• Loki in CrashLoopBackOff with msg="validating config" err="CONFIG ERROR: invalid schema config: validating period_config: tsdb must always have periodic config for index set to 24h": the tsdb index needs period: 24h. Clients then get connection refused or no endpoints.
• failed to flush chunks: ...: the ingester cannot write to object storage; the wrapped error names the bucket, credentials or network problem. Data stays in memory and the WAL until it can.
• too many unhealthy instances in the ring or empty ring: ingesters are not ACTIVE or not found (memberlist cannot join, pods not Ready).

State the cause in one sentence, then the evidence you saw (the client's status, Loki's log line, the config field), then the proposed fix.

Proposed fix

Show each fix to the user before running it: what it changes and how to undo it. Loki's config, its limits and its storage are shared by every client and tenant: never change them yourself; hand them to the user as proposed fixes. A client's own config (its timestamps, its tenant header, its URL) belongs to that client's owners. Prefer fixing the source (the Helm values, the manifests in Git). Write each undo with the values you found.

Change Loki's config in its source and roll it out, for example index.period: 24h, or limits_config.reject_old_samples_max_age for a client that must replay older logs:

Proposed fix: review before running. The agent does not run this.

kubectl apply -f my-loki-config.yaml
kubectl rollout restart deployment/my-loki -n my-namespace
# Undo: kubectl apply -f my-loki-config-before.yaml, then restart my-loki again

After a client's fix rolls out, restart the client to retry at once:

Ask first: the agent runs this only after the user approves it.

kubectl rollout restart deployment/my-app-logger -n my-namespace
# Undo: none needed; the restart only recreates the pods with the same template

Sources

Checked against the Loki 3.7 documentation and source (v3.7.9):

• Troubleshoot log ingestion and request validation and rate limits.
• Troubleshoot operations: rings, memberlist, -verify-config.
• Multi-tenancy, HTTP API, deployment modes, schema and retention.
• Helm install: the OSS chart is now in grafana-community/helm-charts.
• Messages from the source: validation/validate.go, ingester/flush.go and storage/config/schema_config.go. The docs' failed to store chunks: storage unavailable does not exist in 3.7. The refusals quoted here were seen on a Kubernetes 1.37 kind cluster with Loki 3.7.9.