OpenTelemetry Collector Topology

Component names: keep the pre-rename spellings (deliberate)

Collector core v0.144.0 renamed the otlp exporter to otlp_grpc and otlphttp to otlp_http. contrib has made similar renames: k8sattributes → k8s_attributes, resourcedetection → resource_detection, hostmetrics → host_metrics, filelog → file_log, prometheusremotewrite → prometheus_remote_write, deltatocumulative → delta_to_cumulative. In each case the old name was kept as a deprecated alias.

Generated configs use the old names on purpose. They work on every Collector version; the new names fail on any Collector older than the rename. The cost is a startup warning on newer Collectors, e.g. "otlp" alias is deprecated; use "otlp_grpc" instead, once per pipeline. That warning is expected, not a bug: don't "fix" it by switching names in one place. The receiver is still called otlp; only exporters and the contrib components above were renamed.

Revisit when upstream schedules removal of an alias, since a removed alias makes generated configs fail to start. The drift-check job reports the latest Collector release, and the pinned test version (tests/collector-validate.sh) is where to confirm the aliases still load.

Two modes: agent and gateway

Agent mode

Runs on every host/pod alongside the application. Receives OTLP from the app, applies basic processing, forwards to a gateway or directly to a backend.

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317      # app sends here
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 1s
    send_batch_size: 1024
  memory_limiter:
    check_interval: 1s
    limit_mib: 256

exporters:
  otlp:
    endpoint: gateway.internal:4317  # or backend endpoint
    tls:
      insecure: false

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp]
    logs:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp]

Gateway mode

Centralized aggregation tier. Receives from agents, applies tail sampling, exports to backend(s).

Adds tail_sampling processor between memory_limiter and batch:

processors:
  tail_sampling:
    decision_wait: 10s
    num_traces: 100000
    expected_new_traces_per_sec: 10000
    policies:
      - name: errors-policy
        type: status_code
        status_code: {status_codes: [ERROR]}
      - name: slow-traces-policy
        type: latency
        latency: {threshold_ms: 1000}
      - name: debug-policy
        type: string_attribute
        string_attribute:
          key: debug.sampling
          values: ["true"]
      - name: probabilistic-fallback
        type: probabilistic
        probabilistic: {sampling_percentage: 10}

Wire tail sampling into the traces pipeline only (metrics and logs use direct forwarding):

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, tail_sampling, batch]
      exporters: [otlp]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp]
    logs:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp]

Receiver auth (--public — internet-exposed collectors)

The receivers.otlp block above binds 0.0.0.0 with no auth — fine when the collector is only reachable from localhost or a shared private network, a real gap when it isn't. That happens for a deployment shape neither mode's base template assumes: the app runs on a platform with no private-networking path to a collector it doesn't share a host with (e.g. a serverless-container platform that can't run a sidecar, so the collector runs on a separate, internet-reachable host). An unauthenticated receiver there lets anyone who finds the endpoint (trivial — DNS + a standard OTLP port) push arbitrary telemetry through it on the owner's ingest quota, or worse (#107, confirmed in a real deployment: Scaleway Serverless Containers with a collector on a separate Instance, no shared private network).

This is opt-in via --public, never a default: most agent/gateway deployments genuinely are same-host or private-network, and forcing auth on every user would be disruptive for no benefit in the common case. bearertokenauth (open-telemetry/opentelemetry-collector-contrib) is currently beta stability — functionally solid and confirmed working server-side on a receiver (not just client-side on an exporter — verify this against the extension's own README/source if the pin moves, since summaries of it are inconsistent on this point), but note the beta status when recommending it, consistent with how language-maturity flags below-Stable components elsewhere in this plugin. When --public is set, add the extension and wire it into both receiver protocols:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
        auth:
          authenticator: bearertokenauth
      http:
        endpoint: 0.0.0.0:4318
        auth:
          authenticator: bearertokenauth

extensions:
  bearertokenauth:
    token: "${env:COLLECTOR_AUTH_TOKEN}"   # NEVER a literal token in the file

service:
  extensions: [bearertokenauth]
  pipelines: {}   # unchanged — pipelines don't reference extensions directly

A client must then send Authorization: Bearer <token> (the extension's default scheme/header) or the request is rejected before it reaches any pipeline. TLS in front of the collector is a requirement here, not an optional extra (a reverse proxy such as Caddy, or the OTLP receiver's own tls: block): this bearer token is a shared secret sent on every request, and without transport encryption it travels in cleartext across the public internet — anyone who can observe the network path recovers it and defeats the whole point of adding auth to an internet-exposed receiver. Neither --public's generated config nor this skill sets TLS itself (deployment-specific — certs, a proxy, or the receiver's native tls: block are all valid, so the plugin does not pick one), so state this as a REQUIRED step in the next-steps output, not a passing mention. Print in the next-steps output that the user must (1) terminate TLS in front of this collector, (2) generate a random token and set it as COLLECTOR_AUTH_TOKEN on the collector's host, and (3) configure every sending app with OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer%20<same token> — the %20, not a literal space, because this env var's values must be percent-encoded per the OTel spec (W3C Baggage format); a literal space works in some SDKs but the OTel Python SDK specifically rejects it, failing every export with UNAUTHENTICATED.

--public's receiver wiring applies identically to agent and gateway mode — both share this receivers.otlp block. The sender-side instructions do NOT transfer, though: agent mode's senders are apps (SDKs), configured via the OTEL_EXPORTER_OTLP_HEADERS env var; gateway mode's senders are agent Collectors, which need client-side bearertokenauth wired into their own exporters.otlp.auth.authenticator instead — an otelcol exporter does not read OTEL_EXPORTER_OTLP_HEADERS. Printing the app-side instruction for a gateway-mode collector produces guidance the user cannot act on; see /otel-collector's Step 6 for the per-mode text.

Also set the exporter's tls.insecure: false when --public is set. This is a separate leg from everything above — the receiver-auth work is entirely about the inbound hop (sender → this collector); this is about the collector's own outbound hop (this collector → its backend/gateway) — but the two correlate: the base agent template defaults tls.insecure: true on the exporter because it assumes a local-dev shape ("local agent → local sink"), and a --public collector is by definition not that shape — it exists because the deployment has a real network between hosts, so its outbound leg should not default to a local-loopback assumption either. This is a judgment call tied to the flag, not a hard technical requirement of bearertokenauth itself, so state it as generation guidance here (not folded into the receiver auth explanation above) rather than leaving it as an unexplained value only the golden shows.

Cardinality Guardrails (add to agent config)

Always include a transform processor to drop high-cardinality identifiers that are common mistakes. It runs on spans and on metric data points, with the same drop-list in both:

processors:
  transform:
    error_mode: ignore
    trace_statements:
      - context: span
        statements:
          # Drop the canonical high-cardinality identifiers (kept in sync with the
          # semconv-discipline skill and the semconv-lint hook).
          # OTTL uses `where` (not `if`) as the statement-level conditional keyword.
          - delete_key(attributes, "user.id")           where IsString(attributes["user.id"])
          - delete_key(attributes, "session.id")        where IsString(attributes["session.id"])
          - delete_key(attributes, "request.id")        where IsString(attributes["request.id"])
          - delete_key(attributes, "order.id")          where IsString(attributes["order.id"])
    metric_statements:
      - context: datapoint
        statements:
          # The SAME list. A metric dimension is the worst position for an identifier (one
          # retained series per value; sampling cannot help), so this is the more valuable half.
          # Dropping the key merges the series that differed only by it, which is the intended
          # outcome: say so in the generated file so nobody is surprised.
          - delete_key(attributes, "user.id")           where IsString(attributes["user.id"])
          - delete_key(attributes, "session.id")        where IsString(attributes["session.id"])
          - delete_key(attributes, "request.id")        where IsString(attributes["request.id"])
          - delete_key(attributes, "order.id")          where IsString(attributes["order.id"])

service:
  pipelines:
    traces:  { processors: [memory_limiter, transform, batch] }
    metrics: { processors: [memory_limiter, transform, batch] }
    logs:    { processors: [memory_limiter, batch] }

When generating config, check the context JSON for any attributes marked high-cardinality by the brownfield-auditor and add matching delete_key statements to both lists, so they cannot drift apart (#172; before this, repo-specific keys were only ever added for spans).

Logs are deliberately not guarded. A log record is not a time series, and semconv-discipline ranks a log-record attribute as the prescribed home for a per-request identifier, so deleting user.id there would remove debugging data with no cardinality benefit. Add a log_statements list only if a specific backend turns log attributes into indexed labels, and then drop only the keys it indexes.

This guardrail is a backstop, not the fix: the source of truth is still the instrumentation (semconv-lint catches the identifier in code). It only covers the keys on its list.

Cost/Cardinality Budget (metrics, gateway mode, --experimental) — #119

The guardrail above is reactive: it deny-lists attribute NAMES already known to be high-cardinality. A budget is proactive: a growth-rate ceiling on label VALUES, so a new high-cardinality dimension nobody named yet still gets caught before it hits the bill.

Verified against the real otelcol-contrib 0.164.0 binary (otelcol-contrib components), not memory or guesswork — neither of the two components the issue speculated about is right: cardinalitylimit does not exist anywhere in contrib, and groupbyattrs only regroups data points by attribute, it does not limit or drop anything. The real component is cardinality_guardian (processor/cardinalityguardianprocessor), and its stability per the binary's own component list is metrics: Development, traces/logs: Undefined — Development is this plugin's lowest maturity tier (same bar as Node.js/Python logs), so this is metrics-only (the component has no traces/logs support at all — this isn't a choice, it's an upstream constraint) and always gated behind --experimental, never generated by default. (Its own README calls itself "alpha" — trust the binary's self-reported status over the README; re-verify against whatever version is pinned if this is ever revisited.)

It tracks new unique label VALUES per (metric, label) pair per epoch (HyperLogLog++, default epoch 300s) — "a label that grew by 100 in the last epoch" is the problem, not "a label with 50K stable values." Three enforcement modes exist (tag_only, overflow_attribute, strip_and_reaggregate) — generate tag_only, paired with a filter processor that drops anything it tags. The other two modes reaggregate the exceeding series instead of just tagging it, but ONLY for Delta Sums and Gauges — Cumulative Sums, Histograms, ExponentialHistograms, and Summaries silently fall back to tag_only with no reaggregation. OTel SDKs default to Cumulative temporality, so those two modes would silently do nothing for most real traffic while looking like they enforce a cap. tag_only + an explicit drop is the only combination that behaves identically across every metric type — a budget that silently doesn't apply to most of your metrics is worse than no budget, because it reads as enforced when it isn't.

processors:
  cardinality_guardian:
    max_cardinality_delta_per_epoch: <BUDGET>   # from --cardinality-budget <N> — required, never invented
    epoch_duration_seconds: 300                 # upstream default; not exposed as a flag in v1
    enforcement_mode: tag_only                  # see the temporality caveat above before changing this
  filter/cardinality_budget:
    error_mode: ignore
    metrics:
      datapoint:
        - 'attributes["otel.metric.overflow"] == true'

Wire into the gateway metrics pipeline only, between memory_limiter and batch (see the ordering rule below):

service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, cardinality_guardian, filter/cardinality_budget, batch]
      exporters: [otlp]

Gateway only, never agent. The growth-rate detector needs aggregate visibility across every upstream agent to mean anything — an agent-mode instance only ever sees one service's slice of the metric space, which starves the HyperLogLog sketches of signal and makes max_cardinality_delta_per_epoch mean something different per service. Revisit only if a real need for a per-service budget shows up.

Never invent the budget number: max_cardinality_delta_per_epoch comes directly from --cardinality-budget <N> on /otel-collector, with no inferred or guessed default — if the flag is not given, do not generate this processor at all, even under --experimental. The upstream config also has never_drop_labels, metric_overrides, top_offenders_count, max_tracker_count, and estimated_cost_per_metric_month; this generator leaves all of them at their upstream defaults in v1 — expose them only if a real need shows up.

Required exporter configuration

Always include:

1. retry_on_failure block on all exporters
2. sending_queue with in-memory queue (no storage extension — file-backed queue is v1)
3. timeout: 30s on exporters

exporters:
  otlp:
    endpoint: ${env:OTEL_EXPORTER_OTLP_ENDPOINT}
    timeout: 30s
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 30s
      max_elapsed_time: 300s
    sending_queue:
      enabled: true
      num_consumers: 10
      queue_size: 1000

Exporter auth (the outbound hop)

The exporters.otlp block above sends no credentials. That is correct when it forwards to an unauthenticated collector on a private network. Agent mode also "forwards to a gateway or directly to a backend", and every backend's OTLP ingest requires an auth header: without one, each batch is rejected, and because retry_on_failure and sending_queue are on, the data sits in the queue and is then dropped (#171). The failure is silent at generation time and only appears at export time; otelcol-contrib validate passes.

Credentials go in headers: under the exporter, always as an environment reference, never a literal:

exporters:
  otlp:
    endpoint: ${env:OTEL_EXPORTER_OTLP_ENDPOINT}
    headers:
      Authorization: "Bearer ${env:BACKEND_AUTH_TOKEN}"

/otel-collector --exporter-header 'Name=${env:VAR}' emits it. The header names are the backend's contract (the vendor's OTLP-ingest docs), so this plugin does not hard-code them per vendor; an authenticated gateway made by --public uses client-side bearertokenauth instead (see /otel-collector's Step 6).

Regeneration must not drop it. Exporter headers: / auth: are user-owned wiring the templates do not produce, so a --force regeneration keeps them on a same-named exporter and stops, like the receiver-auth guard, if they sit on an exporter the new config lacks.

otelcol-contrib vs otelcol

Always generate configs for otelcol-contrib (the distribution with all receivers, processors, and exporters). The base otelcol distribution lacks several processors used in these templates (e.g. tail_sampling, transform).

Validate command: otelcol-contrib validate --config=<file>

Processor ordering rule

memory_limiter MUST always be the first processor in every pipeline. This is a hard OTel Collector constraint — placing it after other processors means those processors run before the memory check and can cause OOM crashes.

Correct order: [memory_limiter, (optional: transform, tail_sampling, cardinality_guardian, filter/cardinality_budget), batch] — filter/cardinality_budget must come after cardinality_guardian specifically (it drops what that processor just tagged), not merely somewhere before batch.