fluent-bit

# Repository Guidelines

## Preferred Commands
- Configure: `cmake -S . -B build -DFLB_TESTS_RUNTIME=On -DFLB_TESTS_INTERNAL=On`
- Configure on Windows:
  `cmake -S . -B build -DFLB_TESTS_RUNTIME=On -DFLB_TESTS_INTERNAL=On`
- Build: `cmake --build build -j8`
- Test: `ctest --test-dir build --output-on-failure`
- Prefer targeted tests with `ctest --test-dir build -R <name> --output-on-failure`
  when the affected area is known, because the full enabled suite can be slow.
- Windows supports building and running runtime tests. Prefer focused
  `flb-rt-*` targets or CTest matches because the full runtime suite can be
  slow. The GitHub Actions unit-test workflow enables runtime execution only
  for x64 to control CI running time. Do not apply that CI-only restriction to
  local agents or AI cloud builds.
- Run a focused integration test with
  `ctest --test-dir build -R flb-it-opentelemetry --output-on-failure`
- Run the in-tree Python integration suite with:
  `cd tests/integration && ./setup-venv.sh && ./run_tests.py`
- List available Python integration scenarios with:
  `cd tests/integration && ./run_tests.py --list`
- Run locally with `./build/bin/fluent-bit -c conf/fluent-bit.conf`

## Project Structure & Module Organization
Fluent Bit is a C/C++ monorepo built with CMake.

- `src/`: core engine/runtime (`flb_*` components, schedulers, routing, I/O).
- `include/fluent-bit/`: public/internal headers used by core and plugins.
- `plugins/`: input/filter/processor/output plugins (`in_*`, `filter_*`, `processor_*`, `out_*`).
- `lib/`: bundled libraries (e.g., `cprofiles`, `ctraces`, `cmetrics`, `chunkio`).
- `tests/`: integration/runtime tests and fixtures.
- `tests/integration/`: in-tree Python integration test suite for end-to-end
  plugin and protocol validation; introduced from the original
  `github.com/fluent/fluent-bit-test-suite` project.
- `conf/`: sample configurations for local validation.

Keep changes scoped: plugin logic in its plugin directory, shared behavior in `src/` or `lib/`.

## Bundled Library Changes
- Treat `lib/` as bundled third-party or separately maintained code unless the
  specific path is clearly Fluent Bit-owned.
- Before editing bundled library code, ask for explicit user confirmation. If
  the agent environment supports confirmation popups, use one; otherwise ask in
  chat before writing files.
- Prefer changes that can be sent upstream as a focused patch. Keep bundled
  library patches isolated from Fluent Bit glue code, and document the upstream
  project/path in the close-out.
- Do not mix bundled library edits with unrelated Fluent Bit core, plugin,
  documentation, or test changes in the same commit unless the user explicitly
  asks for that structure.

## Build, Test, and Development Commands
- `cmake -S . -B build -DFLB_TESTS_RUNTIME=On -DFLB_TESTS_INTERNAL=On`:
  configure runtime and internal tests, including on Windows.
- `cmake --build build -j8`: compile Fluent Bit and tests.
- `ctest --test-dir build --output-on-failure`: run enabled tests.
- `ctest --test-dir build -R flb-it-opentelemetry --output-on-failure`: run a focused integration test.
- `cd tests/integration && ./setup-venv.sh`: create the local virtualenv for
  the Python integration suite.
- `cd tests/integration && ./run_tests.py --list`: list available Python
  integration scenarios.
- `cd tests/integration && ./run_tests.py`: run the full Python integration
  suite against `build/bin/fluent-bit`.
- `cd tests/integration && FLUENT_BIT_BINARY=/path/to/fluent-bit ./run_tests.py`:
  run the Python integration suite against a specific binary.
- `./build/bin/fluent-bit -c conf/fluent-bit.conf`: run locally with a config.

## Coding Style & Naming Conventions
- Follow Apache-style C conventions used by Fluent Bit.
- Use 4-space tabs/indentation and target 100 chars per line; 120 chars max.
- Always use braces for `if/else/while/do` blocks.
- Put function opening braces on the next line:
  `int fn(void)\n{ ... }`
- Keep short boolean conditions on one line when they fit within 100 chars.
- Wrap conditions only when needed, and break at logical operators (`&&`, `||`);
  do not force one operand per line when readability does not improve.
- Keep short function calls on one line when they fit; avoid splitting each
  argument into separate lines unless line length or clarity requires it.
- Declare variables at the start of functions, not mid-block.
- Prefer descriptive `snake_case` for functions/variables and `flb_*`/`cprof_*` prefixes.
- Use `/* ... */` comments (single or multiline), with wrapped long comments.

## Testing Guidelines
- Add or update tests for behavior changes, especially protocol parsing and encoder/decoder paths.
- Prefer targeted tests close to the changed module (`tests/internal`, plugin runtime tests).
- Prefer focused `ctest -R ...` runs or specific test binaries when the touched area is known.
- Windows supports `tests/runtime`, `flb-rt-*` targets, and runtime CTest
  matches. Configure with `-DFLB_TESTS_RUNTIME=On` and run applicable focused
  runtime coverage. In `.github/workflows/call-windows-unit-tests.yaml`, keep
  runtime execution disabled for x86 and ARM64 unless the workflow scope
  explicitly changes; that exclusion controls GitHub Actions running time only.
  It does not apply to local agents or AI cloud builds.
- Use `tests/integration` when validating end-to-end plugin behavior, network
  protocols, downstream request generation, or local fake-server interactions
  that are awkward to cover in `ctest` binaries alone.
- The Python integration suite is not part of the default CMake `ctest` targets;
  run it explicitly from `tests/integration`.
- Do not skip focused integration coverage for a touched component when that
  component has a corresponding `tests/integration` scenario. Agents must run
  the focused scenario(s) for the touched component before closing the task.
- For touched components covered by `tests/integration`, agents must run the
  focused scenario(s) twice:
  - once normally to verify behavior;
  - once with the platform memory checker enabled to verify memory-safety
    behavior: Valgrind on Linux or Leaks on macOS.
- The default expectation for component verification is:
  `./tests/integration/setup-venv.sh`
  `cmake -S . -B build -DFLB_TESTS_RUNTIME=On -DFLB_TESTS_INTERNAL=On`
  `cmake --build build -j8`
  `tests/integration/.venv/bin/python -m pytest <focused-scenario> -q`
  On Linux, run the memory-safety pass with:
  `VALGRIND=1 VALGRIND_STRICT=1 tests/integration/.venv/bin/python -m pytest <focused-scenario> -q`
  On macOS, run the memory-safety pass with:
  `LEAKS=1 LEAKS_STRICT=1 tests/integration/.venv/bin/python -m pytest <focused-scenario> -q`
- On Windows, use the same `-DFLB_TESTS_RUNTIME=On` configuration and run
  relevant focused runtime and functional integration cases. Valgrind and
  macOS Leaks are normally unavailable on Windows; report that exact
  memory-checker blocker instead of conflating it with test support.
- Run broader test coverage when changing shared lifecycle, routing, storage, or accounting code.
- Validate both success and failure paths (invalid payloads, boundary sizes, null/missing fields).
- You can also run specific binaries from `build/bin` (e.g., `./bin/flb-it-opentelemetry`).
- When changing code covered by `tests/integration`, agents must verify the
  affected scenarios are clean under the platform memory checker. On Linux,
  run `tests/integration/run_tests.py --valgrind --valgrind-strict ...`. On
  macOS, run `tests/integration/run_tests.py --leaks --leaks-strict ...`. Do
  not stop at functional pass/fail if memory errors or leaks remain.
- If a focused integration or platform memory-checker run cannot be executed,
  agents must not silently skip it. They must report the exact blocker in the
  final response (for example: missing binary, missing Python environment,
  unsupported scenario, missing dependency, or infrastructure failure).
- Final task close-outs must include proof of verification:
  - the exact focused integration command(s) run;
  - which platform memory checker was used (Valgrind on Linux or Leaks on
    macOS), or `not run` with the exact blocker;
  - pass/fail status;
  - any concrete blocker if a required run could not be completed.
- Keep generated integration artifacts out of git. Do not commit
  `.venv/`, `.pytest_cache/`, `results/`, or `__pycache__/` under
  `tests/integration`.

## Commit & Pull Request Guidelines
- Prefix commit subjects with the component/plugin name in lowercase, e.g.:
  - `engine: fix flush buffer handling`
  - `in_opentelemetry: profiles: fix ingestion path`
- Keep subject/body lines <= 80 chars.
- Keep each commit scoped to one component/prefix; avoid mixed-area commits.
- When creating commits, sign them with DCO using `git commit -s`.
- During code review, do not report missing `Signed-off-by:` trailers.
  Commit metadata is validated by `.github/workflows/commit-lint.yaml`.
- PRs should include: problem statement, scope, test evidence (`ctest` output), and compatibility notes.
- If behavior changes user output/config, include a short before/after example.
- Target `master` for next major by default; open backport PRs to release branches (`1.x`) when needed.

## Commit Pattern (Branch Practice)
- Follow observed local history style:
  - component/plugin: `component: short imperative description`
  - internal tests: `tests: internal: short imperative description`
  - integration tests under `tests/integration/`:
    `tests: integration: short imperative description`
  - runtime tests/binaries outside `tests/integration/`:
    `tests: runtime: short imperative description`
- The repository commit-prefix linter in
  `.github/scripts/commit_prefix_check.py` is authoritative. When its inferred
  prefix set is narrower than a hand-written nested subject, follow the linter.
  Examples from current repo history:
  - `include/fluent-bit/config_format/flb_cf.h` +
    `src/config_format/flb_cf_yaml.c` => `config_format:`
  - `tests/internal/env.c` => `env:` or `tests:`
  - `tests/internal/fuzzers/config_map_fuzzer.c` => `config_map_fuzzer:` or `tests:`
  - `tests/internal/config_map.c` => `config_map:` or `tests:`
- Agents must follow this same style when proposing commit subjects or
  `git commit` commands; do not invent ad hoc prefixes such as `environment:`
  when the touched files map to an existing component or test area prefix.
- When suggesting commit commands, include DCO signing by default with
  `git commit -s` unless the user explicitly asks otherwise.
- Keep one interface per commit. If an interface touches both `.c` and `.h`,
  commit them together in the same commit.
- Do not bundle different interfaces into one commit just because they support
  the same feature. Core config-map changes, input/output/filter/custom/
  processor plumbing, plugin changes, tests, and documentation must be split
  into separate commits unless they are the same interface.
- Do not mix unrelated interfaces in one commit.
- Do not include `AGENTS.md` or other documentation updates in a code commit
  unless the user explicitly asks for a docs+code combined commit.
- Prefer concise one-line subjects unless extra context is required.
- Detect and avoid bad squash commits. Do not place YAML examples, config lines,
  or multiple subject-like prefix lines in the commit body unless they are
  fenced code blocks.

## Commit Lint Workflow
- When the user asks for git commit commands, provide commands that also verify
  commit-prefix lint before push or PR submission.
- After creating a commit when the user asked for a commit, agents must run the
  repo linter before closing the task and fix any bad commit subject
  immediately.
- Do not treat a local non-PR linter pass as sufficient before push. Outside
  GitHub PR context, `.github/scripts/commit_prefix_check.py` validates only
  `HEAD`, which can miss earlier commits in the branch.
- Run the same checker used in CI:
  `python .github/scripts/commit_prefix_check.py`
- The checker requires `gitpython`. If `python -c 'import git'` fails, install
  it before running the linter:
  `python3 -m pip install gitpython`
- Do not assume a generic `docs:` prefix is acceptable. Check the touched file
  prefixes from repo history and from `.github/scripts/commit_prefix_check.py`
  path inference before choosing a commit subject. For example, changes only to
  `AGENTS.md` must use `agents:`, not `docs:`.
- For pull-request-style validation, use full history and fetch the base branch
  first, matching CI behavior. This is required because the checker can fall
  back to validating only `HEAD` when the base ref is unavailable:
  `git fetch --all --prune`
  `git fetch origin <base-branch>:origin/<base-branch>`
- Before pushing a branch or opening/updating a PR, agents must lint the full
  PR commit range against the base branch, not just the latest commit. Use the
  CI-style environment when possible, for example:
  `GITHUB_EVENT_NAME=pull_request GITHUB_BASE_REF=<base-branch> python .github/scripts/commit_prefix_check.py`
- If a commit mixes component code and integration tests, do not assume a
  local `HEAD`-only lint pass proves the earlier code commit is acceptable.
  Validate the whole branch range before push/PR submission.
- When giving commit-command sequences to the user, include a final lint step
  that checks the PR range and mention the `gitpython` install step if it may
  be missing locally.

## Agent Action Limits
- Do not open issues, pull requests, or remote branches unless the user explicitly asks.
- Do not rewrite git history, amend commits, or force-push unless the user explicitly asks.
- Do not revert user changes outside the requested scope.
- Do not edit bundled libraries under `lib/` without explicit confirmation.
- Prefer minimal patches that avoid unrelated formatting or refactoring churn.

## Agent Playbook (Pipeline Architecture Primer)

### Runtime model (mental map)
- Fluent Bit moves data through: input -> chunk -> router -> task ->
  filter/processor -> output -> engine result handling.
- Routing is per output instance; one chunk can fan out to many routes.
- Route state is independent (success/retry/drop can differ per output).

### Data units and boundaries
- A **signal** is the high-level type: logs, metrics, traces, profiles, blobs.
- A **record/event** is the logical payload unit inside a signal.
- A **chunk** is the persisted/queued container (often MessagePack-backed).
- A **task** is the engine execution unit for a chunk across routes.
- Never assume "one chunk = one route" or "one serialized event = one log
  record" in shared code.

### Component responsibilities
- Inputs (`plugins/in_*`) create/append data and trigger ingestion.
- Input chunk layer (`src/flb_input_chunk.c`) manages lifecycle, routing masks,
  storage pressure, and drop/release behavior.
- Router (`src/flb_router*.c`) resolves tag/signal matches to outputs.
- Task layer (`src/flb_task.c`) tracks per-route state and retries.
- Filters (`plugins/filter_*`) run on matching streams before output flush.
- Processors (`plugins/processor_*`) can run in input/output contexts depending
  on configuration and may mutate/drop payloads.
- Outputs (`plugins/out_*`) serialize/protocol-encode and return flush result.
- Engine (`src/flb_engine.c`) applies final retry/drop accounting and task
  teardown.

### Signal-aware behavior rules
- Shared paths must branch correctly by `event_type` (logs vs non-logs).
- Some logic is meaningful only for logs (record-level semantics), while
  metrics/traces/profiles/blobs may follow different serialization/counting.
- Group/metadata markers can exist as serialized events; treat them as
  transport/data-shape artifacts unless the interface explicitly requires them.

### Counting and metrics guidance
- Separate these concepts when reviewing code:
  - serialized events in a buffer
  - logical records after processing
  - per-route processed/retry/drop counters
  - byte accounting (chunk bytes vs route-effective bytes)
- Prefer route-aware values when updating route metrics.
- Preserve explicit zero values; use clear sentinel values for "unknown".

### Retry/drop semantics
- `FLB_OK`: route succeeded.
- `FLB_RETRY`: route keeps task/chunk for retry scheduling.
- `FLB_ERROR`: route failure/drop path.
- Final chunk release happens only when all active routes are resolved.

### Storage/backlog interaction
- In-memory and filesystem backlog paths may use different code paths; validate
  both when touching chunk/task lifecycle.
- Backlog-loaded chunks must preserve route state and accounting parity with
  live-ingested chunks.

### Review checklist before patching
- Trace one full path for affected signals: input -> chunk -> task -> output ->
  engine completion.
- Verify fan-out behavior (single chunk, multiple outputs).
- Verify processing behavior (drop/modify/no-op) in both input and output
  processor contexts.
- Verify empty payload behavior (outputs should not crash on zero records).
- Verify metrics/counters for success, retry, and drop paths.

### Testing strategy
- Use `tests/internal` for core lifecycle/accounting logic.
- Use `tests/runtime` for plugin-level behavior and end-to-end semantics,
  including Windows runtime targets supported by the active toolchain and host.
- Add regression tests for:
  - mixed signals
  - processor drop/modify paths
  - multi-route fan-out
  - backlog + live ingestion parity

More agent context in fluent/fluent-bit

One other file this repository gives its agents.