dora

# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## Project Overview

Dora (AI-Dora, Agentic Dataflow-Oriented Robotic Architecture) is a 100% Rust framework for building real-time robotics and AI applications. It provides 10-17x faster latency than ROS2 via zero-copy shared memory and Apache Arrow data format. Supports Rust, Python (PyO3), C, and C++ nodes.

## Build Commands

```bash
# Build all (Python packages require maturin, exclude them for normal builds)
cargo build --all --exclude dora-node-api-python --exclude dora-operator-api-python --exclude dora-ros2-bridge-python

# Build specific package
cargo build -p dora-cli
cargo build -p dora-daemon

# Check all
cargo check --all

# Test all (excluding Python)
cargo test --all --exclude dora-runtime-python --exclude dora-node-api-python --exclude dora-operator-api-python --exclude dora-ros2-bridge-python

# Test single package
cargo test -p dora-core

# Lint
cargo clippy --all --all-targets

# Format
cargo fmt --all

# Format check (CI uses this)
cargo fmt --all -- --check

# Run examples
cargo run --example rust-dataflow
cargo run --example benchmark --release

# Install CLI locally
cargo install --path binaries/cli --locked

# Run a dataflow
dora run examples/rust-dataflow/dataflow.yml
dora run examples/python-dataflow/dataflow.yml --uv --stop-after 10s
```

## Workspace Layout

- **Rust edition 2024; MSRV and default workspace package metadata live in `[workspace.package]` in the root `Cargo.toml`.** Most crates inherit the workspace version via `version.workspace = true`, but a few (the `examples/error-propagation/*` samples and `xtask`) pin their own version independently.
- Python packages use PyO3 0.29 and are built with **maturin**, not cargo

### Key crates

| Path | Crate | Role |
|------|-------|------|
| `binaries/cli` | dora-cli | CLI binary (`dora` command) - build, run, stop dataflows |
| `binaries/daemon` | dora-daemon | Spawns nodes, manages local shared-memory/TCP communication |
| `binaries/coordinator` | dora-coordinator | Orchestrates distributed multi-daemon deployments |
| `binaries/runtime-api` | dora-runtime-api | Language-neutral operator runtime SDK (event loop, node harness, `OperatorRunner` trait) |
| `binaries/runtime-shared-lib` | dora-runtime-shared-lib | Shared-library (C ABI) operator runtime backend; shipped in the `dora` CLI (`dora runtime`) |
| `binaries/runtime-python` | dora-runtime-python | Python (PyO3) operator runtime backend; shipped in the Python wheel (`dora.start_runtime()`) |
| `libraries/message` | dora-message | All inter-component message types and protocol definitions |
| `libraries/core` | dora-core | Dataflow descriptor parsing, build utilities, Zenoh config |
| `apis/rust/node` | dora-node-api | Rust API for writing custom nodes |
| `apis/rust/operator` | dora-operator-api | Rust API for writing in-process operators |
| `apis/python/node` | dora-node-api-python | Python node API (PyO3) |
| `libraries/extensions/mavlink2-bridge` | dora-mavlink2-bridge | MAVLink 2 ↔ Apache Arrow conversion (common dialect) |
| `binaries/mavlink2-bridge-node` | dora-mavlink2-bridge-node | Daemon-spawnable MAVLink 2 bridge (TCP/UDP/serial) |

## Architecture

```
CLI  -->  Coordinator  -->  Daemon(s)  -->  Nodes / Operators
              (distributed)    (per machine)    (user code)
```

- **CLI <-> Coordinator**: WebSocket (port 6013) for build/run/stop commands
- **Coordinator <-> Daemon**: WebSocket for node spawning and dataflow lifecycle
- **Daemon <-> Daemon**: Zenoh for distributed cross-machine communication
- **Daemon <-> Node**: TCP — the reliable path; payloads are copied into a `DataMessage::Vec`
- **Node <-> Node**: direct zenoh data plane — zenoh shared memory at/above `ZERO_COPY_THRESHOLD` (4KB, zero-copy), heap-buffered `put` below it
- **Data format**: Apache Arrow columnar format throughout (zero serialization overhead)

### Dataflow specification

Dataflows are defined in YAML files. Nodes declare inputs (subscribing to other nodes' outputs) and outputs. Built-in timer nodes: `dora/timer/millis/<N>`, `dora/timer/hz/<N>`.

### Communication patterns

User-facing patterns (see `docs/patterns.md`):
- **Topic**: default pub/sub dataflow
- **Service**: request/reply via `request_id` metadata key; helpers: `send_service_request()`, `send_service_response()`
- **Action**: goal/feedback/result via `goal_id`/`goal_status` metadata keys; supports cancellation

Internal transport:
- **flume** channels: bounded MPSC for internal event routing
- **tokio** async runtime with full features
- **Zenoh**: pub-sub for remote/distributed nodes
- **UHL Clock** (`uhlc`): hybrid logical clock for distributed causality

## Pre-commit Quality Gates (MANDATORY)

**Every commit MUST pass these gates before pushing.** This is a BLOCKING REQUIREMENT — do not skip any step. Remote CI is slower than your laptop and has limited capacity; catching failures locally saves 5-15 minutes per round-trip.

> The bar below is the **Class A** (low-risk) baseline. Behavior
> changes and high-risk subsystem edits have additional requirements
> — see [`docs/agentic-qa-policy.md`](docs/agentic-qa-policy.md)
> (#1634) for the per-class expectations and the PR validation
> summary template.

### Step 1: Run /review

Run the `/review` skill on your changes before committing. This catches structural issues, security problems, and logic errors that tests and clippy miss.

### Step 2: Run /simplify

Run the `/simplify` skill to check for unnecessary complexity, code duplication, or inefficiency in the changed code.

### Step 3: Local CI (fmt + clippy + tests)

```bash
# 1. Format — CI rejects formatting diffs
cargo fmt --all -- --check

# 2. Clippy with warnings-as-errors — CI uses -D warnings
cargo clippy --all --all-targets \
  --exclude dora-node-api-python \
  --exclude dora-operator-api-python \
  --exclude dora-ros2-bridge-python \
  -- -D warnings

# 3. Tests — at minimum the affected crate; full workspace before push
cargo test --all \
  --exclude dora-node-api-python \
  --exclude dora-operator-api-python \
  --exclude dora-ros2-bridge-python \
  --exclude dora-runtime-python \
  --exclude dora-cli-api-python \
  --exclude dora-examples

# 4. Example targets — `cargo check --all` does NOT build `[[example]]`
# targets, so signature breaks in `dora_cli::build` / `dora_cli::run`
# can ship silently otherwise. CI gates on this (#1680).
cargo check --examples

# 5. Only if you touched a PyO3 crate (apis/python/cli, apis/python/node, apis/python/operator,
# libraries/extensions/ros2-bridge/python, binaries/runtime-python): their unit
# tests are excluded from the `cargo test --all` above because the test binaries
# link libpython, so run them explicitly. CI runs the same command in ci.yml's
# `contract-tests` job.
make qa-test-python

# Quick single-crate check while iterating:
# cargo test -p <crate-name>
```

**Shortcut**: `make qa-fast` runs fmt + clippy + supply-chain audit + unwrap budget + secret-files + typos + publish-graph + package-includes + Cargo.lock freshness + the no-compile half of the 1.x compatibility gate + nightly-reporting wiring in ~30 seconds. Use it as a sanity check before every commit. Then run the full `cargo test` above before `git push`.

If you add new example dataflows, also run `./scripts/smoke-all.sh --rust-only` to verify smoke tests pass.

### Enforcement

All three steps are required for every commit. Do NOT push without completing them. If any step fails, fix the issue before proceeding. The order matters: /review and /simplify catch design-level issues that are cheaper to fix before running the full test suite.

### Do NOT run the full QA suite on every commit/push

The deeper QA gates — `make qa-full`, `make qa-deep`, `make qa-nightly`, `make qa-release-gate`, `make qa-mutation-audit`, `make qa-examples`, `make qa-coverage`, `make qa-mutants`, `make qa-semver`, `make qa-pgo` — are designed for **focused investigation and pre-release gating**, not the per-commit loop. Do not run them routinely, and do not add them to remote CI:

- `make qa-full` (qa-fast + full tests + coverage) — ~5-10 minutes. Run before a significant push if you want extra confidence; coverage is too slow for every push.
- `make qa-deep` (qa-full + mutation testing + semver) — ~15 minutes. The **target** Tier 1 local gate. Today's CI PR gate only runs the fast subset (fmt/clippy/typos/audit/unwrap-budget + tests); `qa-deep` adds coverage, adversarial review, diff-scoped mutation, and semver — kept laptop-only because they're too slow for every PR (see `docs/plan-agentic-qa-strategy.md` §5). Alias: `make qa-tier1`, kept for back-compat.
- `make qa-nightly` (qa-deep + proptest@1000 + miri + example-smoke + ci-nightly-jobs) — ~3-4 hours. **Full parity with `.github/workflows/nightly.yml`** (#1707, #1710, #1716). After the #1716 rebalance (plus cluster-record-replay from #2013), nightly.yml has **27 test jobs** (`cli-tests` was split into `cli-tests` + `cli-tests-python` in #2742; re-counted in #2999, where the previous 23 counted neither `hub-smoke` nor the two `ros2-zenoh-*` jobs, and `multi-daemon-late-subscriber` is new; `wheel-smoke` added with the `dora-rs-cli` entry-point fix). example-smoke covers the **4 example-backed** jobs (smoke-suite, log-sinks, service-action, streaming); `scripts/qa/ci-nightly-jobs.sh` drives **20** with platform-aware dispatch (record-replay, cluster-smoke, cluster-e2e [Linux, needs `openssh-server`], cluster-record-replay [Linux, needs `openssh-server`], topic-and-top, cpu-affinity [Linux], redb-backend, daemon-reconnect [Linux], state-reconstruction, multi-daemon-late-subscriber [Linux], test-cross-platform [macOS+Windows], examples, cli-tests **and cli-tests-python** — one local `cli-tests` invocation covers both halves, bench-example, cross-check, ros2-bridge [Linux, basic checks — no ROS distro], ros2-zenoh-humble, ros2-zenoh-kilted, msrv, kani-proofs [skipped if Kani not installed]). Of the rest, `hub-smoke` and `wheel-smoke` have no local driver entry at all (`wheel-smoke` needs maturin plus two interpreters; build the wheels by hand if you need it). A green local `qa-nightly` on platform X predicts a green CI nightly for platform X's jobs. **Requires both `uv` and Python 3.12** (both preflighted; the script fails fast with a specific install hint for whichever is missing — `curl -LsSf https://astral.sh/uv/install.sh \| sh` for uv, `uv python install 3.12` for the interpreter, matching the GHA `actions/setup-python` step at `.github/workflows/nightly.yml:56`). example-smoke creates a scratch venv at `target/qa-nightly-venv` and installs `-e apis/python/node` into it so Python nodes use the workspace bindings (not PyPI `dora-rs`, whose message format has drifted from the workspace — #1710). The CI-jobs script installs the CLI into a scratch dir (won't clobber `~/.cargo/bin/dora`) and bails if port 6013 is in use; cpu-affinity + daemon-reconnect skip on non-Linux. Skips miri if `cargo +nightly miri` isn't installed. **Does not** include full-repo mutation testing — that's split into `qa-mutation-audit` because it takes 10-18 hours on this workspace.
- `make qa-release-gate` (qa-deep + semver) — the automatable subset of Tier 3. The non-automatable parts (independent security audit, dogfood campaign, migration validation) are documented in `docs/plan-agentic-qa-strategy.md` §7 but not locally gateable.
- `make qa-mutation-audit` — ~10-18 hours. Full-repo `cargo-mutants` across 6 critical crates. Deliberate test-quality audit; run before a release or when investigating a specific crate, not every nightly.
- `make qa-examples` — ~15-20 min. Runs all **smoke-eligible** example dataflows end-to-end via `scripts/smoke-all.sh`. Skips examples that need CUDA, ROS2, webcam, multi-machine deploy, C/C++ toolchains, or interactive CLI (run `scripts/smoke-all.sh -h` to see the SKIP list). Orthogonal to the qa-fast/full/deep ladder: those targets `--exclude dora-examples` to keep per-commit / pre-push budgets tight. Run this when you want actual dataflows exercised (after touching node/operator APIs, CLI subcommands, or the descriptor surface). Pass flags via `ARGS`, e.g. `make qa-examples ARGS="--rust-only"` or `make qa-examples ARGS="-v"`.
- `make qa-coverage` — generates an lcov report locally. Run when you want to see what's covered.
- `make qa-mutants` — mutation testing, slow. Run when investigating test quality on a specific file or on the PR diff.
- `make qa-semver` — the Rust-API half of the breaking-change check. Run before publishing to crates.io; `make qa-breaking` runs it as one step among the others.
- `make qa-breaking` — the dora 1.x compatibility gate: every surface the 1.0 guarantee freezes, checked against the last release tag (`docs/api-rust.md`, "Stability scope at 1.0"). **Not a deep gate** — its no-compile half is in `qa-fast` and in PR CI, taking ~2 s. The default (full) form adds `cargo-semver-checks` and rebuilds the CLI/schema snapshots to prove they are current; run that when you touched the CLI, the descriptor, `dora-message`, or a node API. `make qa-breaking-update` re-records the snapshots after an addition.
- `make qa-kani` — ~5-10 min cold, <1 min warm. Formal verification: runs the `#[kani::proof]` harnesses (Kani/CBMC model checker) in `dora-message` (auth `constant_time_eq`). A passing proof is exhaustive over the harness state space, not sampled. Run when touching files with `#[cfg(kani)]` proof modules and in pre-release gating; install via `make qa-kani-install`. See [`docs/formal-verification.md`](docs/formal-verification.md).
- `make qa-pgo` — ~30-40 min. Profile-Guided Optimization measurement: builds `dora-cli` with `cargo-pgo` (instrument → train on `examples/benchmark/` → optimize), then runs the benchmark twice (baseline vs PGO) and prints a side-by-side comparison with a +5%-throughput-geomean verdict. Run on your target platform — PGO results don't transfer across OS/arch. macOS-arm64 first-measurement showed +25% throughput geomean, latency unchanged (see [#331](https://github.com/dora-rs/dora/issues/331)). **Do not add to CI** — the build pipeline cost is release-pipeline scope, not per-commit. Install via `make qa-pgo-install`.

**Remote CI is deliberately kept lean** — only the fast hard gates (fmt, clippy, test on Linux, typos, supply chain audit, unwrap budget, publish-graph, package-includes, lockfile freshness, breaking changes, E2E, contract tests, benchmark regression, license check) — so it stays fast (~30-45 min critical path) and runner capacity is not a bottleneck. Do not expand PR CI with slow jobs. See [`docs/qa-runbook.md`](docs/qa-runbook.md) for when and how to use each deep gate locally.

### Remote CI jobs

**PR CI (`.github/workflows/ci.yml`, ~30-45 min, #1716):** Linux-only. Blocks merge.
- `cargo fmt --all -- --check`
- `cargo clippy --all --all-targets -- -D warnings` (excluding Python packages) — `--all-targets` so tests, benches, and example binaries are linted too
- `cargo test --all` on **ubuntu-latest only** (excluding Python packages)
- `cargo check --all-targets` on the three PyO3 crates, so their `#[cfg(test)]` modules are at least type-checked on every PR (`cargo check --all` covers lib targets only)
- E2E tests: `ws-cli-e2e` + `fault-tolerance-e2e`
- Semantic contract tests (`tests/example-smoke.rs::contract_*`), plus the PyO3 crates' unit tests (`make qa-test-python` — this is the one job that sets up a Python interpreter, so it's where those tests run)
- Benchmark regression check (criterion baseline caching)
- Typo checking via `crate-ci/typos` (config: `_typos.toml`)
- Supply-chain audit (`cargo-audit` + `cargo-deny`)
- Unwrap-budget check (production `.unwrap()` / `.expect(` ratchet)
- Committed-credential check (`make qa-secret-files`, a step in the `Check` job): fails if git tracks a file whose name says it holds a secret. .gitignore is not enough on its own — it does not apply to files arriving through an import or a squash-merge of another repo's history, which is how `.adora-token` reached the public repo (#2194)
- crates.io publish-graph check (`make qa-publish-graph`, a step in the `Check` job): no published crate may depend on a `publish = false` one, and the ordered publish list in `release.yml` must name every dependency before its dependents (#3304)
- Breaking-change gate (`make qa-breaking ARGS="--fast"` + `cargo-semver-checks`): every surface dora 1.x freezes — the C header, the cxx bridge, the dataflow YAML schema, the postcard wire format, the `dora` command snapshot, the Python floor, and the Rust API of the covered crates — diffed against the last release tag
- Build-time include check (`make qa-package-includes`, a step in the `Check` job): every `include_str!` / `include_bytes!` target of a publishable crate must be a git-tracked file inside that crate, since `cargo package` ships nothing else. `dora-operator-api-c` read its cmake templates from the sibling `apis/c/node/` crate, which built here and broke `cargo install dora-cli` on 1.0.0 (#3400)
- Cargo.lock freshness (`make qa-lockfile`, a step in the `Check` job, before `cargo check --all` because that command rewrites a stale lock in place and then passes): the committed lock must already satisfy every workspace manifest, and the gate names the drifting packages rather than reprinting cargo's message. It lives in a required status, not only `qa-fast`, because a new `[[package]]` block merges cleanly into a tree whose workspace version has since moved — so the invalid lock exists only in the merged result, which `push: trunk-merge/**` is what makes visible (#3512)
- Nightly reporting wiring (`make qa-ci-reporting`, its own checkout-only job): asserts every `nightly.yml` job is actually watched by `file-issue-on-failure` and `close-issue-on-success`, and that none of them is silently advisory, so a regression cannot go unreported (#2987)
- License check (`cargo-lichking`)
- Rust toolchain pinned to 1.97.1 (see `.github/workflows/ci.yml`)

**Nightly CI (`.github/workflows/nightly.yml`, ~3-4h, daily 06:40 UTC):** Broader coverage, does NOT block PRs. Auto-files an issue on failure (`nightly-regression` label).
- Test on macOS + Windows
- Examples end-to-end (all 3 platforms, incl. C/C++/CMake/Arrow/multi-daemon)
- CLI Tests (all 3 platforms, incl. template projects + Python Dynamic Node + error-event)
- Bench Example (all 3 platforms)
- Cross-compilation matrix (8 targets)
- ROS2 Bridge Basic Checks (ubuntu-latest; cargo check + msg-gen + Python bridge, no ROS distro — full ROS2 examples are release QA via `scripts/ros2dev.sh qa`)
- MSRV check (`cargo-hack`)
- Plus the integration smokes: record/replay, cluster (lifecycle smoke + ssh end-to-end), topic-and-top, cpu-affinity, redb-backend, daemon-reconnect, state-reconstruction, and the Kani formal-verification proofs (`kani-proofs`)

**Label automation:** two API-only workflows (no runner build) keep the PR queue labels honest.
- `needs-rebase.yml` keeps `needs-rebase` in sync with GitHub's merge-conflict state: it labels a PR when it conflicts with `main` and removes the label as soon as it stops conflicting — on the PR's own pushes, on every push to `main` (which rescans all open PRs, so a conflict resolved by someone else's merge clears too), and via a daily sweep as a safety net.
- `waiting-for-review.yml` swaps `waiting-for-author` for `waiting-for-review` when a PR moves again: new commits on the head branch (whoever pushed them), or a comment from the PR author. The swap is one-directional — `waiting-for-author` stays a manual maintainer decision, a PR that does not already carry it is left untouched, and a draft keeps it until `ready_for_review`.

**Developer guidance:** for non-Linux verification before merge, run `make qa-test` or `make qa-examples` locally; `make qa-nightly` covers the full nightly matrix (except ROS2) locally in ~3-4 hours.

## Test-Driven Development

All new features and bug fixes must follow the RED-GREEN-IMPROVE cycle:

1. **RED**: Write a failing test that defines the expected behavior
2. **GREEN**: Write the minimum code to make the test pass
3. **IMPROVE**: Refactor while keeping tests green

### Which test tier to write first

| Change type | Start with | File/location |
|-------------|-----------|---------------|
| New library function | Unit test | `#[cfg(test)]` in same file |
| New coordinator/daemon behavior | Integration test | `binaries/coordinator/tests/` |
| New CLI command or flag | Smoke test (networked) | `tests/example-smoke.rs` using `run_smoke_test()` |
| New dataflow feature | Smoke test (both modes) | `tests/example-smoke.rs` using both `run_smoke_test()` and `run_smoke_test_local()` |
| Bug fix | Regression test | Whichever tier reproduces the bug |
| New example dataflow | Smoke test entry | Add to `tests/example-smoke.rs` and `scripts/smoke-all.sh` |
| Change to `docker/slim/Dockerfile` | Container deployment smoke | `docker/slim/smoke.sh`, run via `make qa-docker-slim` |

### Workflow

```bash
# 1. Write failing test, verify it fails
cargo test -p <crate> <test_name>

# 2. Implement until test passes
cargo test -p <crate> <test_name>

# 3. Refactor, then verify everything still passes
cargo test -p <crate>
cargo clippy -p <crate> -- -D warnings
cargo fmt --all -- --check

# 4. Run smoke tests if the change touches CLI/coordinator/daemon
cargo test --test example-smoke -- --test-threads=1
```

### Smoke test patterns

Two helpers are available in `tests/example-smoke.rs`:

- `run_smoke_test(name, yaml, timeout)` -- networked: `dora up` + `dora start --detach` + poll + `dora stop` + `dora down`
- `run_smoke_test_local(name, yaml, stop_after_secs)` -- local: `dora run --stop-after`

New example dataflows should have tests in both modes. Use `Once` guards to share build steps across tests.

For quick local validation: `./scripts/smoke-all.sh`

### References

- Full testing guide: `docs/testing-guide.md`
- Smoke tests: `tests/example-smoke.rs`
- E2E tests: `tests/ws-cli-e2e.rs`
- Fault tolerance tests: `tests/fault-tolerance-e2e.rs`

## Conventions

- Format with `rustfmt` default settings before submitting PRs
- Discuss non-trivial changes in a GitHub issue or Discord first
- Don't fix unrelated warnings in PRs
- Python node API is distributed via PyPI (`dora-rs`); CLI is distributed via crates.io and GitHub Releases
- Release profile `dist` uses fat LTO

More agent context in dora-rs/dora

2 other files this repository gives its agents.