Python SDK

Python wrapper around the Rust core via PyO3 and maturin.

Client code patterns

When writing or reviewing client code that uses this SDK, follow the cross-SDK performance flow from the root CLAUDE.md:

• Idiomatic flow: ingest in a loop with ingest_record_offset(), or prefer ingest_records_offset() for bulk data, then call flush() once to confirm durability. ingest_record_offset() returns as soon as the record is queued; the SDK sends it and tracks its acknowledgment in the background. Do not feature ingest_record_nowait() in examples: it spawns a detached task and is not safely synchronized with flush().
• In async code, an AckCallback is a good way to track durability without blocking.
• wait_for_offset() blocks until a specific offset is acknowledged — use it to confirm a specific record before continuing. Acks are ordered, so waiting on the LAST offset confirms all prior records. Avoid calling it after every record in a tight loop, since that limits throughput to one record per round-trip.
• ingest_record() is deprecated — prefer the offset-based APIs.

Structure

python/
├── rust/src/lib.rs   # PyO3 bindings (the Rust ↔ Python bridge)
├── zerobus/          # Python package (pure-Python layer over _zerobus_core)
├── tests/            # pytest test suite
└── pyproject.toml    # Build config (maturin)

The native extension module is _zerobus_core, exposed as three submodules:

• zerobus._zerobus_core.sync — Synchronous API
• zerobus._zerobus_core.aio — Async API (Python async/await)
• zerobus._zerobus_core.arrow — Arrow Flight support

Build commands

Run from python/:

• make dev — Create venv, install deps, build Rust extension
• make build-rust — Release build of the Rust extension
• make develop-rust — Debug build (faster iteration)
• make test — pytest with coverage
• make lint — pycodestyle + autoflake
• make fmt — black + autoflake + isort

Features

Avro record format support is always compiled in — the avro feature is enabled on the databricks-zerobus-ingest-sdk dependency in python/rust/Cargo.toml. The avro pip extra installs fastavro, which is required only for encoding dict records; pre-encoded Avro bytes need no extra.

FFI boundary: PyO3

PyO3 handles memory management automatically via Python reference counting. Key considerations:

• GIL release: Async operations release the GIL, allowing concurrent Python threads. The tokio runtime is initialized at module import via pyo3_async_runtimes::tokio::init().
• PyO3 API: The bindings use PyO3 0.29 and the Bound<'py, T> API. Use Python::attach (not with_gil), Python::detach (not allow_threads), and cast (not downcast). Py<T> is not unconditionally Clone — use clone_ref(py) under an attached interpreter.
• Object lifetime: PyO3 classes are ref-counted by Python's GC. No explicit free needed, but close() should still be called to flush pending records before GC cleanup.
• Error mapping: Rust ZerobusError maps to ZerobusException / NonRetriableException Python exception classes.
• Serialization: Proto descriptors and record payloads cross as Python bytes. JSON records cross as Python str. No extra serialization layer — PyO3 handles the conversion.

Breaking change rules

The public Python API is defined by the classes and functions in zerobus/. Changes here follow semver:

• Removing a class, method, or parameter is breaking.
• Deprecate with warnings.warn(..., DeprecationWarning) first.
• The PyO3 binding layer (python/rust/src/lib.rs) is internal; refactoring it is safe as long as the Python-facing API doesn't change.

Performance notes

• PyO3 has minimal overhead for bytes/str payloads (no copy for immutable buffers).
• Async API avoids GIL contention — prefer it for high-throughput ingestion.
• Batch ingestion (ingest_records) amortizes the Python→Rust call overhead.

Changelog and documentation

• Every PR must update python/NEXT_CHANGELOG.md under the appropriate section if it changes user-facing behavior.
• Update python/README.md if the change affects usage, setup, or API surface.
• Add or update examples in python/examples/ for new or modified APIs.
• Add docstrings for all new public classes/methods. Update type stubs (_zerobus_core.pyi) if the native module interface changes.

Release

• Version is derived from python/rust/Cargo.toml via maturin (no separate Python version file).
• Tag: python/v<semver> → triggers release-python.yml → builds wheels for all platforms/Python versions → publishes to PyPI.
• On version bump PR: update the version in python/rust/Cargo.toml, move NEXT_CHANGELOG.md contents to CHANGELOG.md, reset NEXT_CHANGELOG.md.

Config

• Python >= 3.9 required; tested through 3.14 (free-threaded builds are not supported)
• Rust >= 1.88 required for the wrapper crate (Tonic 0.14.6 MSRV)
• Line length: 120 (black)
• Formatter: black, isort