agentleFS
matt82198matt82198/aesop/state_store/CLAUDE.md

aesop / state_store

CLAUDE.md2 starsChanged 31 days agoMITView on GitHub

Optimistic Concurrency Control (Phase 2, 2026-07-21): append(..., expected_version=N) asserts "stream is at version N" and succeeds only if true. Atomicity: version check + append both run under BEGIN IMMEDIATE, so there is no TOCTOU window. Failure mode: version mismatch raises ConcurrencyConflict(expected, actual) WITHOUT writing any event (fail-closed). Retry protocol: caller re-reads the stream, takes the new version from ConcurrencyConflict.actual_version (or re-counts events), retries with expected_version=new_version. Backward compatible: expected_version=None (default) disables OCC; old code is unchanged and unaffected. Use case: multiple…

The file

We split this CLAUDE.md into the rules it gives an agent

2 rules. About 0 are common, 1 is distinctive, and 1 is very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈1

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 1

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Append-only: never mutate/delete events.

    line 52
  2. Very specific:

    Run from repo root. npm run test:py runs ALL Python suites (includes state_store).

    line 61

≈ “Common” is an estimate: at least 1 in 5 of the 27,641 CLAUDE.md files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
CLAUDE.md

state_store/ — event-sourced state layer (SQLite WAL, projections, git-as-export)

Universal rules (every domain)

  • Feature branch only, never main; every push gated by python tools/secret_scan.py --staged exit 0.
  • Tests never pollute cwd or global git config; temp dirs only; dummy secrets are runtime-concatenated, never literal.
  • In worktrees use ABSOLUTE paths under the worktree for every write.
  • Domain docs stay minimal-but-complete; update this file in the same PR as code it describes.

Purpose & Status

Durable substrate moving aesop's coordination/state off git (which cannot scale to a team due to single-writer control files, hot-file merge conflicts, no transactions/concurrency). State becomes an append-only event log with per-stream versioning; current state is a projection; git is demoted to a rendered, diffable export. Status (2026-07-14): additive prototype. The live ui/ tracker path is UNCHANGED. Full architecture & migration design: docs/TEAM-STATE.md.

API Surface (state_store.api.StateAPI)

Facade over EventStore + projections; backend swap seam (currently SQLite WAL).

  • append(stream, event_type, payload, actor="system", expected_version=None) → int: Append one event; return its new per-stream version. OCC support (Phase 2): if expected_version is provided, the append succeeds ONLY if the stream's current max version equals expected_version; otherwise raises ConcurrencyConflict WITHOUT writing (fail-closed, atomic). Enables writers to serialize reads: "I read version N; I will append only if it's still N when I try."
  • get(stream) → list: Return all events in stream ascending by version.
  • get_since(stream, after_version) → list: Return events with version > after_version. Enables snapshot + tail-replay for any stream.
  • close(): Release the underlying thread-local cached connection. Safe to call multiple times; the next operation lazily reopens.
  • project(view) → dict: Fold the same-named stream through its projector into current state. Registered views: "tracker" (via project_tracker).
  • Exceptions: ConcurrencyConflict(expected_version, actual_version) — raised by append() when OCC check fails; carries both versions so caller can rebase and retry.

Concurrency Model & Measured Safety

Multi-writer safe via SQLite WAL + atomicity: PRAGMA journal_mode=WAL (many readers, serialized writers via write lock); PRAGMA busy_timeout=5000 (retry 5s on contention before erroring); BEGIN IMMEDIATE in append() makes read-max-version-then-insert atomic so two writers never collide or duplicate a version. Measured safety (2026-07-18 spike, single-box): 4 concurrent writers x 800 events (3200 total), 0 lock errors, ~704 events/sec — a single-host micro-benchmark; real-world wave throughput averages ~100 events/sec.

Optimistic Concurrency Control (Phase 2, 2026-07-21): append(..., expected_version=N) asserts "stream is at version N" and succeeds only if true. Atomicity: version check + append both run under BEGIN IMMEDIATE, so there is no TOCTOU window. Failure mode: version mismatch raises ConcurrencyConflict(expected, actual) WITHOUT writing any event (fail-closed). Retry protocol: caller re-reads the stream, takes the new version from ConcurrencyConflict.actual_version (or re-counts events), retries with expected_version=new_version. Backward compatible: expected_version=None (default) disables OCC; old code is unchanged and unaffected. Use case: multiple orchestrators coordinating on multi-instance state (distributed tracing, multi-writer audit log) use OCC to prevent lost updates when racing to extend the same stream.

Module Layout

  • read_api.py — read_text_file(path) (module-level, read-only, None on OSError) is the facade for reading markdown control files (STATE.md, BUILDLOG.md), used by tools/boot_state.py. ReadAPI facade over state surfaces; read-only. Delegates to existing parsers: tracker snapshot, orchestrator status, heartbeat freshness via tools/common, ledger rows via tools/fleet_ledger. Never forks logic.
  • write_api.py — WriteAPI(state_dir) facade for tracker mutations (state consolidation write path). Exposes tracker CRUD and markdown operations: tracker_append_item(item_dict), tracker_update_status(item_id, new_status, note), tracker_update_item(item_id, update_data) (general-purpose patch), tracker_archive_item(item_id) (soft-delete), and rebuild_projection(). Retry-safe create (11bececf532c): when the projection render raises WriteConflict after the item_created event is already appended, a retry of the identical create reuses that durable event (_find_pending_create) instead of appending a second copy under a fresh id.
    • Markdown write path (Inc 1+2): write_state_md(content), append_buildlog(line), ensure_buildlog_exists(header=...) (idempotent create, never overwrites; header lets migrated legacy writers keep their historical header byte-for-byte), rebuild_state_md(content, force). File written atomically first, event (state_md_written/buildlog_entry) appended only on success. Migrated callers (Inc 2): tools/buildlog.py, tools/ensure_state.py, tools/eod_sweep.py.
    • tracker_ methods*: All append events atomically AND render tracker.json projection via materialize_tracker() under a file lock (tempfile + os.replace). Fail-closed: event append failure blocks projection write.
    • OCC (Optimistic Concurrency Control): Detects concurrent modification before atomic write: if on-disk hash differs from both start-of-operation hash and computed projection hash, raises WriteConflict (fail-closed). Corrupt JSON on disk also raises WriteConflict (fail-closed, not fail-open). Baseline hash captured at operation START (before event append) so the check window covers the entire operation.
    • ID collision detection: tracker_append_item with explicit id rejects duplicates (raises ValueError before appending) to prevent duplicate events for the same logical item.
    • Self-healing recovery: rebuild_projection() force-renders from the event store, bypassing OCC, to recover orphaned events (event in store, missing from projection).
    • Snapshot optimization (2026-08-03): _project_tracker() now uses project_tracker_with_snapshot() (O(n) tail-replay) instead of project_tracker() (O(n²) full replay). Reduces per-mutation projection cost; gracefully falls back to full replay when snapshot missing/corrupt. OCC WriteConflict guard unchanged and load-bearing.
    • _file_lock() acquisition race (2026-10-06, PR #856 CI escape): the advisory lock uses a .lock file deleted on release, so a losing thread's os.open() of that filename can transiently raise PermissionError on Windows while the winning thread is concurrently closing/unlinking it — that was only retried for the subsequent msvcrt.locking() byte-range call, not for os.open() itself, so the raw PermissionError escaped instead of resolving to the expected WriteConflict. Fixed by retrying the whole acquisition (open + lock) up to 20 times with a short backoff; proven against tests/test_writeapi_markdown.py::test_concurrent_write_race_prevention (previously flaky on windows-shard).
  • store.py — EventStore(db_path, busy_timeout_ms=5000): append-only SQLite log with thread-local connection pooling. Writes (append, save_snapshot) take a per-database-file in-process lock so threads in one process never race SQLite's coarse busy handler; a write that still cannot get the SQLite lock within the busy window raises StoreBusyError (subclass of sqlite3.OperationalError; fix for the windows-shard "database is locked" flake, branch fix/state-store-busy-timeout). append(stream, type, payload, actor, expected_version=None) returns new version or raises ConcurrencyConflict on OCC mismatch; read(stream) / read_since(stream, after_version) / read_all() return event rows; close() releases the cached connection. Corrupt JSON payloads are skipped with stderr log; snapshot read/write for tail-replay optimization.
  • init.py — Public exports: EventStore, StateAPI, ConcurrencyConflict, project_tracker, export_tracker, ingest_tracker_json.
  • projections.py — project_tracker(events): folds item_created / item_updated / item_archived into the full tracker.json shape, preserving first-seen order. Duplicate item_created rule (fix cffdf93d567f, 2026-10-05): a second item_created for an id already in the fold is a no-op on fields that are already set (only backfills still-null fields) — it never overwrites state folded from a later item_updated/item_archived. ~38 real items carry such a stray duplicate; a blind overwrite here resurrects closed items on every full re-render. The event is never dropped from the append-only log; the no-op is surfaced via a stderr warning at fold time.
  • api.py — StateAPI(db_path): the backend swap seam (currently SQLite WAL). Callers use this only; backend implementation hidden. Passes through OCC, connection lifecycle (close()), and tail-read (get_since()) support transparently.
  • export.py — export_tracker(api, out_path): render the projection back to a git-tracked JSON snapshot (indent=2, ascii-escaped to match the live file).
  • ingest.py — ingest_tracker_json(api, path): backfill one item_created per existing item; validates event structure at boundary. Accepts both tracker shapes (canonical {"items": [...]} and legacy bare list [...]), matching wave_scheduler.load_tracker_items.
  • identity.py — Multi-instance identity: get_instance_id() returns ephemeral form (hostname:pid:nonce); get_identity_with_epoch(state_root) returns durable (stable_id, epoch) persisted to $AESOP_STATE_ROOT/instance-id. Epoch is the monotonic boot counter used as the multibox fencing token. Acquisition increments it (atomic read-modify-write: temp file + fsync + os.replace), so every boot on a state root gets a strictly higher epoch than every prior boot; the value is cached per process, so repeated calls within one process return one epoch. Fail-closed on corruption: a corrupt existing file — torn/partial JSON, empty, missing keys, or a non-integer/<1 epoch that cannot be incremented monotonically — raises IdentityCorruptionError. Fail-closed on persist failure: if the bumped epoch cannot be written, raises EpochPersistError (subclass of IdentityCorruptionError, so existing handlers still catch it) instead of handing back a reused epoch. Fail-open on creation only: fresh box (no prior file) with an unwritable directory falls back to ephemeral (allows solo mode). Distinction preserves monotonicity: a stale crashed instance cannot claim epoch=1 after restart. release_own_stale() raises NotImplementedError — prior-epoch claim reclamation needs the Inc 5 lease-backend coordination; it previously returned an unconditional True, so a caller believed reclamation had happened when nothing ran. Until Inc 5, prior claims are reclaimed by TTL expiry.
  • paths.py — canonical_claim_path(path, repo_root=None, case_policy=...): THE claim-key canonicalizer, and the only place claim-path normalization logic lives. Folds separators to / via posixpath (never os.path, which on Linux treats \ as an ordinary filename character and would leave a Windows-style path uncollapsed), collapses ./.., NFC-normalizes Unicode, then case-folds per case_policy: "platform" follows os.name, "insensitive"/"sensitive" are host-independent. Idempotent.
  • coordination.py — Lease-by-append claims for multi-writer coordination: try_claim(store, resource, instance_id, ttl) / release / current_holder / fold_claims / compact_claims via fail-closed event appends. Accepts a StateAPI (.get) OR a raw EventStore (.read) — RS3-W fix: try_claim previously required .get() so every EventStore claim fail-closed to False (dead gate). TTL expiry is ENFORCED at fold time: a claim past ts + ttl is ignored/reclaimable (crashed holders cannot hold forever; legacy events without ts/ttl never expire). RS5 F2: a try_claim whose post-append read/fold raises retracts its own claim_requested (best-effort) before returning False — a failed claim attempt never strands a phantom holder for a full TTL. Claims-stream compaction: compact_claims(store) snapshots the active claim events for O(tail) reads instead of O(total). It keeps every claim_requested the fold would still count (unexpired, not followed by a same-instance release), each with its own ts/ttl, never only the first claim per holder (item 71b37d195787: a release+re-claim lost the live holder once the first TTL lapsed, letting a rival claim R). Prevents concurrent orchestrators from colliding. Snapshot optimization (2026-08-03): try_claim() now uses _read_claims_compacted() (snapshot + tail-replay, O(n)) instead of _read_claim_events() (O(n²) full scan on every parallel dispatch). Enables fleet to scale to larger claim histories without mutation cost explosion.
  • lease_claims.py — LeaseStore(db_path, case_policy=None, config=None): atomic file-path leasing for multi-instance coordination using a dedicated SQLite leases table (not event-sourced, for atomicity + TTL efficiency). API: claim(paths, instance_id, ttl_seconds)→lease_id (atomically claims set of paths; raises LeaseConflict if any held by another active instance), renew(lease_id, instance_id, ttl_seconds) (extends TTL; fails if not holder), release(lease_id, instance_id) (releases lease; frees paths for others), get_holder(paths). Deterministic time injection (clock parameter) enables testing without sleeps. Expiry-steal-on-claim: leases past deadline are reclaimable immediately at claim time (no background thread). Callers must size ttl_seconds to cover the work it guards: tools/multi_dispatch.py uses claim_ttl_seconds(--timeout, lease_ttl) (timeout + 60s margin, floor = multibox lease_ttl), since a crashed holder never releases and its lease then expires at that TTL. All operations fail-closed: exceptions propagate.
    • Claim keys are host-independent. _normalize_path(path, case_policy=None) is the SINGLE normalization entry point used for both storage and conflict checking, and delegates to paths.canonical_claim_path. It must never infer the policy from os.name: with case_policy="platform" the same file keyed as runner.py on Windows and Runner.py on Linux, so _check_conflicts (exact match) missed and BOTH instances were granted the claim — split-brain. Any guard test proving host-independence must call _normalize_path (calling canonical_claim_path directly with a policy production never passes proves nothing).
    • Case policy resolution (resolve_case_policy(config)): explicit case_policy= arg → config["multibox"]["case_policy"] → $AESOP_CLAIM_CASE_POLICY → DEFAULT_CASE_POLICY ("insensitive"). An unrecognized policy raises ValueError (fail-closed) — a silent fallback would change the keyspace unnoticed, the same bug class again. When no explicit policy or config is given, LeaseStore defers resolution to call time so env changes apply without restart.
    • Why "insensitive" is the default: it is the only policy that is identical on every host (what multibox requires), and it errs toward OVER-collision. A claim keyspace may safely collide two distinct files (the loser waits); it may never fail to collide two names for the SAME file (both instances write it). Case-sensitive single-box semantics remain available via LeaseStore(db, case_policy="sensitive") or multibox.case_policy.
  • claim_backend.py — ClaimBackend protocol + LocalLeaseBackend(db_path, config=None) adapter over LeaseStore; get_backend(db_path, config) returns the backend when multibox.enabled=True, else None (legacy advisory path). Config is passed through to LeaseStore so multibox.case_policy reaches the claim keyspace.

Invariants

  • Append-only: never mutate/delete events; state changes are new events.
  • Per-stream version is 1-based and gapless (enforced atomically).
  • git as export, not source: nothing here reads git for state.
  • Round-trip fidelity: ingest → project → export reproduces the same items (tested against the real state/tracker.json).
  • Every writer appends events; projections are never patched (GAP, 2026-10-05): tools/tracker_guard.py's cmd_enforce() used to close zombie items by writing state/tracker.json directly, with no item_updated event. Because tracker.json is RE-RENDERED FROM THE EVENT LOG on every WriteAPI call anywhere (any mutation re-projects the whole tracker), an un-journaled lane change is correct only until the next unrelated write, at which point it silently resurrects to whatever the event log still says — exactly the "DRIFT DETECTED" tools/state_rebuild.py --check reports for tracker.json when this happens. Fixed: cmd_enforce() now closes items through WriteAPI.tracker_update_item(); tracker_guard.py --reconcile-journal backfills the item_updated events lost to the old direct-patch history (idempotent; proven on a copy of the real log: 46 drifted items -> 0 -> 0 on a second run). Same incident also surfaced that write_api.py's _render_tracker_atomic() hand-rolled its own json.dumps() instead of calling the canonical materialize_tracker() (state_store/materialize.py) — byte-different (no trailing newline) from what state_rebuild.py --check compares against, so EVERY tracker.json write drifted against that gate regardless of content. Now unified: the writer calls materialize_tracker() too, so there is exactly one renderer for the view.

CI Isolation & Concurrency Gotcha

SQLite tests deadlock under parallel CI shards (false positive; no code defect). When running the unittest suite under parallel CI shards, multiple test files may contend on filesystem-level WAL locks. Solution: Use _retry_on_db_lock(func, max_retries=3, delay=0.1) wrapper for DB initialization and appends; apply exponential backoff. Real fix = per-shard DB isolation (future work). On CI re-run, the shard passes.

Test Commands

Run from repo root. npm run test:py runs ALL Python suites (includes state_store). Individually: python -m unittest tests.test_state_store (core API, concurrency, round-trip) | .test_state_store_occ (Phase 2 OCC multi-process: exactly-one-succeeds, no-write-on-conflict, retry-convergence, backward-compat) | .test_state_store_concurrency (Phase 1 multi-process coordination: claims, leases) | .test_state_store_hardening (corrupt event handling, input validation) | .test_state_store_snapshots (snapshot read/write, tail-replay).

Agent Lifecycle Events (Wave-29)

New event types (additive, appended by UI collectors on agent phase changes), all with payload {agent_id, timestamp}: agent_dispatched (dispatch start), agent_working (work in progress — thinking/tool-use), agent_done (completion), agent_stalled (stall/error detected). Projection: project_agent_lifecycle(events) folds these into per-agent lifecycle state with transition history (state + timestamp), letting the Activity view show agents entering/leaving states over time.

Multi-instance coordination (MVP)

New module: instance_projection.py — event-sourced instance lifecycle tracking. Event types: instance_registered, instance_heartbeat, instance_failed, file_claim_requested, file_claim_released. Projection tables: instances (id, hostname, pid, status, heartbeats) and file_claims (instance+paths). Core API: register_instance(), heartbeat(), claim_files(), release_files(), list_active_instances(), detect_stale_instances(), get_claimed_files(), get_all_claimed_files(). Stale detection uses a configurable threshold (default 300s); crashed instances' claims become reclaimable after TTL. Fail-closed: every operation degrades to False or an empty collection on error. Instance coordination is a prerequisite for multi-machine orchestration (team-scale single-project development); used by tools/instance_manager.py (CLI) and tools/multi_dispatch.py (dispatch guard). Claim-read fail-closed: get_all_claimed_files_strict() raises ClaimStateUnknown on a read error (logged with the store path); driver/wave_loop.py advisory check uses it and skips the item (the lenient get_all_claimed_files() still returns {} for its other callers).

Multibox Increment 1 (2026-08-02) — canonical repo-path normalization

Inc 1 fixes defect (b): _normalize_path() was host-platform-dependent, causing heterogeneous-box split-brain. Two instances (Windows + Linux) would canonicalize the same path differently and both claim it (47c967b P0 recurrence). New module: paths.py — canonical_claim_path(path, repo_root=None, case_policy="platform"|"insensitive"|"sensitive") -> repo-relative (if repo_root given), forward-slash-only, .. collapsed, NFC-normalized Unicode, case-folded per policy (not os.name). lease_claims._normalize_path now thin alias with case_policy="platform" for backward compatibility (all 18 existing tests pass untouched). Tests: 22 new in test_state_store_paths.py (four 47c967b regressions through canonical form, heterogeneity guard with monkeypatched os.name, Unicode NFC/NFD equivalence, separator idempotence) + 2 new heterogeneity regressions in test_lease_claims.py. Invariant: same path normalizes identically whether running on Windows or Linux (when case_policy specified); default "platform" preserves exact byte-for-byte behavior.

Multibox Increment 2 (2026-08-02) — atomic dispatch claims seam (TOCTOU fix)

Inc 2 fixes defect (a): multi_dispatch TOCTOU (time-of-check to time-of-use) race. check_conflict() + claim_files() were separate ops with no lock; concurrent claims on same path both succeeded. New module: claim_backend.py — ClaimBackend protocol (claim/renew/release/holder) + LocalLeaseBackend adapter over LeaseStore (atomicity inherited via BEGIN IMMEDIATE). get_backend(config) returns LocalLeaseBackend if multibox.enabled=True, else None (advisory path). tools/multi_dispatch.py updated: calls backend.claim() atomically when flag on (exit 1 on ClaimConflict, no record written), keeps legacy check_conflict+claim_files byte-for-byte when flag off. instance_projection.claim_files() docstring: marked advisory-only (projection/dashboard feed, not mutual exclusion). Canonical paths in lease_claims.py (2026-08-03): case policy now configurable via explicit-arg > multibox.case_policy > $AESOP_CLAIM_CASE_POLICY > "insensitive" (was hard-coded "platform"); resolves heterogeneous-box split-brain (B1 defect, 47c967b) and vacuous heterogeneity guard (B2 defect). Tests: 10 new in test_multi_dispatch_claim.py (TOCTOU concurrent-claims race, conflict-no-record-written, flag-off legacy path) + 22-test contract suite in test_claim_backend.py (reusable by Inc 4a FsClaimLog). Invariant: exactly one concurrent claimant succeeds; loser gets ClaimConflict with fail-closed record.

Increment 1 (state consolidation, 2026-07-30) — canonical materializer + state_rebuild

Inc 1 consolidates all view rendering to ONE place. materialize.py: canonical pure projector functions for all views (tracker.json, orchestrator-status.json, STATE.md, ledger); each view is (projection_dict) -> bytes (deterministic, idempotent, testable). Callers render each view directly via its own materialize_*() function (materialize_tracker, materialize_orchestrator_status, materialize_state_md, materialize_ledger); there is no materialize_all() umbrella — it was removed 2026-10-06 (inert-code cleanup) after a repo-wide grep found zero callers (every write path, including WriteAPI, already called the view-specific functions directly under its own lock). tools/state_rebuild.py: CLI --all | --view NAME | --check rebuilds and verifies materialized views; --check is the CI drift gate (renders to memory, diffs disk, exit 1 on drift). ui/collectors.py (refactored): the independent save_tracker() + .tracker-render-dirty self-heal was DELETED; all tracker CRUD now routes through WriteAPI, which calls materialize_tracker() under the same lock, while load_tracker() still reads the materialized tracker.json as a cache that mutations keep in sync. export.py (thin alias): DEPRECATED; now wraps materialize_tracker() for backward compatibility. DOC DRIFT FIX: state_store/CLAUDE.md and ui/CLAUDE.md both claimed tracker.json is "git-committed" — it is not; corrected to "materialized view, git-ignored, rebuildable" (R2 rule). Outcome: single canonical render path (materialize + WriteAPI), OCC-protected CRUD, deterministic views; baseline delta ~0-2 keys (this increment buys safety, not ratchet points).

State drift gate (F-02, 2026-10-14): CI job state-drift-gate (.github/workflows/ci.yml) runs state_rebuild --all then --check on a fresh state root, fail-closed. A stored STATE.md (state_markdown stream) is materialized verbatim; only the no-body fallback embeds a timestamp, which --check masks. Every DB path resolves via state_store/paths.py::resolve_db_path.

Increment 2 (state consolidation, 2026-07-30) — orchestrator_status into the store

Inc 2 moves orchestrator_status into the event store. projections.py: project_orchestrator_status(events) folds phase_changed, activity_changed, status_cleared and historical meta/phase_set into the byte-compatible orchestrator-status.json shape {id, role, activity, phase, updated_at}. api.py: registered project_orchestrator_status in the _PROJECTORS dict (already exported by __init__.py). write_api.py: set_orchestrator_status(activity, phase, id, role) and clear_orchestrator_status() append events FIRST then materialize the view atomically — fail-closed, an event-append failure blocks the projection write — via the _project_orchestrator_status() / _render_orchestrator_status_atomic() helpers. read_api.py: read_orchestrator_status() reads projection-first (if DB present), falls back to the materialized file if the DB is absent, preserving fail-open-to-None and future-timestamp-is-stale (age < -120) semantics. materialize.py: materialize_orchestrator_status(projection) renders the projection to bytes (indent=2, newline-terminated, deterministic). tools/orchestrator_status.py: CLI delegates to WriteAPI (set/clear commands); stdout stays byte-identical for shell tests ([OK] Status updated: ..., [OK] Status cleared). tools/healthcheck.py: routed _check_orchestrator_status → _check_orchestrator_status_api to use ReadAPI (projection-first with file fallback).

Event types (appended to the orchestrator_status stream): phase_changed payload {phase, timestamp, actor} (new, Inc 2); activity_changed payload {activity, timestamp, actor} (new, Inc 2); status_cleared payload {} (new, Inc 2); meta / phase_set payload {phase} (historical, from reconcile.py --resolve; folded forward). Byte-compatibility: the projection renders byte-identical to the current orchestrator-status.json shape, and views written by materialize_orchestrator_status() are deterministic and idempotent. Baseline movement: 2 keys retired (43 → 41); materialization views (status-json, tracker-json, state-md-write) are legitimately appended to baseline because they write derived caches from projections.

Increment 3 (durable identity + epoch fencing, 2026-08-02) — multibox coordination baseline

Core API: get_identity_with_epoch(state_root) persists durable identity (stable_id, epoch) to $AESOP_STATE_ROOT/instance-id as JSON. Epoch is a monotonic boot counter; simulated restart increments it. Distinction (fail-closed on corruption): Fresh box (no prior file) creates epoch=1; corrupt existing file raises IdentityCorruptionError (fail-closed) to preserve epoch monotonicity for Inc 4+ fencing. Fail-open exception: only OSError on FRESH box creation falls back to ephemeral (allows solo mode on unwritable state root). Corrupt file when prior id existed is a hard error; caller must handle recovery. Tests: fresh-box creation, corrupt-with-prior-file raises, epoch monotonicity across restarts, AESOP_STATE_ROOT respected, backward-compat shape. release_own_stale(state_root, stable_id, prior_epochs) API placeholder for Inc 5 coordinated reclaim. Export: IdentityCorruptionError added to state_store.init public API.

Multibox Increment 4a (2026-08-02) - FsClaimLog: shared-filesystem lease-by-append

New module: fs_claim_log.py carries the claim log (and ONLY the claim log) onto a shared FS; state.db stays on local disk because SQLite WAL needs an mmap-backed -shm index coherent only within one host. Records are immutable JSON, one file per event, named <lamport>-<epoch_ms>-<instance_id>-<uuid4>.json - unique by construction, so the backend uses NO filesystem mutual-exclusion primitive at all (no advisory file locks, no exclusive-create, no hardlink tricks); instance_id is sanitized in the filename only (':' is illegal on Windows), the true value is always read from the record body. Kinds: claim_requested | claim_released (tombstone) | heartbeat (renew). fold_fs_claims(records, now, max_skew, detail=False) is a PURE function over a list of dicts - no FS, no clock, no sleeps - and is the entire correctness surface; it mirrors coordination.fold_claims: lowest sort key (lamport, epoch_ms, instance_id, uuid) wins each path (deterministic total order, no clock sync needed for ordering), TTL enforced at fold time so a crashed holder is reclaimable, a tombstone releases the lease it names and a heartbeat cannot resurrect it, a heartbeat extends the deadline without changing the sort key, legacy records with no ttl never expire, and max_skew is only ever ADDED to epoch_ms/1000 + ttl so a skewed peer clock stalls throughput but can never double-grant. FsClaimLog implements the Inc 2 ClaimBackend protocol, so tests/test_claim_backend.py::ClaimBackendContractTests runs against it unmodified (imported and re-parametrized, not copied); claim() runs the settle-window protocol - write request + fsync, sleep settle_seconds (must exceed the measured p99 cross-box directory-visibility delay; Inc 0 measures, Inc 7 gates), force a fresh listing, fold, grant iff this exact record wins every requested path, else append our OWN tombstone and raise ClaimConflict (RS5 F2 retract: a losing request never strands a phantom holder for a full TTL). renew() APPENDS a heartbeat and never mutates a record; release() appends a tombstone; injectable clock/sleep make every path testable without real sleeps; all keys go through canonical_claim_path(case_policy="insensitive") (over-colliding costs throughput, under-colliding costs correctness). Corrupt records fail CLOSED, unlike store.py: a truncated/unparseable/empty record might be somebody's live claim on the path we are about to take, so it folds into a live claim by an unknown holder covering every path (reserved key FS_UNKNOWN_PATH = "*", holder FS_UNKNOWN_HOLDER = "<unknown>") until mtime + default_ttl + max_skew; an unreadable directory folds the same way, because an I/O error must never read as "nothing is held". Tests (67 in tests/test_fs_claim_log.py): 26 pure-fold table cases (sort-key precedence at each of the four levels, tombstones, expiry/reclaim, ttl-less legacy, skew lengthens-never-shortens, heartbeat extension, corrupt blocks then expires, order-independence), 25 tmpdir cases with an injected clock and settle=0 (record shape, filename uniqueness, lamport monotonicity incl. adopting a peer's high-water mark, self-tombstone on loss, settle window observed, source-level assertion that no FS locking primitive is used), the 11 imported Inc 2 contract tests, and 5 replaying the 47c967b split-brain regressions (separator, case, renew-on-expired, renew-on-released) through FsClaimLog. Flag: reachable only when multibox.transport == "shared-fs" (Inc 7). Deferred to Inc 4b (now SHIPPED, see below): temp-name + os.replace + parent-dir fsync durability, the measured skew matrix, and compact(retain_seconds) GC.

Multibox Increment 4b (2026-08-02) - FsClaimLog durability + skew bounds + GC

Inc 4b closes the three things 4a deliberately left open, all in fs_claim_log.py. Durability: _append no longer writes the final name directly - it writes a temp name in the same directory (<final>.json.tmp; os.replace is only atomic within one filesystem), flush() + os.fsync(fileno) to make the BYTES durable, os.replace to publish the name atomically, then _fsync_dir() to make the directory ENTRY durable. That order is the contract: a peer must never see a record name whose bytes are not yet durable, and no reader may ever see a partially written .json. On POSIX an fsynced file can still be absent from a peer's listing after a crash (the entry is a separate write), so _fsync_dir does os.open(dir, O_RDONLY) + os.fsync(fd); Windows refuses to open a directory as a file, so that raises and falls through to _flush_dir_windows() - CreateFileW with FILE_FLAG_BACKUP_SEMANTICS + FlushFileBuffers via ctypes, best-effort and silent on failure because os.replace is already atomic there (MoveFileEx/REPLACE_EXISTING) so a refused dir-sync degrades durability, never correctness. Two Win32 gotchas, both caught by asserting success against a REAL directory instead of a mock: FlushFileBuffers requires GENERIC_WRITE - with GENERIC_READ the handle opens fine and the flush then fails ERROR_ACCESS_DENIED (5), i.e. the helper degrades to a silent no-op, precisely the failure a best-effort path hides best (this was a live defect found by the test, not a hypothetical); and argtypes/restype must be declared or the returned HANDLE is truncated to a C int on 64-bit. The temp suffix is deliberately NOT .json, so _read_records skips it: a crash mid-write leaves a file that was never published, no grant was ever made from it, and ignoring it reproduces the exact pre-write state - this is not a weakening of 4a's fail-closed rule, a truncated .json WAS published and still blocks every path.

Clock skew - writer side (4a shipped the reader/fold side). Ordering stays lamport-keyed and skew-immune; only TTL uses wall clock. Two guards: (1) _writer_epoch_ms() clamps our stamp so it can never regress - an NTP step backwards would otherwise publish a record that looks OLDER than one we already wrote, shortening our own lease; clamping upward only LENGTHENS, the safe direction. (2) _assert_peer_skew_within_bound() raises ClockSkewError (new; fail-closed, nothing written, nothing granted) before claim() writes anything, when a peer record is stamped past now + max_skew_seconds. Rationale: the fold's + max_skew only makes "skew lengthens, never shortens" true INSIDE the bound - a peer BEHIND by more than the bound would have its live lease expired early by our fold, a double-grant. That direction is undetectable from a record alone (an old-looking stamp is indistinguishable from an old record), but the symmetric AHEAD direction is directly observable, so we fail closed on the observable half and leave measurement of both halves to Inc 0 preflight / Inc 7 startup gating. Our OWN records are exempt (guard 1 clamps them forward by design). Skew matrix proven: for every |skew| <= max_skew the computed deadline epoch_ms/1000 + ttl + max_skew is >= the true deadline, so no early expiry ever; the past-bound cell is asserted as the falsifiable case (it DOES expire early), which is what makes max_skew a load-bearing measured precondition rather than decoration.

GC: compact(retain_seconds=0.0) -> int deletes only what it can PROVE dead past the full bound ttl + max_skew_seconds + settle_seconds + retain_seconds; anything unprovable is kept forever. Group-atomic: all records of a lease (request + heartbeats + tombstone) go together or not at all - deleting a tombstone while its request survives would RESURRECT a released claim, and deleting a request while its heartbeats survive would silently shorten a live lease; deleting the whole group is safe because a peer seeing no records concludes exactly what a peer seeing the tombstone concludes (path free). Never live: a lease still winning a path in the current fold is untouched (belt-and-braces on top of the deadline test), and the + max_skew term keeps that true for a skewed peer. Never unprovable: no group containing a ttl-less legacy record is ever collectable. Tombstoned leases are held to the same full bound (a tombstone frees the path at FOLD time, not at GC time) so no deletion depends on how fresh a peer's listing is. Corrupt records have no readable deadline, so mtime is the only evidence: collected only past mtime + default_ttl + max_skew + settle + retain, the same bound at which the fold stops treating them as a live unknown-holder claim. Idempotent and concurrency-safe: a file another compactor already removed is not an error; an unreadable directory collects nothing. Tests (42 in tests/test_fs_claim_log_durability.py, 4a's 67 unchanged and still green): 12 durability cases asserting the call order over a recording os shim (fsync -> replace -> open/fsync/close, plus the refuse-dir-open platform branch, same-directory temp name, temp invisible to fold, truncated .json still blocks) - the proof is the ORDERING, since on tmpfs no assertion about a real fsync's effect is possible; 6 pure skew-matrix cells + 7 writer-guard cases; 17 GC cases (never-delete-live incl. the skewed-clock and heartbeat-extended cases, settle margin, ttl-less legacy, group atomicity, corrupt-by-mtime both sides of the bound, idempotence, concurrent-removal race, and compact never changes the answer of the fold).

Multibox Increment 5 (2026-10-05) - failover: stale detection, primary election, generation fencing

New module: failover.py implements three coordination pieces for multi-instance primary election and stale-instance detection. (1) Tier-S heartbeats (HeartbeatDir): instead of appending events (which would bloat a shared log), heartbeats are single files per (instance_id, epoch), atomically REPLACED every heartbeat (instances/<instance_id>.<epoch>.hb). Directory stays bounded by fleet size forever; same fsync ordering as Inc 4b claim log. detect_stale_instances() gains optional transport-aware source; 300s default unchanged. (2) Primary election (elect_primary): the primary is simply the holder of a reserved resource orchestrator_lock, taken through the same claim protocol as any file — same settle window, deterministic sort key, TTL expiry at fold time. No separate election algorithm, no consensus, no quorum; the shared log is the arbiter. A primary that stops renewing has its lock expire and any live instance may take it. (3) Generation fencing (Increment 5 F1-F3): a generation counter (monotonic per state root, like epoch) prevents a partitioned primary from resuming after reunion — on comeback, a write carrying the old generation raises FencedWriteError (state_store/failover.py, assert_fenced) once a newer generation exists (fail-closed). Tests: 15 in tests/test_failover.py covering heartbeat directory lifecycle, stale instance detection, primary election determinism and TTL expiry, generation fencing fail-closed semantics, and heterogeneous transport (shared-fs vs advisory log). Flag: election and fencing reachable when multibox.transport == "shared-fs" (Inc 7); heartbeat dir always available for monitoring/debugging. Invariants: only one instance holds orchestrator_lock at any fold time; TTL prevents zombie holders; generation prevents split-brain resumption.

Pyflakes cleanup (2026-10-06, Guardrail G15): removed genuinely-unused imports/locals flagged by tools/pyflakes_gate.py from api.py, identity.py, lease_claims.py, write_api.py; zero behavior change, full suite green.

Orchestrator goal stream (2026-10-15)

tools/goal_state.py appends goal_set events to stream orchestrator_goal (append-only, never patched); its project_goal folds the latest one, and tools/boot_state.py reads it as the GOAL section of the canonical boot brief.

Task-event stream (2026-10-15)

tools/ingest_task_events.py tails the brain's ~/.claude/state/task-events.jsonl (written by the brain hook; this code only consumes it) into stream task, event type task_event, in the tracker DB (resolve_db_path(state, "tracker")). Idempotent via cursor state/task_events.cursor.json plus sha256 event_id dedupe; fail-closed (exit 3) on malformed or truncated input; kind: "internal" tags harness suggestion echoes (empty agent_type AND user-voice or slash-command summary). The store ts is the ingest time; the source time is payload.ts.

Tracker adopt mode (2026-10-16) — disk-only items into the log

WriteAPI.tracker_adopt_disk_items(actor, reason, ids=None, dry_run=False) is the only sanctioned path for items present in tracker.json but absent from the event log (the set the OCC guard refuses on). Each adopted item gets an item_created carrying its full disk record plus adopted_from_disk, adopt_reason, adopt_actor, adopt_ts (those four fields are therefore present on the rendered item); a non-open status also gets an item_updated transition. Refused unless: non-empty reason (non-dry-run), every id disk-only, and the set covers ALL disk-only items (a partial adopt would make the render drop the rest). The fold is verified against the disk records before any append and again before the render. CLI: tools/tracker_guard.py --adopt-disk-items --reason TEXT [--ids a,b] [--dry-run]. Tests: tests/test_tracker_adopt.py.

Next (cutover, follow-up — NOT this increment)

Phase 1 (middle): Tracker dual-read (StateAPI for CRUD, export job keeps tracker.json rendered). Phase 2 (cutover complete): Flip all readers to API; remove git fallback. Phase 3 (optional, contingent on team scale needs): Backend swap behind StateAPI (e.g. Postgres). Not scheduled; single-box SQLite is sufficient for current throughput (~100 ev/s real-world vs ~704 ev/s measured ceiling). See docs/MULTI-INSTANCE-ROADMAP.md for the decision tree.

Map of all domains: /CLAUDE.md

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.