CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Project Status

EKOS has an implemented Rust (2024 edition) Cargo workspace — this is not a design-phase repo. Read devlogs/devlog_*.md (numbered chronologically — the highest-numbered file is the latest, devlog_329.md as of this writing) before starting non-trivial work: they are the project's long-term memory and record what shipped, why, and what was learned. TODO.md tracks the phase-by-phase roadmap; RFCs are split across two locations for historical reasons, not a meaningful distinction — docs/rfcs/ (repo root) has 0001–0024, ekos/docs/rfcs/ has 0025+ (highest: 0251; 0213–0218 are reserved by the Oracle/iDempiere plan for Parts II–V, 0217 still unwritten). Check both directories for the highest existing number before picking one for a new RFC — two RFCs have already collided on the same number once (0027) from sessions that only checked one location.

The Cargo workspace root is ekos/, not the repo root — there is no top-level Cargo.toml. benchmark/ and tests/integration/ are separate Cargo workspaces that depend on the ekos/ crates by path.

Commands

# Main workspace (run from ekos/, or pass -p <crate>/--manifest-path from elsewhere)
cd ekos
cargo build --workspace
cargo test --workspace
cargo test -p ekos-ledger                    # one crate, e.g. ekos-ledger, ekos-plugin-git
cargo test -p ekos-ledger some_test_name     # one test
cargo clippy --workspace -- -D warnings      # CI fails on any warning
cargo fmt --check                            # CI checks formatting, doesn't fix it
../scripts/cognitive-complexity.sh           # functions over Sonar's complexity limit (S3776), no scan needed
../scripts/test-install.sh                   # install.sh against a local fake release (RFC 0179)
../deploy/e2e/package.sh                      # full package on Linux end to end: setup --answers, the console
                                             # (local sign-in, settings + restart, pipeline), the wizard in a
                                             # browser, uninstall — HOME is a scratch dir. RFC 0185
../scripts/package-release.sh                # offline install package (Linux x86_64, glibc + static musl,
                                             # SHA256SUMS, install.sh, INSTALL.md) → dist/; committed tree only
../deploy/e2e/run.sh                         # the production image end to end: Dex sign-in, two tenants,
                                             # roles, gateway, an MCP client, every page in a browser,
                                             # backup/restore. Needs `docker build -t ekos:e2e .`
                                             # first; nothing else covers real OIDC or the image. No model call
../scripts/ossie-validate.sh                 # RFC 0245: exports → Ossie's own validate.py (export, ingest +
                                             # re-export, review round trip, planted conflicts); needs uv
../scripts/audit.sh                          # RustSec advisories + stale-lockfile check, all 3
                                             # workspaces (CI: audit.yml, also weekly). Run it
                                             # locally too: `[skip ci]` commits bypass CI

# Integration tests (separate workspace, depends on ekos/ crates by path)
cd tests/integration && cargo test

# Benchmarks (separate workspace, Criterion)
cd benchmark && cargo bench
cargo bench --bench ledger_write             # one benchmark file

# CLI (binary crate name is `ekos`, package is crates/cli)
cargo run -p ekos -- init
cargo run -p ekos -- build && cargo run -p ekos -- recover && cargo run -p ekos -- resolve \
  && cargo run -p ekos -- compile && cargo run -p ekos -- commit   # full pipeline, in order
cargo run -p ekos -- doctor
cargo run -p ekos -- project add shop --source <dir>   # RFC 0178: config + state under ~/.ekos,
cargo run -p ekos -- --project shop build               # sources never written to
cargo run -p ekos -- home backup                        # RFC 0183: the home as one .tar.zst;
cargo run -p ekos -- home restore <file>                # replaces nothing without --force
cargo run -p ekos -- ask "<question>"        # compiled REASON answer (--classic / --explain)
cargo run -p ekos -- query find "<text>" --mode hybrid   # SEARCH: BM25 + vector, RRF-fused (RFC 0118/0125)
cargo run -p ekos -- ekl "FIND Object WHERE kind = 'Table'"  # Enterprise Knowledge Language
                                             # (entities are Object / Relationship only)
cargo run -p ekos -- identity scan          # cross-system candidate matches (RFC 0029)
cargo run -p ekos -- marketing publish      # devlog -> tweet -> approval -> X (RFC 0030)
cargo run -p ekos -- marketing post <file> --confirm   # a file's text to X as written (no LLM, no 280 cap)
cargo run -p ekos -- mcp serve --workspace <dir>
cargo run -p ekos -- docs generate --layout curated --output doc   # README/Architecture/API/
                                             # SequenceDiagrams + per-entity pages (RFC 0035/0037/0042)
cargo run -p ekos -- simulate <scenario.yaml>       # World Engine: load + run a scenario (RFC 0047-0055)
cargo run -p ekos -- replay <scenario.yaml>         # read back a previously recorded simulation, read-only
cargo run -p ekos -- eval bench prepare --suite bird-mini-dev   # RFC 0240/0241: BIRD Mini-Dev → one
cargo run -p ekos -- eval bench run --suite bird-mini-dev --cell g --split tune   # project per DB;
                                             # cells in evals/bench/<suite>/configs/ (non-Ollama needs --allow-metered)
cargo run -p ekos -- sql check q.sql --database pg      # RFC 0242/0243: lint + read-only probes;
cargo run -p ekos -- sql generate "<question>" --sqlite f.db   # agent over the ledger schema
cargo run -p ekos -- semantics gaps                 # RFC 0170 (`[semantics] enabled = true`): business-semantics
cargo run -p ekos -- export linkml --status hypothesis --out model.yaml   # hypotheses → draft LinkML
cargo run -p ekos -- export ossie --status all --out m.ossie.yaml   # RFC 0245: Apache Ossie model;
cargo run -p ekos -- import ossie m.ossie.yaml --dry-run            # edits → review decisions

CI (.github/workflows/ci.yml) runs build+test+clippy+fmt from ekos/ and cargo bench from benchmark/ on every push/PR to main. Match these locally before pushing.

High-Level Architecture

EKOS is a compiler for enterprise knowledge, not a database or document store. It observes enterprise systems without interpreting them, compiles those observations through deterministic passes into a Canonical Knowledge Model, and stores the result in an append-only ledger where every conclusion carries the evidence it was derived from.

Enterprise Systems → Observation Layer → Knowledge Compiler → Canonical Knowledge Model (CKM)
                                                                          ↓
                              AI/Apps ← Knowledge Runtime (read-only) ← Semantic Knowledge Ledger

This maps directly onto the ekos init/build/recover/resolve/compile/commit CLI pipeline: each verb is a compiler stage, run in that order, writing artifacts the next stage consumes.

Crate map (ekos/crates/)

RFC 0246 consumers all read ekos_ossie::knowledge::resolve(objects, relationships, current_files) (metrics rewritten over tables by an AST walk, joins as column pairs, skipped with reasons — never guessed):

• sqlgen/src/ossie.rs (Schema::with_ossie: synonyms, unique_keys, is_time, declared_joins beside foreign_keys — read every join through all_joins(), metrics); the agent offers the metric tool only via protocol_with(.., metrics), so BIRD prompts stay byte-identical; lint L012;
• dbt-gen/src/semantic.rs (ekos export dbt-semantic; MetricFlow needs a time dimension on a measure's own model, else skipped);
• business semantics (ossie_meanings source ossie, ossie_gloss never makes a gap; the commit step drops non-current Ossie objects first);
• runtime/src/retrieval.rs::synonym_entities (keyword searches, ResolvedEntity.via);
• EKOS Migrate (ossie_keys → DocCheck source ossie; tier V2Metrics/v2m in migrate-validate/src/metrics.rs, with a planted offset control; live test in tests/live_tiers.rs) | | runtime | Read-only state reconstruction and context projection. ai.rs is the ekos ask grounding+citation pipeline; retrieval.rs is RFC 0121 query understanding (understand → QueryType); reason.rs is RFC 0123's Query Plan IR + rules planner + typed EvidenceSet; retrieval_eval.rs is RFC 0126's CI-gated retrieval quality harness. RFC 0177: a search hit's claim shows the window of its text that matched the query (reason.rs::matched_window), a symbol's source text falls back to its own evidence fragment (a commit records a ka: knowledge-artifact id, which holds no file text), a requested dependents traversal from M::self includes the dependents of every M::… import-path object, and Runtime::excluding_from_search exists for ekos eval run only. | | ekl | Enterprise Knowledge Language — parser.rs + interpreter.rs for the ekos ekl query command. A field that is not a built-in column is read from properties (only the fields a query names are copied into its rows; a.b is a nested path), and a WHERE/ORDER BY/GROUP BY field no candidate has is EklError::UnknownField (RFC 0175). Supports AS OF <ts> / COUNT / GROUP BY (RFC 0096) and SEMANTIC 'text' [LIMIT k] retrieval candidate sets (RFC 0124). | | docs-gen | Deterministic Markdown/HTML rendering from the compiled ledger (RFC 0035) — render_object_page (--layout objects, one page per significant object) and render_readme/render_architecture/render_api/render_sequence_diagrams (--layout curated, RFC 0037/0042) — zero LLM calls; render_dependency_risk_report/render_onboarding_guide/render_findings_memo (--layout solution-architect, RFC 0090) for a team-handoff bundle; estate.rs (--layout estate, RFC 0214): a database estate as wiki pages in the lakehouse-docs page format, pure over objects + relationships + review scope; a column profile gives counts only (never min/max), and text is shown only when every source it came from has a known licence (licence_allows_text; the CLI commands/docs_estate.rs detects licence files and writes pages, keeping manual sections and deleting only pages with the ekos_estate_version: marker); --prose (opt-in) is the one path that layers an LLM overview on top, via ekos ask's own grounding+citation pipeline. render_architecture_summary's Executive Summary renders real Custom("Risk") objects (RFC 0094) and an ArchitectureConfidence score (RFC 0095, a small local struct mirroring ekos_recovery::EvaluationReport rather than a real cross-crate dependency — cli does the translation) when the caller has real signal, an honest placeholder otherwise — never a fabricated 100%. | | dbt-gen | Renders the Transformation IR (RFC 0027) into executable dbt SQL models with ref() semantics | | marketing | Devlog → tweet → human approval → X publish (RFC 0030) — auxiliary tooling outside the compiler pipeline, not a CompilerPass/Observer | | simulation | World Engine (RFC 0047-0055) — auxiliary, opt-in, deliberately separate from the compiler pipeline above: action.rs/decision.rs/simulation.rs (a closed 12-action vocabulary, a provider-independent DecisionEngine trait, a deterministic round-based loop with seeded priority/resource conflict resolution), scenario.rs (YAML scenario/agent definitions, ekos simulate's loader), forum.rs (channels/replies/likes/follows/shares), replay.rs (a durable per-ledger event log + point-in-time reconstruction, ekos replay), ingest.rs (world.sources: real documents via the actual localdocs connector + LocalDocAnalyzerPass, no LLM). Writes through &dyn KnowledgeStore directly, the same access level commit.rs has — Runtime stays read-only throughout, unmodified by this crate. Every RFC in this crate builds additively on existing KIR/ledger primitives (Custom() escape hatches, properties conventions) rather than inventing new storage | | session | RFC 0151 agent session memory: inbox.rs (capped, redacted, append-only JSONL inbox — the only thing an agent can write, never a ledger handle), map.rs/commit.rs (deterministic inbox → Session/SessionClaim KIR + ekos session commit), anchor.rs (exact-match anchors + per-kind staleness fingerprints), read.rs (recall/brief, staleness verdicts), lifecycle.rs (human-only confirm/reject/supersede — mcp.rs must never call it, a test enforces it), capture.rs (redacted transcript slices + strict-JSON LLM extraction that is only ever a T0 proposal), eval.rs (deterministic proxy eval). Every Runtime read path — not just find_objects/retrieve — hides Session/SessionClaim, and runtime::graph_export filters them separately because it reads the store directly; a guard test (every_object_returning_read_path_has_an_audited_session_decision) fails CI on a new unaudited pub fn. An edge counts as session memory only when an endpoint does — AnchoredTo/ObservedIn are a cheap pre-filter, never the decision | | migrate, migrate-validate, migrate-dq, migrate-target-clickhouse, migrate-approval, migrate-report, migrate-logic, pg-live, plpgsql | EKOS Migrate (RFC 0154-0167), ekos migrate *, opt-in via [migrate] enabled = true, PostgreSQL → ClickHouse. migrate = project/unit model + append-only state machine; pg-live = the third raw-content entry point (live PG catalog + profiling, runs its own redaction, sync postgres driver on a dedicated thread — never inside the tokio runtime); migrate-validate = RFC 0155 canonical serialization + RFC 0156 tiers V1-V3 with planted controls; migrate-dq = rules measuring affected rows, plus doc_claims (RFC 0172 Phase 3: checkable claims in column comments, and in Markdown/Confluence sentences naming one table.column — recorded at recover as column_mentions on Section/Page by recovery/src/column_mentions.rs — → DQ.CONSIST.DOC findings and doc_vs_data conflicts at migrate assess, kept in .ekos/conflicts/migrate.json so a commit never drops them); migrate-target-clickhouse = type map, design, statement classifier and the guarded executor; migrate-approval = computed risk + evidence-pinned, human-only approvals (commands/mcp.rs must never reach it — a source-scan test enforces it); migrate-report = compiled, citation-verified report; migrate-validate's dialects are PostgreSQL, ClickHouse, Trino and Oracle (RFC 0216, a lakehouse source: needs an AL32UTF8 database; UnitPlan::digest_columns hashes a row over column digests because an Oracle string stops at 4000 bytes; Trino's row string is a flat chunked concat, since the Trino read guard re-prints every query and a || chain recursed past a thread's stack — the SQL is pinned in tests/snapshots/); plpgsql = PL/pgSQL parser + procedural IR (RFC 0163), consumed by recovery's plpgsql_analyzer; not yet lowered into the Transformation IR (RFC 0164). migrate-logic = RFC 0164 Phase 1, pure (no LLM, no connection): lower.rs is a whitelist walker — every match ends in an arm that blocks by name, so add a rule there rather than passing a construct through — that turns a PostgreSQL CREATE VIEW into a ClickHouse one and must read back identically through the ClickHouse grammar; invention.rs reduces statements to atoms (tables, columns, literals, functions, comparisons) and rejects a candidate with an atom its source lacks; gate.rs is the fidelity gate; triggers.rs the per-class proposals. CLI: commands/migrate_logic.rs (ekos migrate lower / check-candidate), which re-reads definitions from source files by recorded span and therefore applies RFC 0043 exclusion + redaction itself. The engine differences the rewrites assume are measured in crates/migrate-logic/tests/clickhouse_semantics.sql (live suite). v5.rs is differential execution of a lowered view behind a Sandbox trait (fixture from the view's own comparisons, compared columns crossed, branch coverage measured on the source engine, not_run is never a pass); the engines are pg-live/src/sandbox.rs::PgSandbox — the only writable PostgreSQL session, a separate type from the read-only PgSource, one rolled-back transaction per run — and a ClickHouse database that is always dropped; ekos migrate v5 refuses a connection without sandbox = true or one that is a project's source. reconstruct.rs builds the prompt and judges a model's answer (gate before the call, anti-invention after); the call is in the CLI and needs --allow-metered unless the provider is Ollama. Proposals are Custom("MigrationProposal"), keyed by source object, always written status: proposed — nothing writes another status, and commands/mcp.rs does not reach any of this. Live suites: tests/v5_live.rs (planted defects) and tests/clickhouse_semantics.rs. Not built: V5 for routines, approval of proposals. Migrate kinds are built from constants, which every_pipeline_custom_kind_is_registered cannot see — every_migrate_kind_has_a_registry_row covers them. Live suites: docker-compose.migrate.yml + EKOS_MIGRATE_LIVE=1 | | cli | commands/ — one file per CLI subcommand, dispatched from crates/cli/src/app.rs (main_with; bin/ekos.rs is a shim); home.rs + commands/project.rs (RFC 0178): a project in the EKOS home (<home>/data/<tenant>/<project>/{ekos.toml,state/}), selected with the global --project/--tenant; main_with then loads that config strictly, sets EkosConfig::location and changes into [workspace] root. EkosConfig::ekos_dir is the only function that decides the state directory — never build a state path from cwd (the session inbox and default output directories did; see session::inbox_at, EkosConfig::output_base), and a project run must leave the source tree byte-identical (tests/project_home.rs). Part 2: [[sources]] names a project's inputs; EkosConfig::observe_bases is the only function that turns [[sources]] or [observe] paths into directories and keys (a source's key is its name — the RFC 0044/0079 project field), semantic/src/sources.rs attributes each object (source/sources, stamped on File objects at build and on compiled objects in commit.rs::write_objects; an object with neither must be treated as not visible by a source-restricted reader), commands/source.rs edits ekos.toml through toml_edit; also hosts the MCP server (commands/mcp.rs) and extension.rs, the RFC 0149 extension seam (EkosExtension: extra observers, recover passes, a post-commit step, MCP tools) that out-of-tree builds plug into. RFC 0191 widened it: commands() + command_classes() + run_command() add whole command groups (main_with parses the built-in tree plus Extensions::extend_cli, which refuses a taken name or a leaf without a catalogue class; an extension command becomes the skipped Commands::Extension variant and goes through the normal project/config setup; main_with_args exists for tests and installs the global logger, so call it once per test binary), and mcp_tool_is_scoped() lets an extension's own tool reach a restricted reader (built-ins are decided by SCOPED_TOOLS/SCOPE_REFUSED_TOOLS whatever an extension says). EKOS Lakehouse (RFC 0190) is the first such extension, in its own public repo alexeyban/ekos-lakehouse; its kinds are the Lakehouse* rows in custom_kinds::REGISTRY. RFC 0198: console_pages() declares console tabs as data (ConsolePage/Panel/View/Action in extension.rs); commands/extension_pages.rs checks them for ekos extensions --json (a panel runs only a read catalogue command with its own flags, an action needs no argument and is not local), and the console checks again (web/api/app/extensions.py, routes/extensions.py, pages/ExtensionPage.tsx — readproc.read_checked_json is only for argv extensions.panel_argv accepted). RFC 0207 (pages v3): ConsolePage.params are selections filled from a source panel's table rows; Panel.params append them as --name=value to a value option of the panel's read command (checked on both sides, ≤512 chars, no NUL); Action.binds/flags/confirm; views code/graph/markdown (the UI's own renderer, pages/markdown.tsx — never HTML strings); the page types derive Default, so construct with ..Default::default(). RFC 0226: Action.inputs are (argument, label) value arguments the person types (a reason, a note), checked like binds. POST /cli refuses --allow-metered unless the tenant allows metered calls. RFC 0200: EkosExtension::java_decompiler() / Extensions::java_decompiler() (the private ekos-binary build offers a JavaDecompiler; the public build none), CommandContext/RecoverContext carry extensions; Task::Translate ([llm.tasks.translate]); the Pentaho translator itself lives in ekos-lakehouse (crates kettle, kettle-expr, kettle-rel, pentaho-dbt), its kinds are the Pentaho* registry rows. RFC 0199: setup::Machine::program is the running binary's name, so an extension binary installs and starts the console as itself; scripts/package-full.sh takes EKOS_PACKAGE_PROGRAM/EKOS_PACKAGE_INSTALL_MD; migrate::ConnectionRef::secret_file (pg-live refuses a file others can read) for jobs that do not inherit a shell's variables. EkosExtension::binary() gives an extension binary its own name and version (--version prints <name> <version> (EKOS <x>)). The console registers home projects at start-up and on every project list (web/api/app/home_projects.py), so one created while it runs appears without a restart. RFC 0170: commands/semantics.rs (ekos semantics list|show|gaps|eval|confirm|reject|edit + the commit step) and commands/export.rs (ekos export linkml, default --status confirmed), commands/import.rs (ekos import linkml — YAML edits → review decisions by ekos_id, human-only like semantics confirm); the web console's Semantics tab (web/api/app/routes/semantics.py + semantics_write.py, web/ui/src/pages/Semantics*.tsx/Linkml.tsx) runs those CLI commands under the write role, never MCP; its Conflicts tab (RFC 0174, routes/conflicts.py + conflicts_write.py, pages/Conflicts.tsx) does the same for ekos conflicts resolve. RFC 0179: a registered workspace whose path is <EKOS_HOME>/data/<tenant>/<project> is a home project, and web/api/app/home.py is the only place that builds an ekos command line for a workspace (command, mcp_command) or names its state directory — a new subprocess call in the console must go through it, or it runs against the wrong directory for a project. RFC 0180 (console access control): web/api/app/auth.py only turns a request into a Principal (OIDC user, API token, or development static token); web/api/app/authz.py::authorize is the only place a role is compared, and every route reaches it through authz.authenticated / allow(action) / allow_tenant(action) / allow_platform — tests/test_authz_routes.py walks main.ROUTERS and fails on a route that declares none, or on a new route that decides its action in the handler without being added to the test's DYNAMIC list. A token principal is denied review, admin and platform inside authorize whatever its bindings say (review stays human-only at this layer too). A workspace or tenant the caller has no role in is 404, never 403. RFC 0181 in the console: a role binding may carry a rule; authz.scope_for turns a principal's bindings into the scope, deps.mcp_for_workspace returns a client that attaches it to every MCP call (the only way a route gets one), and allow(action, scoped=True) marks the few routes safe for someone who sees part of a workspace — every other workspace route answers them 403, and test_only_the_listed_routes_are_open_to_a_restricted_reader pins the list. RFC 0182: routes/mcp_gateway.py is the MCP endpoint for agents (POST /mcp/<tenant>/<project>, mounted outside /api — main.MOUNTS); it builds each forwarded call from the tool name and arguments only (a client's _meta never reaches the server), decides the needed right from mcp_policy.action_for, and audits every call. mcp_policy.SCOPED_TOOLS must equal SCOPED_TOOLS in commands/mcp.rs (a test reads the Rust source). The supervisor starts a project's server on first use (acquire), stops idle ones and caps how many run. RFC 0183: commands/home_backup.rs is ekos home backup|restore (a project is copied under fact_ledger::hold_write_lock; restore unpacks to a staging directory, accepts plain files and directories at checked paths only, and moves anything it replaces to trash/, never deleting). In the console every ekos subprocess gets its tenant's environment from envs.for_path / envs.for_tenant (a base list plus the variable names an owner gave that tenant, while allow_metered is on) — _proc.spawn / run_streaming must always be passed env=, and test_every_process_the_console_starts_is_given_an_environment scans for a call without it. RFC 0184: console settings also come from <home>/control/console.toml (precedence: environment, keychain for secret fields, file, default — Settings.settings_customise_sources); every Settings field needs a row in console_file.FIELDS (a test compares the sets); routes/platform.py (owner only) edits the file with tomlkit and stores secrets through app/keychain.py (keyring; entries env:<NAME> / console:<field> under service ekos), which never returns a value; tests get an in-memory keychain from an autouse fixture and must never touch the real one. envs.for_tenant fills a granted variable from the keychain. RFC 0185: commands/console.rs (launcher; stop/restart are files control/stop-requested/restart-requested that app/lifecycle.py watches when supervised — no cross-process signals), commands/setup.rs (answers → pure plan → apply through the Effects trait, so tests never start services; control/install.json is what uninstall undoes), setup_server.rs + setup.html (the wizard, token on every request). RFC 0188: uninstall_server.rs + uninstall.html (ekos uninstall --ui; the steps shown come from the pure setup::uninstall_plan, and a test holds them equal to what uninstall does — change both together); the console's POST /api/platform/uninstall (local mode, install.json present) runs uninstall --ui --detach, which under the systemd login service goes through systemd-run --user because stopping the unit kills every process in its cgroup. RFC 0189: a chained command's stage argv is Command.stage_args, its stages per workspace Command.stages_for (home projects skip init), each stage limited by the setting run_stage_timeout_seconds (single commands keep Command.timeout). The console's CI runs on Python 3.12 (Linux), macOS and Windows: tests must not assume /bin/..., POSIX permissions, #! fake binaries or / in paths — skip on Windows with the reason when they must. Never build a path for display or comparison with Rust's canonicalize (Windows \\?\): use home::canonical_dir. RFC 0186: a new CLI subcommand needs a row in commands/catalogue.rs::CLASSES (read / write / review / admin / external / local — who may run it from the web; a test fails without one); the console runs catalogue commands only through routes/cli.py + app/catalogue.py::build_argv (never a shell, never an unlisted argument, reviewer forced to the signed-in person); config sections derive schemars::JsonSchema for ekos config schema --json. RFC 0187: app/chat/ (providers over httpx, agent.turn) + routes/chat.py; the model is offered only tools with mcp_policy.action_for(tool) == "read" that mcp_gateway._Caller.may allows, through mcp_for_workspace (the scoped client); the chat code must never name a review tool (a test scans it); a metered provider (providers.is_metered) needs the tenant's allow_metered; tests use a scripted provider, the package check a local Ollama model — never a paid call. RFC 0251: platform_auth = "aca" — Azure Container Apps' authentication signs people in, auth._from_platform reads X-MS-CLIENT-PRINCIPAL-IDP/-NAME (subject <idp>:<name>; an API token still wins; no session, no static token), Settings.signs_in replaces oidc_enabled wherever the meaning is "people sign in"; signed_in_group adds a group every user principal (never a token) is in, in authz._bindings. Local mode (mode = "local"): LoopbackOnly middleware, one-time codes in control/local-signin/ (local_signin.py), principal kind local (an owner). docs/deployment.md is the operator's guide and docs/operator-workflow.md the seven-step workflow's (RFC 0209–0227); keep both in step with these RFCs. Schema changes to the console database go in models.MIGRATIONS (append a function; create_all never alters a table). The production image is the root Dockerfile (no BuildKit-only syntax: it must build with the classic builder) + deploy/docker-compose.yml; Dockerfile.dev and web/docker-compose.yml are development files. RFC 0232 (clouds): the image is built with the ekos crate's object-storage feature (ekos home backup --to / restore --from, commands/home_remote.rs; S3, Azure Blob, GCS — never the distributed feature), its entrypoint is scripts/ekos-entrypoint.sh (files in EKOS_SECRETS_DIR become variables) and it ships scripts/ekos-git-sync.py; web/api/app/cloud.py is the console's instance lease (instance_lease, one writer over a shared home) and scheduled backups (backup_url, run with cloud.platform_env, never a tenant's environment); deploy/terraform/modules/{common,azure,gcp,aws} × compute = vm | container | kubernetes and deploy/helm/ekos (one replica, Recreate) — CI's deploy job runs terraform fmt/validate and helm lint/template. freshness.rs (RFC 0171): ekos build records a per-file (path, size, mtime) manifest in the same walk as its fingerprint (observation_sdk::source_manifest / fingerprint_of — the hash must stay byte-identical, a test pins it), a successful ekos commit promotes it to .ekos/committed-manifest.json, and check diffs it against the disk for ekos status/doctor/freshness, FRESH001 at compile/commit and MCP (ekos_status block + a note on read results, memoised in StoreCache). Metadata only — never a raw-content entry point (the opt-in [freshness] content-hash = true stores a SHA-256 per file in the manifest so a touch is not a change: add_hashes at build, diff_with hashes only same-size/moved-time files at check, RFC 0043-excluded paths are never read; nothing from it reaches the artifact store or ledger); EKOS's own state dir is always excluded | | common | Shared utilities — ContentHash, zstd compression, and redaction (RFC 0043: built-in, non-disable-able secrets/PII pattern table + excluded-file globs, the single module both build.rs and recover.rs's direct-file-read blocks call before anything reaches the artifact store or ledger; a reader that re-reads a workspace file after build for a model or the ledger goes through redaction::read_redacted — exclusion refusal, then the whole file redacted — and git blame (commands/semantics.rs::GitBlame) redacts each commit summary, RFC 0043 amendment) | | scheduler, sql-dialect-sdk | Pass scheduling primitives; the SqlDialectParser trait every plugins/sql-dialect-* crate implements (RFC 0031) |

Connectors (ekos/plugins/)

Each plugin implements Observer from observation-sdk and is registered independently in ekos/Cargo.toml's workspace members. Real/tested: file, git, github, confluence, localdocs (PDF/DOCX/text/Markdown/HTML/email), pentaho (.ktr/.kjb, RFC 0027), crypto, python (real AST + PySpark DataFrame chains into the Transformation IR, RFC 0038/0040), rust (real AST + the first real Calls function-call graph, RFC 0041), java (RFC 0209: hand-written declaration parser in recovery/src/java_syntax.rs, JavaType/JavaMethod; app_model.rs turns iDempiere-family generated I_/X_ classes into ModelTables — a code claim, never merged into a database Table; java_sql.rs folds SQL built in method bodies; generated files get no JavaMethod objects; the kinds are built from constants, so every_java_kind_has_a_keyed_registry_row guards them), elixir/javascript (real AST + Calls recovery), clickhouse (real HTTP client, schema metadata + the opt-in live NL-to-SQL query engine, RFC 0056). sql-dialect-{postgres,mysql,mssql,snowflake,databricks,clickhouse,trino,oracle,sqlite} implement the SqlDialectParser trait (RFC 0031; trino is RFC 0190 — GenericDialect plus a preprocess that blanks time travel, ALTER TABLE … EXECUTE and TABLESAMPLE; the dbt analyzer falls back to it; oracle is RFC 0210 — GenericDialect plus a line-preserving preprocess that blanks SQL*Plus commands and whole PL/SQL units, and rewrites Oracle-only clauses, listed in the crate docs). Oracle scripts are split by crates/plsql's script module (Oracle quoting: '' escapes, backslash is plain, q'[…]'; units end at a / line; block comments do not nest — PostgreSQL's do, so never mask Oracle text with ekos-plpgsql's scanner). crates/plsql also reads PL/SQL routines, triggers and packages into RFC 0163's ProcedureIr, handing each BEGIN … END to ekos_plpgsql::parse_body with Origin { flavor: Flavor::PlSql } (PL/SQL-only statement forms are recognised only under that flavor, so PL/pgSQL parsing never changes); recovery/src/plsql_analyzer.rs writes them through plpgsql_analyzer::recover_parsed_routines, so both languages produce identical Procedure/ProcedureStatement objects, and an Oracle trigger's inline body is a Procedure named after it plus a Trigger whose function names it. recovery/src/schema_history.rs (RFC 0210) turns every <yyyyMMddHHmm[ss]>_<description>.sql migration script into a keyed SchemaChangeSet + one SchemaChange per DDL operation, classified from tokens (sqlparser 0.53 reads Oracle MODIFY c NOT NULL as a column of type NOT), including iDempiere's PostgreSQL INSERT INTO t_alter_column rows; semantic/src/schema_history.rs links changes to tables at commit (Changes, unique name only). RFC 0211: crates/ora-live is the fourth raw-content entry point (live Oracle catalog; driver oracle behind the crate feature oracle, off by default; Instant Client on LD_LIBRARY_PATH at run time) — reads DBA_* (through ALL_* a read-only user sees 0 of 48 functions and the counts still reconcile), inside SET TRANSACTION READ ONLY, reconciled against *_OBJECTS. [[databases]] in ekos.toml (the password's variable name, never its value) → plugins/oracle observer at build → OracleCatalogPass at recover → objects with origin: live, which identity never merges and name lookups (procedure_lineage::NameIndex::add_object, data_lineage) use only for names no file defines. RFC 0235: kind = "postgres" (plugins/postgres, through pg-live, reconciled) and kind = "clickhouse" (plugins/clickhouse/src/catalog.rs, readonly=1) emit the same artifacts (observation_sdk::live_catalog); recovery/src/oracle_catalog_analyzer.rs is one pass per CatalogDialect (Oracle's ids and pass name unchanged), and ekos database test reports the same JSON for every kind, counting compiled objects from ckm/model.json. semantic/src/live_conflicts.rs compares ModelTables / db/oracle routines with them at commit (RFC 0172 conflicts with an explanation: pending_migration by time and release horizon, translation_table; mandatory_but_nullable carries a measured null count from the profile). Profiles: ora-live::profile ([[databases]] profile = p0|p1|p2) — a text column is never a MIN/MAX argument, PII rules are the shared ekos_common::pii. Live sources run outside the build fingerprint (build.rs::read_databases, once per build, failure fatal); the catalog pass's output sets KnowledgeContent::supersedes_earlier_runs, so compile keeps only its newest read (multi-input passes otherwise accumulate every past output and merge them), and each catalog artifact lists the objects it read so a dropped table leaves. A pass that calls another analyzer's functions must fold that analyzer's LOGIC_VERSION into its cache key (two stale-cache bugs). RFC 0212: semantic/src/write_behaviour.rs derives one keyed Custom("WriteBehaviour") per table at commit (update class, delete classes, change log, citations — not evidence, which EKL reserves) from footprint update_sets / delete_shapes, recovery/src/java_po.rs PO delete receivers (method-local declarations only; test code counted apart), live cascading FKs and the opt-in [[databases]] dictionary (ora-live/src/dictionary.rs, AD_Table flags); statements in migration scripts are ignored. RFC 0219: ekos_common::tickets::find (exact [tickets] keys only — never a generic ABC-123); the git analyzer emits keyed Custom("Ticket") objects + Touches and sets supersedes_earlier_runs (any pass that reads "all of X" should); semantic/src/tickets.rs adds TicketFor at commit. [observe] git-max-commits (default 500) joins the build fingerprint key only when changed (build.rs::observer_settings_key) — an observer setting that changes output must invalidate the cache. File objects live in the ledger, not the CKM. RFC 0220 run logs: ekos_common::run_log::emit(stage, msg, &data) is a run-log line (target ekos_run_log); crates/cli/src/run_log.rs is the layer — the pipeline commands (and, RFC 0218, every command whose catalogue class is write/review/admin/external, by Cli::command_path: run_log::for_class) write JSON lines to EKOS_RUN_LOG or <state>/runs/<stamp>-<cmd>.jsonl (warnings copied redacted), ending with llm (model calls, counted from ekos_model_call events that MeteredLlmProvider, the LLM cache and the embedding providers emit — a new model-call path must emit one) and run finished. The terminal filter turns both targets off. The console sets EKOS_RUN_LOG=<runs>/<id>.jsonl (runner.with_run_log) and serves /api/runs/<id>/events. RFC 0221: localdocs lifts Markdown front matter into a front_matter field (Document properties; title/source_url/ source_revision/licence also on sections) — a change to an observer's output needs PIPELINE_LOGIC_VERSION bumped; sections record tickets (with_ticket_keys) and semantic::tickets links Sections; scripts/wiki-snapshot.py (MediaWiki export/API → Markdown with provenance; tests in scripts/tests). Testbed: docker-compose.oracle.yml + deploy/fixtures/idempiere-oracle.sh; live suite EKOS_ORACLE_LIVE=1. SqlAnalyzerPass stamps dialect on every Table, and conflicts compares claims only within one dialect (generic = unknown, compared with all) — engine twins of one script are two databases, not a dispute. init --detect writes one [[recover.sql.dialect-rules]] per engine-named directory (oracle/, postgresql/, …). Not in this repo: RFC 0148's compiled .NET/JVM binary decompiler (readers, observer, analyzer pass, [binary-reconstruction], ekos_binary_explain) lives in the private alexeyban/ekos-binary workspace as an RFC 0149 extension with its own ekos binary. Public EKOS keeps only the [binary-reconstruction] config schema, the Binary* custom-kind registry rows and identity's binary narrowing, so a ledger a private build wrote still resolves and queries here. Public crates must never depend on it.

Scaffolded proof-of-concept only (mock API shapes, not exercised against live accounts): salesforce, sap, fabric, snowflake (oracle is live since RFC 0211).

crate_topology_analyzer/cicd_analyzer (RFC 0042, in recovery, not plugins/) and the dependency-scan block in recover.rs are a second, separate raw-content entry point — they walk observe_paths and call std::fs::read_to_string directly rather than going through an Observer/ArtifactStore round-trip. Both entry points (the Observer-based one and this direct one) independently run RFC 0043's redaction pass before any content is persisted — see common's crate-map entry.

Key invariants (enforced by review, not just convention)

• The ledger is append-only — knowledge is never modified in place.
• Every semantic conclusion must be traceable to Evidence (one of the four primitives: Object, Relationship, Event, Evidence).
• The Runtime is read-only — it reconstructs state, never modifies it.
• AI systems consume knowledge through the Runtime only (in practice, through ekos mcp serve, RFC 0013) — they never touch raw enterprise systems directly.
• Compiler passes must be deterministic and side-effect-free.
• Artifacts are content-addressable (id + checksum + metadata + dependencies + version).
• Secrets/PII are never observed or stored (RFC 0043) — a built-in redaction baseline (ekos_common::redaction) runs at every raw-content entry point; ekos.toml's [security] section can only extend it, never disable it. Because the ledger is append-only, this has to be a prevention control, not a cleanup step — there is no way to un-commit something already ledgered (confirmed: no object-level delete/tombstone exists anywhere in the codebase).

AI agent access (MCP)

ekos mcp serve --workspace <dir> exposes the Runtime read-only over stdio via newline-delimited JSON-RPC 2.0 (also --tcp <addr> for multiple clients on one server, RFC 0115). Tools: ekos_search (mode lexical/vector/hybrid, RFC 0124/0125), ekos_query / ekos_retrieve (compiled fact+graph answer and the inspectable query plan / evidence set, no LLM — RFC 0124), ekos_ekl, ekos_neighborhood, ekos_state, ekos_dependents, ekos_impact (multi-hop, RFC 0018), ekos_diff, ekos_status, ekos_transformation_explain/ekos_transformation_diff (Transformation IR, RFC 0028), ekos_architecture_evaluate/ekos_architecture_drift/ekos_architecture_diff (RFC 0065/0107-0109), the gated ekos_clickhouse_query (live NL-to-SQL, off by default, RFC 0056), the opt-in ekos_session_note/ekos_session_recall/ekos_session_brief (RFC 0151, [session-memory]; the note tool writes only the inbox file, never the ledger), the read-only ekos_signoff_status (RFC 0213, the workflow bar and sign-off states), the read-only ekos_ossie_export (RFC 0245, the compiled schema as an Apache Ossie model), the opt-in ekos_sql_generate (RFC 0250, [sql-generate] mcp: the first MCP tool that calls a model — sql_check::agent_generate, ekos sql generate's agent with read-only probes; allow-metered for a non-Ollama provider; a daily cap counted in <state>/sqlgen/usage.json before the model is called; in SCOPE_REFUSED_TOOLS; the console gateway's mcp_policy.METERED_TOOLS needs the tenant's allow_metered and spends a per-person/per-token daily allowance, and the chat never offers it), the read-only ekos_xref_check (RFC 0233, the commit-time integrity index plus a one-name resolve; in SCOPE_REFUSED_TOOLS), the read-only ekos_conflicts (RFC 0172, where sources disagree; ekos_state also carries open_conflicts, and an ekos_query/ekos_retrieve claim about a disputed attribute carries disputed: true — RFC 0173, EvidenceSet::mark_disputed in runtime/src/reason.rs with the lookup supplied by commands/conflicts.rs::flag_disputed, since the Runtime cannot know which conflicts are current; ekos ask gets the same lookup through AiRuntime::with_disputes, and a label conflict carries a second Disputes link to its column's table whose attribute property — columns.<col>.meaning — is what the lookup uses for that table), the opt-in read-only ekos_semantics_lookup/ekos_semantics_gaps (RFC 0170, [semantics]; every result carries its review status in words — there is no MCP tool that confirms a definition), and the two write-capable review tools ekos_identity_review (confirm/reject a candidate identity match — cross-system, RFC 0029, or same-source fuzzy, RFC 0063; the tool doesn't distinguish origin, only Custom("SameAs"); an RFC 0032 treasury AuthorizedBy is human-only, ekos treasury confirm|reject → commands/treasury.rs::review_authorization, issue #16) and ekos_architecture_review (confirm/reject an LLM-classified crate role, RFC 0109). Both record the decision as an event citing evidence with the optional reviewer/note arguments (issue #16). The gated ekos_clickhouse_query also writes: one evidence + ClickHouseQueryExecuted event per live query (RFC 0056 audit; id derived from database, SQL and result hash, issue #11). Every other tool is read-only, going through Runtime only; mcp.rs's only_listed_functions_can_write_the_ledger pins the set of functions that can write. This repo's own .claude/skills/ekos-knowledge and .claude/skills/memory skills consume this server. Runnable demos live under demo/ (ledgersmb-linkml/, session-memory/, binary-demo/); the old twelve-act walkthrough was removed in the 2026-10-10 cleanup (it is in git history before that date).

LLM-backed passes

recovery's analyzer passes that call an LLM (document semantics, some SQL/Git recovery) go through the LlmProvider trait (recovery/src/llm.rs), with anthropic.rs and ollama.rs implementations selected via [llm] in ekos.toml. Treat this as a Claude-relevant provider boundary — check recovery/src/llm.rs before assuming request/response shapes.

Mandatory Development Workflow

Every task follows this sequence — do not skip steps:

1. Design — write an RFC in docs/rfcs/NNNN-<topic>.md before any code
2. Architecture Review — validate against the compiler model above
3. Interfaces — define public traits and types first
4. Tests — write tests before implementation
5. Implementation
6. Refactoring
7. Documentation — every public API must be documented
8. Integration
9. Benchmark — required for performance-relevant changes (see benchmark/)
10. Merge

No feature is implemented until its RFC is accepted. RFCs are authored just-in-time, before the phase that needs them, not all up front (see TODO.md Phase -1 for the rationale).

PR Checklist

Every PR must satisfy: tests passing, documentation, benchmarks (performance-relevant changes only), no public API break, compiler diagnostics, logging, examples.

SonarCloud after every push to main

.github/workflows/sonar.yml analyses main after every push: coverage, then the scan, then the quality gate. After each commit you push to main, finish the job:

1. Wait for the SonarCloud run of that commit: gh run list --workflow sonar.yml --commit <sha>, then gh run watch <id> --exit-status.

2. Read the result. These need no token:

   • the gate: curl -s 'https://sonarcloud.io/api/qualitygates/project_status?projectKey=alexeyban_EKOS';
   • the open new-code issues: curl -s 'https://sonarcloud.io/api/issues/search?componentKeys=alexeyban_EKOS&inNewCodePeriod=true&resolved=false&ps=500';
   • the per-file uncovered new lines (api/measures/component_tree … metricSort=new_uncovered_lines).

   First check that the analysed revision (api/project_analyses/search?project=alexeyban_EKOS&ps=1) is your commit.

3. Fix every issue in code you touched, and any new bug, vulnerability or security hotspot, then push again (which re-runs the analysis). Before pushing, run scripts/cognitive-complexity.sh locally: it catches S3776 before Sonar does.
4. An issue that is deliberate is not silenced in code without a reason.
   • Plain # NOSONAR / // NOSONAR goes on the line, with the reason on the line above.
   • An accept in SonarCloud (issues/do_transition transition=accept plus a comment) is a write to an external service. Write it as a script for the person to run.
5. Report the gate and anything left open, with the reason.

A failing coverage step (not the scan) is a CI fault: fix the workflow first. The first run failed on the runner's Python 3.14; UV_PYTHON is pinned for that reason.

Coding Rules

• Rust 2024 edition
• Zero unsafe unless formally justified in an RFC
• No global mutable state
• Dependency injection through traits (Observer, LlmProvider, CompilerPass, ArtifactStore, …)
• Every artifact must be serializable
• Pure functions wherever possible
• Reproducible builds

Devlog Rule

devlogs/devlog_N.md files are the project's long-term memory. They are the primary source of truth for project history, architecture decisions, production incidents, and non-obvious knowledge. Treat them as the first thing to read, not the last thing to write.

After any session with significant changes, generate a new devlogs/devlog_N.md.

Significant = any of: new feature shipped, bug found and fixed, architecture decision made, non-obvious knowledge captured, production incident, or a set of PRs merged in one session. Minor chores (typo fixes, dependency bumps) alone do not warrant a devlog entry — fold them into the next substantive one.

Filename

Increment from the highest existing devlogs/devlog_N.md: devlog_43.md → devlog_44.md, etc.

Required sections

# Devlog N — <short title>

**Date:** YYYY-MM-DD
**PRs:** #N, #N+1, …
**Branch:** <branch> → <target> (merged / squash-merged)

---

## Summary
<2–5 sentence overview: what changed and why it mattered>

---

## PR #N — <title>

### Problem / motivation
### What was built  (table of components if >3 items)
### Implementation details worth remembering
### Decisions (alternatives considered, why this choice)

(repeat for each PR)

---

## Knowledge Captured
<Non-obvious facts, gotchas, SDK quirks, production behaviour, or patterns
 that should not be re-discovered from scratch. Each item should answer:
 "What would a future developer need to know to avoid the same mistake?">

---

## Files Changed
| File | Change summary |

What belongs in "Knowledge Captured"

• SDK/library quirks that aren't in the docs
• Production-only behaviour
• Decisions with non-obvious rationale
• Cost/latency benchmarks discovered in practice
• Debugging techniques that were hard to find

After writing the devlog

1. Also update TODO.md to tick off completed items for the day's work
2. Update README.md if any user-facing behaviour changed
3. Update docs/generated/ekos-self-documentation.html — the human-readable, comprehensive capabilities reference published at alexeyban.github.io/EKOS/generated/ekos-self-documentation.html. If the devlog shipped a new capability, a new CLI command/flag, a new MCP tool, or changed how an existing one behaves, add or update that capability's own section there — this page is meant to stay a complete, current, human-readable description of the whole project and every aspect of it, not a point-in-time snapshot. A devlog that's purely an internal fix/refactor with no user-visible capability change doesn't need an update here (same bar as the README update in step 2). Pushing this file (it lives under docs/) redeploys the live page automatically via .github/workflows/pages.yml.
4. Commit everything in one PR: chore: devlog_N, README + TODO + capabilities doc update for <topic>