analytics/AGENTS.md

Python 3.13 analytics app executing docs/ideas/analytics-platform-plan.html Phase 0: dlt extraction → ClickHouse warehouse → dbt models → Cube semantic layer → Superset dashboards, orchestrated by Dagster. Runs as the clickhouse / dagster-* / cube / superset* compose services; nothing here gates the app services. The product/architecture overview with screenshots is README.md; this file is the working guide.

The golden rule — structure and metrics, never document bodies

The warehouse must never receive CV/JD/chat text, run input payloads (run.kwargs), store values, or checkpoint channel state. All extraction shaping lives in nextrole_analytics/dlt_sources/transforms.py (pure functions), and the contract is pinned by tests/test_transforms.py + tests/test_message_explode.py — extend the tests before shipping any new extracted field. Review transforms.py first on any extraction change.

Tooling

• Package manager: uv only (uv add, uv run); never pip install. Second uv virtual project — independent lockfile from backend/.
• Lint/format: uv run ruff check --fix / uv run ruff format (line length 100, select = ["ALL"] — config in analytics/pyproject.toml, kept in sync with backend's ignore list).
• Type check: uv run ty check (pre-commit hook analytics-typecheck gates on it).
• SQL (dbt models): uv run sqlfluff format dbt/ → uv run sqlfluff fix dbt/ → uv run sqlfluff lint dbt/ (ClickHouse dialect, jinja templater with dbt stubs + project macros; config in analytics/pyproject.toml under [tool.sqlfluff.*], path exclusions in analytics/.sqlfluffignore). Deliberately light: the curated core rule group only, lowercase keywords/identifiers, 100-char lines; CP03/CP05 are excluded because ClickHouse functions/types are case-sensitive camelCase. Pre-commit runs fix + lint on analytics/**/*.sql. Use -- noqa: <RULE> for deliberate exceptions (see fct_message's self-aliases) — never widen the exclude list to silence one line.
• Tests: uv run pytest from analytics/.
• Version lockstep: dagster-* integration libs (0.29.x) exact-pin dagster core (1.13.x) — bump together; dbt-core stays <1.11 until dbt-clickhouse raises its ceiling; Python ceiling <3.14 comes from dagster-dbt.

Layout

• nextrole_analytics/ — the Dagster code location. dlt_sources/ (extraction + transforms), assets_dlt.py / assets_dbt.py / translators.py / schedules.py / definitions.py, run_pipeline.py (manual dlt run).
• dbt/ — dbt project: models/sources.yml (bronze raw_app___* / raw_auth___*), models/staging/ (views + the stg_app__messages first-seen ledger), models/marts/ (star schema: fct_run, fct_message, fct_thread, fct_session, dim_user, dim_assistant), seeds/model_prices.csv (editable token prices → est_cost_usd).
• Metadata contract ("dbt owns tables, Cube owns metrics"): every mart column carries an agent-facing description in dbt/models/marts/marts.yml (grain, semantics, value domains, caveats); shared prose lives in dbt/models/docs/shared.md as {% docs %} blocks — extend a block rather than restating column meaning across models. Metric definitions, formulas, synonyms, units, example questions and caveats live on Cube members/cubes as description + meta (exposed verbatim by /cubejs-api/v1/meta). Don't write "input_tokens: input tokens" — say what the number means, what it excludes, and when it misleads.
• cube/model/ — Cube cubes + views (every member described: the future analytics agent's vocabulary via /cubejs-api/v1/meta).
• superset/ — image (drivers), superset_config.py, and assets/dashboard_export/ (the seeded "NextRole Overview" bundle imported by the superset-init one-shot).
• dagster/ — dagster.yaml (Postgres-backed instance) + workspace.yaml.
• clickhouse/ — dev low-memory server config + per-query limits.

Local dev loop

• No hot reload for pipeline code: after editing anything under nextrole_analytics/ or dbt/, run docker compose restart dagster-webserver dagster-daemon dbt-docs (the boot commands re-run dbt parse for a fresh manifest and dbt docs generate for the docs site). Cube reloads its model dir on its own in dev mode; Superset bundle changes need docker compose up -d superset-init to re-import.
• Trigger the whole pipeline now (instead of waiting for the hourly schedule): docker compose exec dagster-daemon dagster asset materialize --select "*" -m nextrole_analytics.definitions — or click "Materialize all" in the Dagster UI.
• Manual dlt-only run (bronze only): docker compose exec dagster-daemon python -m nextrole_analytics.run_pipeline.
• dbt by hand (inside the container, env is preset): docker compose exec dagster-daemon dbt build --project-dir /deps/next-role/analytics/dbt --profiles-dir /deps/next-role/analytics/dbt.
• Import-time gotcha (mirrors the backend's config import): importing nextrole_analytics.definitions needs ANALYTICS_SOURCE_PG_URI set and the dbt manifest present (dbt parse first). Host-side dagster dev therefore needs both; the compose services guarantee them.
• UIs (host ports from .env): Dagster http://localhost:<DAGSTER_LOCAL_PORT>/, Superset http://localhost:<SUPERSET_LOCAL_PORT>/ (login SUPERSET_ADMIN_*), Cube Playground http://localhost:<CUBE_LOCAL_PORT>/, dbt docs http://localhost:<DBT_DOCS_LOCAL_PORT>/ (the dbt-docs service runs dbt docs generate against the warehouse at boot — no local dbt needed; nextrole_analytics/dbt_docs_patch.py then labels the artifacts' blank ClickHouse database so the Database tab groups as clickhouse → schema → table), ClickHouse http://localhost:<CLICKHOUSE_HTTP_LOCAL_PORT>/play.

Warehouse layout

One ClickHouse database per layer: bronze in ${CLICKHOUSE_DB} (raw_app___*, raw_auth___* — deduped by dlt's merge write disposition, so staging reads them plain), staging views in ${CLICKHOUSE_DB}_staging, marts in ${CLICKHOUSE_DB}_marts (MergeTree, ordered by (date, owner)). dlt incremental cursors persist in the destination (_dlt_pipeline_state), so container recreation never re-backfills. The transient raw_*_staging___* tables are dlt merge machinery — ignore them.

Known limitation — message event time

Source messages carry no timestamp: stg_app__messages stamps first_seen_at at capture, so token/cost time-series are accurate to the schedule cadence going forward, while history backfilled before the pipeline existed lumps at deploy date (thread_created_date is carried for re-bucketing). Langfuse traces (a later blueprint phase) replace this with real per-message timing.

Superset dashboard bundle

superset/assets/dashboard_export/ is a hand-maintained Superset export bundle (databases → datasets → charts → dashboard, cross-referenced by fixed UUIDs; connection URIs carry __..._URI__ placeholders that superset-init fills from env). To change a chart: edit it in the Superset UI, export the dashboard, and copy the updated chart YAML back into the bundle — don't hand-tune params blind.