CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Tracely is trace-native CI/CD for AI agents: production trace → failure detection → regression test → CI/CD gate. The trace is the source of truth; evals, clusters, cases, gates and trends are all derived from it. There are no hand-authored datasets.

A uv workspace (backend, workers, sdk) plus a pnpm Next.js app (frontend) and a Nextra docs site (docs).

Commands

make install                     # uv sync --all-packages --all-extras + pnpm install
make infra-up                    # clickhouse, postgres, redis, minio (docker)
make migrate                     # ClickHouse DDL runner + Alembic (Postgres)
make seed                        # default project + ingest key `tracely_dev_key`
make backend / workers / frontend    # three terminals: FastAPI :8000 · Celery · next dev
make demo                        # populate the whole product (traces, clusters, cases, gates)

Tests — no infra required, ~6s:

uv run pytest -q backend/tests sdk/tests        # what CI runs
uv run pytest -q backend/tests/test_gate_eval.py::test_name -x     # single test
uv run ruff check . && uv run ruff format .
cd frontend && pnpm test        # vitest; pnpm test:watch; pnpm build type-checks (tsc) + lints

Alembic: cd backend && uv run alembic revision -m "…" / alembic upgrade head. ClickHouse migrations are *.up.sql files in backend/tracely/infrastructure/clickhouse/ddl/ applied by python -m tracely.infrastructure.clickhouse.migrations.

Whole stack in Docker: docker compose up -d --build --wait → UI on :3001, backend on :8000 (remap with TRACELY_WEB_PORT / TRACELY_BACKEND_PORT). make frontend runs plain next dev (:3000) — use cd frontend && pnpm dev -p 3001 to match Docker.

Architecture

Write path (deliberately mirrors Langfuse, reimplemented in Python):

SDK/OTLP → POST /v1/traces → S3 blob (durable FIRST) → Redis/Celery
  → worker: otel/ mapping → registry upsert → ClickHouse events
  → evaluate_run_task (countdown=4, debounces late spans) → scores + structural clustering

Five stores: ClickHouse (events one row per span + scores, ReplacingMergeTree) · Postgres + pgvector (registry: projects, keys, agents, cases, gates, clusters, evaluators, users, monitors, annotations) · S3/MinIO (raw OTLP body = source of truth, plus regression fixture bundles) · Redis (Celery) · pgvector (failure embeddings).

backend/tracely/ is one package with three roles (FastAPI app, shared domain, Celery tasks) and strict layering:

workers/ is a deployable shim that imports tracely.workers.tasks. sdk/ is the instrumentation SDK and the tracely simulate / gate / replay CI CLI. simulate is the scenario path (one agent, --agent a,b, or --all = every agent with an enabled scenario); the fan-out is CLI-side — one GateRun per agent, aggregated into one commit status and one PR comment.

Hard rules

• No SQL in api/routers/. ClickHouse reads go through infrastructure/clickhouse/async_reader.py (async, for the API) or trace_reader.py (sync, for workers/services); Postgres through infrastructure/db/repositories.py; auth lookups through auth/queries.py. Adding a query means adding a function there, not inlining SQL in a router. (health.py's SELECT 1 probe is the sole exception.)
• Every LLM call goes through infrastructure/llm/provider.py (run_structured_agent / run_text_agent — LangChain create_agent on OpenRouter, OpenAI as fallback). Never construct a client elsewhere. LLM modules are lazy-imported and llm_enabled() gates them: with no key the pipeline must still run (judge/FI/meta-analysis degrade, not crash). Customers bring their own key: inside provider.use_project_key(project_id) only that workspace's OpenRouter key applies — never the server-wide OPENROUTER_API_KEY/LLM_JUDGE_API_KEY/OPENAI_API_KEY. A project with no key configured is exactly an LLM-disabled deployment. Any new project-scoped LLM entry point wraps in use_project_key and checks llm_enabled() inside the wrap. The one exception is the in-app assistant (services/assistant_service.py): it explains and operates our product, so the model runs on our key via provider.use_server_key() (through use_assistant_key, which falls back to the workspace's own key only on a deployment with no key of ours, i.e. a self-host) — an explicit scope, not an unwrapped call, because under REQUIRE_PROJECT_LLM_KEY an unscoped call fails closed as a forgot-to-wrap bug. Anything else that spends the server key on a customer's behalf needs the same deliberate seam. Note the split: only the model is on our key. The assistant's tools re-enter our own routers carrying the caller's own credentials (api/internal_client.py), so we pay for tokens without widening anyone's reach.
• Agents reach the product through the routers, never the DB. Both the MCP server (api/mcp_server.py) and the in-app assistant's tools (services/assistant_tools.py) call internal_client.api_call, which re-issues the request against our own ASGI app with the caller's authorization/x-tracely-key/x-tracely-project headers. That leaves get_project_id as the only place scoping happens and the routers' own 4xx bodies as the only place a bad write is rejected — handed back to the agent, that error text is what lets it correct itself. Assistant tools return their failures as text (langgraph re-raises anything that isn't a bad-args error, which would kill the stream mid-answer), and no assistant tool argument may be named config (StructuredTool._arun swallows it).
• Everything is scoped by project_id, resolved from Authorization: Bearer <ingest-key> via api/auth. Every read and write takes it.
• An ingest key ingests and reads; it never destroys or sets secrets. Routes that wipe data, delete what people built, set the OpenRouter key or re-point an agent endpoint carry dependencies=[Depends(require_user)] (api/auth.py) — a machine principal has no role and is refused. Dev mode (no human auth, prod refuses to boot in it) is the one place an ingest key acts as OWNER. A leaked CI key must cost traces, not the workspace.
• Every customer-supplied URL Tracely will call goes through infrastructure/net.assert_public_url — at save time and right before the request. The worker shares a network with ClickHouse's HTTP port (which runs SQL from a POST body), so an unchecked agent endpoint or monitor webhook is a cross-tenant read. Off outside prod (self-hosters run agents on localhost) unless ALLOW_PRIVATE_URLS says otherwise.
• Ending a session means bumping users.token_version. Sessions are stateless JWTs carrying it as tv; _resolve_local_jwt compares the two. Password change and reset bump it (and hand back a fresh token, or the caller logs themselves out). Nothing else can revoke a session early.
• An invite never logs anyone into an existing account. The raw invite token is shown to the inviter; when the invited email already has an account, accept_invitation verifies that account's password before consuming the token.
• Writes are idempotent. Deterministic ids (score id = uuid5(trace_id:name:span_id)) + ReplacingMergeTree mean re-ingest/re-eval converges instead of duplicating. Anything sampled must be deterministic per (trace_id, score_name) for the same reason (domain/evaluation/targeting.py).
• One verdict policy. A trace/turn/session fails iff it has a FAIL on a non-advisory evaluator. Python: domain/evaluation/verdict.py. Its SQL twin lives in async_reader (name NOT IN {adv:Array(String)}), fed by api/advisory.py. Change both together or the badge, threads dot, and trends disagree.
• Frontend fetch pattern: Server Components (pages) call app/lib/api.ts directly; Client Components fetch a thin proxy under app/api/* which re-issues with the Bearer key server-side. TRACELY_KEY / TRACELY_API must never reach the browser — do not add an env: block to next.config.mjs.
• New backend env var? Add it to config.py (Settings) and the x-app-env anchor at the top of docker-compose.yml — .env alone won't reach the containers.
• Emulated conversations have four non-obvious invariants. Break any one and the gate goes quietly green instead of failing loudly:
  1. Span ids are base64, never hex. parse_otlp_traces_json runs protobuf's json_format.Parse, whose canonical mapping decodes bytes as base64 — hex does not raise, it silently yields a 24-byte id nothing can look up (domain/simulation/emit.py).
  2. Turns are ingested and evaluated inline, not via Celery. The gate task already owns the worker's only slot under --pool=solo --concurrency=1; enqueuing work and waiting for it deadlocks.
  3. Driving and grading are separate tasks (run_scenario_gate → countdown → grade_scenario_gate). The customer's spans arrive as ordinary OTLP, so their ingest is queued behind the gate — releasing the worker between phases is the only thing that lets them land before grading.
  4. The attack judge is inverted. For an ADVERSARIAL scenario, goal achieved = attack succeeded = FAIL. Without it the goal only generates turns and nothing judges the outcome, so a fully-successful attack passes.
• Internal runs must never be evaluated. Tracely records its own work — an evaluation, a scenario run — as a trace (domain/introspection.py), listed in the Traces tab behind the Evals filter chip. Every span carries internal_kind (eval|sim|assistant — the in-app agent records each of its turns, tools and all), and three things depend on it: the ingest hop skips scheduling evaluation for those trace ids, EvaluationService.evaluate_trace refuses them outright (monitors and manual re-runs don't go through ingest), and _REAL in async_reader keeps them out of every list, count and metric (the sole opt-in is sessions_overview(include_internal=True), which the toggle asks for). Lose the first two and grading an eval run records another eval run, for ever; lose the third and a project's trace count doubles the day evaluators are switched on.
• SKIP scores are dropped before the conversation roll-up. rollup_verdict reads any score as evidence a run was graded, so a conversation whose only scores were skipped would report PASS having checked nothing (domain/simulation/aggregate.py).

Gotchas

• The Celery worker does not hot-reload. After touching worker/eval/failure-intel/otel-mapping code: docker compose restart worker (backend/frontend are volume-mounted and do reload).
• Celery runs --pool=solo --concurrency=1 locally on purpose — numba/UMAP/HDBSCAN in failure intelligence are fork-fragile.
• AUTH_MODE is dev (open) | local (email/password + JWT, needs SESSION_SECRET) | clerk. The frontend's NEXT_PUBLIC_AUTH_MODE must match the backend's. Prod refuses to boot with AUTH_MODE=dev or a seeded tracely_dev_key.
• OTLP arrives in three message conventions (structured / OpenInference-flattened / OpenLLMetry legacy); otel/messages.py normalizes them. Fix I/O rendering bugs there, not in the frontend renderers.
• uv sync --frozen in CI catches lock/extra drift — regenerate uv.lock when touching deps.

Where the detail lives

Each folder has a thorough README; read it before changing that area rather than re-deriving from code.