traceway

# CLAUDE.md - Traceway Project

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## Project Overview

Traceway is an error tracking and monitoring platform consisting of:
- **Frontend**: SvelteKit 2 dashboard application with Svelte 5
- **Backend**: Go/Gin API server with ClickHouse database
- **CLI**: Go/Cobra command-line client for the backend HTTP API (`/cli`)

---

## Code Style

- **No pointless comments**: Do not add comments that simply describe what the code does. The code should be self-explanatory. Only add comments when explaining non-obvious "why" decisions.
- **No `py-4` in dialog form content**: Do not add `py-4` on the content wrapper inside `AlertDialog` or `Dialog` components — it creates too much blank space between the form and the action buttons.
- **Dialog button labels & toasts**: For form dialogs, use descriptive button labels with icons instead of generic "Create"/"Update". The `{Entity}` is always a platform entity, capitalized (Project, Widget, Dashboard, Channel, Rule, Token, Invitation, ...). Create actions: `<Plus icon> New {Entity}` with `variant="success"`. Update actions: `<Check icon> Update {Entity}` with the default (primary) variant. Delete/revoke/remove confirm buttons: `<Trash2 icon> Delete {Entity}` (or `Revoke {Entity}` / `Remove {Entity}`) with `variant="destructive"`. After success, show `toast.success('Successfully created the {Entity}', { position: 'top-center' })` for creates and `'Successfully updated the {Entity}'` for updates. The button should only be `disabled` during the loading state — never disable it to enforce validation; let the backend return 422 and show the error in the dialog instead.

---

## Quick Start

### Development Commands
| Component | Command | Description |
|-----------|---------|-------------|
| Frontend | `cd frontend && npm run dev` | Dev server (port 5173) |
| Frontend | `npm run build` | Production build |
| Frontend | `npm run check` | TypeScript checking |
| Frontend | `npm run lint` | `prettier --check . && eslint .` (CI gate; `npm run format` fixes the prettier half) |
| Backend | `cd backend && go run ./cmd/traceway` | API server (port 8082) |
| Backend | `cd backend && govulncheck ./...` | Vulnerability scan (default tags only); CI also scans the other storage build-tag combos (`.github/workflows/backend-vulncheck.yml`) |
| CLI | `cd cli && just build` | Builds `bin/traceway` |
| CLI | `cd cli && just test` | Runs unit tests |
| CLI | `cd cli && just check` | Lint + test + vulncheck + skill drift + contract tests (pre-commit gate; CI enforces it for `cli/`) |
| CLI | `cd cli && just smoke-test` | Live E2E (needs `TRACEWAY_SMOKE_*` env vars) |
| Any | `nix develop` (or direnv via `.envrc`) | Dev shell with the Go and Node versions read from `backend/go.mod` and `.nvmrc`, plus `just`, `golangci-lint`, `govulncheck` (`flake.nix`; shells `default`, `backend`, `frontend`, `oxc`). Put `JWT_SECRET` in the gitignored `.envrc.local` |
| Any | `./scripts/check-node-pins.sh` | Asserts every Dockerfile's `ARG NODE_VERSION` default matches `.nvmrc`; run by `release-traceway.yml` before it builds |

**Node's version lives in `.nvmrc`** (a bare major, currently `26`) and nothing else derives it independently: every `setup-node` step uses `node-version-file: .nvmrc`, `flake.nix` reads it for the dev shells, and `release-traceway.yml` passes it to all five image builds as `--build-arg NODE_VERSION`. The Dockerfiles each carry an `ARG NODE_VERSION=<major>` default because a bare `docker build` cannot read `.nvmrc`; that default is the one value that can drift, which is what `scripts/check-node-pins.sh` exists to catch (it also fails on a literal `node:<major>` tag reintroduced in a `FROM`). To move Node, edit `.nvmrc` and the ARG defaults, then run the script. `frontend/package.json`'s `engines.node` is deliberately left as a floor (`>=22`) rather than a pin — it is what npm enforces on consumers, and `node-version-file` pointed at it resolves a range to the *newest* release, which is the drift `f95424cc` was fixing. `testing/devtesting-nestjs/Dockerfile` is out of scope: it pins the runtime of a sample third-party app, not anything Traceway builds.

Set `JWT_SECRET` (min 32 characters) before running the backend, including the All-in-One image. Former public signing keys are rejected. The All-in-One shell script leaves `.env` loading to the backend so Docker-provided values win. Embedded `Run(opts...)` never reads `JWT_SECRET`, since it runs inside the host app's process where that name usually belongs to the host: it takes `WithJWTSecret`, else `TRACEWAY_JWT_SECRET`, else a random key per start; set a stable secret to preserve dashboard sessions across embedded restarts. The All-in-One image's supervisord sends the backend's output to the container's stdout/stderr, so `docker logs` shows a startup rejection. `SQLITE_PATH` sets the database location, defaulting to `./traceway.db` in the working directory.

CI on pull requests is opt-in. `backend.yml`, `backend-vulncheck.yml`, `cli.yml`, `cli-lint.yml`, `cli-contract.yml`, `frontend.yml` and `plugin-scanner.yml` listen for `pull_request: types: [labeled]` and every job is gated on `github.event.label.name == 'ci'`, so a PR runs nothing until a maintainer applies the `ci` label (only collaborators can), and applying any other label runs nothing either. Each application validates the PR as it is at that moment: to validate later pushes, remove and re-add the label. The same workflows still run on pushes to `main` that touch their paths (`backend/**`, `cli/**`, `frontend/**`, `skills/traceway/**` for `cli.yml`, `.nvmrc` for `frontend.yml`, their own workflow file; `plugin-scanner.yml` runs on every push to `main` because it scans the whole repo), on their daily schedules (the two vulnchecks), and from the Actions tab via `workflow_dispatch`.

Action versions are watched by `.github/dependabot.yml` (the `github-actions` ecosystem only; npm and Go modules are deliberately not watched there). It opens a weekly PR per group — `actions/*` in one, every third-party action in the other — and deliberately does **not** auto-apply `ci`, so a bot PR passes the same collaborator gate as any other. Label the first-party group to validate it; `golangci/golangci-lint-action` is reachable the same way via `cli-lint.yml`. The rest of the third-party group lands on `release-*.yml`, `benchmark-*.yml` and `traceway-autofix.yml`, which no label reaches — and `workflow_dispatch` is **not** a dry run on the release path: dispatching `release-docs.yml` or `release-website.yml` deploys to Cloudflare for real, and the other `release-*.yml` workflows publish real assets, so dispatching one from an unreviewed bot branch ships it. Validate those by reading the action's release notes instead, and dispatch only where the target is disposable. `traceway-autofix.yml` is the in-repo caller of the reusable `autofix.yml` (composite actions `.github/actions/autofix-prepare` and `autofix-publish` around the agent step; `docs/pages/learn/auto-fix.mdx` documents the contract); its `workflow_dispatch` takes an `issue_number` and runs the whole loop from the dispatched branch, which is the one pre-merge validation path for the `claude-code-action` pin — the most privileged action here — but it runs the agent for real, so dispatch it only at a scratch issue. Note too that labelling a PR which touches no gated workflow yields zero checks, which reads like a pass but is silence. Every updatable pin is a bare major tag, so Dependabot only ever proposes major bumps here. The exception is `plugin-scanner.yml`, which pins both actions to commit SHAs because the HOL scanner scores unpinned actions down; Dependabot bumps those SHAs along with their version comments. It uses the scanner config in `.plugin-scanner.toml`, whose `ignore_paths` cover fixtures, demos and build output plus three reviewed production files (the read-only token mask in both `project.repository.go` copies and `KindPassword` in `cli/internal/state/state.go`). The HOL registry's own scan ignores that file and still fails on syntax-highlighter strings in the committed `backend/static/frontend` bundle; that is accepted.

### Tech Stack
- **Frontend**: SvelteKit 2.49, Svelte 5.45, Tailwind CSS v4, shadcn-svelte, Vite 7
- **Backend**: Go 1.26, Gin 1.11, ClickHouse, PostgreSQL (in `backend/go.mod` the `go` line is the floor importers inherit and the `toolchain` directive is what builds it — keep the two distinct)
- **CLI**: Go 1.26, Cobra 1.10, separate Go module (`github.com/tracewayapp/traceway/cli`); justfile entrypoints, covered by the root `flake.nix` dev shell
- **Client SDK**: Go 1.25, Gin middleware support

### lit Library (SQL mapper)
- **Import**: `github.com/tracewayapp/lit/v2` (currently v2.0.5)
- **Purpose**: Lightweight generic CRUD operations on top of `database/sql`. Supports PostgreSQL, MySQL, SQLite, and DuckDB drivers (`lit.PostgreSQL`, `lit.MySQL`, `lit.SQLite`, `lit.DuckDB`). Traceway registers models against `db.Driver` (SQLite or PostgreSQL depending on build tags); the DuckDB telemetry backend deliberately keeps `lit.SQLite` for reads since DuckDB accepts `?` placeholders.
- **Docs for AI use**: the lit repo ships a skill at `skills/lit/SKILL.md` plus `llms.txt` with the full API contract and pitfall checklist.

#### Model Registration (required before use)
All models are registered centrally in `models/models.go` via `models.Init(driver lit.Driver)`, called at boot with `db.Driver`:
```go
func Init(driver lit.Driver) {
    lit.RegisterModel[User](driver)
    lit.RegisterModel[Project](driver)
    // ...all models registered here
}
```
Repository-local result models (e.g., aggregate structs only used in one repo) can use file-level `init()` instead. Irregular table names use `lit.RegisterModelWithNaming` with a struct embedding `lit.DefaultDbNamingStrategy` (see `escalationPolicyNaming` in `models.go`).

#### Naming Conventions
- Fields: CamelCase → snake_case (`FirstName` → `first_name`)
- Consecutive uppercase: stay together (`HTTPCode` → `http_code`)
- Tables: pluralize + snake_case (`User` → `users`)
- Override via struct tag: `lit:"custom_name"`

#### Core CRUD Operations
All lit functions take a `lit.Executor` as the first argument. Both `*sql.Tx` and `*sql.DB` qualify: main-DB repositories pass the `*sql.Tx` from the transaction middleware, telemetry repositories pass `db.TelemetryDB` directly.

| Function | Description |
|----------|-------------|
| `lit.Insert[T](tx, &entity)` | Insert, returns auto-generated int ID |
| `lit.InsertUuid[T](tx, &entity)` | Insert with auto-generated UUID |
| `lit.InsertExistingUuid[T](tx, &entity)` | Insert with pre-set UUID |
| `lit.Select[T](tx, query, args...)` | Retrieve multiple records (returns `[]*T`) |
| `lit.SelectSingle[T](tx, query, args...)` | Retrieve one record (returns `*T`) |
| `lit.Update[T](tx, &entity, "id = $1", id)` | Update (auto-prepends WHERE) |
| `lit.UpdateNative(tx, "UPDATE table SET col = $1 WHERE ...", args...)` | Raw SQL update for partial/single-field changes |
| `lit.Delete(tx, "DELETE FROM table WHERE id = $1", id)` | Delete records (any exec statement) |
| `lit.SelectNamed[T]` / `lit.SelectSingleNamed[T]` / `lit.UpdateNamed[T]` | `:name` placeholder variants taking a `lit.P{...}` map; render per-driver, used for dialect-neutral queries |
| `lit.DeleteNamed(driver, tx, query, lit.P{...})` | Named delete/exec. Gotcha: takes the driver as the FIRST argument |
| `lit.ParseNamedQuery(driver, query, lit.P{...})` | Renders `:name` to positional; escape hatch for `QueryContext`/`RowsAffected` via `database/sql` |

#### Transaction Helper (`pgdb.ExecuteTransaction`)
All PostgreSQL operations should use `ExecuteTransaction` for automatic commit/rollback:

```go
// ExecuteTransaction[T] wraps a function in a transaction
// - Commits on success, rolls back on error or panic
// - Returns (T, error) directly - no pointer wrapping

project, err := pgdb.ExecuteTransaction(func(tx *sql.Tx) (*models.Project, error) {
    // All repository calls receive the transaction
    return transactional.ProjectRepository.FindById(tx, id)
})
```

#### Transactional Middleware (`middleware.Transactional`)
For auth flows and routes requiring transaction context throughout the request lifecycle, use the `Transactional` middleware:

```go
// In routes.go - wrap routes that need transaction context
api.POST("/register", middleware.Transactional, authController.Register)
api.POST("/login", middleware.Transactional, authController.Login)

// In controller - retrieve transaction from Gin context
func (c *AuthController) Register(ctx *gin.Context) {
    tx := db.GetTx(ctx)  // Get transaction from context (db package, not middleware)

    // Use tx for all repository calls
    user, err := transactional.UserRepository.FindByEmail(tx, email)
    if err != nil {
        ctx.JSON(500, gin.H{"error": err.Error()})
        return  // Transaction auto-rolls back on non-success status
    }

    ctx.JSON(201, user)  // Transaction auto-commits on 200/201/303
}
```

**Auto-commit/rollback behavior:**
- Commits on status codes: 200, 201, 303
- Rolls back on all other status codes or panics

**Body buffering:** `Transactional` reads the request body into memory (cap `maxTransactionalBodyBytes`, 8MB) before it calls `db.DB.Begin()`, answering 413/408/400 without a transaction when that fails; the handler's later bind is served from memory. This is what keeps a slow-drip body from holding the single SQLite main-DB connection, and it covers every transactional route without each one remembering a special middleware. Routes that want a tighter cap put `middleware.BufferAuthBody` (64KB) in front of `Transactional`, which then skips the body it finds already buffered; the anonymous auth endpoints do this. The self-transacting OAuth routes (`/auth/device/*`, `/auth/token`, `/auth/logout`) use `BufferAuthBody` on its own for the same cap. `UseAppAuth` caps every dashboard route's body at the same 8MB with `MaxBytesReader`, so the non-transactional telemetry reads are bounded too. Every plain JSON bind site in the controllers answers through `middleware.RejectBindError(c, err, fallback)`, which maps `MaxBytesError` to 413 and the body guard's timeout to 408 with fixed messages and otherwise returns 400 with the fallback (the ingest routes use `middleware.RejectIngestBindError`, the same mapping except that the timeout becomes 503 + `Retry-After`, because OTLP exporters retry 503 but treat 408 as permanent; the Traceway SDKs re-queue a failed batch whatever the status); the guard itself swaps the raw deadline error (a `*net.OpError` naming the listener address) for `middleware.ErrBodyTimedOut` before it leaves `Read`, so no handler can echo the address.

**Preference:** For CRUD controller methods, always prefer using `middleware.Transactional` in the route + `db.GetTx(ctx)` in the controller over `pgdb.ExecuteTransaction`. The middleware approach keeps controllers flat, avoids nested closures, and follows the established pattern. (The tx getter lives in the `db` package: `db.GetTx(ctx)`, not `middleware.GetTx`.)

#### Repository Pattern
Repositories accept `*sql.Tx` to participate in transactions:
```go
func (p *projectRepository) FindById(tx *sql.Tx, id uuid.UUID) (*models.Project, error) {
    return lit.SelectSingle[models.Project](
        tx,
        "SELECT id, name, token, framework, created_at FROM projects WHERE id = $1",
        id,
    )
}
```

#### PostgreSQL Specifics
- Uses `$1, $2, $3` placeholders (not `?`)
- Tables must have an `id` column
- Always pass `*sql.Tx` from `ExecuteTransaction` to lit functions

#### Common Pitfalls

**Always initialize all struct fields with defaults:**
When using `lit.Insert`, all struct fields are included in the INSERT statement, overriding database defaults. Always set fields like `CreatedAt` explicitly:

```go
// CORRECT - set CreatedAt explicitly
user := &models.User{
    Email:     email,
    Name:      name,
    CreatedAt: time.Now().UTC(),
}

// WRONG - CreatedAt remains zero value (0001-01-01)
user := &models.User{
    Email: email,
    Name:  name,
}
```

**lit.Update WHERE clause:**
The `lit.Update` function automatically includes `WHERE` in the generated SQL. Do not add `WHERE` yourself:

```go
// CORRECT - just the condition
lit.Update(tx, &user, "id = $1", user.Id)

// WRONG - results in "WHERE WHERE id = $1"
lit.Update(tx, &user, "WHERE id = $1", user.Id)
```

#### Custom Result Models for Aggregates
For queries that return aggregated or computed values (not direct table rows), create a custom result model:

```go
// Define a result model for the query output
type CountResult struct {
    Count int `lit:"count"`
}

// Register in models.Init() (or file-level init() if repo-local)
lit.RegisterModel[CountResult](lit.PostgreSQL)

// Use in repository
func (r *userRepository) CountByOrganization(tx *sql.Tx, orgID uuid.UUID) (int, error) {
    result, err := lit.SelectSingle[CountResult](
        tx,
        "SELECT COUNT(*) as count FROM users WHERE organization_id = $1",
        orgID,
    )
    if err != nil {
        return 0, err
    }
    if result == nil {
        return 0, nil
    }
    return result.Count, nil
}
```

#### Handling "Not Found" Cases
**IMPORTANT:** When using lit/PostgreSQL, do NOT check for `sql.ErrNoRows`. The lit library returns `nil` when no record is found, not an error. Always check for `nil` instead:

```go
// CORRECT - check for nil
user, err := transactional.UserRepository.FindByEmail(tx, email)
if err != nil {
    return nil, err  // actual database error
}
if user == nil {
    // record not found - handle accordingly
    return nil, errors.New("user not found")
}

// WRONG - do not use sql.ErrNoRows with lit
user, err := transactional.UserRepository.FindByEmail(tx, email)
if err == sql.ErrNoRows {  // This won't work with lit!
    // ...
}
```

#### Handling "Not Found" Cases (ClickHouse)
**IMPORTANT:** ClickHouse queries behave differently from lit/PostgreSQL. ClickHouse returns `sql.ErrNoRows` when no record is found. Always use `errors.Is()` to check:

```go
// CORRECT - ClickHouse returns sql.ErrNoRows
exception, err := exceptionRepo.GetByHash(projectID, hash)
if errors.Is(err, sql.ErrNoRows) {
    // record not found - handle accordingly
    return nil, errors.New("exception not found")
}
if err != nil {
    return nil, err  // actual database error
}

// Summary of error handling:
// - lit/PostgreSQL: check `if result == nil` (no error returned)
// - ClickHouse: check `errors.Is(err, sql.ErrNoRows)`
```

### Environment Variables (Backend)
```
JWT_SECRET=<min 32 char secret for JWT signing>   # required in the standalone binary and every server image, including All-in-One. Missing, short and former public keys exit 1 with an actionable message. Docker's environment takes precedence over .env. Embedded Run(opts...) ignores JWT_SECRET (it belongs to the host app there): it uses WithJWTSecret, else TRACEWAY_JWT_SECRET (an explicitly empty value fails validation), else a random key per start, so set a stable value to preserve sessions across restarts. Retired public keys are matched by SHA-256 digest in jwt.service.go so no secret literal sits in scanned code.
DB_TYPE=                              # sqlite | postgres. Defaults to sqlite in every build EXCEPT -tags transactional_pg, where an empty value still selects PostgreSQL. Without that tag SQLite is the only supported main DB (the migration runner applies SQLite-dialect migrations unconditionally), so the default is set once in config.defaultDBType rather than at each DBType call site.
PORTS=80,8082                         # comma-separated HTTP listen ports. Every port is bound up front; a port that fails to bind (typically :80 as a non-root user) logs a warning (and reports via CaptureException when monitoring is configured), and startup only fails if NO port binds. Order is not significant: a listener that later stops serving takes the process down whichever position it holds.
SQLITE_CACHE_SIZE_MB=512              # PRAGMA cache_size of the SQLite TELEMETRY DB in MB, per connection (the main DB keeps SQLite's 2MB default; no effect on DuckDB or ClickHouse telemetry). The telemetry pool opens up to 4 connections (`telemetryMaxOpenConns`, db_sqlite_open.go) and modernc's allocator rounds each ~4.4KB page buffer up to an 8KB slot, so resident memory is about 2x the value per connection: worst case 8x the value, 4GB at the default. A cache fills from ingest, from the retention DELETE scan at startup and every hour, and from reads of pages still in the WAL (write transactions and WAL pages bypass the 1GB mmap window), and it is held for as long as the pooled connection lives. At the default a 1GiB container is OOM-killed on a 24h dashboard or under sustained ingest (issue #396). Rule of thumb: container MB / 32 (16 for 512MB, 32 for 1GB, 64 for 2GB). A non-positive or unparsable value falls back to 512 (`config.SizeMB`). The effective value is logged at startup, and the embedded `Run(opts...)` path honors the env var through `applyEnvOverrides`. Operator docs: `docs/pages/server/sqlite.mdx#memory-sizing`.
SQLITE_PATH=                          # main DB path, defaults to ./traceway.db in the working directory. The telemetry DB derives from it (_telemetry.db, or _telemetry.duckdb under -tags telemetry_duckdb).
APP_BASE_URL=                         # public origin of this server (e.g. https://traceway.example.com). Used as the OAuth issuer / device verification URL and SSO redirect base, and to absolutize the deep links in notifications. If unset, the device-auth + well-known endpoints derive it per-request from the Host / X-Forwarded-* headers; set it explicitly behind a proxy that doesn't forward Host.
TRUSTED_PROXIES=                      # comma-separated IPs/CIDRs whose X-Forwarded-For/X-Real-IP gin trusts for c.ClientIP() (every per-IP rate limiter keys on it, and `/api/report` stores it on every session as `client.ip`). Unset = loopback only, so no peer can spoof its IP. Private ranges are deliberately NOT trusted by default: on a Docker/k8s network every peer has a private address, the clients included when the port is published directly, so trusting RFC1918 would let any of them forge XFF past every per-IP limiter. Set it to the CIDRs of whatever sits in front (a proxy container's compose network, the ingress pod range, the CDN ranges); otherwise every visitor shares one rate-limit bucket keyed on the proxy. The value REPLACES the default, so it must name every hop in the chain, not just the outermost (a CDN-only list leaves a private ingress untrusted and disables XFF entirely); `*` trusts every peer. A malformed entry exits 1 at startup with a FATAL message, before the database is opened (`trustedProxyNets`, cmd/run.go). The first request carrying X-Forwarded-For/X-Real-IP from a peer outside the list logs a one-time warning naming that peer (`warnOnUntrustedForwardedFor`, cmd/run.go), which is how a forgotten proxy hop shows up. Parsing lives in `Cfg.TrustedProxyList` (config.go), covered by config_test.go.
TRUSTED_PROXY_HEADER=                 # optional header taken verbatim as the client IP on every request (gin TrustedPlatform: X-Real-IP behind ingress-nginx, CF-Connecting-IP behind Cloudflare); it wins over TRUSTED_PROXIES and turns the untrusted-forwarder warning off. Only safe when every request provably passes through a proxy that overwrites the header.
UPLOAD_MAX_CONCURRENT=4               # concurrent /api/sourcemaps/upload + /api/symbols/upload requests; excess waits 30s then gets 503. Separate from INGEST_MAX_CONCURRENT so a CI upload burst cannot starve telemetry ingest. Each upload holds a whole file in memory while parsing, so this bounds peak upload memory at roughly concurrency x largest file. It is a count-based gate, not a memory budget: on a memory-capped container size it from the limit rather than trusting the default. Same gate and bounded waiting room as ingest, its own instance. The source-map warm-up that follows an upload (`services.GenerateTWArtifacts`) runs outside this gate on its own 2-worker pool with a 64-job queue: the stale `.tw` artifact is deleted and the cache invalidated inline, the rebuild is queued, identical pending jobs are coalesced, and a full queue drops the warm-up (rate-limited `CaptureException`) because lookups build a missing artifact on demand anyway.
REPORT_MAX_BODY_MB=64                 # cap on the DECOMPRESSED /api/report and /api/profiles/ingest body (middleware.UseGzip bounds both the raw and gunzipped stream, so gzip cannot raise it). Deliberately far above the 10MB OTLP cap: session-replay frames carry rrweb segments and are a different size class, and the shipped browser SDKs re-queue a rejected batch forever, so a reachable cap is a permanent wedge rather than shed load. Overruns answer 413.
CLICKHOUSE_SERVER=localhost:9000
CLICKHOUSE_DATABASE=traceway
CLICKHOUSE_USERNAME=default
CLICKHOUSE_PASSWORD=
CLICKHOUSE_TLS=false
POSTGRES_HOST=localhost
POSTGRES_PORT=5432
POSTGRES_DATABASE=traceway
POSTGRES_USERNAME=traceway
POSTGRES_PASSWORD=
POSTGRES_SSLMODE=disable

# DuckDB telemetry backend (only with -tags telemetry_duckdb build; see "DuckDB Telemetry Backend" below). The embedded Run(opts...) path honors these three and DUCKDB_RETENTION_DAYS through applyEnvOverrides.
DUCKDB_MEMORY_LIMIT=                  # e.g. 4GB. Unset = DuckDB auto-tunes (~80% RAM). Set explicitly in memory-capped containers to avoid OOM.
DUCKDB_THREADS=                       # e.g. 4. Unset = DuckDB auto-tunes (= cores). Cap in constrained/shared environments.
DUCKDB_CHECKPOINT_THRESHOLD=          # e.g. 256MB. Unset = DuckDB default (16MB). Raise under sustained ingest to reduce WAL checkpoint stalls; costs a larger WAL and longer restart replay.

# Ingest admission gate (all telemetry ingest endpoints: /api/report, /api/profiles/ingest, /api/otel/*)
INGEST_MAX_CONCURRENT=                # max concurrently processed ingest requests. Unset = 2×CPU cores, min 4. Bounds ingest memory so overload sheds load with 503s instead of the process being OOM-killed (on DuckDB an OOM death is followed by a minutes-long WAL-replay stall on restart). The gate (`newAdmissionGate` in `middleware/ingest_admission.go`, a buffered-channel semaphore) also bounds waiters at 4×capacity (min 16); beyond that a request gets an immediate 503 instead of parking a goroutine and a timer for the wait window.
INGEST_ADMISSION_WAIT_SECONDS=5       # how long a request may wait for a slot before the 503 + Retry-After; 0 = reject immediately when saturated

# Email
EMAIL_PREVIEW_ENABLED=false           # "true" registers GET /api/email-preview[/:template], rendering every email template with sample data for design review. Off by default; never enable in production.

# Notifications
NOTIFICATION_POLL_SECONDS=60          # polled rule evaluation interval; minimum 5, invalid values fall back to 60
ONCALL_POLL_SECONDS=30                # on-call escalation worker interval; minimum 5, invalid values fall back to 30. Kept separate from NOTIFICATION_POLL_SECONDS so raising rule-evaluation intervals never delays paging. A buffered Wake() channel makes freshly opened pages notify L1 near-instantly regardless of this interval.
OUTBOX_POLL_SECONDS=15                # notification outbox drain interval; minimum 5, invalid values fall back to 15. The outbox (backend/app/outbox, notification_outbox table in the main DB) is the persist-then-send layer for ALL notifications: rule dispatch and the escalator only enqueue (with an adapter-config snapshot) inside their transactions; the drain worker sends with retries (backoff 1m/5m/15m/60m, 5 attempts, then terminal failed + CaptureException). Crash-safe at-least-once: stale 'sending' rows are reclaimed after 5 min, cancelled rows can never resurrect (guarded status transitions), ack/resolve cancels queued page deliveries via outbox.CancelByKey. Cooldown and event-rule dedup record at enqueue commit (the durable promise), and fired_notifications is written at the terminal outcome. /api/health/deep exposes an `outbox` block; `traceway.outbox.*` metrics are emitted when monitoring is on; terminal rows are pruned daily (sent/cancelled 7d, failed 30d).

# Synthetics (synthetic uptime monitoring: backend/app/synthetics, /synthetics frontend route)
SYNTHETICS_POLL_SECONDS=15            # scheduler tick for due checks; minimum 5, invalid values fall back to 15. The scheduler enqueues due checks into the check_runs queue (main DB, outbox-style guarded claims, advisory lock 824737004) and records expired queued runs as `missed` in telemetry — a probe is never executed late. In-process executors claim http/tcp runs always, browser runs only when mode=embedded.
SYNTHETICS_BROWSER_MODE=off           # off | embedded | remote. Browser checks are real @playwright/test specs executed by spawning Node against a harness dir (no npm/npx at runtime; allowlisted env so user scripts never see server secrets). `embedded` requires the :browser image (Dockerfile.browser, DuckDB base + Node + Chromium) and fails fast at startup otherwise; `remote` queues browser runs for traceway-runner binaries that long-poll /api/runners/poll authenticating with SYNTHETICS_RUNNER_SECRET. Hard-blocked in cloud mode (startup panic + 422 at check creation).
SYNTHETICS_BROWSER_SANDBOX=auto       # auto | bwrap | off. Isolation for the Playwright subprocess in embedded mode and in traceway-runner. The env allowlist alone does not contain a spec: node runs as the same uid as its parent, so unconfined it can read the parent's /proc/<pid>/environ (the runner secret), overwrite node_modules in the shared harness to backdoor later runs, and read a sibling run's directory. `bwrap` gives each run its own pid/ipc/uts/user namespace with a fresh /proc, a read-only harness bind with only that run's dir writable, an empty tmpfs over the .runs container so sibling run dirs are hidden (not exposed by the wholesale harness bind), and no /etc beyond DNS/CA/locale (so a secret in the unit file or EnvironmentFile is unreachable); the network namespace stays shared since probes must reach the internet. `auto` uses bubblewrap when the host can and degrades to off with a warning; explicit `bwrap` fails fast at startup instead. Resolved once via browserexec.ResolveSandbox, which probes by running `node --version` through the real bind set.
SYNTHETICS_RUNNER_SECRET=             # shared bearer credential for the operator's runner fleet; required for remote mode (fail-fast at startup). Runners are deployment infrastructure, NOT tenant entities: no dashboard CRUD, no per-runner tokens. A runner self-registers a liveness row in synthetic_runners under its X-Traceway-Runner-Name on first poll (upsert throttled to 1/min via an in-memory cache; claim identity is "runner:<name>"). Rotation = change the secret + restart runners. Runner-side env: TRACEWAY_URL, TRACEWAY_RUNNER_SECRET, TRACEWAY_RUNNER_NAME (default hostname), RUNNER_WORKERS (default 2, max 16), and optional TRACEWAY_RUNNER_MONITORING (<project_token>@<url>/api/report) which self-instruments the runner with the Traceway Go SDK: default server metrics, a "browser_run" task trace per executed run (tags check_id/run_id/status, server_name = runner name), and CaptureException on infra failures (start/report errors, per-job panics recovered without killing the worker).
HEALTH_DEEP_TOKEN=                    # operator bearer secret gating GET /api/health/deep (its payload is instance-wide: cross-tenant queue depth, runner fleet, storage engine stats). Unset = endpoint disabled with a 401 pointing here.
SYNTHETICS_HTTP_CONCURRENCY=8         # concurrent in-process http/tcp probes
SYNTHETICS_BROWSER_CONCURRENCY=2      # concurrent Chromium instances in embedded mode (~300-500MB each)
SYNTHETICS_ALLOW_PRIVATE_TARGETS=true # "false" rejects checks that resolve to private/LAN addresses, validated at save AND at dial time for both http and tcp probes (netguard.GuardedDialContext dials the vetted IP, DNS-rebinding safe). Forced false in cloud mode regardless of env (tenants must never probe the platform's network). Default allow elsewhere: probing LAN services is a core self-hosted use case. When the guard is off, http probes honor HTTP(S)_PROXY; when on, the proxy is ignored so the guard vets the real target. Check creation also runs controllers.CheckLimitHook (nil = unlimited; cloud caps monitors per org plan).
SYNTHETICS_PLAYWRIGHT_DIR=/opt/traceway-playwright   # Playwright harness dir (node_modules with pinned @playwright/test; docker/playwright/package.json pins the version)
SYNTHETICS_SCREENSHOT_RETENTION_DAYS=30 # browser failure artifacts under STORAGE_PATH/synthetics/ (.png screenshots AND .log Playwright output tails — failed browser runs store both, referenced by screenshot_key/output_key on check_results, served via GET /api/synthetics/screenshot|output?key= with project prefix checks); local storage only, 0 disables the cleanup worker

# Retention (see "Data Retention" section below)
SQLITE_RETENTION_DAYS=30              # 0 to disable; only applies in SQLite mode
DUCKDB_RETENTION_DAYS=30              # telemetry TTL on the DuckDB backend; when set it wins over SQLITE_RETENTION_DAYS there, when unset SQLITE_RETENTION_DAYS applies as fallback; 0 to disable
LOG_RECORDS_MAX_ROWS=                 # optional cap on log_records rows; unset/0 = disabled. NOT a hard limit: a cleanup worker trims to the newest N rows once per minute, so ingest above N rows/minute overshoots the cap between runs. Only applies in SQLite mode (SQLite or DuckDB telemetry)
SESSION_RECORDING_RETENTION_DAYS=30   # 0 to disable; only applies when STORAGE_TYPE=local
PROFILE_ARCHIVE_RAW=false             # native pprof ingest only: write the original pprof bytes to object storage as a lossless archive
PROFILE_RETENTION_DAYS=30             # 0 to disable; on-disk archive TTL, only with PROFILE_ARCHIVE_RAW + STORAGE_TYPE=local

# V2 trace tables move-over (see "Trace Storage (V2 tables)" below)
V2_MOVE_OVER=                         # "true" copies the history of the tables V2 replaced into the V2 tables, in the background, newest day first. Runs at startup and one hour after each pass; resumable (progress in the transactional DB v2_move_over_days, sqlite/0077 + pg/0143), so it can stay set. Finish upgrading old writers before enabling it: completed source-table/date pairs are skipped. With several backend instances set it on exactly one. On SQLite and DuckDB it stops at the telemetry retention window, since older days would be pruned at once.
V2_MOVE_OVER_OLDEST=                  # optional YYYY-MM-DD; days before it stay in the old tables. An unparsable value logs and does not start the move.
V2_MOVE_OVER_WORKERS=1                # source days copied concurrently inside the one enabled instance (max 16; invalid/non-positive = 1). Each day keeps table order; days are scheduled newest first. Increase to 4 for parallel backfill when the telemetry DB has spare capacity. Progress rows and logs update after successful batches roughly every 10s; slow queries/inserts can delay updates. Errors cancel and join all workers before the next hourly pass.

# Session recording uploads (see "Session Recording Uploader" section below)
SESSION_RECORDING_UPLOAD_WORKERS=32   # 0 to disable uploads entirely
SESSION_RECORDING_UPLOAD_QUEUE_SIZE=2048

# Source map symbolicator
SYMBOLICATOR_PARSER=goja              # goja (default) or oxc (requires -tags oxc build, see scripts/build-oxc-shim.sh)
SOURCEMAP_CACHE_TYPE=memory           # memory (default) or disk (mmap-backed .tw cache)
SOURCEMAP_DISK_CACHE_PATH=./twcache   # only used when SOURCEMAP_CACHE_TYPE=disk
SOURCEMAP_DISK_CACHE_MAX_MB=2048      # capacity-based LRU eviction of local .tw files
```

---

## Architecture Overview

### Data Flow
```
Go Application → [traceway SDK] → GZIP POST /api/report → Backend → ClickHouse
                                                              ↓
Dashboard ← [SvelteKit Frontend] ← JSON API ← Gin Controllers
```

### Authentication
Two-tier system:
1. **Client Auth**: Project bearer tokens (SDK telemetry via `Authorization: Bearer <project_token>`)
2. **App Auth**: JWT-based user authentication (dashboard via `Authorization: Bearer <jwt_token>`)

**App Auth credentials.** `UseAppAuth` accepts three credential shapes on the `Authorization: Bearer` header:
- **Dashboard JWT** — issued by `/api/login` / SSO (7-day expiry).
- **Personal access token (PAT)** — opaque `twp_`-prefixed token; looked up by SHA-256 hash in `personal_access_tokens`, resolves to its user. Created/listed/revoked from the account page (`/api/personal-access-tokens*`). Non-expiring or with an optional TTL; `last_used_at` is touched (throttled to 1/min, off the request path).
- **Device-flow access token** — a short-lived (15-min) JWT minted by the CLI's OAuth device flow.

**CLI / OAuth device flow** (`backend/app/services/authserver/`, controllers `device_auth.controller.go` / `wellknown.controller.go` / `pat.controller.go`): RFC 8628 device authorization grant plus rotating refresh tokens. `traceway login` (default) → `POST /api/auth/device/authorize` (client_id allowlisted, per-IP rate-limited, opportunistically prunes expired rows) → user approves at `/device` → the CLI polls `POST /api/auth/device/token` (grant `device_code`; `/api/auth/token` is an equivalent alias) which issues a 15-min access token + 90-day rotating refresh token (family-tracked in `refresh_tokens`). Refresh (`grant_type=refresh_token`) rotates the token atomically and revokes the whole family on genuine reuse; within a 30s grace window a benign concurrent retry is answered with the same rotated token set (from an in-memory rotation cache) instead of `invalid_grant`. `POST /api/auth/logout` revokes a family server-side. Tokens are stored SHA-256-hashed. The grant endpoints **self-manage their transactions** via `db.ExecuteTransaction` (not `middleware.Transactional`) because OAuth returns 400 for normal flow control (`authorization_pending`, reuse-revoke) and those side effects must still commit. `/.well-known/oauth-authorization-server` and `/.well-known/oauth-protected-resource` are served at the **origin root** (registered in `cmd/run.go`, not the `/api` group) per RFC 8414 / 9728. Beyond the device grant, the server implements the **authorization-code + PKCE grant** (S256 only) with **RFC 7591 dynamic client registration** for MCP clients: `POST /api/oauth/register` (rate-limited, open registration of public clients; https/custom-scheme redirect URIs anywhere, plain http only on loopback with port-flexible matching per RFC 8252) -> the client sends the user to `/oauth/authorize` (an SPA consent page like `/device`) -> the page calls `GET /api/oauth/client` + `POST /api/oauth/approve|deny` (approve mints a single-use 5-min `twa_` code bound to user/client/redirect/challenge; the redirect target is validated server-side) -> the client exchanges it at `POST /api/auth/token` (grant `authorization_code`; the code is consumed even on a failed exchange, wrong verifier/client/redirect are all `invalid_grant`). RFC 8707 `resource` params are validated against the issuer origin (`invalid_target`). Expired codes are pruned by the auth-tokens retention worker and opportunistically at approve time.

---

## Frontend (`/frontend`)

### Framework & Build
- **Framework**: SvelteKit 2 with Svelte 5 runes API
- **Styling**: Tailwind CSS v4 with shadcn-svelte components
- **Build**: Vite 7, static adapter with SPA fallback
- **SSR**: Disabled - pure client-side SPA (`ssr = false` in `+layout.ts`)

### Project Structure
```
frontend/
├── src/
│   ├── routes/              # SvelteKit pages
│   ├── lib/
│   │   ├── api.ts           # API client with auth
│   │   ├── state/           # Svelte 5 state management
│   │   ├── components/
│   │   │   ├── ui/          # shadcn-svelte base components
│   │   │   └── traceway/    # Custom Traceway components
│   │   └── utils/           # Helpers (formatting, sorting)
│   └── app.css              # Tailwind + global styles
├── static/                  # Static assets
└── svelte.config.js         # SvelteKit config
```

### State Management (Svelte 5 Runes)

#### Runes Pattern
```typescript
// Use $state() for reactive state
let data = $state<Type>(initial)

// Use $derived() for computed values
let computed = $derived(expression)

// Use $effect() for side effects
$effect(() => { /* reactive code */ })
```

#### State Files
| File | Purpose | Persistence |
|------|---------|-------------|
| `src/lib/state/auth.svelte.ts` | Token auth, login/logout | localStorage |
| `src/lib/state/projects.svelte.ts` | Multi-project management | localStorage |
| `src/lib/state/theme.svelte.ts` | Dark/light mode toggle | localStorage |
| `src/lib/state/timezone.svelte.ts` | UTC/local timezone toggle | localStorage |

#### Singleton Pattern
State files export class instances as singletons:
```typescript
// src/lib/state/auth.svelte.ts
class AuthState {
    token = $state<string | null>(null)
    isAuthenticated = $derived(!!this.token)

    constructor() {
        // Load from localStorage on init
        this.token = localStorage.getItem('token')
    }
}
export const authState = new AuthState()
```

### API Client (`src/lib/api.ts`)

The API client automatically:
- Includes `Authorization: Bearer <token>` header
- Adds `projectId` as a query parameter to all requests
- Handles 401 responses by logging out and redirecting to `/login`

```typescript
// Usage
const data = await api.post<ResponseType>('/endpoint', { body })
const data = await api.get<ResponseType>('/endpoint')
```

### Component Patterns

#### Table System
Tables use shadcn-svelte base components with custom Traceway wrappers:

| Component | Location | Purpose |
|-----------|----------|---------|
| `Table`, `TableHeader`, etc. | `src/lib/components/ui/table/` | Base shadcn components |
| `TracewayTableHeader` | `src/lib/components/traceway/traceway-table-header.svelte` | Adds sorting + tooltips |
| `TableEmptyState` | `src/lib/components/traceway/table-empty-state.svelte` | Empty state display |
| `PaginationFooter` | `src/lib/components/traceway/pagination-footer.svelte` | Pagination controls |

#### Sorting Storage
Table sorting persists to localStorage using a consistent pattern via `src/lib/utils/sort-storage.ts`:

```typescript
// Types
type SortState = { field: string; direction: 'asc' | 'desc' }

// Key format: traceway_sort_{pageKey}
// Example: traceway_sort_issues, traceway_sort_endpoints

// In +page.svelte - load initial state
let sortState = $state<SortState>(getSortState('issues', { field: 'last_seen', direction: 'desc' }))

// After sort change
function onSortClick(field: string) {
    sortState = handleSortClick(field, sortState.field, sortState.direction, 'desc')
    setSortState('issues', sortState)  // Persist to localStorage
}
```

**Available functions (`src/lib/utils/sort-storage.ts`):**
| Function | Description |
|----------|-------------|
| `getSortState(pageKey, defaultState)` | Load sort state from localStorage |
| `setSortState(pageKey, state)` | Save sort state to localStorage |
| `handleSortClick(field, currentField, currentDirection, defaultDirection)` | Toggle sort direction, returns new `SortState` |

#### TracewayTableHeader Component
```svelte
<TracewayTableHeader
    label="Last Seen"
    column="last_seen"
    tooltip="When this issue was last reported"
    {orderBy}
    onclick={() => handleSortClick('last_seen')}
/>
```

### URL State Management

Time range and filters persist in URL query params via `src/lib/utils/url-params.ts`:

```typescript
// Available presets: 30m, 60m, 3h, 6h, 12h, 24h, 3d, 7d, 1M, 3M

// Parse time range from URL (in +page.svelte)
const timeRange = parseTimeRangeFromUrl(timezoneState.timezone, '24h')

// Get resolved Date objects for API calls
const { from, to } = getResolvedTimeRange(timeRange, timezoneState.timezone)

// Update URL with new time range (preserves other params)
updateUrl({ preset: '7d' })
updateUrl({ from: customFrom, to: customTo })  // Custom range
```

**Available functions (`src/lib/utils/url-params.ts`):**
| Function | Description |
|----------|-------------|
| `parseTimeRangeFromUrl(timezone, defaultPreset)` | Parse `TimeRangeParams` from current URL |
| `getResolvedTimeRange(params, timezone)` | Convert params to `{ from: Date, to: Date }` |
| `updateUrl(params, options?)` | Update URL query params, optionally replace history |

### Navigation Utilities

Helper functions for preserving URL params during navigation (`src/lib/utils/navigation.ts`):

```typescript
// Add sticky params (like time range) to href for <a> tags
const href = addStickyParamsToHref('/issues/abc123', 'preset', 'from', 'to')
// Result: "/issues/abc123?preset=24h" (if preset=24h is in current URL)

// Create click handler for table rows that preserves params
const handleClick = createRowClickHandler('/issues/abc123', 'preset', 'from', 'to')
```

**Available functions:**
| Function | Description |
|----------|-------------|
| `addStickyParamsToHref(href, ...stickyParams)` | Returns href with specified params from current URL |
| `createRowClickHandler(href, ...stickyParams)` | Returns click handler that navigates with sticky params |

### Routes
```
/                           Dashboard (protected) - overview metrics
/login                      Login page (public)
/register                   Registration page (public)
/issues                     Issues list with filtering/sorting
/issues/[hash]              Exception details view
/issues/[hash]/events       Exception events timeline
/endpoints                  Endpoint analytics with P50/P95/P99
/endpoints/[endpoint]       Single endpoint details
/tasks                      Background tasks list
/tasks/[task]               Single task details
/spans                      Span explorer: every OTel span of the project, filtered by name, service, kind, status, duration, trace ID and attribute values (time range defaults to 60m)
/spans/[traceId]            Whole trace waterfall across services and across every project of the organization (?at= is required, ?span= highlights one span), with the current project's correlated logs on backend projects
/dashboards                 Dashboards page (tabs of org dashboards; /metrics redirects here)
/monitors                   Monitors (synthetic checks): single sidebar item, TabsRow tabs Monitors | Status Pages | Post-Mortems (?tab=, Status Pages admin-only, Post-Mortems for all members); status pages tab has branding (description, logo upload, custom domain) plus a link to the per-page incidents page (which owns the Record Incident dialog); old /monitors/status-pages redirects to the tab, /monitors/runners to /monitors (runners have no UI — operator infra), bare /monitors/post-mortems to the tab
/monitors/status-pages/[pageId]/incidents  Paginated incident management for one status page (timeline updates, titles, manual resolve/delete, post-mortem links); reads for members, mutations admin-gated in the UI
/monitors/[checkId]         Monitor detail (latency chart, uptime bars, runs w/ result filter, incidents w/ titles + post-mortem links)
/monitors/post-mortems/[id] Post-mortem editor/viewer (@milkdown/crepe WYSIWYG with a Rich text/Markdown raw toggle, tag pills + Add-tags dialog, incident link pill + searchable picker, Activity sheet; write gating follows the effective project role). Project-scoped: opening a doc under a different project 404s. Creation is a title-only dialog (new-post-mortem-dialog.svelte, openable from the tab, monitor detail incidents, and the status page incidents page with an optional incident pre-link) that POSTs immediately and navigates here; /monitors/post-mortems/new redirects to the tab
/status/[slug]              Public status page (light standalone design, no auth, raw fetch, listed in isPublicPath)
/connection                 SDK integration guide
```

### UI Components
Location: `src/lib/components/ui/*`
Uses shadcn-svelte registry with bits-ui primitives. Key components:
- `button`, `card`, `table`, `badge`, `tooltip`
- `select`, `input`, `checkbox`
- `sheet` (slide-out panels), `dialog` (modals)

---

## Backend (`/backend`)

### Architecture
- **Framework**: Gin Gonic HTTP framework
- **Database**: ClickHouse (columnar OLAP for telemetry), PostgreSQL (relational for projects)
- **Port**: 8082
- **Pattern**: Repository pattern with singleton controllers

### Project Structure
```
backend/
├── traceway.go                 # package tracewaybackend - embeddable API (Run, WithPort, ...)
├── cmd/
│   ├── run.go                  # Run() implementation: DB init, routes, server start
│   ├── traceway/main.go        # Binary entry point (package main)
│   └── traceway-runner/        # Synthetics remote runner binary
├── app/
│   ├── controllers/
│   │   ├── routes.go           # Route registration
│   │   ├── dashboard.go        # Dashboard metrics
│   │   ├── auth.go             # Login handler
│   │   ├── projects.go         # Project CRUD
│   │   └── clientcontrollers/
│   │       └── report.go       # Telemetry ingestion (/api/report)
│   ├── repositories/           # ClickHouse queries
│   │   ├── transactions.go
│   │   ├── exceptions.go
│   │   ├── metrics.go
│   │   └── projects.go
│   ├── models/                 # Data structures
│   ├── middleware/
│   │   ├── auth.go             # Token validation
│   │   └── gzip.go             # Request decompression
│   ├── cache/                  # In-memory project cache; on Postgres, replicas refresh each other through LISTEN/NOTIFY (project_cache_changed, sent by the project repository inside the write transaction)
│   ├── pgdb/                   # PostgreSQL connection manager
│   └── migrations/
│       ├── ch/                 # ClickHouse migrations
│       └── pg/                 # PostgreSQL migrations
```

### Middleware Chain Composition

| Route Type | Middleware Chain |
|-----------|-----------------|
| Read-only telemetry | `UseAppAuth, RequireProjectAccess` |
| Write telemetry | `UseAppAuth, RequireProjectAccess, RequireWriteAccess` |
| PostgreSQL CRUD (read) | `UseAppAuth, RequireProjectAccess, Transactional` |
| PostgreSQL CRUD (write) | `UseAppAuth, RequireProjectAccess, RequireWriteAccess, Transactional` |
| Admin org management | `UseAppAuth, RequireAdminAccess, Transactional` |
| Public (auth/invitations) | `RateLimitPerIP`, `BufferAuthBody`, `Transactional` (limiter first so a throttled request never reads a body or opens a transaction) |
| Client SDK ingestion | `CORSReport, UseClientAuth, UseGzip` |
| Synthetic runner API | `UseRunnerAuth` only (shared SYNTHETICS_RUNNER_SECRET bearer + self-registration by X-Traceway-Runner-Name; no Transactional — poll holds the request up to 25s) |
| Public status page | `RateLimitPerIP` only |

**Global middleware** (registered in `cmd/run.go` ahead of the routes): `middleware.GuardBodyReads` (every request body gets the progress guard, 30s idle / 10 min total / 1 KB/s floor after 20s, answering through `middleware.BodyReadError` as 408; routes with their own budget call `GuardBodyRead` again, which re-arms the same guard rather than wrapping twice, so uploads get 30s/30 min, `/api/report` 20s/10 min and OTLP 20s/2 min. The guard clears the socket read deadline the moment the body is complete: Go's background connection read would otherwise time out and cancel the request context of any handler that outlives the idle window. The `http.Server` `ReadTimeout` is a 1h backstop that only applies while a body is still arriving (the guard lifts it for bodiless requests too, so a long-lived handler such as the MCP `GET /mcp` stream is not cancelled after an hour), `ReadHeaderTimeout` is 15s), `configureClientIP` (gin's trusted proxies from `Cfg.TrustedProxyList`, plus `TrustedPlatform` from `TRUSTED_PROXY_HEADER`; a malformed list panics), `warnOnUntrustedForwardedFor` (list mode only: one-time log when a forwarding header arrives from a peer outside `TRUSTED_PROXIES`), and `middleware.SecurityHeaders` (`X-Frame-Options: DENY`, CSP `frame-ancestors 'none'; base-uri 'self'; form-action 'self'`, `Referrer-Policy`, `nosniff`). The two frame headers are skipped for any `/status/*` or `/api/status/*` path so public status pages stay embeddable in an iframe. A custom-domain status host is exempt as a whole: the SPA handler serves nothing but that status page there, so it drops the two headers itself (see the status domains row below).

### API Endpoints

**Client SDK Ingestion**
| Method | Endpoint | Auth | Purpose |
|--------|----------|------|---------|
| POST | `/api/report` | Client | Telemetry ingestion (gzipped) |
| POST | `/api/otel/v1/traces` | Client | OTLP/HTTP trace ingestion |
| POST | `/api/otel/v1/metrics` | Client | OTLP/HTTP metric ingestion |
| POST | `/api/otel/v1/logs` | Client | OTLP/HTTP log ingestion |
| POST | `/api/otel/v1development/profiles` | Client | OTLP/HTTP profile ingestion (development signal) |

**Auth & Registration**
| Method | Endpoint | Auth | Purpose |
|--------|----------|------|---------|
| POST | `/api/login` | None | Dashboard authentication |
| POST | `/api/register` | None | New user registration |
| GET | `/api/has-organizations` | None | Check if any orgs exist (self-hosted only) |
| POST | `/api/forgot-password` | None | Request password reset email |
| GET | `/api/password-reset/:token` | None | Validate reset token |
| POST | `/api/password-reset/:token` | None | Reset password with token |

**CLI Device Auth & OAuth** (see the Authentication section above)
| Method | Endpoint | Auth | Purpose |
|--------|----------|------|---------|
| POST | `/api/auth/device/authorize` | None | Start device flow; returns device/user code + verification URL |
| POST | `/api/auth/device/token` | None | Poll for the token (grant `device_code`); also accepts `refresh_token`. Shares a 60/min per-IP limiter with `/api/auth/token`; over it a `device_code` grant answers the RFC 8628 `400 slow_down` (`RateLimitOAuthTokenPerIP` peeks `grant_type` from the buffered body), so the shipped CLI backs off instead of aborting the login, while every other grant gets a plain `429` + `Retry-After` because `slow_down` is undefined for them and OAuth clients treat an unknown 400 as terminal. Body capped at 64KB by `BufferAuthBody` like the other anonymous auth routes |
| POST | `/api/auth/token` | None | Token endpoint: `device_code`, `refresh_token` or `authorization_code` grant (JSON or form-encoded). Same shared limiter, grant-aware rejection and 64KB body cap as the device token route |
| POST | `/api/auth/logout` | None | Revoke the presented refresh token's family (idempotent) |
| GET | `/api/device` | App | Look up a user code for the approval screen |
| POST | `/api/device/approve` | App | Approve a pending device authorization (tokens carry only the approving user's own role, so no write guard) |
| POST | `/api/device/deny` | App | Deny a pending device authorization |
| POST | `/api/oauth/register` | None | RFC 7591 dynamic client registration (rate-limited) |
| GET | `/api/oauth/client` | App | Resolve a client_id to its display name (consent page) |
| POST | `/api/oauth/approve` | App | Approve an authorization request; mints the code, returns the validated redirect |
| POST | `/api/oauth/deny` | App | Deny an authorization request; returns the error redirect |
| GET | `/.well-known/oauth-authorization-server` | None | RFC 8414 metadata (served at origin root, not `/api`) |
| GET | `/.well-known/oauth-protected-resource` | None | RFC 9728 metadata (served at origin root, not `/api`) |
| GET/POST/DELETE | `/mcp` | Bearer | Streamable HTTP MCP server (origin root); 401s carry a `WWW-Authenticate` resource-metadata challenge |

**Personal Access Tokens**
| Method | Endpoint | Auth | Purpose |
|--------|----------|------|---------|
| POST | `/api/personal-access-tokens` | App | Create a PAT (returns the `twp_` token once) |
| GET | `/api/personal-access-tokens` | App | List the current user's active PATs |
| DELETE | `/api/personal-access-tokens/:id` | App | Revoke a PAT |

**Projects**
| Method | Endpoint | Auth | Purpose |
|--------|----------|------|---------|
| GET | `/api/projects` | App | List projects |
| POST | `/api/projects` | App | Create project (optional `organizationId` body field targets another org; the handler checks the caller's **org role** in the target org and 403s for non-members/`readonly`; creation is org-scoped, so per-project overrides neither grant nor deny it) |
| POST | `/api/projects/source-map-token` | App+Write | Generate source map upload token |

**Dashboard**
| Method | Endpoint | Auth | Purpose |
|--------|----------|------|---------|
| GET | `/api/dashboard` | App | Dashboard metrics |
| GET | `/api/dashboard/overview` | App | Recent issues + top endpoints |
| POST | `/api/stats` | App | Homepage stats |

**Metrics**
| Method | Endpoint | Auth | Purpose |
|--------|----------|------|---------|
| GET | `/api/metrics/application` | App | Application metrics |
| GET | `/api/metrics/stats` | App | Stats metrics |
| GET | `/api/metrics/server` | App | Server metrics |
| POST | `/api/metrics/query` | App | Custom metric queries. Aggregations avg/min/max/sum/count/last plus `rate`, which differences every series (full tag set = identity) against its previous sample, drops resets, and sums the deltas per bucket as a per-second rate; the reported unit gains `/s` (`By/s`), a counter of seconds becomes the ratio `1`, and on ClickHouse `rate` follows `selectTable` like every other aggregation: past 6h it reads the rollups through `maxMerge(max_val)` (a counter only grows, so a rollup bucket's max is its last sample), never `metric_points_1d`, with the chart bucket clamped to the rollup step and the lookback widened by it (`rateSource`). The `traceway-otel-agent` template relies on it (plus `state` filters, `groupBy: server_name`, and the per-source `complement` flag that the widget renderer applies as 1 − value); `backfill.RunOtelAgentDashboardSources` upgrades installed copies whose widgets still carry the original plain-average source. Bounded in `metric_query.controller.go`: 256KB body, max 50 queries and 20 tag filters each, range at most 731 days and `to >= from` (422), buckets widened so no series exceeds 2000 points (the effective `intervalMinutes` is returned at the top level of the response), at most 200 groups per query (the repositories return one extra so the result carries `truncatedGroups: true`), and a 30s deadline for the whole request that answers 504. The discovery endpoints validate their `from`/`to` the same way |
| GET | `/api/metrics/discover` | App | Discover available metrics |
| GET | `/api/metrics/discover/tags` | App | Discover metric tags |
| GET | `/api/metrics/discover/instances` | App | Distinct `server_name` values in a range (dashboard instance filter) |
| PUT | `/api/metrics/registry` | App+Write | Update metric registry entry |

**Dashboards & Templates**

Dashboards are org-owned JSON documents (`{schemaVersion, widgets: [{id, title, widgetType, config}]}`, widget ids are server-generated `w_xxxxxxxx` strings, array order = display order) applied to projects via `project_dashboards`. Dashboard mutations require org role above `readonly` (checked in-handler); project-scoped routes (list/star/reorder/populate) use the standard middleware chains; apply/unapply also check the effective role of each affected project. The old per-project widget-group tables are converted once at startup by `backfill.RunDashboards` (advisory-locked on PG) and retained for rollback.

| Method | Endpoint | Auth | Purpose |
|--------|----------|------|---------|
| GET | `/api/dashboards` | App | Dashboards applied to the project (tab order) |
| GET | `/api/dashboards/library` | App | All dashboards across the user's orgs with applied project ids |
| POST | `/api/dashboards` | App | Create in org (auto-applies to current project unless `applyToProjectIds` given) |
| GET | `/api/dashboards/:id` | App | Meta + widgets (+ per-project `isStarred`, `appliedProjectIds`) |
| PUT | `/api/dashboards/:id` | App | Update name/description and/or full `definition` (the as-code path) |
| DELETE | `/api/dashboards/:id` | App | Delete everywhere (assignments + stars cascade) |
| PUT | `/api/dashboards/:id/apply` | App | Set the full project assignment list |
| DELETE | `/api/dashboards/:id/apply/:projectId` | App | Unassign from one project |
| POST | `/api/dashboards/:id/copy` | App | Copy (also cross-org) with optional apply |
| PUT | `/api/dashboards/reorder` | App+Write | Tab order for a project (explicit id order) |
| POST | `/api/dashboards/:id/widgets` | App | Add widget |
| PUT | `/api/dashboards/:id/widgets/reorder` | App | Reorder widgets (explicit id order) |
| PUT | `/api/dashboards/:id/widgets/:wid` | App | Update widget |
| DELETE | `/api/dashboards/:id/widgets/:wid` | App | Delete widget (+ its stars) |
| PUT | `/api/dashboards/:id/widgets/:wid/star` | App+Write | Star/unstar for the project homepage |
| GET | `/api/dashboards/starred` | App | Starred widgets with homepage layout |
| PUT | `/api/starred-widgets/reorder` | App+Write | Reorder homepage starred widgets (`{ids}` = starred row ids) |
| PUT | `/api/starred-widgets/:id` | App+Write | Update homepage layout (colSpan/size) |
| GET | `/api/dashboards/:id/export` | App | Export one dashboard as JSON |
| GET | `/api/dashboards/export?organizationId=` | App | Export the org bundle |
| POST | `/api/dashboards/import` | App | Import doc/bundle (`mode: create\|upsert`, upsert matches by name) |
| POST | `/api/dashboards/import/grafana` | App | Convert a Grafana export (best-effort, returns `warnings[]`) |
| GET | `/api/dashboard-templates` | App | Marketplace list/search (`search`, `category` params) |
| POST | `/api/dashboard-templates/:key/install` | App | Copy a template into the org and apply |
| POST | `/api/dashboards/populate-defaults` | App+Write | Install the framework-default template set for an empty project |
| GET | `/api/metrics/discover/org` | App | Metric names across all org projects (command palette) |

Templates are DB rows seeded by migrations (`traceway-otel-agent` for the OTel host agent, `golang` for Go SDK apps, `traceway-clickhouse`/`traceway-duckdb` for the telemetry stores of a monitored Traceway instance; SQLite emits no store-specific metrics so it has no template); cloud can insert more rows without a release. The OTLP metric ingest allowlists per-resource identity and grouping tags (`host.name`, `host.id`, `os.type`, `cloud.region`, `container.name`, `k8s.cluster.name`, `k8s.pod.name`, `k8s.node.name`, `postgresql.database.name`, ...) in `otelcontrollers/metric_converter.go` for infrastructure views and custom widgets.

**Endpoints**
| Method | Endpoint | Auth | Purpose |
|--------|----------|------|---------|
| POST | `/api/endpoints` | App | List all endpoints |
| POST | `/api/endpoints/grouped` | App | Endpoint aggregates (P50/P95/P99) |
| POST | `/api/endpoints/endpoint` | App | Single endpoint details |
| POST | `/api/endpoints/chart` | App | Stacked chart data |
| GET | `/api/endpoints/slow` | App | Get slow endpoint threshold |
| POST | `/api/endpoints/slow` | App+Write | Set slow endpoint threshold |
| POST | `/api/endpoints/:endpointId` | App | Endpoint detail view |

**Tasks**
| Method | Endpoint | Auth | Purpose |
|--------|----------|------|---------|
| POST | `/api/tasks` | App | List all tasks |
| POST | `/api/tasks/grouped` | App | Grouped by task name |
| POST | `/api/tasks/task` | App | Single task details |
| POST | `/api/tasks/:taskId` | App | Task detail view |

**Exceptions**
| Method | Endpoint | Auth | Purpose |
|--------|----------|------|---------|
| POST | `/api/exception-stack-traces` | App | Grouped exception list |
| POST | `/api/exception-stack-traces/archive` | App+Write | Archive exceptions |
| POST | `/api/exception-stack-traces/unarchive` | App+Write | Unarchive exceptions |
| POST | `/api/exception-stack-traces/by-id/:exceptionId` | App | Single exception by ID |
| POST | `/api/exception-stack-traces/:hash` | App | Exception by hash |

**AI Traces & Conversations**

One `ai_traces_v2` row per LLM call (any OTLP span with `gen_ai.*` attributes). Each row carries a `conversation_id` resolved at ingest (`gen_ai.conversation.id` -> `session.id` span/resource attr -> trace id), `tool_call_count`/`tool_names` parsed from the completion payload (OpenAI `tool_calls`, Anthropic `tool_use`, OTel output messages, `gen_ai.tool.*` fallback), and `flagged`/`flagged_terms` from an ingest-time word-boundary scan of prompt+completion against per-project selected built-in language packs (`projects.ai_flagged_languages`, default `["en"]`; packs live in `backend/app/services/contentflag/terms/*.txt` — en/de/es/fr/it/pt/sr; empty array = custom terms only) plus per-project custom terms (`projects.ai_flagged_terms`); both edited in the project settings AI tab and cached via the project cache. All three project-creation paths (`Create`, `CreateWithOrganization`, `cmd/seed.go`) must set `AiFlaggedLanguages` explicitly since lit inserts every struct field. `tool_names`/`flagged_terms` are stored comma-separated (values sanitized at ingest); the content matcher lives in `backend/app/services/contentflag/`. Conversation analytics exclude rows with an empty `conversation_id`; user analytics additionally require a non-empty `user_id`.

| Method | Endpoint | Auth | Purpose |
|--------|----------|------|---------|
| POST | `/api/ai-traces/grouped` | App | AI traces grouped by trace name |
| POST | `/api/ai-traces/trace` | App | Calls for one trace name (`?traceName=`) |
| POST | `/api/ai-traces/:traceId` | App | Single call detail + conversation blob |
| POST | `/api/ai-conversations/grouped` | App | Conversations (GROUP BY conversation_id): turns, cost, tokens, tools, models, flagged; filters userId/model/toolName/flaggedOnly/search (search matches conversation id, user, model, tool names, and flagged terms; row-level filters are semi-joins on conversation_id so a match on any turn returns the whole conversation's aggregates); response also carries `thresholds` (range-wide P95 cost/turns for outlier highlighting) and `facets` (models, tools) |
| POST | `/api/ai-conversations/conversation` | App | All turns of one conversation (id in body) ordered by recorded_at, each with its stored input/output payload (capped at 200 turns of payloads), plus stats |
| POST | `/api/ai-users/grouped` | App | Per-user conversation analytics: conversation count, total calls, avg/min/median turns, avg cost per conversation, total cost, flagged conversation count |

Frontend routes: `/ai-traces` (tabs: Traces, Conversations, Users), `/ai-traces/conversations/[conversationId]` (chat timeline with tool calls rendered). Trace names `conversations` and `users` are shadowed by these static routes. Notification rule types `ai_trace_cost` (per call), `ai_conversation_cost` (24h cumulative per conversation), and `ai_flagged_content` (flagged term match, optional term filter) are event-driven; remember both `notification_rule.repository.go` copies list event rule types explicitly.

**Organization Management**
| Method | Endpoint | Auth | Purpose |
|--------|----------|------|---------|
| POST | `/api/organizations` | App | Create an organization owned by the caller; the recovery path for a user removed from their last organization (self-hosted: 422 once one exists; cloud: `OrganizationLimitHook`). Runs `PostRegistrationHooks` like register and SSO finish-setup |
| GET | `/api/organizations/:orgId/settings` | Admin | Get org settings |
| PUT | `/api/organizations/:orgId/settings` | Admin | Update org settings |
| GET | `/api/organizations/:orgId/members` | Admin | List members |
| PUT | `/api/organizations/:orgId/members/:userId` | Admin | Update member role |
| DELETE | `/api/organizations/:orgId/members/:userId` | Admin | Remove member |
| GET | `/api/organizations/:orgId/members/:userId/project-roles` | Admin | List member's per-project role overrides |
| PUT | `/api/organizations/:orgId/members/:userId/project-roles/:projectId` | Admin | Set/clear a per-project role override |

**Organization Overview** (frontend routes `/organization` = Servers, `/organization/issues`, `/organization/monitors`, `/organization/on-call`, `/organization/projects`)

Org-wide read views that fan out over every project of the org, in `organization_overview.controller.go`. `RequireOrganizationAccess` (any org role) gates them; the servers/issues/monitors handlers deliberately skip `Transactional` because they run per-project telemetry queries and must not hold the single-connection SQLite main DB across them (they open their own short `db.ExecuteTransaction` for the project list first). The pure main-DB pages endpoint keeps `Transactional`. The frontend renders the organization routes in the same sidebar shell as project pages, with the sidebar switched to organization items (Servers, Issues, Monitors, On-Call, Projects under an "Organization" label; badges come from `/overview/counts`). The selected organization is the shared `organizationContext` in `frontend/src/lib/state/organization-context.svelte.ts` (the `organizationId` query param, else the first membership), and `src/routes/organization/+layout.svelte` owns the redirects: a single-project org into that project, legacy `?tab=` links to the new paths, and frontend-only orgs from Servers/Monitors to Issues. Login auto-lands on `/organization` when the first org has more than one project (`frontend/src/lib/utils/landing.ts`). A single-project organization never shows the view: `/organization` redirects into that project and the switcher renders the org as a plain label, since the project's own pages already cover everything it would aggregate. When none of the org's projects is a backend project (`isBackendFramework` in `projects.svelte.ts`, i.e. every project is a browser or mobile framework), the Servers and Monitors sidebar items are hidden and `/organization` redirects to `/organization/issues`.

| Method | Endpoint | Auth | Purpose |
|--------|----------|------|---------|
| GET | `/api/organizations/:organizationId/overview/servers` | Member | One row per `server_name` per project over the last 30 min: CPU/memory/disk/network + a CPU sparkline, plus host/os/cloud/k8s metadata lifted from metric tags. Reads OTel hostmetrics (`system.cpu.utilization` state=idle inverted, `system.memory.utilization` state=used, `system.filesystem.utilization` state=used, `system.network.io` by direction) and falls back to the legacy SDK names (`cpu.used_pcnt`, `mem.used`/`mem.total`). A project that fails to read sets `partial: true` instead of failing the request |
| POST | `/api/organizations/:organizationId/overview/issues` | Member | Paginated issues across all projects. Body mirrors `/api/exception-stack-traces` (`fromDate`, `toDate`, `orderBy`, `search`, `searchType`, `pagination`); every project contributes its top `page*pageSize` groups in the requested order and the page is a slice of their merged order, so it equals what one combined project would return. That per-project fetch is capped at 1000 rows (`orgOverviewMaxIssueFetch`): a deeper page answers 422 and `totalPages` is clamped so the footer never offers one; `FindGrouped` orders ties by `exception_hash` on every backend so the merge is stable across pages. Returns `{data, pagination, partial}` with 24h hourly trends attached; a project that fails to read sets `partial: true` like the servers endpoint |
| POST | `/api/organizations/:organizationId/overview/pages` | Member | Paginated on-call pages across all projects: body `{status, search, fromDate, toDate, pagination}` (status active/open/acknowledged/resolved, search over subject + rule name, range on `created_at`); returns `{data, pagination, openPagesCount}` |
| GET | `/api/organizations/:organizationId/overview/counts` | Member | `openPagesCount` + `downMonitorsCount` for the organization sidebar badges |
| POST | `/api/organizations/:organizationId/overview/incidents` | Member | Paginated incidents across monitors and status pages: body `{search, fromDate, toDate, pagination}`; an incident matches when it overlaps the range (`started_at <= to` and unresolved or `resolved_at >= from`), search covers title, monitor, status page, and error message |
| GET | `/api/organizations/:organizationId/overview/monitors` | Member | Every synthetic check in the org with 30-day aggregates; a project whose results fail to read sets `partial: true` like the servers endpoint |

Instance identity is the `server_name` tag, which comes from the OTLP `service.name` resource attribute; grouping by Kubernetes cluster keys off the `k8s.cluster.name` tag and the group selector only offers it when at least one instance carries one. Clicking a row opens `/dashboards?projectId=&server=&preset=30m` (plus `dashboard=` when the project has the `traceway-otel-agent` template applied); `server` scopes every widget query via `scopeTagFilters` in `widget-grid`/`widget-renderer`. Cluster-wide instrumentation manifests live in `examples/kubernetes/`, documented at `docs/pages/learn/kubernetes.mdx`.

**Invitations**
| Method | Endpoint | Auth | Purpose |
|--------|----------|------|---------|
| POST | `/api/organizations/:orgId/invitations` | Admin | Send invitation |
| GET | `/api/organizations/:orgId/invitations` | Admin | List invitations |
| DELETE | `/api/organizations/:orgId/invitations/:id` | Admin | Revoke invitation |
| GET | `/api/invitations/:token` | None | Get invitation info |
| POST | `/api/invitations/:token/accept` | None | Accept (new user) |
| POST | `/api/invitations/:token/accept-existing` | App | Accept (existing user) |

**On-Call** (teams, schedules, escalation policies, pages)

Org-scoped entities in the main DB. Teams (`teams`/`team_members`) own projects one-to-one (`project_teams`, unique on project_id). Schedules (`oncall_schedules`) store PagerDuty-style calendar layers as a JSON `definition` document (rotations daily/weekly/custom, handoff time/day, time-of-day and day-of-week restrictions); one-off overrides are normalized rows (`oncall_overrides`). The pure resolution engine lives in `backend/app/oncall/` (`ResolveRange`/`ResolveAt`, tz-aware calendar math, later layer wins, overrides trump all). Both apply the same stacking, so a schedule puts exactly one person on call at any instant: `ResolveAt` is `ResolveRange` over a single instant, and paging a whole schedule stack (waking the person an override was meant to relieve) is the bug it exists to prevent. Escalation policies (`escalation_policies`) hold JSON steps (`targets` schedule/user/team/channel + `delayMinutes`, `repeatCount`); rules page on-call via the `escalation` notification channel type (config `{"policyId"}`), special-cased in `notifications/dispatch.go` through `RegisterPageOpener` (wired in `cmd/run.go`). A fired rule opens a `pages` row (dedup key `ruleId|dedupToken` with a partial unique index while unresolved; rules without a dedup token dedup at rule level; refires bump `event_count` and never reset the escalation clock). The escalator worker (`oncall/escalator.go`, `ONCALL_POLL_SECONDS`) claims due pages in a transaction (pg advisory lock 824737002 for multi-instance), inserts `page_notifications` rows, resolves targets to users, delivers via each user's `user_contact_methods` (email/slack/pushover/telegram adapter configs; account-email fallback always), then escalates level by level until ack/resolve or exhaustion. `RequireOrganizationAccess` middleware (any org role) gates member-level reads; mutations are org-admin; page acknowledge deliberately requires no write access.

| Method | Endpoint | Auth | Purpose |
|--------|----------|------|---------|
| GET/POST | `/api/organizations/:organizationId/teams` | Member / Admin | List (with members+projects) / create |
| PUT/DELETE | `/api/organizations/:organizationId/teams/:teamId` | Admin | Update / delete |
| PUT | `.../teams/:teamId/members`, `.../teams/:teamId/projects` | Admin | Replace ordered members / owned projects |
| GET/POST | `/api/organizations/:organizationId/schedules` | Member / Admin | List / create |
| GET/PUT/DELETE | `.../schedules/:scheduleId` | Member / Admin / Admin | Detail+overrides / whole-document update / delete |
| GET | `.../schedules/:scheduleId/timeline?from=&to=` | Member | Rendered per-layer + final shifts (max 62 days) |
| POST/DELETE | `.../schedules/:scheduleId/overrides(/:overrideId)` | Member | Create (any member, max 30d) / delete (creator, covered user, or admin) |
| GET | `/api/organizations/:organizationId/oncall/now` | Member | Overview: per team/schedule current + next on-call |
| GET | `/api/oncall/current?projectId=` | App | Owning team + current on-call for a project (issue page) |
| GET | `/api/escalation-policies` | App | Policies of the project's org (channel dialog picker) |
| GET/POST | `/api/organizations/:organizationId/escalation-policies` | Member / Admin | List / create |
| PUT/DELETE | `.../escalation-policies/:id` | Admin | Update / delete (422 while referenced by a channel) |
| POST | `/api/pages` | App | List (POST-body: status open/acknowledged/resolved/active + pagination) |
| GET | `/api/pages/:id` | App | Detail + delivery log |
| POST | `/api/pages/:id/acknowledge` | App (no write gate) | open -> acknowledged, stops escalation; 409 if not open |
| POST | `/api/pages/:id/resolve` | App | open/acknowledged -> resolved; 409 if already resolved |
| GET | `/api/pages/open-count` | App | Sidebar badge count |
| GET/POST | `/api/contact-methods` | App | Own contact methods (self-scoped). Types: email, slack, pushover, telegram, sms. SMS requires Twilio (`TWILIO_ACCOUNT_SID` + `TWILIO_AUTH_TOKEN` + one of `TWILIO_FROM_NUMBER` / `TWILIO_MESSAGING_SERVICE_SID`); without them SMS is not offered at all — the list response carries `smsEnabled: false` so the type picker hides it; create, re-point, resend-code and test answer 422; the escalator drops existing sms methods before its "no methods left" check so those users fall back to the account email; and the adapter errors instead of reporting a delivery nobody received. Disabling and deleting a leftover sms method stay available. Creating/re-pointing an sms method starts code verification; unverified numbers are never paged |
| PUT/DELETE | `/api/contact-methods/:id` | App | Update (incl. enabled toggle) / delete |
| POST | `/api/contact-methods/:id/test` | App | Send a canned test through one method (422 for unverified sms) |
| POST | `/api/contact-methods/:id/verify` | App (rate-limited) | Confirm the 6-digit SMS code (hashed at rest, 10-min expiry, 5-attempt cap). Deliberately **not** under `middleware.Transactional`: a wrong code answers 422, which would roll the consumed attempt back, so the handler manages its own transactions (nesting one under the middleware would also deadlock the single-connection SQLite main DB) |
| POST | `/api/contact-methods/:id/resend-code` | App (rate-limited) | Re-issue the verification code |
| GET/PUT | `/api/user-notification-rules` | App | Per-user notification-rule chains `{high: [{contactMethodId, delayMinutes}], low: [...]}` (PagerDuty-style: the page's urgency picks the chain; steps are enqueued at claim time as scheduled outbox deliveries and cancelled on ack; no chain = all enabled+verified methods immediately). Escalation policies carry `urgency: auto\|high\|low` in their definition (auto: critical -> high); pages store the resolved urgency |
| GET/POST | `/api/ack/:token` | None (rate-limited) | Tokenized no-login acknowledge: per-delivery `twk_` tokens (SHA-256-hashed on page_notifications), GET = read-only summary (scanner-safe), POST = idempotent ack recorded as `acknowledged_via='link'` attributed to the delivery's recipient; 404 after resolve. Frontend page: `/ack/[token]` |

**Monitors** (user-facing name for synthetic uptime checks; engine in `backend/app/synthetics`, frontend under `/monitors`, see the SYNTHETICS_* env block)

Checks (`synthetic_checks`, main DB) are http/tcp/browser probes with per-check interval, timeout, and a consecutive-failure threshold (flap damping). Runs flow through the `check_runs` queue (outbox-style guarded claims, terminal rows deleted, expired queued runs recorded as `missed`); results are telemetry (`check_results`, all three backends, pruned by the SQLite retention worker / 90d CH TTL). State transitions open/resolve `check_incidents` and feed the event-driven notification rule type `check_down` (recovery auto-resolves the page a rule-scoped dedup key opened via `oncall.AutoResolveByDedupKey`; recovery never dispatches to escalation channels). The recovery notice carries `Recovered` on `NotificationMessage`: `dispatch` leaves its `info` severity alone instead of applying the rule's, and the Slack adapter colours it green. The notify hook runs post-commit with no ambient tx (SQLite single-connection). Remember: `notification_rule.repository.go` (both copies) lists event rule types explicitly, and `check_down` is one of them.

**Incidents, incident updates & post-mortems.** `check_incidents` covers both auto and manual incidents: auto rows carry `check_id`+`project_id` (nullable now), hand-recorded ones carry only `status_page_id` (org admins record them from a status page for outages monitors missed; they never affect uptime numbers). All incidents can be given a public `title` and statuspage.io-style timeline updates (`incident_updates`: status investigating/identified/monitoring/update/resolved + message; auto open/resolve seeds empty-message investigating/resolved updates in the same tx in `synthetics/result.go`, and the probe `error_message` stays internal, never in the public payload). A `resolved` update closes a manual incident; auto lifecycle stays owned by `ProcessOutcome` (manual resolve/time-edit/delete of auto incidents answer 422). The public `/api/status/:slug` payload gained `pastIncidents` (90d union of the page's checks' auto incidents + its manual ones, titles defaulting to "<check name> is down", nested updates, cap 30, no ids/error messages); incident mutations invalidate every status-page cache slug of the org. Post-mortems (`post_mortems`, **project-scoped**: `project_id` NOT-NULL-in-practice with ON DELETE CASCADE, backfilled by migration from the linked incident's project else the org's oldest project) are internal markdown documents, never public: optional 1:1 incident link (`incident_id` unique where set, `ON DELETE SET NULL` so documents survive monitor/incident deletion; linkable incidents are the project's own autos plus the org's manual status-page incidents — cross-project autos answer 422), JSON-array `tags` (models.StringSlice), LIKE search over title/content/tags, and an append-only edit log (`post_mortem_events`: action created/updated + JSON `changes` field list, recorded only when something actually changed; served as the Activity panel). Routes are project-scoped (`/api/post-mortems*`, standard RequireProjectAccess/RequireWriteAccess chains, org resolved from the project for incident validation); incident mutations are org-admin since they are public content. Frontend: Post-Mortems tab on `/monitors` (all members; creation is a title-only dialog that creates the document immediately and opens the editor page), full-page WYSIWYG editor at `/monitors/post-mortems/[id]` (`@milkdown/crepe`, dynamically imported, app font stack overrides the theme's serif), paginated incident management page at `/monitors/status-pages/[pageId]/incidents` (which owns the Record Incident dialog).

| Method | Endpoint | Auth | Purpose |
|--------|----------|------|---------|
| GET/POST | `/api/synthetics/checks` | App / App+Write | List / create checks (422 validation incl. browser-mode + cloud gates) |
| GET/PUT/DELETE | `/api/synthetics/checks/:id` | App / +Write / +Write | Detail+incidents / update (type immutable; pausing drops queued runs) / delete (auto-resolves the check's open pages) |
| POST | `/api/synthetics/checks/:id/run` | App+Write | Run now (422 if a run is already queued; OnCommit wake) |
| POST | `/api/synthetics/overview` | App | Checks + per-range aggregates (uptime, avg latency) |
| POST | `/api/synthetics/checks/:id/results` | App | Paginated run history (telemetry read, no Transactional; optional `status` filter up/down/missed + fromDate/toDate) |
| POST | `/api/synthetics/checks/:id/series` | App | Bucketed uptime/latency series (telemetry read) |
| GET | `/api/synthetics/screenshot?key=` | App | Streams a failure screenshot; key prefix-checked against the project |
| GET | `/api/synthetics/output?key=` | App | Streams a failed browser run's stored Playwright output (.log keys only, same prefix check); "logs" link in the run history UI |
| GET | `/api/synthetics/open-count` | App | Down-check count for the sidebar badge |
| POST | `/api/runners/poll` | Runner (shared secret) | Instance-wide long-poll claim of browser runs (25s hold on the wake channel, NO Transactional, excluded from tracewaygin self-monitoring); self-registers the runner's liveness row |
| POST | `/api/runners/results/:runId` | Runner (shared secret) | Report an outcome (MaxBytesReader 6MB incl. screenshotBase64; idempotent: lost/missing claim = 200 no-op; claims keyed "runner:<name>") |
| GET/POST/PUT/DELETE | `/api/organizations/:organizationId/status-pages(/:id)` | Member / Admin | Status page CRUD (slug `[a-z0-9-]{3,60}` unique, checkIds validated against the org; branding fields `description`, `customDomain` unique hostname) |
| POST | `/api/organizations/:organizationId/status-pages/:id/logo` | Admin | Upload a PNG/JPEG logo (raw body, 1MB cap, sniffed content type; SVG rejected as scriptable) stored via storage.Store under `statuspages/<id>/logo` |
| GET | `/api/status/:slug/logo` | None (rate-limited) | Streams a public page's logo |
| GET | `/` on a custom domain | None | Status domains (CNAMEd vanity hosts, TLS at the operator's proxy). The SPA handler in `cmd/run.go` looks the Host up with `controllers.StatusPageForHost` (public pages only, never the `APP_BASE_URL` host) and, on a match, serves `index.html` with a `traceway-status-slug` meta tag plus the page's title and description. The frontend reads it in `$lib/status-domain.ts` and the root layout renders only `StatusPageView` at `/`, whatever the auth state. Every other page path on that host 302s to `/`, so the dashboard never loads on a customer's domain; static assets and `/api` still work. Saving a custom domain equal to `APP_BASE_URL`'s host or the admin's current host answers 422 |
| GET | `/api/status/:slug` | None (rate-limited) | Public status payload: per-check status + 90 daily uptime buckets + incidents + `pastIncidents` (date-grouped titles with timeline updates); cached ~30s per slug; private/unknown slugs are both 404; latency values and internal error messages stripped. Frontend page: `/status/[slug]` (in `isPublicPath`) |
| GET | `/api/organizations/:organizationId/incidents` | Member | Org incidents (90d, joined check/status-page names, updatesCount, postMortemId) |
| GET | `/api/organizations/:organizationId/incidents/:incidentId/updates` | Member | One incident + its timeline updates |
| GET | `/api/organizations/:organizationId/status-pages/:id/incidents` | Member | Paginated incident history of one status page (auto incidents of its checks + its manual ones, no time window; standard pagination envelope + `statusPage` meta). Frontend page: `/monitors/status-pages/[pageId]/incidents` |
| POST | `/api/organizations/:organizationId/status-pages/:id/incidents` | Admin | Record a manual incident on a status page (title required, optional first update message, backdatable, optional resolvedAt) |
| PUT/DELETE | `/api/organizations/:organizationId/incidents/:incidentId` | Admin | Edit title (any incident) / times (manual only, empty resolvedAt reopens; 422 for auto) / delete (manual only, 422 for auto) |
| POST/DELETE | `/api/organizations/:organizationId/incidents/:incidentId/updates(/:updateId)` | Admin | Post / delete a public timeline update (a `resolved` update also closes an unresolved manual incident) |
| GET/POST | `/api/post-mortems` | App / App+Write | Paginated list for the project (query params search/page/pageSize + repeatable `tag` params ANDed together) / create in the project (422 on duplicate or cross-project incident link) |
| GET/PUT/DELETE | `/api/post-mortems/:id` | App / +Write / +Write | Full document with author names (contentMd) / update (records a `post_mortem_events` diff row; no-op saves change nothing) / delete |
| GET | `/api/post-mortems/:id/activity` | App | Edit log for the Activity panel (who created/edited what, joined user names) |

**Logs**
| Method | Endpoint | Auth | Purpose |
|--------|----------|------|---------|
| POST | `/api/logs` | App | List/search logs (filters: severity, service, trace, body) |

**Source Maps**
| Method | Endpoint | Auth | Purpose |
|--------|----------|------|---------|
| POST | `/api/sourcemaps/upload` | SourceMap | Upload source map file |

### Data Ingestion Flow (`/api/report`)

1. **Gzip Middleware**: Decompresses request body (SDK sends gzipped data)
2. **Auth Middleware**: Validates `Authorization: Bearer <project_token>`
3. **Parse Frame**: JSON decode into `models.Frame` (transactions, exceptions, metrics)
4. **Batch Insert**: Repository methods insert batches into ClickHouse

```go
// backend/app/controllers/clientcontrollers/report.go
func (c *ReportController) Report(ctx *gin.Context) {
    var frame models.Frame
    ctx.BindJSON(&frame)

    // Insert each data type
    transactionRepo.BatchInsert(frame.Transactions)
    exceptionRepo.BatchInsert(frame.Exceptions)
    metricRepo.BatchInsert(frame.Metrics)
}
```

### Database Schema

#### Tables (ClickHouse)
| Table | Purpose | Partitioning |
|-------|---------|--------------|
| `transactions` | HTTP request metadata | Monthly (`toYYYYMM(timestamp)`) |
| `spans_v2`, `endpoints_v2`, `tasks_v2`, `ai_traces_v2`, `exceptions_v2` | Traces, see "Trace Storage (V2 tables)" below | Daily (`toYYYYMMDD(recorded_at)`) |
| `spans`, `endpoints`, `tasks`, `ai_traces`, `exception_stack_traces` | The tables V2 replaced. History only, read by the move-over | Monthly |
| `metric_records` | Time-series system metrics | Monthly |
| `archived_exceptions` | Archived/resolved exceptions | None |
| `log_records` | OTel logs with 3 attribute maps (resource/scope/log) | Daily (`toDate(timestamp)`) |

> Data retention/TTLs are documented separately under **Data Retention** below.

#### Tables (PostgreSQL)
| Table | Purpose |
|-------|---------|
| `users` | User accounts with email/password |
| `organizations` | Multi-tenant organizations |
| `organization_users` | Junction table linking users to organizations with roles |
| `projects` | Project config + tokens, linked to organizations |
| `project_user_roles` | Per-project role overrides (`user`/`readonly`) for org members |
| `invitations` | Team invitations with token, role, expiry |
| `source_maps` | Uploaded source map files (project, version, storage key) |
| `metric_registry` | Custom metric definitions (type, unit, description) |
| `dashboards` | Org-owned dashboards (name, JSONB definition with widgets, template provenance) |
| `project_dashboards` | Which projects show a dashboard, and tab order |
| `dashboard_templates` | Marketplace templates (key, category, definition), seeded by migrations |
| `starred_dashboard_widgets` | Homepage layout per project (dashboard id + widget id, position, col_span, size) |
| `synthetic_checks` | Synthetic check config + current state (status, consecutive_failures, next_run_at) |
| `check_runs` | Synthetic run queue (queued/claimed only; terminal rows deleted, telemetry is the record) |
| `check_incidents` | Incident spans: auto (nullable check_id+project_id) and manual (nullable status_page_id), plus public title (feeds status pages) |
| `incident_updates` | Public timeline updates on incidents (status + message, seeded on auto open/resolve) |
| `post_mortems` | Internal markdown post-mortems (project-scoped via `project_id` ON DELETE CASCADE, tags JSON array, optional unique incident link with ON DELETE SET NULL) |
| `post_mortem_events` | Append-only post-mortem edit log (action, JSON changes list, user, ON DELETE CASCADE) |
| `synthetic_runners` | Liveness registry for self-registered runner fleet (unique name, version, first/last_seen_at; no credentials, no org scoping) |
| `status_pages` | Public uptime pages (org-scoped slug + selected check ids + branding: description, logo_key, unique custom_domain) |
| `widget_groups` / `widget_group_widgets` / `starred_widgets` | Legacy pre-dashboards tables, retained read-only for rollback until a follow-up drop |

#### ClickHouse vs PostgreSQL Decision Guide
- **PostgreSQL**: Relational/config data needing ACID, frequent updates, JOINs, low volume (users, organizations, projects, invitations, widgets, source maps, metric registry)
- **ClickHouse**: High-volume append-only telemetry with time-series aggregations, batch inserts only (transactions, exceptions, metric points, spans, tasks, sessions)
- Rule of thumb: "Will this data be updated after creation?" → PostgreSQL. "Is this immutable, time-stamped, high-volume data queried with aggregations?" → ClickHouse.

#### SQLite Dual-Database Architecture (self-hosted mode)

In SQLite mode (`DB_TYPE=sqlite`), the backend uses **two separate SQLite databases** mirroring the PostgreSQL/ClickHouse split:

| Database | Variable | File | Purpose | Transactions |
|----------|----------|------|---------|-------------|
| **Main DB** | `db.DB` | `traceway.db` | PostgreSQL replacement — relational/config data | Yes (`middleware.Transactional`, `db.ExecuteTransaction`) |
| **Telemetry DB** | `db.TelemetryDB` | `traceway_telemetry.db` | ClickHouse replacement — append-only telemetry | No — direct inserts without transactions |

Both embedded builds (SQLite and DuckDB telemetry) are single-instance: the databases are local files, and the in-memory project cache only learns about another process's writes through the Postgres LISTEN/NOTIFY listener, which exists only under `transactional_pg`. Running more than one backend instance is supported only on the `transactional_pg telemetry_ch` build (docs: `/server/minimal#running-more-than-one-instance`).

**Main DB tables** (`db.DB` — transactional, uses lit with `*sql.Tx`):
- `users`, `organizations`, `organization_users`, `projects`, `invitations`
- `source_maps`, `metric_registry`, `dashboards`, `project_dashboards`, `dashboard_templates`, `starred_dashboard_widgets` (plus the legacy `widget_groups`/`widget_group_widgets`/`starred_widgets`)
- `notification_channels`, `notification_rules`
- `synthetic_checks`, `check_runs`, `check_incidents`, `incident_updates`, `post_mortems`, `post_mortem_events`, `synthetic_runners`, `status_pages`

**Telemetry DB tables** (`db.TelemetryDB` — non-transactional, uses lit with `db.TelemetryDB` directly):
- `endpoints_v2`, `tasks_v2`, `ai_traces_v2`, `exceptions_v2`, `spans_v2`, `metric_points`
- `session_recordings`, `archived_exceptions`, `slow_endpoints`, `fired_notifications`, `check_results`
- `endpoints`, `tasks`, `ai_traces`, `exception_stack_traces`, `spans`: the tables V2 replaced, kept as history for the move-over

**How to access each database in repository code:**

```go
// Main DB (PostgreSQL replacement) — use transactions via middleware or ExecuteTransaction
// Repositories receive *sql.Tx from middleware.GetTx(ctx) or db.ExecuteTransaction
user, err := lit.SelectSingleNamed[models.User](tx, "SELECT ... FROM users WHERE ...", lit.P{...})

// Telemetry DB (ClickHouse replacement) — use db.TelemetryDB directly, no transactions
results, err := lit.SelectNamed[endpointRow](db.TelemetryDB, "SELECT ... FROM endpoints WHERE ...", lit.P{...})

// Telemetry inserts — no transaction wrapping
for _, item := range items {
    row := modelToRow(item)
    lit.InsertExistingUuid(db.TelemetryDB, &row)
}
```

**Migrations** are split into two directories:
- `backend/app/migrations/sqlite/` — runs on `db.DB` (main)
- `backend/app/migrations/sqlite_telemetry/` — runs on `db.TelemetryDB` (telemetry)

**SQLite-specific type helpers** (`backend/app/repositories/telemetry/sqlitetypes/`):
- `SQLiteTime` — implements `sql.Scanner`/`driver.Valuer` for `time.Time` ↔ SQLite TEXT
- `SQLiteJSONMap` — implements `sql.Scanner`/`driver.Valuer` for `map[string]string` ↔ SQLite JSON TEXT
- Row types (e.g., `endpointRow`, `taskRow`) wrap domain models with these types for lit compatibility

#### DuckDB Telemetry Backend (self-hosted, opt-in)

Built with `-tags telemetry_duckdb` (`CGO_ENABLED=1` required), this is an alternative telemetry store for the same `DB_TYPE=sqlite` deployment: the **main DB stays SQLite** (`db.DB`, relational/config), while the **telemetry DB becomes DuckDB** (`db.TelemetryDB`, columnar). It exists because DuckDB's columnar engine is dramatically faster on the analytics/aggregation reads the dashboard issues — at 10M rows it clears read-probe thresholds that SQLite times out on. Backends are selected on two build-tag axes: `telemetry_ch` / `telemetry_duckdb` / *(none = SQLite telemetry)* for the telemetry store and `transactional_pg` / *(none = SQLite main)* for the relational store. Only three combinations are supported — *(no tags)* dual SQLite, `telemetry_duckdb`, and `transactional_pg telemetry_ch` — enforced by compile-time guard files in `backend/app/db/` (stale `pgch`/`duckdb`/`oltp_*` tags also fail with a rename message). Repositories are organized on the same two axes: telemetry repositories live in per-backend packages `backend/app/repositories/telemetry/{clickhouse,sqlite,duckdb}/` and transactional (relational) repositories in `backend/app/repositories/transactional/{pg,sqlite}/`, each re-exported as singletons through tag-guarded facade files at the axis package root (`telemetry/telemetry_ch.go` etc., `transactional/transactional_pg.go` etc.). Consumers import the facade packages — `telemetry.SpanRepository`, `transactional.UserRepository` — never a backend package directly. Helpers shared by all telemetry backends are in `telemetry/shared/`, the SQLite scan/value types shared by the sqlite+duckdb backends are in `telemetry/sqlitetypes/`, and helpers/types shared by the transactional backends (auth-token hashing/time formats, facade-crossing structs) are in `transactional/shared/`. The `transactional/pg` and `transactional/sqlite` implementations are intentionally kept dialect-neutral (lit `:name` queries rendered per `db.Driver`), enforced byte-for-byte by `transactional/parity_test.go`. Running Postgres requires the `transactional_pg` build: the default build's migration runner applies SQLite-dialect migrations unconditionally, so `DB_TYPE=postgres` without the tag is not a supported combination.

- **Driver:** `github.com/duckdb/duckdb-go/v2` (the official driver; marcboeker/go-duckdb is deprecated). Bundles prebuilt static libs for glibc only — **not musl/Alpine**, so the image uses Debian (`Dockerfile.duckdb`).
- **Opened in** `backend/app/db/db_telemetry_duckdb.go`: telemetry path is the SQLite path with `.db` swapped for `_telemetry.duckdb`. By default DuckDB auto-tunes to the host; `DUCKDB_MEMORY_LIMIT`/`DUCKDB_THREADS`/`DUCKDB_CHECKPOINT_THRESHOLD` (passed through as DSN config options) let operators cap memory/threads so a memory-capped container doesn't read the host's RAM and OOM-kill the backend, and raise the WAL checkpoint threshold (default 16MB) so sustained Appender ingest isn't stalled by frequent checkpoints. `preserve_insertion_order=false` is always set — telemetry reads all have explicit ORDER BY, and dropping the guarantee lets DuckDB parallelize bulk loads and large scans with less memory. The read pool is bounded (`SetMaxOpenConns(duckDBMaxReadConns)`) since each DuckDB connection can use all threads + its own query memory; Appender writes use their own `DuckDBConnector.Connect()` connections and bypass that cap. Exposes `db.DuckDBConnector` (needed for the Appender).
- **Writes use the Appender API**, not `INSERT` (`duckdb.NewAppenderFromConn(conn, "", table)` → `AppendRow(...)` → `Close()` flushes). Upserts still go through `ExecContext` with `ON CONFLICT`. The Appender rejects typed `*string` for nullable VARCHAR — use `nullableString()` in `backend/app/repositories/telemetry/duckdb/helpers.go` (returns untyped `nil` or the dereferenced value).
- **Write-path observability:** a row the Appender rejects is dropped rather than failing the whole frame (the SQLite backend 500s instead), so a poison row cannot wedge the SDK's retry loop. Every drop increments a per-table counter (`db.RecordTelemetryRowDropped`) and fires a rate-limited (1/min per table) `traceway.CaptureException`; Appender flush/connect failures still propagate to the request (500, SDK retries) and increment an insert-failure counter. `GET /api/health/deep` (operator endpoint: requires the HEALTH_DEEP_TOKEN bearer secret, unset disables it; all telemetry backends) exposes `telemetryBackend`, `droppedRows` per table, `droppedRowsTotal`, `insertFailures`, `ingestRejected` (requests turned away by the ingest admission gate), and on DuckDB an `engine` object (db/WAL file bytes, `duckdb_memory()` usage, read-pool in-use/wait stats) alongside its existing ClickHouse fields; it 503s only when the configured telemetry backend is ClickHouse and CH is unreachable (the embedded backends answer 200 with `chReachable:false`). The benchmark loadgen polls it before/after every ramp step and fails any step whose drop delta is nonzero; read-probe fills record cumulative `droppedRows` per fill level. When `MONITORING_TRACEWAY_URL` is set, `monitoring.StartTelemetryDBReporter` also emits `traceway.duckdb.*` metrics every 10s: `rows_dropped.delta`, `insert_failures.delta`, `db_size_mb`, `wal_size_mb`, `memory_used_mb`, `read_pool.in_use`, `read_pool.wait_count.delta`, `read_pool.wait_ms.delta`. The hourly retention worker issues a `CHECKPOINT` after its deletes so retention actually reclaims disk (DuckDB otherwise defers reclamation to the WAL checkpoint threshold).
- **`lit` placeholders:** `db.Driver` stays `lit.SQLite`, which emits `?` — DuckDB accepts these, so no separate driver was needed for reads.
- **Migrations:** `backend/app/migrations/duckdb_telemetry/` (mirrors `sqlite_telemetry/` table-for-table; integer columns are `BIGINT`, JSON is `VARCHAR`, no secondary indexes since it's columnar, with one exception: `0005_create_v2_tables` adds `idx_spans_v2_trace_key` on `spans_v2(trace_key)` for the span graph lookup. While that index exists `ALTER TABLE spans_v2 ADD COLUMN` still works, but `DROP COLUMN` and `ALTER COLUMN ... TYPE` fail with a dependency error, so a migration that needs either must drop the index first).
- **Dialect gotchas vs SQLite** (the read queries differ): native `quantile_cont(col, p)` for P50/P95/P99 instead of fetch-and-sort; `strftime('%s',col)`→`epoch(col)`; time bucketing via `time_bucket(to_seconds(N), col, TIMESTAMP '1970-01-01')` — the explicit epoch origin is required because DuckDB anchors sub-day buckets at 2000-01-03 by default, which would misalign chart buckets against the SQLite backend's epoch-floored buckets for any interval that doesn't evenly divide a day; `json_extract`→`json_extract_string`; `json_each`→`LATERAL unnest(json_keys(x))`; strict GROUP BY needs `ANY_VALUE`/`arg_max`; `SUM` returns HUGEINT (CAST to BIGINT); `CAST(.. AS REAL)`→`CAST(.. AS DOUBLE)`.
- **Tests:** `backend/app/repositories/telemetry/testhelper_duckdb_test.go` (tagged `telemetry_duckdb`) provides `setupTestDB` so the entire existing telemetry test suite runs against an in-memory DuckDB.

#### Trace Storage (V2 tables)

A trace is a trace and a span is a span. Five tables hold traces on every backend: `spans_v2`, `endpoints_v2`, `tasks_v2`, `ai_traces_v2`, `exceptions_v2` (migrations `ch/0085`-`0089`, one `CREATE TABLE` each; `sqlite_telemetry/0025_create_v2_tables`; `duckdb_telemetry/0005_create_v2_tables`). They sit next to the tables they replaced (`spans`, `endpoints`, `tasks`, `ai_traces`, `exception_stack_traces`), which nothing reads or writes any more except the move-over and retention. `migrations/v2_tables_test.go` holds the V2 migrations to `CREATE TABLE`/`CREATE INDEX` on `_v2` tables only.

- **Ids.** Every row carries the ids its span arrived with, as lowercase hex: `trace_id` (32 characters), `span_id` (16 for OTel, 32 for a native run or span), `parent_span_id` (`''` for a root). Custom linked-trace identity is retired. Startup cleanup migrations remove `linked_trace_id` and its indexes from all four V2 entity/exception tables; OTLP span links remain intact. Entities and exceptions keep a UUID `id` for URLs and notifications. For OTel rows it is `otelOccurrenceID` = `uuid.NewSHA1(projectId, traceId + spanId)`, so a retried export produces the same id. Nothing about identity lives in the attributes JSON, and ingest adds no `traceway.otel.*` keys.
- **Identity schema cleanup.** Forward migrations `ch/0091`–`0099`, `sqlite_telemetry/0026`, `duckdb_telemetry/0006`–`0007` replace `sessions.distributed_trace_id` with canonical non-null string `trace_id` (empty = no association), remove V2 `linked_trace_id` columns/indexes, and drop the unused `profiles.distributed_trace_id` column/index. Profiles keep their existing `trace_id` / `span_id`. ClickHouse synchronously materializes session IDs before dropping the source column. SQLite/DuckDB commit each migration and its version atomically; DuckDB adds the session non-null constraint in the next transaction. Stop old backends before the first upgraded start and restore a pre-upgrade database for rollback: old session/profile SQL is incompatible. The five old trace tables remain for the move-over.
- **Ingest** (`otelcontrollers/trace_converter.go`) validates and indexes spans once per resource, then returns canonical spans, entity projections, exceptions and the invalid-ID count together. Each span is promoted when its own kind and attributes make it an entry point, and each exception event becomes an `exceptions_v2` row with the span's trace id and span id. Canonical span attributes are rendered directly from the OTLP payload at storage time. Trace and log JSON IDs are normalized at the decoder boundary (`otelcontrollers/otlp_json.go`); converters consume decoded bytes without a UUID/base64 repair roundtrip. No owner resolution, no caches, no reads. `trace_type` on an exception is stored only when that span itself was promoted (native rows say it themselves). Every entity is recorded at its span's **start**, OTel tasks included. Spans are inserted **before** the rows promoted from them, and a transient storage error answers 503 + `Retry-After` (`abortIngestStorage`, `telemetry.IsTransientStorageError`) so OTLP exporters retry. Retries are append-only; the span reader deduplicates per span.
- **Entry points.** An HTTP SERVER span becomes an endpoint unless a local span above it already is one (`isEntryPoint`), which is what keeps a SERVER span nested under another (Next.js under the HTTP instrumentation) from counting as a second request. This is the one check that looks beyond the span, and only inside its `ResourceSpans` block. The walk up the parent chain stops at a remote parent per the OTLP span flags (`0x100` has-is-remote, `0x200` is-remote) and at a CLIENT or PRODUCER span. A missing parent has an unknown kind even when flags mark it local, so it cannot suppress an HTTP entry point. Separately exported nested HTTP spans can create extra projections.
- **Native `/api/report` rows** go to the same tables (`clientmodels.ClientTrace.RootSpan`, `ClientSpan.ToSpan`): a run is stored as the root span of a trace, `span_id` = the run's id, `trace_id` = the run's id. Its spans hang under it, and a native exception sits on the run's span. The report wire format is unchanged: an exception's `traceId` names its owning run, `distributedTraceId` is ignored on runs, exceptions and sessions until the JS SDK supports canonical trace propagation, and native run/span IDs remain UUIDs.
- **`spans_v2`** keeps the lossless payload of accepted OTLP spans (`resource_pb`/`scope_pb`/`span_pb` on ClickHouse, `otlp` on SQLite and DuckDB; `shared.AssembleOtelPayload` joins them back) next to the readable columns, built once by `OtelSpanRow.NestedValues`: `span_attributes` is a string map (a `Map(LowCardinality(String), String)` on ClickHouse, a JSON object elsewhere) filled by `shared.StringAttributes` the way the OpenTelemetry Collector's ClickHouse exporter does it, and `resource`, `scope`, `events`, `links` are JSON text. A span that did not arrive as OTLP gets a payload synthesized from its available fields (`NewOtelSpanRow`), including string attributes and service/scope metadata. Native UUID span IDs remain in the API; their exported span/parent IDs use a stable eight-byte hash mapping with original IDs carried in `traceway.native.*` attributes. Native synthesis cannot recover metadata never sent by the SDK. On ClickHouse `recorded_at` is `DateTime64(6)` on endpoint/task/exception tables and `DateTime64(3)` on AI traces (the waterfall's origin is the entity's `recordedAt`) and a plain partition time on `spans_v2`.
- **Reads.** A waterfall is `spans_v2` by project and trace, the subtree under the entity's `span_id`: `telemetry.SpanRepository.FindGraph(ctx, shared.NewSpanLookup(projectId, traceId, spanId, recordedAt))`, answered by `shared.FindOtelGraphs` (topology first, attributes second). A limit never fails the page: the result carries `spanGraphStatus` (`complete`/`partial`/`unavailable` plus reasons) and spans may have `attributesOmitted`. The subtree includes spans that are entities of their own, such as a CONSUMER span started inside the request.
- **Distributed trace card** (`controllers/distributed_trace.controller.go`, `POST /api/distributed-traces/:traceId`, hex or dashed). `FindByTraceIds` on the four entity and exception repositories matches only `trace_id IN (...)` across projects the user can open. Every returned node belongs to the requested trace; W3C propagation makes browser and backend share that identity. Nodes are nested by one read of the trace's topology, without attributes (`SpanRepository.FindTraceParents`): each node walks up until it meets another node's span (`parentEntitySpanId`). An exception is attached to the nearest node above its span, else it is a node of its own.
- **Exceptions.** A detail page reads `exceptions_v2` by trace and keeps those on its own span or a span of its subtree (`findOccurrenceExceptions`). `telemetry.FindExceptionOwner` does the walk the other way, from the exception's span up to the nearest endpoint, task or AI trace of the same project. The exception responses return it as `relatedEntity`, and notification text uses it (`resolveExceptionOwner`). A failed task, for the task failure rule, is a task with an exception on its own span.
- **Every span read is bound in time.** A lookup reads `recorded_at` within 24 hours either side of the entity it starts from (`shared.OtelLookupWindow`, `shared.TraceWindowBounds`; entity lookups by trace id use 48 hours), which on ClickHouse prunes the read to a few daily partitions. A lookup that names no time is anchored on now. The cost is deliberate: a span recorded more than 24 hours away from its entity is not shown on that entity's page. Tests that store spans under a fixed timestamp must pass that timestamp as the lookup time, never the wall clock.
- **Orphans.** A span whose parent never arrived has no place in a subtree. The entity promoted from the trace's root span (the one with no parent) shows such spans of its own project and trace, so they are visible exactly once, and the waterfall marks them "Parent span is unavailable". Once the parent arrives they nest normally.
- **Span explorer** (`controllers/span_explorer.controller.go`, frontend `/spans`). `POST /api/spans/search` (App auth + project access) pages the spans of a project in a required time range of at most 31 days. Body: `fromDate`, `toDate`, `orderBy` (`start_time desc|asc`, `duration desc|asc`), `serviceName`, `name` (case-insensitive contains), `traceId` (32 hex), `kind` (0-5), `status` (0-2), `minDurationMs`, `maxDurationMs`, `attributeFilters` (at most 10 exact `{key, value}` matches on `span_attributes`, value required since a ClickHouse Map answers `''` for a missing key), `pagination`. The response adds `services`, the busiest 100 service names in the range. Validation problems and a search that hits its 15s read limit both answer 422, which the page shows as a warning callout, not an error. The query builder is `shared.OtelSearchQueries` with one small dialect struct per backend. `GET /api/spans/traces/:traceId?at=<RFC 3339>` returns the whole trace (`shared.FindOtelTrace`) and 404s when nothing is recorded within 24 hours of `at`. **It reads every project of the current project's organization**, not only the current one (`organizationProjects` in the controller, one lookup per project, one query): services of one trace usually report to different projects, and organization membership is what grants read access to each. Other organizations of the same user stay out. The response names the projects that hold spans (`projects`), each span keeps its own `projectId`, the page links spans across projects (`buildSpanTree(..., acrossProjects)`), and the popover loads attributes with the span's own project id. Search stays per project. Unlike the detail pages it sends **no attributes**: each span carries only `dbStatement`, a 240 character preview of `db.query.text` (or `db.statement`) that the row label needs, and the page fetches one span's attributes from `GET /api/spans/traces/:traceId/spans/:spanId/attributes?at=` when its popover opens. One query serves both cases of the row cap (`FindTraceOutline` on each backend, ordered by `shared.OtelImportanceOrder`): under `MaxOtelGraphRows` every span comes back, over it the `LIMIT` keeps errors first, then entry points (no parent, SERVER or CONSUMER), then the slowest, and the status is `partial` with reasons `row_limit` and `most_important`. Ordering by duration also tends to keep the path above a span, since a parent usually outlasts its children. Responses are not compressed by the backend: 20,000 spans are about 13 MB of JSON, about 1.1 MB once a proxy gzips them. This is the one place browser and mobile projects see their spans, since they have no endpoint or task pages.
- `GET /api/otel/spans/:traceId/:spanId?at=<RFC 3339>` (App auth + project access) returns one stored span (16 or 32 hex span id) as an OTLP `ExportTraceServiceRequest` protobuf. `at` is required (400 without it) for the same 24 hour rule.
- **Frontend: large traces.** A trace over 500 spans opens with every node that has more than 50 children folded (`initiallyCollapsed` in `utils/span-tree.ts`); smaller traces open fully expanded. A folded row shows a pill with the number of hidden spans and an error marker when one of them failed, and the header has Expand all and Collapse all. The waterfall keeps what was clicked as the difference from that default, so there is no effect to reset. Pages must hold span payloads in `$state.raw`: a deep `$state` proxy over 20,000 spans took four minutes to read in the browser, against 0.4 seconds raw. The payload is only ever replaced, never edited.
- **Frontend.** `span-waterfall.svelte` names the service on each row and colours bars by service once a trace crosses services (one colour per row otherwise), marks error spans and missing parents, renders rows 500 at a time so a 20,000 span trace cannot stall the page, caps the name column to half the visible width on narrow screens, and links to the whole trace in the explorer. The span popover shows service, kind, status and scope (`utils/span-fields.ts`). Span payloads are held in `$state.raw`. The tree is keyed `traceId:spanId` (`utils/span-tree.ts`), and the issue pages build their related entity from the backend's `relatedEntity`.
- `GET /api/health/deep` also reports `spanPartitionTimeFallbacks`: spans whose source start was missing or unrepresentable, so their partition time is the ingest time (`shared.OtelStorageTimes`).
- **Move-over** (`repositories/telemetry/move_over.go`, started by `retention.startMoveOver` when `V2_MOVE_OVER=true`). Copies the old tables into the V2 tables inside the backend (an embedded DuckDB file cannot be opened by a second process), UTC days scheduled newest first with configurable concurrency (`V2_MOVE_OVER_WORKERS`, default 1, cap 16), each day keeping table order, recorded in the transactional DB `v2_move_over_days` (migrations `sqlite/0077`, `pg/0143`) so it stops and resumes. The scheduler runs at startup and one hour after each pass ends, including failures; passes do not overlap. `started` is committed before telemetry writes, `done` after all batches succeed, with no main-DB transaction held during the copy. `moved_rows` counts scanned source rows in the current attempt, including already-copied rows skipped during recovery; it resets on resume. Progress and logs update after successful batches roughly every 10s (a slow query or insert can delay this). A failed worker cancels the pass, and all workers are joined before return; separate backend instances are still not coordinated. Finish upgrading old writers first: completed table/date pairs are not revisited. A resumed day skips rows already written. It reads through `LegacyRepository` (one per backend, the old ids parked in the V2 model fields, see `shared/legacy.go`) and writes through the V2 repositories, so the id mapping lives in one place: the trace id comes from the `traceway.otel.trace_id` attribute the old ingest kept, else the old `distributed_trace_id`, else the row id; an old zero padded span UUID becomes its last 16 hex characters; spans and exceptions find their old owner by id, nearest in time, and take its trace; each entity also gets its own span row. Slices are halved when a read comes back full, so no day is sorted, and only a single overfull second is paged by stable full-row order and offset. Operator notes: `docs/pages/server/upgrade-v2-tables.mdx`.

ClickHouse `ch/0090_add_span_version` adds a materialized SHA-256 fingerprint over length-prefixed protobuf components. All span readers rank by duration then this fingerprint, so topology never reads payload columns for new parts. Old V2 parts calculate it on read until merged or explicitly materialized; no rewrite is scheduled by migration. The column costs 32 uncompressed bytes per physical row.

#### V2 review corrections

Legacy history paging uses stable full-row ordering and offsets only within overfull one-second slices; source tables must be immutable after cutover. Recovery checks project, occurrence ID, trace/span IDs and destination-precision timestamps. Rejected canonical rows fail the native/migration facade, so a day cannot be marked complete after silent row rejection. An old preview's incorrectly completed day can be set back to `started` for checked recovery; see `docs/pages/server/upgrade-v2-tables.mdx`.

Span search, topology, attributes and export choose one retry version (longest duration, then the stored SHA-256 `span_version` on ClickHouse or stored protobuf bytes on SQLite/DuckDB), with UTF-8 byte budgets applied in SQL. Native report spans are written before projections. OTLP storage returns rejected span identities so their endpoint/task/AI/exception projections and conversation uploads are omitted while reporting partial success. HTTP entry-point classification suppresses only an observed local HTTP ancestor, never an absent parent's assumed kind. Fallback AI conversation IDs retain the historical dashed UUID spelling. Log search accepts `traceId` and optional `spanId`, scoped to the selected project and time window. Retired `distributedTraceId` and `excludeTraceId` selectors return 400. The trace log panel has one paginated list with stale-response protection.

#### Data Retention

Retention is handled in several different ways depending on the deployment.

**0. Main-DB auth prune — `retention.Start` worker** (`backend/app/retention/auth_tokens.go` + `oauth_sessions.go`, sharing `startDBPruneWorker` in `prune_worker.go`). Runs in **all modes** (Postgres and SQLite) against `db.DB`: once at startup, then every 24h, deleting expired/consumed auth rows. `auth_tokens` prunes expired `device_authorizations` (also pruned opportunistically on every `/api/auth/device/authorize` call, so the daily worker is a backstop there), `refresh_tokens` that are expired, revoked, or used more than 30 days ago (used rows are kept a month for replay detection, then dropped to bound per-user growth), plus revoked-or-expired `personal_access_tokens` (PAT expiry was otherwise only enforced lazily at read time). `oauth_sessions` prunes expired SSO login sessions. No env var — always on. (Refresh-token families and PATs also have explicit revoke paths: `POST /api/auth/logout` and the account PAT UI.)

**1. ClickHouse — `TTL` clauses on the table itself.** Only a few tables have a TTL; everything else is kept indefinitely (operators can drop monthly partitions manually if needed).

| Table | Retention | Source migration |
|-------|-----------|------------------|
| `metric_points` (raw) | **7 days** | `0034_add_ttl_metric_points.up.sql` |
| `metric_points_1m` (1-min rollup) | **30 days** | `0035_add_ttl_metric_points_1m.up.sql` |
| `metric_points_1h` (1-hour rollup) | **1 year** | `0036_add_ttl_metric_points_1h.up.sql` |
| `log_records` | **90 days** | `0075_increase_ttl_log_records.up.sql` (raised from 30d set in `0045_create_log_records.up.sql`) |
| `profiling_samples` (the bulk) | **30 days** | `0068_add_ttl_profiling_samples.up.sql` |
| `profiles` (slim metadata) | **30 days** | `0069_add_ttl_profiles.up.sql` |
| `profiling_stacks` (dedup table) | **30 days** | `0070_add_ttl_profiling_stacks.up.sql` |
| `check_results` (synthetic check probes) | **90 days** | `0083_add_ttl_check_results.up.sql` |
| All other CH tables (the five V2 trace tables and the five they replaced, `transactions`, `sessions`, `session_recordings`, `fired_notifications`, `archived_exceptions`, `slow_endpoints`, etc.) | **No TTL, retained indefinitely** | none |

The three profiling tables share a 30-day TTL keyed on each table's time column (`start_time` / `recorded_at` / `last_seen`). `profiling_stacks` is a `ReplacingMergeTree(last_seen)` dedup table, so `last_seen` is bumped on every re-ingest that references a stack — a stack only ages out once it has gone unreferenced for the full window, which is exactly when its samples have also expired, so no sample is ever left pointing at a dropped stack.

**2. SQLite — `retention.Start` worker** (`backend/app/retention/sqlite.go`). In SQLite mode, neither `db.DB` nor `db.TelemetryDB` has any built-in expiry, so a background worker fires once at startup and then every hour and runs a `DELETE FROM <table> WHERE <time_column> < cutoff` against each telemetry table.

| Variable | Default | Notes |
|----------|---------|-------|
| `SQLITE_RETENTION_DAYS` | `30` | TTL in days. Set to `0` to disable the worker entirely. Has no effect outside SQLite mode. |
| `DUCKDB_RETENTION_DAYS` | `30` | Same worker on the DuckDB telemetry backend. When set it takes precedence there; when unset the worker falls back to `SQLITE_RETENTION_DAYS` (selection in `retention.telemetryRetentionConfig`). |

Tables it prunes (and the column used):

| Database | Table | Time column |
|----------|-------|-------------|
| Telemetry (`db.TelemetryDB`) | `endpoints_v2`, `tasks_v2`, `exceptions_v2`, `spans_v2`, `ai_traces_v2`, the five tables they replaced (`endpoints`, `tasks`, `exception_stack_traces`, `spans`, `ai_traces`), `metric_points`, `session_recordings` | `recorded_at` |
| Telemetry | `log_records` | `timestamp` |
| Telemetry | `sessions` | `started_at` |
| Telemetry | `fired_notifications` | `fired_at` |
| Telemetry | `profiling_samples` | `start_time` |
| Telemetry | `profiles` | `recorded_at` |
| Telemetry | `profiling_stacks` | `last_seen` |
| Telemetry | `check_results` | `recorded_at` |

`archived_exceptions` (per-hash flags) and `slow_endpoints` (per-endpoint config) are intentionally skipped — they are not time-series data. `profiling_stacks` *is* pruned (unlike those two) because it holds no user intent — it is a regenerable dedup table whose `last_seen` tracks the most recent referencing sample, so deleting expired stacks is safe.

**2c. Synthetics failure screenshots — `retention.Start` worker** (`backend/app/retention/synthetics.go`). Browser-check failure screenshots written to local disk under `<STORAGE_PATH>/synthetics/` are aged out hourly by mtime, mirroring the session-recording split: the `check_results` rows referencing them are pruned separately (item 2 / CH TTL) and deliberately not coupled. `SYNTHETICS_SCREENSHOT_RETENTION_DAYS` (default 30, `0` disables); no-op unless `STORAGE_TYPE=local`.

**2b. Log row cap — `retention.Start` worker** (`backend/app/retention/log_cap.go`). SQLite mode only (SQLite or DuckDB telemetry), off by default. When `LOG_RECORDS_MAX_ROWS` is set to a positive N, a worker runs once at startup and then every minute and deletes `log_records` rows strictly older than the Nth-newest row's `timestamp` (single portable DELETE with an `ORDER BY timestamp DESC LIMIT 1 OFFSET N-1` subquery; NULL boundary = no-op under the cap, boundary ties are kept). **The cap is best-effort, not a hard limit**: nothing throttles ingest, so between passes the table can exceed N, and sustained ingest above N rows/minute keeps it above the cap permanently — document it to users as a cleanup task with a 1-minute window, sized with headroom for peak log volume. Bounds log disk usage independently of `SQLITE_RETENTION_DAYS`; disk reclamation still happens via the hourly retention pass / DuckDB WAL checkpointing.

**3. On-disk session recordings — `retention.Start` worker** (`backend/app/retention/recordings.go`). Session recordings written to local disk (`STORAGE_TYPE=local`) accumulate under `<STORAGE_PATH>/recordings/`. A second worker walks that directory once at startup and then every hour and removes files whose `mtime` is older than the TTL, then prunes any directories left empty. The worker is a no-op when `STORAGE_TYPE=s3`.

| Variable | Default | Notes |
|----------|---------|-------|
| `SESSION_RECORDING_RETENTION_DAYS` | `30` | TTL in days. Set to `0` to disable the worker. Only runs when `STORAGE_TYPE=local` (default). |

The DB rows in `session_recordings` are pruned by the SQLite retention worker (above) or by ClickHouse TTL — they are intentionally not coupled to the disk cleanup. Controllers that read recordings already log a non-fatal `traceway.CaptureException` when a referenced file is missing.

**4. On-disk raw profile archives — `retention.Start` worker** (`backend/app/retention/profiles.go`). When `PROFILE_ARCHIVE_RAW` is enabled, the **native pprof ingest path** (`/profiles/ingest`) writes each upload's original pprof bytes to `<STORAGE_PATH>/profiles/<projectId>/<yyyymmdd>/<id>.pprof` (recorded on `Profile.StorageKey`) as a lossless archive for download / re-ingest / PGO — off the read path. The OTLP endpoint does not archive (its rows carry an empty `StorageKey`). A worker walks that directory once at startup and then every hour and removes files whose `mtime` is older than the TTL, then prunes empty directories. It reuses the same generic age-based cleanup as the recordings worker (`runDirAgeCleanup` / `isSafeStorageSubdir`) and is a no-op unless `PROFILE_ARCHIVE_RAW` is on **and** `STORAGE_TYPE=local`.

| Variable | Default | Notes |
|----------|---------|-------|
| `PROFILE_ARCHIVE_RAW` | `false` | Master switch for the raw archive. When off, no blob is written and the disk worker does nothing. |
| `PROFILE_RETENTION_DAYS` | `30` | TTL in days for the on-disk archive. Set to `0` to disable the disk worker. Only runs when `PROFILE_ARCHIVE_RAW` is on and `STORAGE_TYPE=local`. |

The `profiles` DB rows (and their `storage_key`) are pruned by the SQLite retention worker / ClickHouse TTL above — not coupled to this disk cleanup, mirroring the session-recording split.

**5. Main-DB outbox prune — `retention.Start` worker** (`backend/app/retention/outbox.go`, same `startDBPruneWorker` scaffolding as item 0). Runs in **all modes** against `db.DB`: once at startup, then every 24h, deleting terminal `notification_outbox` rows — `sent`/`cancelled` older than 7 days, `failed` older than 30 days. `pending` and `sending` rows are never pruned, so nothing undelivered is dropped. No env var — always on. The durable record of what was notified lives in `fired_notifications` and `page_notifications`, which this does not touch. Note that `pages` and `page_notifications` themselves are currently retained indefinitely.

#### Session Recording Uploader

Session recording segments arriving on `/api/report` are not uploaded inline. The handler enqueues each segment onto a bounded worker pool (`backend/app/recordings/uploader.go`, started from `cmd/run.go` next to `retention.Start`). Workers drain the queue and write the body via `storage.Store.Write` (S3 or local disk); successful writes are handed to a single batcher goroutine that calls `SessionRecordingRepository.InsertAsync` once per ~1000 rows or every 2 s, whichever comes first — single-row inserts are an anti-pattern for ClickHouse. Enqueue is non-blocking: when the queue is full the segment is dropped (newest-first) so a burst of `/api/report` traffic cannot spawn unbounded goroutines or saturate S3.

| Variable | Default | Notes |
|----------|---------|-------|
| `SESSION_RECORDING_UPLOAD_WORKERS` | `32` | Concurrent uploaders. Set to `0` to drop every segment (uploads disabled). |
| `SESSION_RECORDING_UPLOAD_QUEUE_SIZE` | `2048` | Max queued segments. Overflow is dropped. |

Observability (emitted every 10s via `traceway.CaptureMetric`):

| Metric | Type | Meaning |
|--------|------|---------|
| `traceway.recordings.queue_depth` | gauge | Segments currently waiting in the channel. |
| `traceway.recordings.in_flight` | gauge | Workers mid-upload. |
| `traceway.recordings.uploaded` | counter | Successful S3/local + DB writes since startup. |
| `traceway.recordings.dropped` | counter | Segments dropped on overflow since startup. |
| `traceway.recordings.failed` | counter | Upload or DB-insert errors since startup. |

Sustained drops also fire a rate-limited (1/min) `traceway.CaptureException` so overload is visible in the issues feed without flooding it.

**Session end is derived on read.** The browser SDK's closing payload is unreliable (a keepalive body over 64 KB is rejected on unload, and frozen mobile tabs never fire `pagehide`), so the stored `sessions.ended_at` is only a hint. Each segment row carries its client-side `ended_at` (`ch/0100`, `sqlite_telemetry/0027`, `duckdb_telemetry/0008`), and every session read joins the latest segment activity and applies `shared.ResolveSessionEnd`: an explicit end is kept only when it lands within `SessionIdleTimeout` (15 min, the SDK's inactivity timeout) after the last activity, otherwise the session ends at its last activity; it is still live while a segment arrived (server time) within the timeout. Every session is capped at `shared.SessionMaxSpan` (65 min: the SDK's 60 min cap plus slack) because `@tracewayapp/frontend` <= 1.2.0 kept uploading segments under an ended session id for the tab's whole lifetime (18h "sessions"); the end and duration are clamped to it (sessions without segments too), and when segments run past it the session ends at the last segment inside it (`span_activity`, computed per backend against the session start; the usual 15 min rule then honours the SDK's own close), not at the cap, because an idle tab uploads nothing until it wakes hours later (sleep/wake, or 1.2.0's zombie uploads) and resumes under the same id; `Session.CutoffAt` carries the resulting cut-off, a session whose segments run past it is never live (compared on the client clock, so skew between browser and server cannot end a live session), and the session detail and `GET /api/sessions/:id/recording` drop exceptions and segments past `CutoffAt`, anchored on the stored session start rather than the `startedAt` query param (`shared.SegmentsWithinSession`). `HasRecording` is set when any segment exists, and the frontend labels an old session with neither as "No recording" (`utils/session-duration.ts`). Sorting the list by duration uses a SQL copy of the same rule in each backend (`sessionDurationSortKey`); `TestSessionRepository_EndFromRecordingActivity` holds the SQL order to the Go rule. `GET /api/sessions/:id/recording?startedAt=` bounds the segment lookup to the session's window, and segments are read from storage 16 at a time.

#### Users, Organizations & Projects

- **Users to organizations is many-to-many** via `organization_users`, one role per membership. A user joins additional organizations through invitations (`POST /api/invitations/:token/accept-existing` for existing accounts) and, in cloud mode, registration. Login/register/`LoginBundle` return every membership as `Organizations[]` with roles; the frontend keeps them in `authState.organizations` and groups the navbar project selector by organization when there is more than one.
- **Projects belong to exactly one organization** (`projects.organization_id`). Org membership grants read access to all of the org's projects.
- **Per-project role overrides** live in `project_user_roles(project_id, user_id, role)` with role `user` or `readonly`. Overrides only apply to members whose org role is `user` or `readonly`; owners and admins always have full access to every org project. Override rows are kept when a member's org role changes (they are inert for owner/admin), and are deleted when the member is removed from the org or the project is deleted. Managed from Settings > Team Members (expand a member row) via `GET/PUT /api/organizations/:orgId/members/:userId/project-roles(/:projectId)`; `PUT` with `role: "default"` clears the override and is accepted for any member regardless of org role (so stale rows on members promoted to admin can still be cleaned up); setting `user`/`readonly` is rejected with 422 for owners and admins.
- **Effective project role** (`ProjectRepository.GetEffectiveRole`): the org role if `owner`/`admin`, otherwise the override if present, otherwise the org role. `/api/projects` returns it as `role` on each project and masks `token`/`sourceMapToken` when it resolves to `readonly`; the frontend derives write gating from it (`isProjectReadonly` in `projects.svelte.ts`).

#### Organization Roles
| Role | Description |
|------|-------------|
| `owner` | Full access, can manage organization |
| `admin` | Full access to projects |
| `user` | Standard access to projects (can be overridden per project to `readonly`) |
| `readonly` | Read-only access, cannot create projects or archive exceptions (can be overridden per project to `user`) |

Middleware enforcement: `RequireProjectAccess` checks org membership (read access, unaffected by overrides); `RequireWriteAccess` blocks writes when the **effective project role** is `readonly`; `RequireAdminAccess` requires org role `owner`/`admin` for the `:organizationId` route param.

#### Key Columns - transactions
```sql
project_id UUID,
timestamp DateTime64(3),
trace_id String,
endpoint String,           -- normalized: "GET /api/users"
duration_ms Float64,
status_code UInt16,
app_version String,
server_name String,
-- Indexes: bloom_filter(trace_id), tokenbf_v1(endpoint)
```

#### Key Columns - exception_stack_traces
```sql
project_id UUID,
timestamp DateTime64(3),
hash String,               -- normalized hash for grouping
type String,               -- error type (e.g., "RuntimeError")
value String,              -- error message
stacktrace String,         -- full stack trace
tags Map(String, String),  -- contextual tags from scope
```

### Database Migrations

**CRITICAL RULES:**
1. `migrations/ch/` and `migrations/pg/` files must contain **exactly ONE SQL statement**. Both run through `golang-migrate`, and the ClickHouse driver is constructed with `MultiStatementEnabled: false` (`migrations_telemetry_ch.go`), so a second statement fails. `pg/` follows the same rule for symmetry.
2. `migrations/sqlite/`, `migrations/sqlite_telemetry/` and `migrations/duckdb_telemetry/` run through `runMigrationsOn` in `migrations.go`, which splits on semicolons outside string literals (`splitStatements`, covered by `split_test.go`). A file there may hold a `CREATE TABLE` plus its indexes — that is the existing convention, e.g. `sqlite/0043_create_pages.up.sql`.
3. Only create `.up.sql` files (no down migrations)
4. Use sequential numbering: `NNNN_description.up.sql`

**Example - Adding two columns requires TWO files:**
```
backend/app/migrations/ch/
├── 0013_add_app_version_to_transactions.up.sql
│   └── ALTER TABLE transactions ADD COLUMN app_version String DEFAULT ''
├── 0014_add_server_name_to_transactions.up.sql
│   └── ALTER TABLE transactions ADD COLUMN server_name String DEFAULT ''
```

### Exception Hash Normalization

The backend normalizes stack traces before hashing to group identical errors despite different runtime values. This happens in `backend/app/controllers/clientcontrollers/client.controller.go`.

**Normalization Steps** (`ComputeExceptionHash`, applied in this order, and skipped entirely when `isMessage` is true):
1. Strip the message from `Caused by:` lines, keeping the class name (`causedByRe`)
2. Strip the message from the error line, keeping the error type (`errorMessageRe`)
3. Collapse JS SDK function-name lines (ending in `()`, directly above a 4-space-indented `file:line:col` location line) to `<fn>` so resolved function names never affect grouping; anchoring on the location line keeps Go traces (tab-indented file lines) untouched (`jsFuncLineRe`)
4. Remove URL origins such as `https://cdn.example.com` (`urlOriginRe`)
5. Remove absolute file paths, keeping `filename:line` (`absolutePathRe`)
6. Drop the column from any frame whose line number is 2 or higher (`laterLineColRe`)
7. Remove `@v1.2.3` module version suffixes (`versionRe`)
8. Replace hexadecimal addresses with `<hex>` (`hexRe`)
9. Replace UUIDs with `<uuid>` (`uuidRe`)
10. Replace runs of 5 or more digits with `<id>`, unless preceded by a colon or another digit (`largeNumberRe`)
11. Replace email addresses with `<email>` (`emailRe`)
12. Replace IP addresses, with an optional port, with `<ip>` (`ipRe`)
13. Replace Go goroutine ids with `goroutine <n>` (`goroutineRe`)
14. Drop line numbers from Java, Kotlin and Scala frames (`javaLineNumRe`)
15. Collapse `... 12 more` to `... more` (`javaEllipsisRe`)
16. Collapse runs of spaces and tabs, then runs of newlines (`spacesRe`, `newlinesRe`)
17. Trim, hash with SHA-256, truncate to 16 hex chars

**Not normalized:** timestamps and ANSI color codes. Two otherwise identical traces that differ only in an embedded timestamp, or only in `\x1b[31m` escapes, produce two separate Issues. Add a regex to the block in `client.controller.go` if that ever matters.

**Note:** the column is kept only on line 1 (step 6). Minified bundles put everything on line 1, so there the column is the only frame disambiguator when no source map matched, while on real source lines the column is noise. Function names are excluded (step 3) because they are derived from the location and change as symbolication improves.

**Result:** Same logical error gets same hash, even if:
- Error message contains different user IDs
- Stack trace has different memory addresses
- File paths differ between environments

### Repository Patterns

#### Singleton Pattern
Repositories are exported as package-level singletons, re-exported per storage axis through the facade packages `app/repositories/transactional` and `app/repositories/telemetry`:
```go
// backend/app/repositories/transactional/sqlite/user.repository.go (and the pg/ twin)
var UserRepository = userRepository{}

// backend/app/repositories/transactional/transactional_sqlite.go (tag-guarded facade)
var UserRepository = sqliterepo.UserRepository

// Usage in controllers
user, err := transactional.UserRepository.FindByEmail(tx, email)
spans, err := telemetry.SpanRepository.FindByTraceId(projectId, traceId)
```

#### Batch Insert (ClickHouse)
```go
func (r *TransactionRepository) BatchInsert(txns []models.Transaction) error {
    batch, _ := r.db.PrepareBatch(ctx, "INSERT INTO transactions ...")
    for _, txn := range txns {
        batch.Append(txn.ProjectID, txn.Timestamp, ...)
    }
    return batch.Send()
}
```

#### Aggregation with Quantiles
```go
// P50, P95, P99 percentiles
query := `
    SELECT
        endpoint,
        count() as count,
        quantile(0.5)(duration_ms) as p50,
        quantile(0.95)(duration_ms) as p95,
        quantile(0.99)(duration_ms) as p99
    FROM transactions
    WHERE project_id = ? AND timestamp BETWEEN ? AND ?
    GROUP BY endpoint
    ORDER BY count DESC
`
```

### Error Handling Pattern

**IMPORTANT:** When handling errors in controllers, always use `c.AbortWithError` with `traceway.NewStackTraceErrorf` instead of `c.JSON` with a generic error message. This ensures proper error tracking with stack traces.

```go
// CORRECT - Use AbortWithError with descriptive reason
projectId, err := middleware.GetProjectId(c)
if err != nil {
    c.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("RequireProjectAccess middleware must be applied: %w", err))
    return
}

// WRONG - Do not use c.JSON for internal server errors
if err != nil {
    c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
    return
}
```

**Key points:**
- The reason should describe the actual cause (e.g., "RequireProjectAccess middleware must be applied")
- Use `%w` to wrap the original error for proper error chaining
- This pattern applies to all 500 Internal Server Error responses
- For client errors (400, 404), `c.JSON` with an error message is acceptable

**Non-stopping errors:** For errors that should not abort the request (e.g., optional feature failed to load), use `traceway.CaptureException` to report them instead of `log.Printf`:

```go
// CORRECT - Report non-stopping errors via traceway
if err != nil {
    traceway.CaptureException(fmt.Errorf("failed to read session recording (key=%s): %w", key, err))
}

// WRONG - Do not use log.Printf for errors
if err != nil {
    log.Printf("Failed to read session recording (key=%s): %v", key, err)
}
```

**Validation error conventions:**
- `400 Bad Request`: Malformed requests, missing required params, type errors
- `422 Unprocessable Entity`: Business validation in form dialogs (name too long, duplicate name, required field empty). Return `c.JSON(422, gin.H{"error": "descriptive message"})`. The frontend `api.ts` extracts 422 error messages — dialogs catch and display them inline.

**Summary:**
- **Stopping errors** (abort the request): `c.AbortWithError(status, traceway.NewStackTraceErrorf("reason: %w", err))`
- **Non-stopping errors** (continue serving): `traceway.CaptureException(fmt.Errorf("reason: %w", err))`
- **Validation errors** (user-facing): `c.JSON(422, gin.H{"error": "message"})` for form validation
- **Always** wrap errors with `traceway.NewStackTraceErrorf` or `fmt.Errorf` using `%w` — never discard the original error

### Emails

Every email Traceway sends is a hardcoded template in `backend/app/services/emailtemplates/` (embedded with `go:embed`, parsed once at init). The copy of an email lives in its own `.gohtml` file; the payload is the only thing it interpolates. There is no shared layout document, so changing one email cannot reshape the others, and every file is a complete HTML document styled like the rest.

Templates: `new_error`, `error_regression`, `check_down`, `check_recovered`, `alert` (the threshold email every rate/latency/apdex/throughput/task/impact/cost rule sends), `ai_flagged`, `page` (on-call escalation), `test` (both test buttons), `invitation`, `password_reset`.

- **One send path** — `services.SendEmail(ctx, services.Email{...})` in `backend/app/services/email.service.go` is the only way mail leaves the process: it renders the template, wraps it as `multipart/alternative` (plaintext part first, quoted-printable), and talks SMTP. With `SMTP_ENABLED` off it logs the plaintext instead. `services.RenderEmail` is the render-only half, used by the preview endpoint.
- **`services.Email`** carries the chrome every template shares (`Title`, `Badge`/`BadgeColor` via `EmailColor*`, `URL`, `Footer`, `LogoURL` defaulting to `{APP_BASE_URL}/traceway-mark.png`) plus `Template` and `Data`, the typed payload reached as `{{.Data.X}}`. `Text` is the plaintext alternative and must stand on its own.
- **Notification payloads** live on `models.NotificationMessage.Email` (`models.NotificationEmail`: a template name plus one typed struct — `EmailException`, `EmailCheck`, `EmailAlert`, `EmailFlagged`, `EmailPage`, `EmailTest`). The builders in `notifications/messages.go` attach it; `services.NotificationEmail` turns a message into the `Email` at send time (severity chip, absolute link, "why you got this" footer). It rides through the notification outbox as JSON, so the shape is a wire format: add fields, never repurpose them. Only the email adapter reads it — Slack/Telegram/SMS/webhook and `fired_notifications` still see `Body` alone, so `Body` must always stay complete.
- **Adding an email**: add `emailtemplates/<name>.gohtml`, add its payload struct, and call `SendEmail` with `Template: "<name>"`. For a new rule type that only needs a sentence and a link, reuse `alert` by filling `models.EmailAlert`.
- **Preview**: `EMAIL_PREVIEW_ENABLED=true` registers `GET /api/email-preview` (index) and `GET /api/email-preview/:template`, which renders any template with sample data (`?format=text` shows the plaintext alternative). Off by default, so it is never a route in production. Sample data lives in `controllers/email_preview.controller.go`.

---

## Native `/api/report` Protocol

There is **no Traceway Go SDK**. It was retired, and every backend, Go included, instruments with
OpenTelemetry and exports over OTLP/HTTP. The docs carry no Go SDK pages, and `docs/public/_redirects`
301s the old `/client/sdk` and `/client/*-middleware` URLs to `/client/otel`. Do not reintroduce them.

The native protocol is **deprecated**. Traceway is OpenTelemetry first, and the public docs do not
describe `/api/report` as a way to send data: there is no protocol page (`/protocol` 301s to
`/client/otel`) and `docs/pages/learn/data-flow.mdx` is written around OTLP. Do not document it again.
The URL still shows up in the docs in one role only, as part of the connection string the browser and
mobile SDKs are configured with.

The endpoint itself is still served, because those SDKs (`@tracewayapp/*`, the iOS and Android
libraries, Flutter) speak it and the backend uses it to report its own telemetry. This section is the
engineering reference for keeping it working. Its rows land in the same V2 tables as OTLP: a run is
stored as the root span of a trace with its spans under it (see "Trace Storage (V2 tables)").

### Data Format (Frame)

Clients send data as gzipped JSON. The wire shape is `ReportRequest` in `backend/app/controllers/clientcontrollers/client.controller.go` wrapping `CollectionFrame` from `backend/app/models/clientmodels/`:

```json
{
  "appVersion": "1.2.3",
  "serverName": "myapp-host-1",
  "collectionFrames": [
    {
      "traces": [
        {
          "id": "5b8e1a2f-3c4d-4e5f-8a9b-0c1d2e3f4a5b",
          "endpoint": "GET /api/users",
          "duration": 45200000,
          "statusCode": 200,
          "recordedAt": "2024-01-15T10:30:00Z",
          "isTask": false,
          "attributes": {},
          "spans": []
        }
      ],
      "stackTraces": [
        {
          "stackTrace": "RuntimeError: connection refused\n  at ...",
          "recordedAt": "2024-01-15T10:30:00Z",
          "attributes": {"user_id": "123"},
          "isMessage": false,
          "isTask": false
        }
      ],
      "metrics": [
        {
          "name": "cpu.used_pcnt",
          "value": 45.2,
          "recordedAt": "2024-01-15T10:30:00Z",
          "tags": {}
        }
      ]
    }
  ]
}
```

Notes:
- Timestamps are `recordedAt` (RFC 3339), never `timestamp`. `duration` is a Go `time.Duration`, i.e. integer nanoseconds (45200000 = 45.2ms).
- Endpoints vs tasks share the `traces` array, split by `isTask`. `CollectionFrame` also carries `sessionRecordings` and `sessions`.
- Each entry in a trace's `spans` may carry an optional `parentSpanId` (UUID) and `attributes` (string map). A span that names no parent hangs under the run's own span. The ids are stored as 32 hex characters.
- **Unknown fields are silently ignored**: a payload in the wrong shape (e.g. top-level `metrics`) still returns 200 but inserts nothing. When hand-crafting test payloads, confirm ingestion landed via `POST /api/metrics/query` (or the relevant list endpoint) instead of trusting the status code.

---

## Common Patterns

### Adding a New API Endpoint

1. **Add model** in `backend/app/models/`
   ```go
   type NewEntity struct {
       ID        uuid.UUID `json:"id"`
       Name      string    `json:"name"`
       CreatedAt time.Time `json:"created_at"`
   }
   ```

2. **Add repository** in `backend/app/repositories/`
   ```go
   func (r *NewEntityRepository) GetAll(projectID uuid.UUID) ([]models.NewEntity, error) {
       // ClickHouse query
   }
   ```

3. **Add controller** in `backend/app/controllers/`
   ```go
   func (c *NewEntityController) List(ctx *gin.Context) {
       entities, err := repo.GetAll(projectID)
       ctx.JSON(200, entities)
   }
   ```

4. **Register route** in `backend/app/controllers/routes.go`
   ```go
   api.GET("/new-entities", newEntityController.List)
   ```

5. **Add frontend API call** in `frontend/src/lib/api.ts` or directly in page

**POST body convention:** List/search endpoints use POST with a JSON body containing filters and pagination:
```go
type ListRequest struct {
    ProjectId  string           `json:"projectId"`
    FromDate   string           `json:"fromDate"`
    ToDate     string           `json:"toDate"`
    OrderBy    string           `json:"orderBy"`
    Search     string           `json:"search"`
    Pagination PaginationParams `json:"pagination"`
}
```

**Paginated response:** Use `PaginatedResponse[T]` from `routes.go` for all paginated endpoints:
```go
type PaginatedResponse[T any] struct {
    Data       []T        `json:"data"`
    Pagination Pagination `json:"pagination"`
}

type Pagination struct {
    Page       int   `json:"page"`
    PageSize   int   `json:"pageSize"`
    Total      int64 `json:"total"`
    TotalPages int64 `json:"totalPages"`
}

type PaginationParams struct {
    Page     int `json:"page" binding:"min=1"`
    PageSize int `json:"pageSize" binding:"min=1,max=100"`
}
```

### Adding a New Frontend Page

1. **Create route folder**: `frontend/src/routes/new-page/`

2. **Add page component**: `+page.svelte` with the standard loading pattern:
   ```svelte
   <script lang="ts">
     import { onMount } from 'svelte'
     import { api } from '$lib/api'
     import { projectsState } from '$lib/state/projects.svelte'
     import { ErrorDisplay } from '$lib/components/ui/error-display'
     import { getErrorMessage, getErrorStatus } from '$lib/utils/errors'

     let data = $state<DataType[]>([])
     let loading = $state(true)
     let error = $state('')
     let notFound = $state(false)

     async function loadData() {
       loading = true
       error = ''
       try {
         const response = await api.post<ResponseType>('/endpoint', payload, {
           projectId: projectsState.currentProjectId ?? undefined
         })
         data = response.data || []
       } catch (e) {
         if (getErrorStatus(e) === 404) {
           notFound = true
         } else {
           error = getErrorMessage(e) || 'Failed to load data'
         }
       } finally {
         loading = false
       }
     }

     onMount(() => { loadData() })
   </script>

   {#if loading}
     <LoadingCircle size="xlg" />
   {:else if notFound}
     <ErrorDisplay status={404} title="Not Found" description="..." onRetry={() => loadData()} />
   {:else if error}
     <ErrorDisplay status={400} title="Error" description={error} onRetry={() => loadData()} />
   {:else}
     <!-- Content -->
   {/if}
   ```

3. **Add data loading** (optional): `+page.ts` for URL params
   ```typescript
   export const load = async ({ params }) => {
       return { param: params.id }
   }
   ```

4. **Add navigation** in `src/lib/components/app-sidebar.svelte`

### Adding a New Metric to Dashboard

1. **Ensure SDK captures metric** (or add to `traceway.go` metrics collection)

2. **Add repository query** to each telemetry backend's `metric_point.repository.go` under `backend/app/repositories/telemetry/{clickhouse,sqlite,duckdb}/`
   ```go
   func (r *metricPointRepository) GetNewMetric(ctx context.Context, projectId uuid.UUID, from, to time.Time) ([]models.TimeSeriesPoint, error) {
       // Query metric_points with the backend's dialect
   }
   ```

3. **Add to dashboard controller** in `backend/app/controllers/dashboard.go`

4. **Frontend auto-renders** from API response (metrics dashboard uses dynamic rendering)

### Adding a Database Column

1. **Create migration file** (remember: ONE statement per file!)
   ```
   backend/app/migrations/ch/0015_add_new_column.up.sql
   ```
   ```sql
   ALTER TABLE transactions ADD COLUMN new_column String DEFAULT ''
   ```

2. **Update model** in `backend/app/models/`

3. **Update repository queries** to include new column

4. **Run migrations**: Backend runs migrations automatically on startup

### Adding Table Sorting to a Page

1. **Import and add state**:
   ```typescript
   import { getSortState, setSortState, handleSortClick } from '$lib/utils/sort-storage'
   import type { SortState } from '$lib/utils/sort-storage'

   let sortState = $state<SortState>(getSortState('page-key', { field: 'default_column', direction: 'desc' }))
   ```

2. **Add sort handler**:
   ```typescript
   function onSortClick(field: string) {
       sortState = handleSortClick(field, sortState.field, sortState.direction, 'desc')
       setSortState('page-key', sortState)
   }
   ```

3. **Use TracewayTableHeader**:
   ```svelte
   <TracewayTableHeader
       label="Column"
       column="column_name"
       orderBy={`${sortState.field} ${sortState.direction}`}
       onclick={() => onSortClick('column_name')}
   />
   ```

4. **Pass to API call** - convert to backend format: `"column asc"` or `"column desc"`:
   ```typescript
   const orderBy = `${sortState.field} ${sortState.direction}`
   ```

### Adding a New Framework

**Backends do not get a new framework value.** OpenTelemetry is the single backend integration path, and `opentelemetry` is the only backend option in the project-creation picker (plus the preselected default). A new backend language or web framework is a *documentation and Connection-page* change, never a new project framework:

1. **Connection page targets** — `frontend/src/lib/utils/otel-setup.ts`: add the framework to the matching `OTEL_TARGETS` language entry (or add a new language target) and its setup steps
2. **Docs guide** — `docs/pages/client/otel/<framework>/`: create `_meta.json` and `index.mdx`, then list it in `docs/pages/client/otel/_meta.json` and the "Next Steps" block of `docs/pages/client/otel/index.mdx`
3. **Docs OTel language table** — `docs/pages/client/otel/index.mdx`: add a row
4. **Docs framework picker** — nothing to do. `docs/components/FrameworkPicker.jsx` carries exactly one backend card, **OpenTelemetry**, pointing at `/client/otel`, so it mirrors the dashboard's project-creation picker. Do not add a per-framework backend card; the guide is discovered from `/client/otel` instead.
5. **Combobox search keywords** — `frontend/src/lib/components/framework-combobox.svelte`: append the name to the OpenTelemetry entry's `keywords` so searching for it still finds the right option
6. **README** — add to the supported-frameworks table if it warrants a row

Only a **browser or mobile** framework needs a real new framework value, because those use platform SDKs with their own Connection-page content:

1. **Backend** — `backend/app/controllers/project.controller.go`: add to `validFrameworks` and update the validation error message
2. **Frontend state** — `frontend/src/lib/state/projects.svelte.ts`: add to the `Framework` union, `FRAMEWORK_LABELS`, and `MOBILE_FRAMEWORKS`/`FRONTEND_FRAMEWORKS`
3. **Frontend combobox** — `frontend/src/lib/components/framework-combobox.svelte`: add an entry to the `Browser` or `Mobile` group with `keywords`
4. **Frontend icon** — `frontend/src/lib/components/framework-icon.svelte`: add the icon mapping
5. **Framework code** — `frontend/src/lib/utils/framework-code.ts`: add install command, integration snippet, label, code language, and testing routes
6. **Connection page** — `frontend/src/routes/connection/+page.svelte`: add highlight language mapping and install description
7. **Dashboard page** — `frontend/src/routes/+page.svelte`: add highlight language mapping
8. **Docs** — `docs/pages/client/<framework>/`, `docs/pages/client/_meta.json`, `SDK_OPTIONS` + `FOLDER_SDK` in `docs/components/SdkContext.jsx`, `SDK_VISIBILITY` in `docs/theme.config.jsx`, `SDK_QUICK_START` in `docs/components/SdkSelector.jsx`, and a `FrameworkPicker.jsx` card

Values removed from the picker (`gin`, `fiber`, `chi`, `fasthttp`, `stdlib`, `custom`, `nextjs`, `nestjs`, `express`, `remix`, `hono`, `cloudflare`, `symfony`, `laravel`, `django`) stay valid in `validFrameworks` and in `FRAMEWORK_LABELS` — existing projects still carry them and must keep rendering.

More agent context in tracewayapp/traceway

4 other files this repository gives its agents.