latitude-llm

# AGENTS.md

High-level guide for coding agents working in this repository.

## Product scope

Multi-tenant LLM observability platform. The repo is a **pnpm** workspace orchestrated with **Turbo**.

At a glance: **`apps/*`** own transport (middleware, authn, mounting); **`@repo/operations`** owns the public API's operation contracts (validation, public schemas, routing to use-cases) shared by HTTP/MCP/SDK/CLI and in-process agent tools; **`packages/domain/*`** own business rules and ports; **`packages/platform/*`** implement infrastructure adapters; **`@repo/utils`** holds cross-cutting pure helpers. Telemetry and control data flow through **Postgres**, **ClickHouse**, **Redis**, and object storage, with **organization-scoped** access everywhere at the boundary.

## Repo-wide conventions

- Organization-scoped Redis or cache keys must start with the organization prefix: `org:${organizationId}:...`. Put the org id first so tenancy is obvious and keyspaces stay consistently partitioned.
- **Code comments are rare, and one line when they exist.** Default to no comment — naming and structure should carry the meaning; a comment explaining *what* code does is a signal to rewrite the code, not to annotate it. Write a comment only for something the code cannot show: a trap (code that looks wrong, removable, or simplifiable but breaks something — ordering requirements, magic values, workarounds for framework/library quirks), a non-obvious external constraint, or contract docs where the file's existing convention documents every sibling (e.g. JSDoc on every method of a port). Never narrate the change you just made, restate what the next line does, paraphrase a well-named identifier, add section-header comments, or explain design rationale in the code — rationale belongs in the commit or PR description. When editing existing code, don't add comments describing your edit, and feel free to delete comments your change made stale or redundant. A comment describes the code as it stands, not the development process that produced it: never anchor one to transient context like a rollout phase, ticket id, or "new"/"now" framing (e.g. `// Phase 2 (LAT-749): ...`) — that reads as stale the moment the phase lands; state what the code does and put the ticket in the commit/PR. The only comments that may name a ticket are actionable `TODO`/`FIXME` markers for work not yet done.
- Never invoke `tsc` directly. Typechecking goes through `tsgo` via the package `typecheck` script — use `pnpm --filter <pkg> typecheck` for one package or `pnpm typecheck` for the whole workspace. `tsc` would diverge from CI.
- ClickHouse migrations must be created with `pnpm --filter @platform/db-clickhouse ch:create <migration_name>`. Do not create ClickHouse migration files manually.
- **PR base branch follows the branch's origin.** Branches forked from `development` (v2 work, default) PR into `development`; branches forked from `latitude-v1` (v1 maintenance) PR into `latitude-v1`. Never start a branch from `main` without confirmation — if the working branch is based on `main`, or the user asks to branch from `main`, confirm with the user first. Detect an existing branch's base by testing ancestry in order: if `git merge-base --is-ancestor origin/latitude-v1 HEAD` is true → `latitude-v1`; else if `git merge-base --is-ancestor origin/development HEAD` is true → `development`; else the branch is likely based on `main` (or something unusual) — stop and confirm with the user before opening a PR. When the user asks to start a new branch, take the base from their wording (mentions of v1 → `latitude-v1`; otherwise default to `development`).
- Production deploys use the single-branch release-tag flow: `development` is trunk and deploys to staging by default; production is triggered only by pushing a `vX.Y.Z` tag that points at the latest `origin/development` commit. Before tagging a production release, update `CHANGELOG.md` with a human-readable diff of the code being pushed to production since the previous production deploy, focusing on the major aspects rather than every commit. Use `scripts/release.sh [version]` to fetch and tag the latest `origin/development` commit; without a version it bumps the latest `vX.Y.Z` tag to the next patch version, with `--minor` / `--major` available for larger bumps. Do not promote by merging `development` into `main`.
- **Keep the product cleanly OSS and self-hostable.** Shipped runtime dependencies must be permissively licensed (MIT/Apache-2.0/BSD/ISC) — no AGPL/SSPL/source-available code in the application bundle; audit a new dependency's license (and its transitive additions) before adding it. The self-host object store is **SeaweedFS** (Apache-2.0) — never reintroduce MinIO or any AGPL store as a bundled default. Keep every infra dependency **isolatable and bring-your-own-able** (dedicated schema/db/namespace/bucket; Redis keys namespaced under `latitude:`) so a self-hoster can swap any bundle for a managed instance. Rationale + full audit: [`dev-docs/licensing.md`](dev-docs/licensing.md); self-host architecture: [`dev-docs/self-hosting.md`](dev-docs/self-hosting.md).
- **Private partner endpoints live under `/v1/private/*` and are plain Hono routes.** They serve vetted, staff-registered partners, are authenticated by per-partner HMAC request signing (never a bearer token), and must stay off every generated surface — use a plain `app.post(...)`, never `app.openapi`/`createRoute`, never `defineOperation`. Every pre-scope refusal returns an identical `401 {"error":"unauthorized"}` so the surface can't be used to enumerate partner ids; only a scope failure gets a distinct 403. Details: [`dev-docs/partners.md`](dev-docs/partners.md).
- **Two image registries, distinct jobs.** Public self-host images go to **Docker Hub `latitudedata/<service>`** — env-neutral, multi-arch, for the six build targets (`api`, `ingest`, `workers`, `workflows`, `web`, `migrations`) — tagged to the release flow: `:X.Y.Z` + `:latest` on a release (git `vX.Y.Z` with the `v` stripped) are the stable tags self-hosters pin, `:development` is the trunk edge tag. **GHCR** (`latitude-<env>-<service>`, sha-tagged) stays Latitude's **private** own-deploy registry and is never the self-host source.

## How to use this guide

1. Skim the **skill glossary** below and open the skill that matches your task.
2. Read that skill's `SKILL.md` in full before editing code in that area.

Detailed policies, command examples, and code samples live under **`.agents/skills/<skill-name>/SKILL.md`**. Load narrow skills instead of memorizing the entire monorepo at once.

**Index coverage:** The glossary lists **every** skill in `.agents/skills/` (one row per `*/SKILL.md`, **28** total), ordered **alphabetically by folder name**. When you add or remove a skill folder, update this table in the same change.

## Skill glossary

| Skill | Path | Use when |
| --- | --- | --- |
| **Agentation watch mode** | [.agents/skills/agentation-watch-mode/SKILL.md](.agents/skills/agentation-watch-mode/SKILL.md) | Agentation annotation watch loops, continuous feedback handling, or when the user says **`watch mode`** and wants annotations acknowledged, fixed, and resolved as they arrive |
| **Analyze problem** | [.agents/skills/analyze-problem/SKILL.md](.agents/skills/analyze-problem/SKILL.md) | Investigating a bug, task, or reported issue to explain behavior, root cause, proposed fix, and verification steps before implementation |
| **API endpoints (HTTP, MCP, SDK, CLI)** | [.agents/skills/api-endpoints/SKILL.md](.agents/skills/api-endpoints/SKILL.md) | Adding or changing API operations in **`@repo/operations`**, **`defineOperation`**, **`openapi.json` / `mcp.json` regen**, **`OperationModule`** manifests, **`group`/`sdkMethod`/`rateLimitTier`**, **`defineToolset`** agent toolsets, writing **field descriptions** that propagate to the **TS + Python SDKs**, **MCP tool**, and **`latitude` CLI** consumers |
| **Architecture and boundaries** | [.agents/skills/architecture-boundaries/SKILL.md](.agents/skills/architecture-boundaries/SKILL.md) | Layering, web vs public API, **app layout** (clients, routes, logging), ports/adapters, **web-standard APIs in domain/shared/utils**, multi-tenancy, DDD layout, anti-patterns, **machine-facing MCP/API product surfaces** |
| **Background jobs and events** | [.agents/skills/async-jobs-and-events/SKILL.md](.agents/skills/async-jobs-and-events/SKILL.md) | **Queues/workers**, **domain events**, side effects **outside** HTTP handlers, task payload design, debounce/dedupe, delayed job semantics, **domain event naming**, **publisher–consumer decoupling** |
| **Authentication** | [.agents/skills/authentication/SKILL.md](.agents/skills/authentication/SKILL.md) | **Better Auth**, sessions, web session helpers, org context on session, **`@domain/auth`** flows |
| **Backoffice** | [.agents/skills/backoffice/SKILL.md](.agents/skills/backoffice/SKILL.md) | Staff-only `/backoffice` features, **`createAdminServerFn`** factory, admin guards + RLS-bypass path, `@domain/admin` feature-folder layout |
| **Better Auth best practices** | [.agents/skills/better-auth-best-practices/SKILL.md](.agents/skills/better-auth-best-practices/SKILL.md) | **Better Auth** server/client setup, DB adapters, sessions, plugins, env (`auth.ts`); email/password, OAuth; **better-auth.com** API reference |
| **CI watchdog** | [.agents/skills/ci-watchdog/SKILL.md](.agents/skills/ci-watchdog/SKILL.md) | Watching GitHub PR checks, monitoring CI status, diagnosing failures from logs, and looping on fixes until checks pass |
| **Code style and TypeScript** | [.agents/skills/code-style/SKILL.md](.agents/skills/code-style/SKILL.md) | Biome, imports, strict TS, naming, **Zod-first shared contracts**, literal-union enums, named constants, generated files |
| **Create PR** | [.agents/skills/create-pr/SKILL.md](.agents/skills/create-pr/SKILL.md) | Creating PRs, writing PR descriptions, summarizing changes, and preparing a reviewable pull request |
| **ClickHouse** | [.agents/skills/database-clickhouse/SKILL.md](.agents/skills/database-clickhouse/SKILL.md) | Parameterized CH queries, Goose migrations, append-only migration rules |
| **Postgres and SqlClient** | [.agents/skills/database-postgres/SKILL.md](.agents/skills/database-postgres/SKILL.md) | Drizzle schema, RLS, SqlClient, migrations (Drizzle Kit), no-FK rules, repository mappers |
| **Documentation and specs** | [.agents/skills/docs/SKILL.md](.agents/skills/docs/SKILL.md) | **`dev-docs/*.md`** (domain), **`docs/`** (ADRs, Mintlify), **`specs/*.md`**, durable documentation sync, spec structure, promoting stable knowledge into `dev-docs/` |
| **Effect and errors** | [.agents/skills/effect-and-errors/SKILL.md](.agents/skills/effect-and-errors/SKILL.md) | `Effect` composition, `Data.TaggedError`, `HttpError`, boundary error handling |
| **Environment configuration** | [.agents/skills/env-configuration/SKILL.md](.agents/skills/env-configuration/SKILL.md) | **`LAT_*` / `VITE_LAT_*`**, `.env.example`, **`parseEnv` / `parseEnvOptional`** |
| **Fix Datadog issues** | [.agents/skills/fix-datadog-issues/SKILL.md](.agents/skills/fix-datadog-issues/SKILL.md) | Finding, triaging, and fixing production errors from **Datadog Error Tracking** (`plugin:datadog:mcp`); picking which issue to work (occurrence/trend/recency, v2-only, prod-only), root-causing in code, reproducing with tests, commenting on the issue, and opening a PR to `development` |
| **GitHub issues** | [.agents/skills/gh-issue/SKILL.md](.agents/skills/gh-issue/SKILL.md) | Creating clear, actionable GitHub issues for bugs, features, and improvements, optimized for LLM/actionability |
| **Humanizer** | [.agents/skills/humanizer/SKILL.md](.agents/skills/humanizer/SKILL.md) | Editing or reviewing prose (docs, PR and commit copy) to remove AI-writing tells such as em dashes, rule of three, promotional language, and filler, so it reads naturally |
| **Managing maintenance windows** | [.agents/skills/managing-maintenance-windows/SKILL.md](.agents/skills/managing-maintenance-windows/SKILL.md) | Enabling, disabling, verifying, or preparing production maintenance mode, which redirects **`console.latitude.so`** to the Better Stack status page with the Pulumi `enableWebMaintenanceRedirect` switch |
| **Mintlify docs preview** | [.agents/skills/mintlify-preview/SKILL.md](.agents/skills/mintlify-preview/SKILL.md) | Running the public **Mintlify** docs site (`docs/`) locally for live preview (`mint dev`); **Node <25 (nvm v22) requirement**, keeping the CLI current, non-default port, page-path mapping |
| **Notifications** | [.agents/skills/notifications/SKILL.md](.agents/skills/notifications/SKILL.md) | Adding a notification **kind**, **group**, or **channel**; in-app + email delivery; `NOTIFICATION_KIND_META` / `NOTIFICATION_GROUPS`; per-user prefs (`users.notification_preferences`); project-level gates (`projects.settings.notifications`); idempotency + cascade-on-`ProjectDeleted` |
| **Production release** | [.agents/skills/production-release/SKILL.md](.agents/skills/production-release/SKILL.md) | Preparing a production release, updating `CHANGELOG.md` from the production diff, and pushing `vX.Y.Z` release tags with `scripts/release.sh` |
| **Review PR comments** | [.agents/skills/review-pr-comments/SKILL.md](.agents/skills/review-pr-comments/SKILL.md) | Loading issue-level and inline PR feedback with GitHub CLI/API, deduping comments, replying in the right thread, and resolving addressed review threads |
| **Temporal workflows** | [.agents/skills/temporal-developer/SKILL.md](.agents/skills/temporal-developer/SKILL.md) | Editing workflows or activities in **`apps/workflows`**, **reordering/inserting activities** in a running workflow, **`patched()` / `deprecatePatch()` / Worker Versioning**, debugging **non-determinism errors**, terminating stuck workflows, replay/history semantics |
| **Testing** | [.agents/skills/testing/SKILL.md](.agents/skills/testing/SKILL.md) | Vitest layers, PGlite/chdb testkit, **`/testing` package exports**, avoiding `vi.mock` for repositories |
| **Toolchain and commands** | [.agents/skills/toolchain-commands/SKILL.md](.agents/skills/toolchain-commands/SKILL.md) | Node/pnpm/Turbo/Vitest/Biome, scripts, filters, CI, `.env.*` setup, **Docker Compose, dev servers, Mailpit** |
| **Web frontend** | [.agents/skills/web-frontend/SKILL.md](.agents/skills/web-frontend/SKILL.md) | `apps/web` UI, TanStack Start, collections, `@repo/ui`, layout, **`-components/`**, legacy UI reference, **`useMountEffect` policy**, **`useForm` + `createFormSubmitHandler` + `fieldErrorsAsStrings`** for Zod field errors on forms |

## Cursor Cloud specific instructions

The update script runs `scripts/cloud-install.sh` and `scripts/cloud-start.sh` on every session start — dependencies are installed, Docker infra is started, and all databases are migrated and seeded automatically.

For services, ports, health checks, Docker Compose, dev servers, and Mailpit auth flow, see [toolchain-commands skill](.agents/skills/toolchain-commands/SKILL.md). Seeded users include `owner@acme.com`, `admin@acme.com`, etc., all in the "Acme Inc." organization.

**Cloud-only gotchas:**

- **mise activation**: Use `eval "$(mise env)"` (not `mise activate bash`) before running scripts that call `node`/`pnpm` in child shells. `mise activate` only hooks the current shell's prompt; child `bash` invocations won't inherit the PATH.
- `pnpm build` must complete before `pnpm db:up` — migration scripts depend on compiled platform packages.
- The `pnpm install` output may warn about unapproved build scripts (`@swc/core`, `sharp`, etc.). These are non-blocking — pre-built binaries are used.
- The Docker daemon in cloud VMs needs the `fuse-overlayfs` storage driver and `iptables-legacy`; on Docker 29+ you must also set `features.containerd-snapshotter=false` in `/etc/docker/daemon.json` or `fuse-overlayfs` is ignored. This daemon config plus `docker` group membership for the run user are baked into the VM snapshot (not the update script); `cloud-start.sh` only starts the already-configured daemon.

More agent context in latitude-dev/latitude-llm

30 other files this repository gives its agents.