SurfSense

# AGENTS.md

Nearest `AGENTS.md` wins. Edit this file, not `CLAUDE.md` (`CLAUDE.md` is a symlink).

## Overview

Desktop app plus scraper API. Five trees:

| Tree | Role |
|------|------|
| `surfsense_backend` | FastAPI API |
| `surfsense_web` | Next.js hosted UI |
| `surfsense_local` | Electron desktop |
| `surfsense_mcp` | MCP server over the REST API |
| `plugins` | SurfSense's own plugin servers ([plugin host](docs/architecture/plugin-host.md)) and the [registry](docs/architecture/plugin-registry.md). `bundles/` holds the earlier, local design's SDK and tooling, deferred |

The hosted service has been export-only since the 2.0.0 launch on 18 Sep 2026, and its user data is purged on 18 Oct 2026 ([sunset](docs/architecture/sunset.md)). Product direction is local + API.

## Code organization

Applies to **every new file**. Existing code is not a template and not a cleanup ticket.

**Must**

- Organize by feature (vertical slice), not by technical layer.
- One responsibility per file. One responsibility per folder.
- If a responsibility grows sub-responsibilities, promote it to a folder. Each sub-responsibility is its own file.
- Names state what and why, not how.
- Comments and docstrings state intent only. Do not restate the code.
- Keep them short. A line or two. Write only what the code cannot say: a constraint, a rejected alternative, a number that justifies a threshold.

**Must not**

- Do not write essays in docstrings. If it explains how you arrived at the code rather than what the code must honour, delete it.
- Do not put new work in a nearby file because it is convenient. New responsibility, new file.
- Do not add catch-all folders (`utils`, `helpers`, `common`, `misc`, `shared`) unless that name is the product concept.
- Do not layer-split new work (`controllers/`, `services/`, `models/` as the primary tree).
- Do not rewrite existing code to this layout unless the task is that rewrite.

The tree is inconsistent. New work follows this. Old work stays until a task explicitly owns it.

## Docs

[`docs/`](docs/README.md) is the engineering map. `architecture/` says what is true now, one doc per feature; `adr/` says why; `proposals/` holds designs not built yet; `contracts/` holds the frozen interfaces between trees. Task status lives in GitHub issues; docs carry only a proposal's `status` and each architecture doc's Known gaps. `plans/` is business and ops material, not specs.

- Before changing a feature, read its doc in `docs/architecture/` and the ADRs it links.
- If a change alters behaviour a doc describes, update the doc in the same change. Fixing a Known gap deletes its line.
- Run `python scripts/check_docs.py` after editing anything under `docs/` or `plans/`.

## Commands

```bash
# backend
cd surfsense_backend && uv sync
cd surfsense_backend && uv run ruff check .
cd surfsense_backend && uv run pytest -m unit

# web
cd surfsense_web && pnpm install
cd surfsense_web && pnpm dev
cd surfsense_web && pnpm format

# desktop
cd surfsense_local/electron && pnpm dev

# MCP
cd surfsense_mcp && uv sync

# plugin host: SurfSense's own plugins, from plugins/remote/
cd plugins/remote/host && uv sync
cd plugins/remote && uv run --project host python -m host

# bundles (deferred): the manifest rules, the SDK, the surfsense-plugins command
cd plugins/bundles/core/manifest && uv sync
cd plugins/bundles/core/sdk && uv sync
cd plugins/bundles/core/sdk && uv run pyright
cd plugins/bundles/core/cli && uv sync

# compose (dev and self-host, not production)
docker compose -f docker/docker-compose.yml

# hooks
pre-commit run --all-files
```

## Testing

| Area | Command |
|------|---------|
| Backend unit | `cd surfsense_backend && uv run pytest -m unit` |
| Backend integration | `cd surfsense_backend && uv run pytest -m integration` |
| Web unit | `cd surfsense_web && pnpm test:unit` |
| Web e2e | `cd surfsense_web && pnpm test:e2e` |
| MCP | `cd surfsense_mcp && uv run pytest` |
| Plugin host | `cd plugins/remote/host && uv run pytest` |
| Bundles: manifest rules | `cd plugins/bundles/core/manifest && uv run pytest -m unit` |
| Bundles: SDK | `cd plugins/bundles/core/sdk && uv run pytest -m unit` |
| Bundles: SDK against the app | `cd plugins/bundles/core/sdk && uv run pytest -m contract` (starts the backend; `uv sync` it first) |
| Bundles: CLI | `cd plugins/bundles/core/cli && uv run pytest` (starts the backend; `uv sync` it first) |
| Desktop | `cd surfsense_local/electron && pnpm test` |
CI: `.github/workflows/`. New behavior: one failing test, then the minimum code to pass it. Use the `tdd` skill. Tests hit public seams, not internals.

## Pull requests

Follow [CONTRIBUTING.md](CONTRIBUTING.md). The parts agents miss:

- Open PRs against `dev`. `gh pr create` targets the default branch, `main`, unless you pass `--base dev`.
- `gh pr create --body` skips the PR template. Keep its headings: What, Why, `Fixes #`, How to test.
- Commit messages and PR titles are Conventional Commits, `type(scope): summary`. The commitizen hook runs only where it is installed; CI does not run it.
- A new feature starts as a proposal PR in `docs/proposals/`, not as code.
- `surfsense_backend/app/proprietary/` and `plugins/remote/proprietary/` are Business Source License 1.1 and changed only by maintainers: ask one before changing them. A change to `docs/contracts/` needs the owners of both sides.

## Skills

Canonical dir: `.agents/skills/`. `.claude/skills` is a symlink to it.

| Skill | When |
|-------|------|
| `tdd` | New behavior or a bug fix with a testable seam |
| `codebase-design` | Module shape, seam, depth — `tdd` depends on this vocabulary |
| `fastapi` | FastAPI / Pydantic work. Symlink into the installed wheel; repair the **same** root link after a FastAPI or Python bump. Do not keep `surfsense_backend/.agents/`. |
| `frontend-workflow` | Frontend work in `surfsense_web` and `surfsense_local/frontend`: React/Next performance, color tokens, UI polish, motion. Bundles `react-performance/`, `color/`, `polish/`, `motion/`, and `base-ui.md` for Base UI in `surfsense_local/frontend`; routes out to `shadcn`. |
| `shadcn` | UI components in a tree with `components.json`. Standalone — it inspects the project live and grants its own CLI, which only works as a discovered skill. Load it after `frontend-workflow`, not instead of it. |
| `translate` | Interface strings in `surfsense_local`: adding a message in code (`en.json` is generated from it), moving hard-coded text into messages, translating Japanese and German. Holds the key shape, tone and glossary. |

Do not install skill catalogs. Do not add `CONTEXT.md` or a second rules tree.

`frontend-workflow` bundles what were separate skills under `react-performance/`,
`color/`, `polish/`, and `motion/`. Those files keep their old frontmatter, which
is inert — they are reference files, not discovered skills. Skill discovery is one
level deep, so do not nest a new skill inside another skill's folder.

Bundle a skill only if it is static markdown that overlaps the others. Anything
with frontmatter that does work — `allowed-tools`, `` !`command` `` substitution —
must stay top-level, because that frontmatter is inert in a bundled file.

`shadcn/SKILL.md` carries one local addition: a note routing to `frontend-workflow`
first. It is the only edit to that vendored file — restore it after any shadcn
re-sync, or component work bypasses the color palette and polish rules.

`skills-lock.json` lists `vercel-react-best-practices`, which now lives at
`frontend-workflow/react-performance/`. `shadcn` is back at its original path and
its lock entry is valid.

## Boundaries

- Do not carve hosted code out of backend/web until the purge on 18 Oct 2026. The export window and the purge script run on that code until then.
- Do not drop SearxNG, sandbox, OpenSandbox, or zero-cache from compose until a new compose is defined.
- Edit the root `README.md` directly. The draft at `plans/community-local/seo/drafts/README.md` shipped on 18 Sep 2026 and is kept only for history. Anything added to the README replaces something — its length was measured against the eight biggest repos in this category. Make the same change in every `README.<locale>.md`, section for section; [`06-repo-readme.md`](plans/community-local/seo/06-repo-readme.md#the-readme-since-9-oct-2026) has the rules, where each format-table cell is checked, and how to replace the demo video.
- Do not commit, push or open a PR unless asked.

## Security

- Never commit `.env` or secrets.
- Do not put credentials in diffs, logs, or sample payloads.

More agent context in MODSetter/SurfSense

16 other files this repository gives its agents.