AGENTS.md - wet

Python MCP Server: web search, content extraction, library docs, structured extraction. Xem AGENTS.md va README.md de hieu architecture va configuration.

Cau truc

• src/wet/ -- Package chinh (src layout)
  • server.py -- FastMCP server (orchestrator, file lon nhat); always the HTTP MCP endpoint (no stdio mode)
  • cli.py -- wet control plane: server start|stop|status, config, token hash, users path, docs reindex|import|reembed, search, warmup
  • runtime.py -- Bridge to hull-core (settings, auth mode, users store, per-sub namespace)
  • config.py -- Pydantic Settings (singleton)
  • cache.py, db.py, embedder.py, reranker.py -- Infrastructure
  • identity.py -- Coherent browsing identity across the strategy chain
  • docs_import.py, docs_reembed.py -- Offline docs maintenance (no running server needed)
  • setup.py, setup_tool.py -- First-run SearXNG install + warmup logic (MCP-callable)
  • sources/ -- Data source integrations (crawler, docs, searxng, search backends/strategies)
  • alembic/ -- Docs DB migrations
• tests/ -- Mirror source modules

Lenh thuong dung

uv sync --group dev                # Cai dependencies
uv build                           # Build package (hatchling)
uv run ruff check .                # Lint
uv run ruff format --check .       # Kiem tra format
uv run ruff check --fix . && uv run ruff format .  # Fix
uv run ty check                    # Type check (ty lenient config)
uv run pytest                      # Test tat ca (integration excluded by default)
uv run pytest -m integration       # Chi integration tests
uv run pytest tests/test_config.py::test_function_name -v  # Test don le
uv run wet server start            # Chay server (background; --foreground de giu terminal)
uv run wet server status           # pid / lock / HTTP liveness
uv run wet server stop

# Mise shortcuts
mise run setup     # Full dev env setup
mise run lint      # ruff check + ruff format --check + ty check
mise run test      # pytest
mise run fix       # ruff check --fix --unsafe-fixes + ruff format
mise run dev       # uv run wet server start --foreground

Script entry points: wet (wet.cli:main) and wet-mcp (legacy alias). wet search and other server-backed subcommands talk to the running server over streamable HTTP; with no server running they exit 2 and print a hint to run wet server start. wet docs reindex|import|reembed work without a server.

Cau hinh quan trong

• Python 3.13 bat buoc -- 3.14+ KHONG tuong thich do SearXNG
• requires-python = "==3.13.*" trong pyproject.toml
• Ruff: line-length 88, target py313, rules E/F/W/I/UP/B/C4, ignore E501
• ty: lenient (unresolved-import, unresolved-attribute, possibly-missing-attribute all "ignore")

Pytest

• asyncio_mode = "auto" -- KHONG can @pytest.mark.asyncio
• Default timeout: 30 seconds per test
• Integration/live/full tests excluded by default
• addopts = "-m 'not integration and not live and not full'" trong pyproject.toml

Server, bind & auth

• One transport: the HTTP MCP server (wet server start, or python -m wet.server).
• Bind: [server] host / port in ~/.wet/config.toml; WET_HOST / WET_PORT env override (container deployments).
• Per-port lifecycle lock (LifecycleLock("wet", port), lock files under ~/.config/mcp/locks) -- one instance per port, several ports can run side by side.
• [server] auth modes:
  • no-auth -- loopback bind only; the server refuses to start on a non-loopback host.
  • token -- single bearer token; store only its hash (wet token hash <token>).
  • multi -- per-user tokens from [server] users_file (default ~/.wet/users.toml, see wet users path); data is namespaced per sub under ~/.wet/subs/<sub>/.
• Client config points at the HTTP endpoint (see the manual example below).

Env vars

• KHONG co prefix ung dung (day la open-source MCP server)
• LLM/Embed/Rerank/Chat: per-task provider cells [models.embed|rerank|chat|jev_score] in ~/.wet/config.toml (each base_url + api_key + model, OpenAI-spec). OpenRouter is the pre-wired default for every cell.
  • Credentials: api_key in the cell, or HULL_<TASK>_API_KEY env injection (HULL_EMBED_API_KEY / HULL_RERANK_API_KEY / HULL_CHAT_API_KEY / HULL_JEV_SCORE_API_KEY — env wins over the file).
  • OPENROUTER_API_KEY env serves any cell that keeps the OpenRouter default base_url — one key can power every task.
  • Empty embed/rerank cell -> local ONNX (fastretrieval); unkeyed chat cell -> LLM features off.
• Local model cache: FASTRETRIEVAL_CACHE_PATH.
• Search backends (SEARCH_BACKENDS CSV, runtime fallback — thử từng backend, rơi xuống backend kế khi error/empty): searxng (self-host) | tavily | brave | exa | kagi (keyed, KAGI_API_KEY) | openrouter (OPENROUTER_API_KEY, openrouter:web_search server tool) | firecrawl (key optional — không key vẫn thử keyless, FIRECRAWL_API_KEY) | duckduckgo | startpage (credential-free HTML scrape, uvx-safe; datacenter IP có thể bị bot-challenge → chain tự advance).
• SearXNG: WET_AUTO_SEARXNG (default true), SEARXNG_URL (external mode)
• Browser render backends (headless leg of extract): BROWSER_BACKENDS (CSV escalation chain: native | browserless | invisible; empty = native), BROWSERLESS_URL/BROWSERLESS_TOKEN (self-host browserless), STEALTHFOX_BINARY (path to a patched Firefox build for the invisible tier — skips the engine download, seal still verified), CAPSOLVER_API_KEY (optional key-gated captcha tier)
• Identity layer (one coherent person across the chain): WET_IDENTITY_SEED (int pin; default 0 = derive once and persist per namespace under ~/.wet/subs/<sub>/identity.json), IDENTITY_PROFILE_DIR (persistent browser profile dir; default ~/.wet/subs/<sub>/profiles). Requires the optional hull-core[identity] extra (invisible-core); without it strategies keep legacy behavior (warn once). First identity build may download a GeoIP mmdb (egress) and resolve locale/timezone from the egress IP — timezone_mismatch behind a proxy falls back to en-US/UTC, never raises to MCP callers
• Robots policy: RESPECT_ROBOTS_TXT (default false); set true to enforce robots.txt for the ScrapingAgent extract chain and Crawl4AI crawl, sitemap, and list_media actions
• Disable-local toggles (skip heavy in-process fallbacks per capability): DISABLE_LOCAL_BROWSER, DISABLE_LOCAL_SEARCH, DISABLE_LOCAL_EMBED, DISABLE_LOCAL_RERANK
• Docs DB backup is an external concern (e.g. rclone of ~/.wet/); the server has no built-in sync.
• Secrets: skret SSM namespace /wet-mcp/prod (region ap-southeast-1)

Manual config example

wet server start            # binds [server] host/port from ~/.wet/config.toml

{
  "mcpServers": {
    "wet": {
      "type": "http",
      "url": "http://127.0.0.1:<port>/mcp"
    }
  }
}

With auth = "token" or "multi", add the client's bearer token header ("headers": {"Authorization": "Bearer <token>"}).

Release & Deploy

• Conventional Commits. Tag format: v{version}
• CD: workflow_dispatch, chon beta/stable
• Pipeline (cd.yml): benchmark-contract gate -> release (PSR v10) -> publish-pypi (uv publish) -> publish-mcp-registry -> sync-marketplace
• No container image and no Cloudflare deploy are published.

Pre-commit hooks

1. Ruff lint (--fix --target-version=py313) + format
2. ty type check
3. pytest (--tb=short -q --timeout=30)
4. Commit message: enforce feat/fix prefix

Luu y quan trong

• Lazy imports ben trong functions cho heavy deps va tranh circular deps
• MCP tools return error strings (return "Error: ...") -- KHONG raise exceptions
• Graceful fallback chains: Cloud -> Local, Tier 0 -> 1 -> 2 -> 3
• match action: cho tool action dispatch
• asyncio.to_thread() cho wrapping sync operations
• Embedding luu tai 768 dims (default). Doi embedding MODEL = doi vector space -> B2 identity guard CHAN boot (set REINDEX_ON_MODEL_CHANGE=true de re-embed). 768-dim chung chi giu table khong vo schema; KHONG cho mix vector tu 2 model khac nhau (cung dims van rac search).
• Renovate: Python upgrades DISABLED

Protocol & live tests

All suites spawn the real HTTP server (python -m wet.server) on an ephemeral loopback port against a tmp instance home; there is no external E2E driver.

• tests/test_protocol_gate.py -- protocol gate over the production HTTP transport, runs from the source tree (default suite).
• tests/test_live_protocol.py -- -m live (real server + stub SearXNG).
• tests/test_full_live.py -- -m full (real upstream services; keys from skret /wet-mcp/prod).
• tests/test_live_mcp.py -- standalone live script (defines no pytest tests; run it directly).

uv run pytest tests/test_protocol_gate.py
uv run pytest -m live
uv run pytest -m full
uv run python tests/test_live_mcp.py

References: ~/.claude/skills/mcp-dev/references/protocol-test-coverage.md, ~/.claude/skills/mcp-dev/references/secrets-skret.md (per-server credential layout), ~/.claude/skills/mcp-dev/references/multi-user-pattern.md (per-sub isolation).