gnosis-mcp

# Gnosis MCP — Full Reference

> Zero-config MCP server that makes your markdown docs searchable by AI agents.
> SQLite default, PostgreSQL optional. Works with any MCP client — Claude Code, Claude Desktop, Cursor, Zed, opencode, Windsurf, VS Code, Cline.
> PyPI: gnosis-mcp | CLI: gnosis-mcp | Import: gnosis_mcp

## Install

```bash
pip install gnosis-mcp               # SQLite (default, zero config)
pip install gnosis-mcp[embeddings]  # + Local ONNX semantic search (no API key)
pip install gnosis-mcp[postgres]     # + PostgreSQL support
pip install gnosis-mcp[web]          # + Web crawl (httpx + trafilatura)
```

## Quick Setup (SQLite)

```bash
gnosis-mcp ingest ./docs/   # Auto-creates DB + loads markdown
gnosis-mcp search "query"   # Test it works
gnosis-mcp serve             # Start MCP server
```

## Quick Setup (SQLite + Semantic Search)

```bash
pip install gnosis-mcp[embeddings]
gnosis-mcp ingest ./docs/ --embed   # Ingest + embed (downloads 23MB model on first run)
gnosis-mcp serve                    # Hybrid keyword+semantic search auto-activated
```

## Quick Setup (PostgreSQL)

```bash
export GNOSIS_MCP_DATABASE_URL="postgresql://user:pass@localhost:5432/mydb"
gnosis-mcp init-db          # Create tables (idempotent)
gnosis-mcp ingest ./docs/   # Load markdown files
gnosis-mcp check            # Verify connection + schema
gnosis-mcp serve
```

## Editor Config

The same `mcpServers` JSON works in most clients. Add it to the appropriate config file:

| Client | Config File |
|--------|------------|
| Claude Code | `.mcp.json` at the project root |
| Claude Desktop | `claude_desktop_config.json` — `~/Library/Application Support/Claude/` (macOS), `%APPDATA%\Claude\` (Windows) |
| Cursor | `.cursor/mcp.json` |
| Zed | `settings.json` (key is `context_servers`, not `mcpServers`) |
| opencode | `opencode.json` or `~/.config/opencode/opencode.json` (key is `mcp`, command as an array) |
| VS Code (Copilot) | `.vscode/mcp.json` (note: uses `"servers"` not `"mcpServers"`) |
| Windsurf | `~/.codeium/windsurf/mcp_config.json` |
| JetBrains | Settings > Tools > AI Assistant > MCP Servers |
| Cline | Cline MCP settings panel |

SQLite (no env needed):

```json
{
  "mcpServers": {
    "docs": {
      "command": "gnosis-mcp",
      "args": ["serve"]
    }
  }
}
```

PostgreSQL:

```json
{
  "mcpServers": {
    "docs": {
      "command": "gnosis-mcp",
      "args": ["serve"],
      "env": {
        "GNOSIS_MCP_DATABASE_URL": "postgresql://user:pass@localhost:5432/mydb"
      }
    }
  }
}
```

Zed — `context_servers`:

```json
{
  "context_servers": {
    "docs": {
      "command": "gnosis-mcp",
      "args": ["serve"]
    }
  }
}
```

opencode — `mcp`, command as an array:

```json
{
  "$schema": "https://opencode.ai/config.json",
  "mcp": {
    "docs": {
      "type": "local",
      "command": ["gnosis-mcp", "serve"],
      "enabled": true
    }
  }
}
```

Zed takes the same `env` block as the `mcpServers` clients; opencode calls that key `environment`.

## Backends

| | SQLite (default) | SQLite + embeddings | PostgreSQL |
|---|---|---|---|
| Install | `pip install gnosis-mcp` | `pip install gnosis-mcp[embeddings]` | `pip install gnosis-mcp[postgres]` |
| Config | Nothing | Nothing | Set `DATABASE_URL` |
| Search | FTS5 keyword (BM25) | Hybrid keyword+semantic (RRF) | tsvector + pgvector hybrid |
| Embeddings | None | Local ONNX (23MB, no API) | Any provider + HNSW index |
| Multi-table | No | No | Yes (UNION ALL) |

Auto-detection: `DATABASE_URL` set to `postgresql://...` -> PostgreSQL. Not set -> SQLite. Override: `GNOSIS_MCP_BACKEND=sqlite|postgres`.

The `[embeddings]` extra installs: onnxruntime, tokenizers, numpy, sqlite-vec. Default model: MongoDB/mdbr-leaf-ir (23M params, 23MB quantized). Model auto-downloads from HuggingFace via stdlib urllib on first use. Customize with `GNOSIS_MCP_EMBED_MODEL`.

## Tools (9)

Six read tools are always advertised; the three write tools appear in `tools/list` only when writes are enabled.

### Read Tools (always available)

1. **search_docs(query, category?, limit?, query_embedding?, rerank?)** — Search docs using keyword (FTS5/tsvector) or hybrid semantic+keyword search. Returns `highlight` field with matched terms in `<mark>` tags.
   - query: string (required) — search text
   - category: string (optional) — filter by category
   - limit: int (default 5, max configurable) — result count
   - query_embedding: list[float] (optional) — pre-computed embedding for hybrid search (PostgreSQL)
   - rerank: bool (optional) — cross-encoder re-score of the top-N candidates (requires the `[reranking]` extra)

2. **get_doc(path, max_length?)** — Get full document by file path. Reassembles chunks in order.
   - path: string (required) — e.g. "guides/quickstart.md"
   - max_length: int (optional) — truncate at N characters

3. **search_git_history(query, author?, since?, until?, file_path?, limit?)** — Search indexed git commit history (the `git-history` category). Same result shape as `search_docs`.
   - query: string (required) — search text (commit messages, file names, authors)
   - author: string (optional) — filter by author name or email substring
   - since / until: string (optional) — date window (YYYY-MM-DD)
   - file_path: string (optional) — restrict to one file's history
   - limit: int (default 5) — result count

4. **get_related(path, depth?, relation_type?, include_titles?)** — Find related documents via bidirectional link graph.
   - path: string (required)
   - depth: int (default 1) — traversal depth (1 = direct neighbors, max 3)
   - relation_type: string (optional) — e.g. `relates_to`, `content_link`, `git_co_change`
   - include_titles: bool (default false) — include title and category per result

5. **get_context(topic?, limit?, category?)** — Usage-weighted context primer. Without `topic`: most-accessed documents plus corpus stats. With `topic`: search results enriched with access counts.
   - limit: int (default 10) — documents to return
   - category: string (optional) — filter by category

6. **get_graph_stats(category?)** — Knowledge graph topology: orphans, hubs, relation distribution, node and edge counts.

### Write Tools (require GNOSIS_MCP_WRITABLE=true)

7. **upsert_doc(path, content, title?, category?, audience?, tags?, embeddings?)** — Insert or replace document. Auto-chunks at paragraph boundaries. Optional `embeddings` accepts pre-computed vectors (one per chunk).

8. **delete_doc(path)** — Delete document, its chunks, and links.

9. **update_metadata(path, title?, category?, audience?, tags?)** — Update metadata fields on all chunks.

## Resources (3)

- **gnosis://docs** — List all documents with title, category, chunk count
- **gnosis://docs/{path}** — Read document content by path
- **gnosis://categories** — List categories with document counts

## REST API (v0.10.0+)

Enable native HTTP endpoints alongside MCP on the same port. Uses Starlette (bundled with mcp>=1.20, no new dependencies).

Enable: `gnosis-mcp serve --transport streamable-http --rest`
Or set: `GNOSIS_MCP_REST=true`

| Endpoint | Description |
|----------|-------------|
| `GET /health` | `{"status", "version", "backend", "docs", "chunks", "search_stats"}` — `search_stats` is `{"total", "misses", "hybrid", "keyword"}` |
| `GET /api/search?q=&limit=&category=` | `{"results": [...], "query", "count"}` — auto-embeds with local provider |
| `GET /api/docs/{path}` | `{"title", "content", "category", "audience", "tags", "chunks"}` |
| `GET /api/docs/{path}/related` | `{"results": [{"related_path", "relation_type", "direction"}]}` |
| `GET /api/categories` | `[{"category", "docs"}]` |

| Env Variable | Description |
|---|---|
| `GNOSIS_MCP_REST` | `true`/`1`/`yes` to enable REST API |
| `GNOSIS_MCP_CORS_ORIGINS` | `*` or comma-separated origins (e.g. `http://localhost:5174`) |
| `GNOSIS_MCP_API_KEY` | Bearer token required in `Authorization: Bearer <key>` |

## Configuration (Environment Variables)

All settings via GNOSIS_MCP_* environment variables. Nothing required for SQLite.

### Core Settings
- GNOSIS_MCP_DATABASE_URL — PostgreSQL URL or SQLite file path (default: SQLite at ~/.local/share/gnosis-mcp/docs.db)
- GNOSIS_MCP_BACKEND — Force backend: auto, sqlite, postgres (default: auto)
- GNOSIS_MCP_SCHEMA — Database schema, PostgreSQL only (default: public)
- GNOSIS_MCP_CHUNKS_TABLE — Chunks table name, comma-separated for multi-table on PG (default: documentation_chunks)
- GNOSIS_MCP_LINKS_TABLE — Links table name (default: documentation_links)
- GNOSIS_MCP_SEARCH_FUNCTION — Custom search function, PostgreSQL only (default: none)
- GNOSIS_MCP_EMBEDDING_DIM — Embedding vector dimension for init-db (default: 1536)
- GNOSIS_MCP_POOL_MIN — Min pool connections, PostgreSQL only (default: 1)
- GNOSIS_MCP_POOL_MAX — Max pool connections, PostgreSQL only (default: 3)
- GNOSIS_MCP_WRITABLE — Enable write tools: true/1/yes (default: false)
- GNOSIS_MCP_WEBHOOK_URL — URL to POST on doc changes (default: none)

### Embedding
- GNOSIS_MCP_EMBED_PROVIDER — Embedding provider: openai, ollama, custom, or local (default: none — nothing is inferred, so with it unset POST /v1/embed returns 503 "No default URL for provider None" even with the [embeddings] extra installed). Set it explicitly: the published Docker image sets it to local, a plain pip install does not
- GNOSIS_MCP_EMBED_MODEL — Embedding model name (default: text-embedding-3-small for remote, MongoDB/mdbr-leaf-ir for local)
- GNOSIS_MCP_EMBED_DIM — Embedding dimension for local Matryoshka truncation and vec0 table width (default: 384)
- GNOSIS_MCP_EMBED_API_KEY — API key for embedding provider (default: none)
- GNOSIS_MCP_EMBED_URL — Custom embedding endpoint URL (default: none)
- GNOSIS_MCP_EMBED_BATCH_SIZE — Chunks per embedding batch, min 1 (default: 50)

### Tuning
- GNOSIS_MCP_CONTENT_PREVIEW_CHARS — Characters in search previews, min 50 (default: 200)
- GNOSIS_MCP_CHUNK_SIZE — Max chars per chunk, min 500 (default: 2000)
- GNOSIS_MCP_SEARCH_LIMIT_MAX — Max search result limit, min 1 (default: 20)
- GNOSIS_MCP_WEBHOOK_TIMEOUT — Webhook timeout seconds, min 1 (default: 5)
- GNOSIS_MCP_TRANSPORT — Server transport: stdio, sse, or streamable-http (default: stdio)
- GNOSIS_MCP_HOST — Bind address for HTTP transports (default: 127.0.0.1)
- GNOSIS_MCP_PORT — Port for HTTP transports (default: 8000)
- GNOSIS_MCP_LOG_LEVEL — Logging: DEBUG/INFO/WARNING/ERROR/CRITICAL (default: INFO)

### Column Overrides (for existing tables with non-standard names)
- GNOSIS_MCP_COL_FILE_PATH (default: file_path)
- GNOSIS_MCP_COL_TITLE (default: title)
- GNOSIS_MCP_COL_CONTENT (default: content)
- GNOSIS_MCP_COL_CHUNK_INDEX (default: chunk_index)
- GNOSIS_MCP_COL_CATEGORY (default: category)
- GNOSIS_MCP_COL_AUDIENCE (default: audience)
- GNOSIS_MCP_COL_TAGS (default: tags)
- GNOSIS_MCP_COL_EMBEDDING (default: embedding)
- GNOSIS_MCP_COL_TSV (default: tsv)
- GNOSIS_MCP_COL_SOURCE_PATH (default: source_path)
- GNOSIS_MCP_COL_TARGET_PATH (default: target_path)
- GNOSIS_MCP_COL_RELATION_TYPE (default: relation_type)

## Custom Search Function (PostgreSQL)

Your function must accept:
```sql
(p_query_text text, p_categories text[], p_limit integer)
```

And return columns: file_path, title, content, category, combined_score.

Optionally, your function can also accept `p_embedding vector(N)` for hybrid search. Gnosis will try passing it automatically when `query_embedding` is provided.

## CLI

```
gnosis-mcp ingest <path> [--dry-run] [--force] [--embed]   # Load files (.md/.txt/.ipynb/.toml/.csv/.json)
gnosis-mcp crawl <url> [--sitemap] [--depth N] [--include] [--exclude] [--dry-run] [--force] [--embed]
gnosis-mcp serve [--transport stdio|sse|streamable-http] [--host H] [--port P] [--ingest PATH] [--watch PATH]
                 [--rest] [--search-limit-max N] [--content-preview-chars N]
gnosis-mcp search <query> [-n LIMIT] [-c CAT] [--embed]    # Search (--embed for hybrid semantic+keyword)
gnosis-mcp stats                                           # Show document/chunk/embedding counts
gnosis-mcp check                                           # Verify connection + sqlite-vec status
gnosis-mcp embed [--provider P] [--model M] [--dry-run]    # Backfill embeddings (auto-detects local provider)
gnosis-mcp init-db [--dry-run]                             # Create tables (or preview SQL)
gnosis-mcp export [-f json|markdown|csv] [-c CAT]          # Export documents
gnosis-mcp ingest-git <repo> [--since S] [--max-commits N] [--include P] [--exclude P] [--dry-run] [--embed] [--merges]
gnosis-mcp diff <path>                                     # Show what would change on re-ingest
gnosis-mcp --version                                       # Show version
```

## Git History Ingestion

`gnosis-mcp ingest-git <repo-path>` converts git commit history into searchable markdown documents. Zero new dependencies — uses `git log` via subprocess.

```bash
gnosis-mcp ingest-git .                                      # Current repo, all files
gnosis-mcp ingest-git /path/to/repo --since 6m               # Last 6 months only
gnosis-mcp ingest-git . --include "src/*" --max-commits 5    # Filtered + limited
gnosis-mcp ingest-git . --dry-run                            # Preview without ingesting
gnosis-mcp ingest-git . --embed                              # Embed for semantic search
```

- One markdown document per file with meaningful commit history
- Each commit becomes an H2 section with date, author, subject, body
- Stored as `git-history/<file-path>` to avoid collision with source docs
- Category set to `git-history` for scoped searches (`search_docs(query, category="git-history")`)
- Auto-links to source file paths via `relates_to` graph
- Content hashing for incremental re-ingest (skips files with unchanged history)
- `--merges` flag includes merge commits (skipped by default)

## Web Crawl

`gnosis-mcp crawl <url>` fetches and ingests documentation from any website. Requires the `[web]` extra (`pip install gnosis-mcp[web]`).

```bash
gnosis-mcp crawl https://docs.stripe.com/ --sitemap           # Crawl via sitemap
gnosis-mcp crawl https://fastapi.tiangolo.com/ --depth 2      # BFS link crawl with depth limit
gnosis-mcp crawl https://docs.python.org/ --dry-run            # Preview discovered URLs
gnosis-mcp crawl https://docs.example.com/ --sitemap --embed   # Crawl + embed for semantic search
```

- Sitemap.xml discovery (`--sitemap`) and BFS link crawling (`--depth N`)
- robots.txt compliance — respects `Disallow` rules
- ETag/Last-Modified HTTP caching for incremental re-crawl (304 Not Modified)
- Content hashing: skips unchanged pages on re-crawl
- URL path filtering with `--include` and `--exclude` glob patterns
- Rate-limited concurrent fetching (5 concurrent, 0.2s delay)
- SSRF protection: blocks private/internal IPs and checks redirect targets
- Crawled pages stored with URL as `file_path`, hostname as `category`
- Force re-crawl with `--force`, dry run with `--dry-run`, embed with `--embed`

## Ingest

`gnosis-mcp ingest <path>` scans a file or directory for supported files (`.md`, `.txt`, `.ipynb`, `.toml`, `.csv`, `.json`) and loads them into the database. Non-markdown formats are auto-converted using Python stdlib only — zero extra dependencies.

- Chunks by H2 headers (H3/H4 for oversized sections). Never splits inside fenced code blocks or tables
- Parses YAML-like frontmatter for title, category, audience, tags
- Auto-linking: `relates_to` in frontmatter populates the links table (supports comma-separated and YAML list, skips glob patterns)
- Content hashing: skips unchanged files on re-run
- Watch mode: `gnosis-mcp serve --watch ./docs/` auto-re-ingests on file changes (mtime polling + debounce + auto-embed)
- Category inferred from parent directory name
- Title extracted from first H1 heading
- Skips tiny files (<50 chars)
- Use `--dry-run` to preview without writing

## Architecture

```
src/gnosis_mcp/
├── backend.py         # DocBackend Protocol + create_backend() factory
├── pg_backend.py      # PostgreSQL backend — asyncpg, tsvector, pgvector, UNION ALL
├── sqlite_backend.py  # SQLite backend — aiosqlite, FTS5 MATCH + bm25()
├── sqlite_schema.py   # SQLite DDL — tables, FTS5, triggers, indexes
├── config.py          # GnosisMcpConfig frozen dataclass, backend auto-detection
├── db.py              # Backend lifecycle + FastMCP lifespan
├── server.py          # FastMCP server: 9 tools (6 read + 3 write-gated) + 3 resources + webhook helper
├── ingest.py          # File scanner + converters: multi-format, smart chunking (H2/H3/H4), hashing
├── crawl.py           # Web crawler — sitemap/BFS discovery, robots.txt, ETag caching, trafilatura
├── parsers/           # Non-file ingest sources
│   └── git_history.py # Git log → markdown documents per file (commit parsing, grouping, rendering)
├── watch.py           # File watcher: mtime polling, auto-re-ingest on changes
├── schema.py          # PostgreSQL DDL — tables, indexes, HNSW, hybrid search functions
├── embed.py           # Embedding sidecar: provider abstraction (openai/ollama/custom/local)
├── local_embed.py     # Local ONNX embedding engine — stdlib urllib model download
└── cli.py             # argparse CLI: serve, init-db, ingest, ingest-git, crawl, search, embed, stats, export, diff, check
```

Default install deps: mcp + aiosqlite. Optional: asyncpg (via `[postgres]` extra), onnxruntime + tokenizers + numpy + sqlite-vec (via `[embeddings]` extra), httpx + trafilatura (via `[web]` extra). Model download uses stdlib urllib (no huggingface-hub dependency).

## Performance

9,463 QPS on 100 docs (300 chunks), 471 QPS on 10,000 docs (30,000 chunks) — SQLite FTS5 keyword. p95 under 6 ms at 10K corpus. End-to-end through the MCP stdio protocol: 8.7 ms mean, 13.0 ms p95 (v0.13.3, after the mcp SDK 1.27 transport upgrade). 862 tests, 10 RAG eval cases (Hit@5 = 1.00, MRR = 1.00, Precision@5 = 0.75). Install size: ~23MB with `[embeddings]` (ONNX model), ~5MB base. Benchmarks: `gnosis-mcp eval`, `python tests/bench/bench_search.py`, `python tests/bench/bench_rag.py`, `python tests/bench/bench_mcp_e2e.py`. See `docs/benchmarks.md` for methodology.

## License

MIT — https://github.com/nicholasglazer/gnosis-mcp