InfoMesh — Copilot Instructions

Project Overview

InfoMesh is a fully decentralized P2P search engine designed exclusively for LLMs. It crawls, indexes, and searches the web via a peer-to-peer network and exposes results through MCP (Model Context Protocol) — no human-facing UI required.

The static EN/KO project website is in site/, deployed separately by .github/workflows/pages.yml. It is a project introduction and MCP setup guide, not a hosted search interface. Keep asset URLs relative, client-specific MCP configuration correct, EN/KO text paired, and keyboard/reduced-motion support. Its network illustration is conceptual; never imply live peer counts or privacy guarantees for distributed queries. Validate desktop/mobile interactions and run node --check site/app.js plus actionlint after website/workflow changes. Keep social-preview metadata in the initial HTML with absolute HTTPS image URLs, matching PNG dimensions and alt text for OG/Twitter. Verify with uv run pytest tests/test_site.py -q; external previews require public deployment.

Mission: InfoMesh does not compete with commercial search providers. These companies serve human search at massive scale with ads-based monetization. InfoMesh provides minimal, sufficient search capabilities for LLMs — for free, via MCP — democratizing real-time web access for AI assistants without per-query billing. It is a community-driven public utility, complementary to existing search providers.

Core Principles

Tech Stack

Project Structure

infomesh/
├── pyproject.toml
├── infomesh/
│   ├── __init__.py          # Package root (__version__)
│   ├── __main__.py          # CLI entry point (Click app)
│   ├── config.py            # Configuration management (TOML + env vars)
│   ├── services.py          # Central AppContext + index_document orchestration
│   ├── db.py                # SQLite store base class (WAL, lifecycle)
│   ├── hashing.py           # SHA-256 hashing utilities (content_hash, short_hash)
│   ├── security.py          # SSRF protection (URL validation, IP filtering)
│   ├── types.py             # Shared Protocol types (PEP 544)
│   ├── cli/                 # Click CLI commands
│   │   ├── __init__.py      #   CLI app registration
│   │   ├── config.py        #   `infomesh config show/set` commands
│   │   ├── crawl.py         #   `infomesh crawl`, `mcp`, `dashboard` commands
│   │   ├── index.py         #   `infomesh index stats/export/import` commands
│   │   ├── keys.py          #   `infomesh keys export/rotate` commands
│   │   ├── search.py        #   `infomesh search` command
│   │   └── serve.py         #   `infomesh start/stop/status/serve` commands
│   ├── p2p/                 # P2P network layer
│   │   ├── node.py          #   Peer main process
│   │   ├── dht.py           #   Kademlia DHT
│   │   ├── keys.py          #   Ed25519 key pair management + rotation
│   │   ├── routing.py       #   Query routing (latency-aware)
│   │   ├── replication.py   #   Document/index replication
│   │   ├── protocol.py      #   Message protocol definitions (msgpack)
│   │   ├── message_auth.py  #   P2P message authentication (SignedEnvelope, replay protection)
│   │   ├── transport.py     #   Bounded v2 signed stream exchange alongside legacy v1
│   │   ├── peer_profile.py  #   Peer latency tracking & bandwidth classification
│   │   ├── peer_store.py    #   Persistent peer cache (SQLite, survives restarts)
│   │   ├── pex.py           #   PEX (Peer Exchange) gossip protocol for peer discovery
│   │   ├── sybil.py         #   Sybil defense (PoW node ID + subnet limiting)
│   │   ├── load_guard.py    #   NodeLoadGuard (QPM + concurrency limiting)
│   │   ├── mdns.py          #   mDNS local peer discovery (multicast UDP)
│   │   ├── throttle.py      #   BandwidthThrottle (token-bucket upload/download)
│   │   └── index_submit.py  #   Enterprise split: DMZ crawler → private indexer
│   ├── crawler/             # Web crawler
│   │   ├── worker.py        #   Async crawl workers
│   │   ├── scheduler.py     #   URL assignment (DHT-based)
│   │   ├── parser.py        #   HTML → text extraction
│   │   ├── robots.py        #   robots.txt compliance + sitemap + crawl-delay
│   │   ├── dedup.py         #   Deduplication pipeline (URL, SHA-256, SimHash)
│   │   ├── simhash.py       #   SimHash near-duplicate detection
│   │   ├── seeds.py         #   Seed URL management & category selection
│   │   ├── recrawl.py       #   Adaptive auto-recrawl scheduler
│   │   ├── url_assigner.py  #   DHT-based URL auto-assignment
│   │   └── crawl_loop.py    #   Continuous seed-and-crawl loop (extracted from services.py)
│   ├── index/               # Search index
│   │   ├── local_store.py   #   SQLite FTS5 local index
│   │   ├── vector_store.py  #   ChromaDB vector index
│   │   ├── distributed.py   #   DHT inverted-index publish/query
│   │   ├── ranking.py       #   BM25 + freshness + trust + authority scoring
│   │   ├── link_graph.py    #   Link graph + domain authority (PageRank-style)
│   │   ├── snapshot.py      #   Index snapshot export/import (zstd-compressed)
│   │   └── commoncrawl.py   #   Common Crawl data importer
│   ├── search/              # Search engine
│   │   ├── query.py         #   Query parsing + distributed orchestration
│   │   ├── merge.py         #   Multi-node result merging
│   │   ├── cache.py         #   Query result LRU cache (TTL + auto-expiry)
│   │   ├── reranker.py      #   LLM re-ranking (prompt-based post-ranking)
│   │   ├── cross_validate.py #  Query result cross-validation
│   │   ├── formatter.py     #   Search result text formatting (CLI/MCP/dashboard)
│   │   ├── feedback.py      #   Implicit search quality signals (SQLite-backed)
│   │   ├── cjk.py           #   CJK tokenization (bigram/trigram for Chinese/Japanese/Korean)
│   │   └── passage.py       #   Passage extraction, best-snippet selection, highlighting
│   ├── mcp/                 # MCP server (Model Context Protocol) — SRP: 4 modules
│   │   ├── server.py        #   Thin wiring: Server creation, tool dispatch, stdio/HTTP runners
│   │   ├── tools.py         #   Tool schema definitions, filter extraction, API key check
│   │   ├── handlers.py      #   All handle_* functions (search, fetch, crawl, batch, etc.)
│   │   └── session.py       #   SearchSession, AnalyticsTracker, WebhookRegistry
│   ├── api/                 # Local admin API
│   │   ├── local_api.py     #   FastAPI (health, status, config, index stats, credits, metrics, /dashboard)
│   │   └── extensions.py    #   API extensions (OpenAPI spec, Prometheus metrics)
│   ├── credits/             # Incentive system
│   │   ├── types.py         #   ActionType, CreditState, dataclasses, constants
│   │   ├── ledger.py        #   SQLite-backed credit ledger (imports types from types.py)
│   │   ├── sync.py          #   Cross-node credit sync (CreditSyncStore, CreditSyncManager)
│   │   ├── farming.py       #   Credit farming detection (probation, anomaly detection)
│   │   ├── scheduling.py    #   Energy-aware scheduling (off-peak LLM bonus + TZ verification)
│   │   ├── timezone_verify.py #   Off-peak timezone verification (IP cross-check)
│   │   ├── verification.py  #   P2P credit verification (signed entries + Merkle proofs)
│   │   └── github_identity.py #   GitHub email resolution for cross-node credits
│   ├── trust/               # Trust & integrity
│   │   ├── attestation.py   #   Content attestation chain (signing, verification, Merkle root)
│   │   ├── audit.py         #   Random audit system + Merkle proof audits
│   │   ├── merkle.py        #   Merkle Tree (index-wide integrity, membership proofs)
│   │   ├── reputation.py    #   LLM reputation-based trust (EMA quality tracking, grade system)
│   │   ├── scoring.py       #   Unified trust score computation
│   │   ├── detector.py      #   Farming + trust unified detector
│   │   ├── dmca.py          #   DMCA takedown propagation
│   │   └── gdpr.py          #   GDPR distributed deletion records
│   ├── summarizer/          # Local LLM summarization
│   │   ├── engine.py        #   LLM backend abstraction (ollama, llama.cpp, vllm)
│   │   ├── peer_handler.py  #   Inter-peer summarization request handling
│   │   └── verify.py        #   Summary verification (key-fact anchoring, NLI)
│   ├── dashboard/           # Console app UI (Textual TUI)
│   │   ├── app.py           #   DashboardApp (main Textual Application)
│   │   ├── bgm.py           #   BGM player (background music via mpv/ffplay)
│   │   ├── text_report.py   #   Rich-based text report (non-interactive fallback)
│   │   ├── data_cache.py    #   Dashboard data caching layer
│   │   ├── utils.py         #   Dashboard utility functions (format_uptime, format_bytes)
│   │   ├── dashboard.tcss   #   Textual CSS stylesheet
│   │   ├── screens/         #   Tab panes (overview, crawl, search, network, credits, settings)
│   │   └── widgets/         #   Reusable widgets (sparkline, bar_chart, resource_bar, live_log)
│   ├── resources/           # Resource governance
│   │   ├── profiles.py      #   Predefined resource profiles (minimal/balanced/contributor/dedicated)
│   │   ├── governor.py      #   Dynamic resource throttling (CPU/memory monitoring, degrade levels)
│   │   ├── preflight.py     #   Pre-startup disk space + network connectivity checks
│   │   └── port_check.py    #   Network port availability check
│   └── compression/         # Data compression
│       └── zstd.py          #   zstd compression with dictionary support
│   ├── search/              # Search engine (extended in v0.2.0)
│   │   ├── nlp.py           #   NLP query processing (stop-words, synonyms, spelling)
│   │   ├── facets.py        #   Search facets (domain, language, date)
│   │   ├── rag.py           #   RAG output, answer extraction, fact checking
│   │   └── explain.py       #   Score breakdowns for search results
│   ├── crawler/             # Crawler extensions (v0.2.0)
│   │   ├── pdf.py           #   PDF text extraction
│   │   ├── structured.py    #   JSON-LD, OpenGraph, meta tag parsing
│   │   ├── rss.py           #   RSS/Atom feed discovery and parsing
│   │   ├── content_extract.py #  Code block and table extraction
│   │   ├── diff.py          #   Content change detection
│   │   └── lang_detect.py   #   Language detection (script + word frequency)
│   ├── sdk/                 # Python SDK
│   │   └── client.py        #   InfoMeshClient (sync/async)
│   ├── integrations/        # LLM framework integrations
│   │   ├── langchain.py     #   LangChain InfoMeshRetriever
│   │   ├── llamaindex.py    #   LlamaIndex InfoMeshReader
│   │   └── haystack.py      #   Haystack InfoMeshDocumentStore
│   ├── persistence/         # Persistent storage
│   │   └── store.py         #   SQLite key-value store
│   ├── observability/       # Monitoring
│   │   └── metrics.py       #   Prometheus metrics collector
│   ├── errors.py            #   Structured error hierarchy
│   ├── scalability.py       #   Batch ingest & horizontal scaling
│   ├── data_quality.py      #   Quality scoring & validation
│   ├── security_ext.py      #   API keys, RBAC, audit logging
│   ├── security_ops.py      #   API key rotation, audit logging (production)
│   ├── dx.py                #   Developer experience (plugin system)
│   ├── plugins.py           #   Hook-based plugin registry (10 hook points)
│   ├── slo.py               #   Service Level Objectives tracking (5 default SLOs)
│   ├── diagnostics.py       #   System diagnostics (`infomesh doctor`)
│   ├── benchmarks.py        #   Performance benchmarking (`infomesh bench`)
│   └── shutdown.py          #   Graceful shutdown (SIGTERM/SIGINT handlers)
├── examples/                # Python usage examples
│   ├── README.md            #   Examples index
│   ├── basic_search.py      #   Simple search demo
│   ├── crawl_and_search.py  #   Crawl + search workflow
│   ├── credit_status.py     #   Credit ledger inspection
│   ├── fetch_page.py        #   Single URL fetch
│   ├── hybrid_search.py     #   FTS5 + vector hybrid search
│   └── mcp_client.py        #   MCP client integration demo
├── bootstrap/
│   └── nodes.json           # Bootstrap node list
├── seeds/                   # Bundled seed URL lists
│   ├── tech-docs.txt        #   Technology documentation URLs
│   ├── academic.txt         #   Academic paper source URLs
│   ├── encyclopedia.txt     #   Encyclopedia URLs
│   ├── quickstart.txt       #   Quickstart seed URLs (curated starter set)
│   └── search-strategy.txt  #   Search strategy seeds
├── tests/                   # 67 test files (pytest + pytest-asyncio)
└── docs/                    # Documentation (EN + KO)

Coding Conventions

General

• Language: All source code, comments, docstrings, commit messages, and PR descriptions in English.
• No specific company/product names in comparisons: README, docs, and marketing text must never mention specific competing companies or products by name (e.g., no "Tavily", "Brave Search", "OpenAI charges…"). Use generic categories instead ("API-based search providers", "commercial search APIs", "other MCP servers"). This avoids legal risk, keeps the project neutral, and prevents the text from becoming outdated when products change. Mentioning MCP-compatible clients (VS Code, Claude Desktop, Cursor, etc.) as integration targets is allowed — that is factual compatibility documentation, not competitive comparison.
• Python version: 3.12+ — use modern syntax (match/case, type statement, StrEnum, TypeVar defaults).
• Async-first: All I/O-bound code must use async/await with asyncio. Never use blocking I/O in the event loop.
• ⚠️ trio exception: py-libp2p uses trio (not asyncio). All libp2p/P2P code must run under trio.run(). Use a _run_trio() wrapper for tests to avoid asyncio_mode=auto conflict. Phase 2 will need a trio↔asyncio bridge (e.g., anyio or trio-asyncio).
• Type hints: Required on all public functions and class attributes. Use from __future__ import annotations for forward references.

Single Responsibility Principle (SRP)

Every module, class, and function must have one clear responsibility.

• Modules: One module = one concern. Don't mix unrelated logic in a single file. If a module exceeds ~300 lines, consider splitting.
• Classes: Each class encapsulates one actor or concept. A class should have only one reason to change.
  • ✅ RobotsChecker — only checks robots.txt compliance.
  • ✅ Compressor — only handles zstd compression/decompression.
  • ❌ A class that both crawls pages AND indexes them.
• Functions: Each function does one thing. Avoid functions that handle orchestration AND business logic simultaneously.
  • Prefer composing small functions over writing long procedural blocks.
  • Extract repeated patterns (e.g., "crawl → index → optionally vector-index") into named helper functions.
• CLI commands: Thin wrappers that delegate to library code. No business logic in Click handlers — they should only parse arguments, call library functions, and format output.
• MCP tool handlers: Same as CLI — dispatch to service-layer functions, don't inline business logic.
• Dashboard panels: Read data from caches or public APIs. Never access private attributes (_conn, _db) of library classes.

When reviewing code, ask: "If I change X, what else breaks?" If the answer includes unrelated concerns, the code violates SRP and should be refactored.

Style & Formatting

• Formatter: ruff format (default settings, line length 88).
• Linter: ruff check with select = ["E", "F", "I", "UP", "B", "SIM"].
• Import order: stdlib → third-party → local (enforced by ruff/isort).
• Prefer pathlib.Path over os.path.

CI Failure Prevention (Lessons Learned)

The following errors have caused CI failures. Always check for these before committing:

Common pitfalls:

• Adding a new import at the end of an import block instead of in alphabetical order → I001.
• Writing Click help="..." strings that exceed 88 chars → E501. Split into help=("part1 " "part2").
• Copy-pasting code with f-strings but removing the interpolated variables → F541.
• Forgetting to remove debug import subprocess or import pdb → F401.

mypy Type Checking (Lessons Learned)

CI runs uv run mypy infomesh/ --ignore-missing-imports with strict = true. The following patterns have caused mypy failures:

Auto-Release Workflow (Lessons Learned)

The auto-release.yml workflow bumps version, tags, builds, and publishes. The following issues have caused failures:

No Private API Access in Consumers

Library classes (LocalStore, CreditLedger, etc.) expose public methods for data access. Never access private attributes like store._conn or ledger._conn in CLI, MCP, or dashboard code. If a needed query doesn't have a public API, add one to the library class first.

Shared Utilities — No Duplication

Utility functions must exist in exactly one place:

• Dashboard utilities (_format_uptime, _get_peer_id, _is_node_running, _read_p2p_status): use infomesh/dashboard/utils.py.
• Domain-extraction SQL: use LocalStore.get_top_domains() — don't inline raw SQL in dashboard code.
• Node status assembly (store stats + P2P status + credit stats): use services.py orchestration — don't duplicate across CLI, MCP, and dashboard.

Pre-commit Checks (Required)

Before every commit, both lint and format checks must pass:

# Lint check — must show 0 errors
uv run ruff check infomesh/ tests/

# Format check — must show "X files already formatted"
uv run ruff format --check .

# Build check — must produce .whl and .tar.gz without errors
uv build

# Test check — must pass on both Python 3.12 and 3.13
uv run pytest tests/ \
  --ignore=tests/test_vector.py \
  -x -q --tb=short

If lint errors are found, fix them before committing:

• Auto-fix safe issues: uv run ruff check infomesh/ tests/ --fix
• Auto-format lines: uv run ruff format .
• Manual fixes needed for: long help strings (split into multi-line), missing imports (F821), unused variables (F841)

CI enforces all four checks on Python 3.12 and 3.13. A commit that breaks lint, format, build, or tests will fail CI.

Git Commit & Push Policy

• NEVER run git commit or git push — the maintainer handles all commits and pushes manually.
• Copilot's responsibility ends at: code changes, lint/format fixes, and test validation.
• After completing work, report what was changed and confirm all checks pass. Do NOT commit or push.

GitHub Projects Workflow

All tasks are tracked on the "InfoMesh Roadmap" Kanban board (GitHub Projects).

Columns (Status field):

Workflow rules:

1. Before starting work: Check the board (gh project item-list) for prioritized tasks.
2. Starting a task:
   • If no Issue exists, create one (gh issue create) and add it to the project (gh project item-add).
   • Set the item's Status to "In Progress" (gh project item-edit).
3. Code complete:
   • Set Status to "In Review".
   • Report changes and confirm lint/test pass.
4. After maintainer commits & pushes:
   • Set Status to "Done".
5. One task at a time: Only one item should be "In Progress" at any moment (matches the todo-list rule of one in-progress item).
6. Link PRs to Issues: When the maintainer creates a PR, reference the Issue number (e.g., Closes #1).
7. Explicit issue closure: Author-confirmation automation accepts only whole-comment /close, /close resolved, or /close completed commands. Progress/verification wording must never imply closure before maintainer review.

When to create Issues:

Every non-trivial work item must have a GitHub Issue before coding begins. This includes:

Issue template (recommended structure):

## Problem / Motivation
<What's wrong or what's needed — 1-3 sentences>

## Proposed Solution
- [ ] Step 1
- [ ] Step 2
- [ ] Step 3

## Success Criteria
- <How to verify this is done>

## Related Code
- `infomesh/<module>/<file>.py`

Bug report specifics:

When reporting a bug, always include:

1. Reproduction steps — exact commands or code to trigger the bug.
2. Expected behavior — what should happen.
3. Actual behavior — what actually happens (include error messages/tracebacks).
4. Environment — Python version, OS, infomesh version (infomesh --version).

Issue lifecycle:

Discover bug / plan feature
  → gh issue create (with labels + body)
  → gh project item-add (add to Roadmap board)
  → Set Status: "Todo" or "In Progress"
  → Work on it (code + test + lint)
  → Set Status: "In Review"
  → Maintainer reviews, commits, pushes
  → Set Status: "Done"
  → gh issue close (with commit reference)

Prioritization guidelines:

Labels convention:

Naming

• Modules/packages: snake_case
• Classes: PascalCase
• Functions/methods/variables: snake_case
• Constants: UPPER_SNAKE_CASE
• Private members: single leading underscore _name

Error Handling

• Use specific exception types, not bare except:.
• Log errors with structlog or stdlib logging — never print() in library code.
• Network/IO failures must be retried with exponential backoff where appropriate.

Testing

• Framework: pytest with pytest-asyncio for async tests.
• Test files mirror source layout: infomesh/p2p/dht.py → tests/test_dht.py.
• Each public function/method should have at least one test.
• Use fixtures and factories over inline setup.

Dependencies & Package Management

• Package manager: uv — used for all dependency resolution, virtual environments, and project management.
• All dependencies declared in pyproject.toml under [project.dependencies].
• Dev dependencies under [dependency-groups] (PEP 735) or [project.optional-dependencies.dev].
• Pin minimum versions only (e.g., httpx>=0.27), not exact pins.
• Lock file: uv.lock — committed to the repository for reproducible builds.
• No requirements.txt, no pip — use uv commands:
  • uv sync — install all dependencies (creates .venv automatically).
  • uv sync --dev — install with dev dependencies.
  • uv add <package> — add a new dependency.
  • uv add --dev <package> — add a dev dependency.
  • uv run <command> — run a command within the project environment.
  • uv run pytest — run tests.
  • uv run infomesh start — run the application.

Documentation Sync (Required)

Every code change that affects user-facing behavior, API surface, or configuration must be accompanied by corresponding documentation updates. Do not consider a task complete until all relevant docs are updated.

Mandatory update targets:

Rules:

• Bilingual: All documentation exists in both English (docs/en/) and Korean (docs/ko/). Both must be updated simultaneously.
• copilot-instructions.md: This file is the single source of truth for AI assistants. Keep it synchronized with the actual codebase behavior.
• Commit message: Use the docs: prefix for documentation-only changes. When a feature commit includes doc updates, use feat: (the docs update is part of the feature).
• Checklist: Before marking a task complete, verify: (1) EN docs updated, (2) KO docs updated, (3) copilot-instructions updated if applicable.

Architecture Guidelines

P2P / DHT

• Large startup publications must checkpoint per document/keyword (AnyIO for shared code, Trio for queue admission); local DHT awaits may not yield. Verify inbound protocol responsiveness under publication load, not merely service active state, and retain a rollback backup on upgrades.
• GovernedHost contains request/cleanup exceptions per stream, including peer EOF, while propagating Trio cancellation; a disconnected peer must not tear down the host nursery. Signed DHT admission rejection uses bounded logging rather than a full traceback for each rejected publication.
• Publication's asyncio cancellation/timeout must cancel the Trio worker via its token, not merely abandon an executor future. Keep a Trio-side 60-second bound and retain admitted durable repair work. Default concrete advertisements prioritize the route-selected interface over bridge addresses; they still are not public endpoint discovery.
• Peer discovery uses typed libp2p IDs for peerstore lookup and caches only usable addresses. PEX reads complete frames with five-second exchanges and connects at most three discovered peers per round with three-second dial bounds; cache only successful dials. Retry cached peers even after bootstrap success and run the first PEX round on the first maintenance tick. Bind wildcard addresses are expanded to concrete interfaces for identify/status by p2p/addresses.py; network.announce_addresses optionally overrides them with up to 16 numeric TCP endpoints matching the real peer identity. This does not provide NAT traversal, automatic public-IP discovery or external reachability guarantees; mDNS is LAN-only and legacy remote discovery bugs require remote upgrades.
• The DHT is Kademlia-based. Node IDs are 160-bit. Distance = XOR metric.
• Two DHT use cases:
  1. Inverted index: hash(keyword) → list of (peer_id, doc_id, score) pointers, stored on nodes closest to the hash.
  2. Crawl coordination: hash(URL) → the node closest to the hash "owns" and crawls that URL.
• Keyword publications merge pointer lists instead of overwriting existing entries. Nodes republish existing local index documents at startup and publish newly crawled documents after local indexing so bootstrap-hosted content remains discoverable.
• Replication factor: N=3 minimum for all stored data.

Crawling

• Always respect robots.txt — implement strict opt-out compliance.
• Default politeness: ≤1 request/second per domain.
• Crawl-Delay: Honors the Crawl-delay directive in robots.txt. Per-domain delay is applied automatically and capped at 60 seconds.
• Sitemap discovery: Extracts Sitemap: URLs from robots.txt and automatically schedules discovered URLs for crawling.
• Canonical tag: Recognizes <link rel="canonical">. If a page declares a different canonical URL, the crawler skips indexing and schedules the canonical URL instead.
• Retry with backoff: Transient HTTP 5xx errors and network failures trigger up to 2 retries with exponential backoff (1s, 2s). SSRF-blocked URLs are never retried.
• Use trafilatura for content extraction. If trafilatura returns None, skip the page.
• Store raw text + metadata (title, URL, crawl timestamp, language).
• Seed strategy: Bundled curated seed lists by category (tech docs, academic, encyclopedia, etc.) + Common Crawl URL import + DHT-assigned URLs + user crawl_url() submissions + link following.
• Deduplication: 3-layer approach — URL normalization (canonical), exact dedup (SHA-256 content hash on DHT), near-dedup (SimHash, Hamming distance ≤ 3).
• Crawl lock: Before crawling, publish hash(url) = CRAWLING to DHT to prevent race conditions. Timeout after 5 minutes.
• SPA/JS rendering: Phase 0 focuses on static HTML. For JS-heavy pages, use js_required DHT tag to delegate to Playwright-capable nodes (Phase 4).
• Bandwidth limits: Default ≤5 Mbps upload / 10 Mbps download for P2P. Configurable via ~/.infomesh/config.toml. Max 5 concurrent crawl connections per node.
• crawl_url() rate limiting: 60 URLs/hr per node, 10 pending URLs/domain, depth unlimited by default (0=unlimited, configurable).
• Force re-crawl: crawl_url(url, force=True) bypasses all dedup checks (URL, content hash, near-duplicate) to re-crawl previously visited pages. CLI prompts user to re-crawl when URL is already seen.
• Same-domain scope: Link following is restricted to the same domain + path prefix as the starting URL. External links are discovered but not followed.
• Scheduler bounds: A full 10,000-entry queue rejects additions immediately. Domain tracking is capped at 50,000, with stale scans above 40,000 at most once per minute. Per-domain admission is serialized; cancelled queued waits requeue when possible, while cancelled direct admissions and completed crawls release pending counts exactly once. DHT transport failures permit local crawling, but explicit peer lock rejections do not.

Indexing

• Local keyword search: SQLite FTS5 with BM25 ranking.
• Local vector search: ChromaDB for semantic/embedding-based queries. Default model: all-MiniLM-L6-v2. Vector search is optional — FTS5 works standalone.
• Distributed index: publish keyword hashes to DHT after crawling; startup republish advertises existing local documents after restarts.
• index_document refreshes existing URLs with stable IDs via explicit upsert; direct LocalStore.add_document retains insert-only behavior unless update_existing=True. Text updates synchronize compressed content; metadata-only updates must not reindex FTS.
• Crawler roles own the recrawl queue independently of RSS. Every minute enqueue up to 50 due records, process up to five per cycle through domain admission, and update EMA-based intervals. Failed refreshes preserve cached content and advance retry time. Initial and priority crawls retain ETag/Last-Modified; conditional 304 refreshes preserve the body and document ID. Redirects discard conditional headers.
• Search cache keys include mode/backends/reranking and LocalStore.cache_token() for own/external writes. Cache hits still run per-request credits, analytics, session and quota processing. Distributed filters exclude remote records without required metadata.

Search Flow

1. Parse query → extract keywords. CJK queries auto-expand to bigrams.
2. If results sparse, auto-expand with synonyms from built-in dictionary.
3. Search local index first (target: <10ms).
4. Route keyword hashes via DHT to responsible nodes (target: ~500ms).
5. Merge local + remote results, rank by BM25 + freshness + trust.
6. Select best passage as snippet (not just first N chars).
7. Fetch full text for top-N results if needed (target: ~200ms).
8. Record implicit feedback signals (fetch/skip/cite).
9. Return via MCP → total latency target: ~1 second.

infomesh search follows the same local + peer search path by default when P2P is available. Use infomesh search --local or --local-only for offline local-index search.

MCP Tools

The MCP server exposes 5 consolidated tools (v0.3.0 — legacy names still supported for backward compatibility):

web_search parameters: query (required), top_k (default 5), recency_days, domain_allowlist/domain_blocklist, language (ISO 639-1), fetch_full_content, chunk_size (enables RAG output), rerank, answer_mode ("snippets"/"summary"/"structured"), local_only, explain (adds score breakdown).

Authentication: Set INFOMESH_API_KEY env var → all tool calls require api_key parameter.

Transports: stdio (default) or HTTP Streamable (infomesh mcp --http --port 8081).

JSON fetch_page may include source-versioned ai_summary with text, content_hash, model, verification (self_verified/cross_validated), peer_verifications, ai_generated=true and source_url. Cross-validation needs two distinct current replica peers; it is corroboration, not proof of independent computation or truth. Legacy remove_url atomically persists URL exclusion and an owner-withdrawal outbox; it is not a global legal takedown API.

Local LLM Summarization

• Nodes can optionally run a local LLM to generate summaries of crawled content.
• Summaries are stored alongside the full text in the local index and shared via DHT.
• Nodes with local LLM capability can also process summarization requests from other peers.
• Recommended models (in order of preference):
  1. Qwen 2.5 (3B/7B) — best multilingual quality, Apache 2.0 license.
  2. Llama 3.x (3B/8B) — strongest general-purpose, large community.
  3. Gemma 3 (4B/12B) — good size-to-performance ratio.
  4. Phi-4 (3.8B/14B) — strong reasoning/summarization.
• Supported runtimes: ollama (simplest), llama.cpp (lightweight CPU), vLLM (GPU throughput).
• Minimum spec: 3B Q4 quantized model on CPU with 8GB RAM.
• The summarizer module should abstract the LLM backend so different runtimes are interchangeable.
• Energy-aware scheduling: Nodes can configure their local timezone and off-peak electricity hours (e.g., 23:00–07:00). LLM-heavy tasks (batch summarization, processing peer requests) are preferentially scheduled during off-peak windows. Scheduler decisions carry is_off_peak; the ledger applies the 1.5x credit multiplier when LLM actions are recorded with that flag.
• Summary verification: 3-stage pipeline — (1) Self-verification via key-fact anchoring + NLI contradiction detection, (2) Cross-validation by replica nodes independently summarizing, (3) Reputation-based trust from verification history.
• Store content_hash alongside every summary so anyone can verify against the source text.

Credit System

Credits are tracked locally per peer — no blockchain.

Formula

C_earned = Σ (W_i × Q_i × M_i)

• W_i = resource weight for action type i (normalized to resource cost)
• Q_i = quantity performed
• M_i = time multiplier (1.0 default; 1.5 for LLM actions during off-peak hours)

Resource Weights

Weights are normalized so that crawling = 1.0 as the reference unit. All weights reflect approximate relative resource cost (CPU, bandwidth, storage).

Time Multiplier (M)

• All Base actions: M = 1.0 always.
• LLM actions during normal hours: M = 1.0.
• LLM actions during configured off-peak hours: M = 1.5.
• Off-peak window is set per node (default: 23:00–07:00 local time).
• The LLM scheduler marks off-peak decisions with is_off_peak; credit recording applies the 1.5x multiplier when LLM actions are recorded with that flag.

Search Cost

C_search = 0.1 / tier(contribution_score)

Fairness Guarantee

• A node doing only crawling (no LLM) at 10 pages/hr earns 10 credits/hr → 100 searches/hr at worst tier. Non-LLM participants are never resource-starved.
• LLM weights are capped so LLM-related earnings never exceed ~60% of a node's total credits. LLM is a bonus for the network, not a requirement for participation.
• Uptime weight (0.5/hr) rewards always-on nodes regardless of hardware capability.
• Search is never blocked — even with zero credits, nodes can still search (see Zero-Dollar Debt below).
• Nodes with higher contribution scores get higher trust and query routing priority.

Zero-Dollar Debt

When credits are exhausted (balance ≤ 0), the node enters a grace/debt cycle:

• Debt is measured in credits, not money. No credit card, no dollars, no subscription.
• Recovery: earn credits through normal contribution (crawling, hosting, uptime).
• When balance returns to positive, the debt state resets to NORMAL.
• Constants: GRACE_PERIOD_HOURS = 72.0, DEBT_COST_MULTIPLIER = 2.0.

Cross-Node Credit Sync

When a user runs InfoMesh on multiple devices with the same GitHub email, each device has its own local ledger. The credit sync system (infomesh/credits/sync.py) enables those nodes to discover each other and display aggregated credit stats.

• Protocol: PROTOCOL_CREDIT_SYNC = "/infomesh/credit-sync/1.0.0" with message types CREDIT_SYNC_ANNOUNCE (72) and CREDIT_SYNC_EXCHANGE (73).
• Privacy: Only SHA-256 hash of the email exchanged on wire (never plaintext).
• Flow: On connect → announce email_hash → if match → exchange signed summaries → store → aggregate.
• Periodic: Re-exchange every 300 seconds with known same-owner peers.
• Stale sweep: Summaries older than 72 hours purged automatically.
• DoS limit: Max 20 peer summaries stored per owner.
• Display: MCP status/credit_balance tools include a network sub-object when multiple same-owner nodes are synced. Dashboard and text report show network totals.

Content Integrity & Trust

• Content attestation: On crawl, compute SHA-256(raw_response) + SHA-256(extracted_text), sign with peer private key, publish to DHT.
• Random audits: ~1/hr per node. 3 audit nodes independently re-crawl a random URL and compare content_hash against the original node. Mismatch = trust penalty.
• Proof-of-Audit: Auditors must submit the actual content hashes they obtained from their independent re-crawl. Cross-validation compares hashes across auditors — any auditor whose hash diverges from the majority is flagged as suspicious.
• Audit signatures: Each AuditResult includes an auditor_signature field (Ed25519) over the canonical audit evidence, proving the auditor produced the result.
• Unified trust score: Trust = 0.15×uptime + 0.25×contribution + 0.40×audit_pass_rate + 0.20×summary_quality. Tiers: Trusted (≥0.8), Normal (0.5–0.8), Suspect (0.3–0.5), Untrusted (<0.3).
• Tamper detection: 3x audit failures → network isolation. Source URL is always re-crawlable as ground truth.
• Transport isolation: TrustStore.is_isolated(peer_id) checks the DB isolation flag. Isolated peers are rejected at message verification level before any processing.

Resource Governance

• Long-running worker entry points (_serve, MCP, standalone crawl) should opt in to AppContext(..., apply_os_priority=True) or apply ResourceGovernor.apply_os_priority() before heavy initialization.
• Interactive control-plane commands (dashboard/status/settings) must not lower their own process priority, because BGM and TUI responsiveness should not inherit crawler worker throttling.
• Dashboard Network's P2P/DHT row uses content-driven height with no fixed maximum; preserve borders and wrapped status text at narrow terminal widths, using the outer scroll container for short viewports. The displayed replication factor is a configured target, not verified replica count.
• Dashboard BGM prefers mpv with gapless looping and buffered audio, attempts a non-interactive best-effort mpv install on first playback when bgm_auto_install_mpv=true, then falls back to ffplay; bgm_idle_stop defaults to false so auto-started BGM does not stop/restart during intermittent crawling unless explicitly enabled.
• Node startup uses infomesh.runtime.StartupLock plus PID validation to prevent duplicate _serve processes for the same data directory; infomesh stop waits for graceful SIGTERM shutdown and clears only the matching PID file.
• Long-running _serve writes runtime_status.json in the data directory; admin /status, /health?detail=1, and /metrics expose the latest runtime heartbeat, degradation level, throttle factor, and process RSS.
• ResourceGovernor exposes public state via state, degrade_level, throttle_factor, cpu_percent, memory_percent, process_memory_mb, state.process_memory_limit_mb, and state.process_memory_ratio; crawl loops must use those public fields rather than private attributes.
• Linux workers apply both CPU nice and best-effort ionice; failures are non-fatal.
• Runtime PID checks use exact InfoMesh _serve arguments and treat zombies as exited; status/PID writes use unique temporary files. Non-finite or far-future heartbeat timestamps are stale. Admin request limits use monotonic time.
• Dashboard audio preparation/downloads run in a cancellable Textual thread worker; cancellation prevents later playback. Audio downloads are bounded to 32 MiB with socket/deadline limits, restart attempts remain capped across automatic retries, and stopped child processes are reaped. Do not use preexec_fn in the threaded dashboard.
• Dashboard ffplay music/effects use one decoder and one filter thread with video/subtitles disabled. Admit one effect at a time, at least two seconds apart; skip bursts without replacing active effects. Reset the five-restart budget only after a 60-second live-process check or explicit playback. POSIX orphan cleanup runs once per player instance, requiring same-user PID-1 adoption and exact player/asset paths; never terminate another live dashboard's audio.
• WSL startup reads the effective wslinfo --networking-mode. Mirrored mode checks port-specific inbound TCP Hyper-V rules in ActiveStore and prints scoped administrator commands, never NAT portproxy setup. Unknown modes skip auto-configuration. NAT writes require Windows administrator rights, terminating PowerShell/native exit handling and state readback; disabled/partial/wrong-endpoint settings are not success. Linux sudo cannot elevate Windows commands; rule presence does not prove external reachability.
• CI pins UV_PYTHON per matrix entry; publishing selects stable remote tag refs without annotated-tag dereferences, stops on branch push failure, and confines write/OIDC permissions to publishing jobs. Run actionlint when workflows change.
• MCP dispatch enforces governor decisions for remote search, LLM, new crawls, uncached fetch and read-only URL removal. P2P QueryRouter enforces NodeLoadGuard before reading inbound requests and bounds the full operation and shielded stream cleanup.
• Crawler shared GETs use BoundedAsyncClient and a pre-request SSRF hook for all redirects. Robots 5xx/network errors deny crawling; seed/feed requests also use robots, scheduler admission and governor pause checks. PinnedNetworkBackend validates all DNS answers and dials numeric IPs while retaining the original TLS SNI/Host. Environment proxies are disabled on this crawler path.
• Shared GET decoding bounds gzip/deflate output, wire bytes and cumulative redirect bodies under a 30-second total deadline; unsupported encodings fail closed. DNS checks must not block the event loop. Cancelled queue waits requeue when capacity permits; saturated requeues release/log rather than leak. Priority cancellation preserves work; zero indexer ACKs never count as delivered.
• MCP web-search input limits are enforced and advertised: top_k 1..100, chunk_size 100..10000, recency_days 1..36500, real boolean flags, finite ordered dates, 2/3-letter language codes, at most 100 normalized hostnames. Invalid inputs return INVALID_PARAM; compare API keys as bytes to avoid Unicode TypeError.

P2P Message Authentication

• Parallel v1/v2: Search and replication register /2.0.0 alongside /1.0.0. p2p/transport.py uses the actual libp2p Ed25519 identity, binds the signer to the stream's remote peer ID, and signs protocol/recipient/direction plus both 32-byte stream challenges. A fresh exchange handles restart/concurrency without global replay counters. v1 compatibility is deliberate, not authenticated.
• Negotiation: Offer v2 then v1 in a single new_stream call. Never retry v1 after v2 authentication, parsing, or exchange failure. Bound frames, handshakes, operation deadlines and shielded stream cleanup.
• Primitive verification: Check isolation/key/time and Ed25519 signature before recording nonce state. Reject non-finite timestamps and invalid nonce ranges. The existing isolation callback is not automatically a node-wide enforcement pipeline.
• Protocol type: MessageType.SIGNED_ENVELOPE = 100 in protocol.py with encode_signed_envelope() / decode_signed_envelope() helpers.
• Replay protection: NonceTracker maintains per-peer highest-seen nonce; any nonce ≤ previous is rejected.
• v2 replay protection: SignedChannel additionally binds both fresh challenges and allows only one request/response per stream. peer_authenticated in ranked JSON is true only for verified remote v2 search, never for v1/DHT pointers, and does not imply content truth. Default replica storage/repair requires v2 identity/hash/positive-ACK checks; only explicitly supplied legacy storage callbacks permit v1 replication.
• Runtime maintenance: ReplicaStore bounds owner-scoped replicas, repair work, tombstones and signed audit evidence. Maintenance repairs five due documents per cycle, propagates durable owner withdrawals, and performs hourly replica/source audits. Replica corruption can isolate a peer; changing web content is inconclusive. Report under-replication instead of promising N=3 without available peers/capacity.
• Withdrawal fairness: Owner deletion batches visit at most 20 connected peers and persist the last peer per URL. Advance through sorted, unique peers across five-minute retries/restarts; do not repeatedly slice the same first 20 peers. Preserve the cursor when no peers are available.
• Replica payload accounting: Charge retained source summaries on retries and include independent summary/model bytes in the shared quota. Serialize completion admission across SQLite connections; replacement frees only discarded content. Versioned one-time counter repair preserves old data, even if over budget. replica_bytes is a UTF-8 payload measure, not a physical SQLite/WAL size cap. Restart all writers on upgrades.
• Additional protocols: GovernedHost meters Trio application streams, including Kademlia, and checks persistent isolation. Ping/PEX/credit-sync/index-submit have v2 adapters; credit/index-submit writes reject v1. New /infomesh/ DHT storage requires publisher/key/identity-bound signatures and bounded keyword ingress; unsigned legacy read hints remain unverified.
• Enterprise split: Search nodes own a default index-submit receiver. Crawler index_submit_peers must contain libp2p multiaddrs; the asyncio/Trio bridge requires v2, matching URL, storage and durable publication ACK. Legacy HTTP destinations fail closed. Apply configured peer ACL, document/index size and governor constraints.
• Summary jobs: Async AppContext owns cancellable governor/off-peak-aware local and independent replica jobs (180-second DB lease, 120-second timeout, version guards and bounded retry). Replica generation receives only source text, then cross-checks the original; signed reports are deduplicated by peer/version. Idempotent trust/reputation receipts and local/peer credit receipts prevent double application. Durable outboxes recover reward and optional-vector deletion work after restart.
• Summary publication: Persist a version-bound publication outbox with each saved summary and backfill existing current summaries once. Retry five due records per cycle, defer failures/zero counts for five minutes, and ACK only matching versions; generation may be disabled during recovery. Require durable replica-queue admission before network publication. Opt-in INFOMESH_TEST_LLAMA_URL enables the actual llama.cpp lifecycle test; full vector tests need the vector dependencies/model.
• Legal authority: network.trusted_deletion_keys is an empty-by-default list of 32-byte Ed25519 public keys in lowercase hex. Local POST /legal/notices and v2 relays verify the original DMCA/GDPR signer, not the forwarding peer. Atomic exclusion/outbox, all-owner replica blocks and original-signature relays survive restart. Do not promise a 24-hour deadline for unreachable peers or physical deletion from a disabled vector backend.
• Audit aggregation: Preserve raw/extracted hashes in new/startup publication; DHT attestation is signed. Bind source recrawl responses to one audit ID, require distinct selected peers, bound pending/completed audit state and evidence bytes. Treat changed source content as inconclusive, never automatic proof of cheating.
• Framing and state: Bound contiguous receive buffers, exact prefix lengths, unique map keys, integer message types and map bodies. Reserve channel start before any await, permit only one receive attempt, and enforce protocol-specific request/response kinds. The conformance document's final checkpoint supersedes historical open-item tables.

Network Security

• Sybil defense: PoW node ID generation (~30 sec on avg CPU) + max 3 nodes per /24 subnet per DHT bucket.
• Eclipse defense: ≥3 independent bootstrap sources + routing table subnet diversity + periodic routing refresh.
• DHT poisoning defense: Per-keyword publish rate limit (10/hr/node) + signed publications + content hash verification.
• Credit farming prevention: New node 24hr probation (higher audit frequency) + statistical anomaly detection + raw HTTP hash audits.
• Key management: Ed25519 key pairs in ~/.infomesh/keys/, rotation via infomesh keys rotate, revocation via signed DHT records.

Legal Compliance

• robots.txt: Strictly enforced.
• Copyright: Store full text as cache only; return snippets in search results.
• GDPR: Provide option to exclude pages with personal data. Distributed deletion via signed DHT records. Deletion records are persisted in SQLite (_GDPRStore) so obligations survive node restarts. Blocklist is also persisted.
• ToS: Maintain a blocklist of sites that prohibit crawling. Auto-detect ToS patterns beyond robots.txt.
• DMCA: Signed takedown propagation via DHT; nodes must comply within 24 hours. Takedown records are persisted in SQLite (_TakedownStore) — notices, acknowledgments, and propagation records survive restarts. A node cannot evade DMCA obligations by restarting.
• fetch_page(): Paywall detection, cache TTL (7 days), max 100KB per call, attribution required.
• LLM summaries: Label as AI-generated, include content_hash linking to source, always provide original URL.

Development Phases

The phase table records historical component milestones, not verified end-to-end runtime conformance. Consult docs/en/13-design-conformance.md and its Korean counterpart for the current integration evidence and operating constraints. The eight tracked runtime paths are connected, with operator-authorized legal relays and signed independent peer-summary reports. Approved v1/read compatibility, capacity-dependent N=3, offline deletion and physical-model quality are not guarantees.

MCP Web Search API — Integration & Keywords

InfoMesh functions as an MCP-based web search API that any LLM-powered agent or IDE can integrate with. It is a real-time, open-source alternative to paid web search APIs.

What InfoMesh Provides (as an MCP Search Tool)

• Web search for AI agents — search the open web from any MCP-compatible client (VS Code Copilot, Claude Desktop, Cursor, Windsurf, Continue, Cline, etc.)
• Real-time web crawling API — crawl arbitrary URLs on demand and index them instantly
• Full-page text fetching — retrieve the full extracted text of any web page by URL
• Offline / local search — search the locally cached index without internet access
• Network statistics API — query node health, peer count, index size, credit balance
• RAG support — chunked, source-attributed output for retrieval-augmented generation
• Answer extraction — direct answer extraction with confidence scores
• Fact checking — cross-reference claims against indexed sources
• Score explanations — transparent BM25/freshness/trust breakdowns

MCP Tool Specifications

IDE / Client Integration

InfoMesh integrates with any MCP-compatible client via stdio or HTTP transport:

{
  "mcpServers": {
    "infomesh": {
      "command": "infomesh",
      "args": ["mcp"],
      "env": {}
    }
  }
}

For HTTP transport (containers/remote): infomesh mcp --http --port 8081

Supported clients: VS Code (GitHub Copilot), Claude Desktop, Cursor, Windsurf, Continue, Cline, any MCP-compatible LLM agent.

Search API Comparison Keywords

InfoMesh can be considered as a free, open-source, decentralized alternative for:

• Commercial web search APIs (per-query billing, API-key required)
• SaaS real-time web access for LLMs (per-query billing)
• Proprietary MCP web search tools
• Grounding / retrieval-augmented generation (RAG) data source
• Commercial web crawler APIs

Key differentiators:

• No API key required — fully decentralized, no central server
• No per-query billing — earn search credits by contributing (crawl, host, uptime)
• Privacy-first — search queries never recorded centrally
• Self-hosted — runs on your own machine, your own index
• MCP-native — designed specifically for LLM tool-use integration
• Offline-capable — local index works without internet
• Community-driven — open source, peer-to-peer, no vendor lock-in