MeshLLM Agent Notes

These instructions apply under mesh/ in addition to the root AGENTS.md. Paths and just commands below are relative to the workspace root. Mesh depends on Skippy; do not move Skippy-native behavior into Mesh crates.

Key docs

Public Website

The maintained website source lives in mesh/website/. Eleventy writes generated static-hosting output into root docs/; authored documentation lives under mesh/docs/ and skippy/docs/. Do not hand-edit generated website files.

just website-build # build website and crate API docs into root docs/
just website-dev   # Eleventy dev server on port 8765
just website-clean # remove generated output
just crate-docs    # regenerate published crate API docs

The build runs Tailwind, Eleventy and Pagefind. It copies website source assets and root install.sh, install.ps1 and install.md into the generated tree. mesh/website/src/assets/site.generated.css is generated by Tailwind.

Building and packaging

• just mesh builds only the backend-neutral MeshLLM host and console. It consumes Skippy's separately packaged runtime and does not compile a backend into the host.
• Bare just builds Skippy first, then MeshLLM. just release-build does the same for release products. just release-host-build builds only the release host.
• just bundle creates a portable product tarball. just release-bundle vX.Y.Z <output> packages the release host and one selected runtime under native-runtimes/.
• just ui-dev starts Vite; just ui-clean clears stale UI installs; just stop, just test, and just auto operate on the local MeshLLM runtime.

A release binary left in target/release/mesh-llm may predate the latest debug build. Its presence does not prove it contains current code; rerun just release-build when in doubt. For serious testing and deployment, use a composed release product.

The host dependency policy is enforced by cargo xtool native verify-host-dependencies. Release, installer, SDK, native-package, and image lanes must not bypass it or copy backend libraries beside the host. For isolated runtime discovery, set MESH_LLM_NATIVE_RUNTIME_BUNDLE_DIR="$PWD/dist/native-runtimes" and a fresh MESH_LLM_NATIVE_RUNTIME_CACHE_DIR. Discovery never searches the working directory. Do not reintroduce an external llama-server or rpc-server lane.

npm "Exit handler never called" error

If just build fails on the UI step with npm error Exit handler never called!, run just ui-clean and then just build. This can follow branch switches that leave node_modules inconsistent with the lockfile.

Workspace Crates

Product crates live under mesh/crates/. The most important crates:

Shipped binary and CLI surface:

• mesh-llm/ — shipped binary; main.rs builds the Tokio runtime, lib.rs owns run_main (CLI parse → one-shot command dispatch via its commands/ module → runtime handoff). No domain logic here.
• mesh-llm-cli/ — Clap types, argument parsing, serve/client surface normalization. No handlers.
• mesh-llm-commands/ — user-facing command handlers (auth, gpus, update, skills, agent launchers like goose/pi/opencode/claude, plugin, benchmark, model packaging).
• mesh-llm-tui/ — terminal UI and progress output surface.
• mesh-llm-events/ — shared runtime event and output contracts (OutputEvent, log formats).

Host and client runtimes:

• mesh-llm-host-runtime/ — the host-side monolith. Owns runtime orchestration, mesh, inference, networking, management API, plugins, models, system integration. This is where most changes land.
• mesh-client/ (mesh-llm-client) — lighter parallel client surface with its own inference/, network/, models/, mesh/ modules. Used as a dev/test surface and for client-only deployments.
• mesh-llm-node/, mesh-llm-embedded-runtime/ — embeddable node primitives and in-process full-node embedding API.
• mesh-llm-config/ — configuration parsing and validation (~/.mesh-llm/config.toml).
• mesh-llm-ui/ — React web console and embedded asset crate (shadcn/ui patterns, see https://ui.shadcn.com/llms.txt).
• mesh-llm-console-server/ — static file server for embedded console assets.

Shared foundations:

• mesh-llm-types/ — shared model/capability types used across crates.
• mesh-llm-protocol/ — wire protocol types and protobuf bindings.
• mesh-llm-routing/ — routing primitives shared across host and client.
• mesh-llm-system/ — machine-local hardware, benchmark, autoupdate, process helpers.
• mesh-llm-identity/ — owner identity and envelope crypto primitives.
• mesh-llm-plugin/ — plugin runtime/DSL primitives.
• mesh-llm-plugin-manager/ — plugin package management (catalog, install, store).
• mesh-llm-skills/ — agent skill data model and installer primitives.

SDK and API surface:

• mesh-llm-sdk/ — Rust SDK facade for clients and embedded serving.
• mesh-llm-api-server/, mesh-llm-api-client/ — public Rust SDK APIs for embedding nodes / client-only use.
• mesh-llm-ffi/, mesh-llm-nodejs/ — FFI bindings and Node.js native addon.
• mesh-mixture-of-agents/ — Mixture-of-Agents fan-out/arbitration engine.

Tools and benchmarks:

• mesh-llm-test-harness/ — integration test harness binary.

This list covers the crates you are most likely to touch; check mesh/crates/ and each crate's Cargo.toml description for anything not listed.

Other product directories:

• mesh/website/ — Eleventy source for the public website; builds into root docs/.
• mesh/docs/design/ — Mesh architecture, protocols, and testing docs.
• mesh/docs/plugins/ — Plugin architecture docs and plans.
• mesh/docs/specs/ — Focused Mesh behavior specs.
• mesh/sdk/ — SDK packaging for Node, Swift, Kotlin.
• mesh/deploy/fly/ — Fly.io deployment (console + API client apps).
• mesh/evals/ — Mesh benchmarking and evaluation scripts.
• tools/relay-fly-legacy/ — Archived self-hosted iroh relay reference; production uses services.iroh.computer.

Module Structure Rules

These rules apply primarily inside mesh/crates/mesh-llm-host-runtime/src/ (the main host monolith), and by analogy inside mesh/crates/mesh-client/src/. New peer crates should still follow the semantic-ownership principles below.

The host-runtime crate root should stay minimal.

• Keep mesh/crates/mesh-llm-host-runtime/src/lib.rs slim — it is a small entry point, not a junk drawer.
• New code should go into an existing domain directory when possible.

Use semantic ownership for module placement. Inside mesh/crates/mesh-llm-host-runtime/src/:

• runtime/ — top-level process orchestration, startup/runtime coordination, runtime instance, capacity, split planning, proxy lifecycle.
• network/ — request routing, proxying, tunneling, relay/discovery networking, request-affinity logic, endpoint rewrite, target health, OpenAI transport glue.
• inference/ — model-serving logic, election, launch, pipeline, MoE behavior, embedded skippy integration.
• system/ — machine-local environment and platform concerns (hardware detection, benchmarking, self-update, local system integration).
• models/ — model catalog, resolution, downloads, local model storage, model metadata.
• mesh/ — peer membership, gossip, heartbeats, identity, peer state, mesh node behavior.
• plugin/ — plugin host, plugin runtime, transport, config, MCP bridge support.
• plugins/ — concrete in-tree plugins (currently blobstore/; most plugins like blackboard, openai-endpoint, and flash-moe/ln are external packages installed via mesh-llm plugins install).
• api/ — management API surface and route handling.
• protocol/ — wire protocol types, encoding/decoding, conversions.
• runtime_data/ — runtime data collection, API views, status snapshots.
• crypto/ — host-side crypto helpers.

CLI ownership rule.

• Clap types, argument parsing, and surface normalization belong in mesh/crates/mesh-llm-cli/.
• User-facing command handlers belong in mesh/crates/mesh-llm-commands/ (or the shipped binary's mesh/crates/mesh-llm/src/commands/ dispatch layer for wiring).
• Domain modules in mesh-llm-host-runtime should not own Clap parsing or top-level command dispatch.
• Domain modules may expose reusable functions that command handlers call.

Current structure notes.

• Request-affinity code belongs with networking/routing behavior (network/affinity.rs), not system/.
• Plugin MCP support belongs inside mesh-llm-host-runtime/src/plugin/, not as a separate root module.
• Model command handlers belong in mesh-llm-commands/ (or mesh/crates/mesh-llm/src/commands/ for dispatch wiring); host-runtime models/ should stay domain-focused.
• The shipped binary crate (mesh/crates/mesh-llm/) carries CLI dispatch wiring only; do not move domain logic into it.

Key Source Files

Host runtime (main monolith — mesh/crates/mesh-llm-host-runtime/src/):

• lib.rs — crate entry; exposes the runtime entrypoints (run_runtime_initialized, initialize_host_runtime) called from mesh/crates/mesh-llm/src/lib.rs.
• runtime/mod.rs — top-level startup flows, runtime orchestration, command dispatch.
• runtime/instance.rs — per-instance runtime directory management: InstanceRuntime, pidfiles, flock liveness, scoped orphan reaping, local instance scanning.
• runtime/local.rs — local model startup loop.
• runtime/discovery.rs — discovery loops and auto-mode coordination.
• runtime/proxy.rs, runtime/proxy/ — HTTP proxy lifecycle from the runtime side.
• runtime/capacity.rs, runtime/split_planning.rs, runtime/context_planning.rs — placement/sizing decisions.
• mesh/mod.rs — Node struct, mesh_id, peer management.
• mesh/gossip.rs — gossip wire format and peer state updates.
• mesh/heartbeat.rs — heartbeat publishing and freshness.
• inference/election.rs — host election, tensor split calculation.
• inference/skippy/ — embedded staged runtime integration.
• inference/pipeline.rs — inference pipeline coordination.
• inference/virtual_llm.rs — virtual LLM (inter-model collaboration).
• network/proxy.rs — HTTP proxy: request parsing, model routing, response helpers.
• network/router.rs — request classification, model scoring, multimodal routing.
• network/nostr.rs — Nostr discovery, score_mesh(), smart_auto().
• network/tunnel.rs — TCP ↔ QUIC relay (RPC + HTTP).
• network/affinity.rs — request-affinity tracking.
• network/target_health.rs — target health tracking.
• network/openai/ — OpenAI transport glue.
• api/mod.rs, api/routes/ — management API (:3131): /api/status, /api/events, /api/discover.
• models/catalog.rs — model catalog, HuggingFace downloads.
• models/capabilities.rs — multimodal/vision/audio/reasoning capability inference.
• models/resolve/ — model reference resolution.
• plugins/blobstore/mod.rs — request-scoped media object storage for multimodal.
• plugin/ — plugin host, runtime, transport, config, MCP bridge (external plugins install via mesh-llm plugins install).

Shipped binary and CLI (mesh/crates/mesh-llm/src/, mesh/crates/mesh-llm-cli/src/, mesh/crates/mesh-llm-commands/src/):

• mesh-llm/src/main.rs — builds the Tokio runtime (custom stack size via MESH_TOKIO_STACK_SIZE) and calls mesh_llm::run_main().
• mesh-llm/src/lib.rs — run_main: CLI parse, one-shot command dispatch, runtime handoff.
• mesh-llm/src/commands/ — dispatch wiring from parsed Command values to handlers.
• mesh-llm-cli/src/parser.rs — Clap surface, serve/client arg normalization, advanced help.
• mesh-llm-commands/src/ — user-facing handlers (auth, gpus, update, skills, agent launchers, plugin, benchmark).

Peer Inference Execution Boundary

• Local-user HTTP ingress may select a mesh host and retry eligible peers.
• Incoming QUIC HTTP inference is already assigned to this host: execute the named local model (including a registered serving plugin) or return an error to the caller. Never forward it to another mesh peer, including on local failure, missing model, automatic/virtual model selection, or routing headers.
• Incoming QUIC inference must never reconnect to the main Mesh HTTP API port (9337 by default, including configured replacements). Use in-process execution ingress; if it is unavailable, return an error. Do not restore a TCP fallback. Headers are not proof of local origin. Model-name aliases and multiple hosts advertising the same model are normal and must not weaken the boundary.
• Keep real-QUIC regression tests proving local execution, local-unavailable rejection without a second hop, and continued local-user HTTP routing when changing ingress or target selection.

Mesh Protocol Compatibility

Mesh compatibility across versions is critical. Nodes in the wild run different versions and must interoperate.

• The mesh supports mixed-version operation: QUIC ALPN mesh-llm/1 (protobuf) and mesh-llm/0 (legacy JSON) nodes coexist. Do not break this.
• Gossip fields, stream types, and protobuf schemas must be additive. New fields should be optional and ignored by older nodes. Do not repurpose or remove existing fields.
• When adding new gossip fields, stream types, or changing wire format, explicitly consider what happens when an older node receives the new data and when a newer node talks to an older peer.
• Capability advertisement (vision, audio, multimodal, reasoning, tool_use, moe) is gossiped to all peers and consumed by routing, the API, and the UI. Changes to capability semantics affect the whole mesh, not just the local node.
• If a change would break mixed-version meshes, explicitly flag it as a breaking protocol change and ask the developer before proceeding.
• Test compatibility by running the current branch against a released binary on a second node. Verify gossip, routing, and inference work across the version boundary.

Plugin Protocol Compatibility

When iterating on the plugin protocol, always consider protocol compatibility.

• If a protocol change may be breaking, explicitly ask the developer whether the change is intended to be breaking.
• If the change is not intended to be breaking, the previous version of the plugin protocol must continue to be supported.
• Do not silently ship plugin protocol changes that strand older plugins or hosts without confirming that outcome is acceptable.

UI Notes

For changes in mesh/crates/mesh-llm-ui/, use components and compose interfaces consistently with shadcn/ui patterns. Prefer extending existing primitives in src/components/ui/ over ad-hoc markup.

Terminal dashboard integrity

The dashboard renders to the controlling terminal (/dev/tty, CONOUT$), never to fd 1 or fd 2, and while it owns the screen those two descriptors are redirected into it (mesh/crates/mesh-llm-tui/src/output/console_capture.rs). Stray println!/eprintln!, inherited child stderr, and native llama.cpp output therefore arrive as dashboard events instead of painting over the frame.

Consequences worth knowing before changing this area:

• Do not point the TUI backend, or the enter/exit escape writers, at io::stderr(). Capture would then redirect the dashboard into its own pipe.
• Anything restoring the terminal (panic hooks, emergency writers) must release the capture before writing, or the message goes into the pipe.
• Capture cannot intercept a write to the tty by a process that did not inherit our descriptors. R is the repair for that case: a physical clear plus a ratatui diff invalidation. A logical Clear widget does not fix backend desynchronization.
• Do not perform that clear on every draw. It repaints from blank and reads as a black blink; measured on an idle dashboard it was ~2.6 full repaints/s.

Prefer OutputEvent or tracing in runtime code regardless — captured lines have no level and are shown with a stdout context. When adding a tracing target that must reach the dashboard, add a directive for it in runtime_tracing_subscriber: EnvFilter::from_default_env() defaults to ERROR, so an unlisted target's warn! is dropped before the writer sees it.

Testing

Read mesh/docs/design/TESTING.md before running tests. It has all test scenarios, remote deploy instructions, and cleanup commands.

Testing matters more than usual in this project because:

• Nodes run on different machines with different hardware and OS versions. Bugs that don't reproduce locally can appear in real deployments.
• The mesh protocol is a distributed system — gossip, election, and routing interact across nodes. Single-node unit tests don't catch protocol-level regressions.
• The public mesh at meshllm.cloud runs continuously. Breaking changes that pass local tests can take down live inference for real users.
• Multimodal, MoE splitting, and multi-model routing all have complex interaction paths that are hard to reason about statically.

When making changes that touch gossip, routing, proxy, election, or capability advertisement, test against at least two nodes before merging. The deploy checklist below is not optional.

Confidence Testing (multi-node, when warranted)

For changes that affect routing, MoA, gossip, the OpenAI surface, agent harnesses, or anything multi-node, validate with these three shapes before declaring a branch ready:

1. 2-node private mesh — start one node with mesh-llm serve --model <big> --port 9337 --console 3131, grab its invite token from the JSON log, and start the second node with mesh-llm serve --gguf <small.gguf> --port 9447 --console 3145 --join <token>. Confirm peers=1 on both consoles and /v1/models returns the union. Exercises QUIC tunnelling and cross-node routing.
2. Public mesh as a client — mesh-llm client --auto from a workstation. Confirm discovery_joined plus a structured client-ready event (passive_mode, status=ready, role=client) in the log and an inference call against a mesh-advertised model returns. Exercises the read-only routing path agent users hit.
3. Agent harness — run at least one existing client smoke (scripts/ci-goose-smoke.sh, scripts/ci-opencode-smoke.sh, or scripts/ci-pi-smoke.sh) against the local proxy with both model=auto and model=mesh to catch tool-call and reducer regressions that simple curl checks miss. See mesh/docs/AGENTS.md for client setup and the direct tool-call reliability probe.

Mesh package validation

For changes to mesh-llm-host-runtime or the shipped mesh-llm binary, use -p mesh-llm for build/check (it pulls in the host runtime) and -p mesh-llm-host-runtime for focused tests.

Running mesh-llm locally

Default the launch to a normal foreground run (TUI visible) unless you have a specific reason to suppress UI surfaces. Most observation/debug tasks do not need the TUI suppressed.

• mesh-llm client --auto — normal foreground run with the TUI. Use this by default.
• --log-format json — emits machine-parseable JSON log lines. Use this when you want to programmatically read events.
• --headless — disables the embedded web UI, not the TUI. The TUI still draws. Only use --headless when you are intentionally avoiding the management web console — it is not the way to get a quiet background run.
• --no-console — fully disables the management console (HTTP API on the console port).
• nohup … & with a foreground binary that draws a TUI will appear to run but often exits or behaves oddly when the TUI cannot attach to a terminal. Prefer letting the developer launch the binary in their own terminal and observing via /api/status, --log-format json, or by reading stderr.

Do not reach for --headless to "go quiet" — that is a recurring mistake. If you want quiet output, use --log-format json and parse what you need.

Rust validation

• After Rust changes, run just no-console-print. There is no allowlist. It forbids println! / eprintln! / print! / eprint! and direct io::stdout() / io::stderr() handles in product code. Route operational output through mesh_llm_events::emit_event or tracing, human-facing CLI prose through mesh_llm_events::console_out / console_err, and a --json payload through mesh_llm_events::machine_out. Only the console output facility may hold a terminal handle; its roster is CONSOLE_OUTPUT_OWNERS in tools/xtask/src/no_console_print/scope.rs.
• If a change is reachable from the shipped binary, run at least cargo check -p mesh-llm and cargo clippy -p mesh-llm --all-targets -- -D warnings.
• Changes to proto/, any Mesh protocol/ module, mesh-llm-host-runtime/src/mesh/gossip.rs, mesh-llm-host-runtime/src/mesh/mod.rs, routing, election, or API serialization need cargo test -p mesh-llm-host-runtime --lib; do not stop at build-only validation.

UI validation

• Use the repo's supported workflow and run just build.
• If just build fails on the UI step with npm error Exit handler never called!, run just ui-clean and then rerun just build.

Deployment and cleanup

Deploy to Remote

just bundle    # /tmp/mesh-llm-bundle.tar.gz — single mesh-llm binary
# scp bundle to remote, tar xzf, then on macOS: codesign -s - mesh-llm && xattr -cr <dir>

For the full per-platform deploy flows, see the repo skills .agents/skills/deploy-macos/, .agents/skills/deploy-linux-gpu/, and .agents/skills/deploy-windows/.

Cleanup

Clean shutdown removes the instance's runtime directory automatically. Prefer the scoped runtime-aware commands first:

mesh-llm stop
just stop

Those paths use the runtime metadata under ~/.mesh-llm/runtime/ to stop the tracked mesh-llm instance and its child servers cleanly.

If an instance is wedged badly enough that the scoped stop path cannot reach it, fall back to an emergency kill:

pkill -f mesh-llm

Running mesh-llm in the Background (for Testing)

When running mesh-llm serve from an agent for testing, the process is non-interactive — it just runs. There is no interactive prompt or TUI to worry about. Use standard backgrounding:

bash -c './target/debug/mesh-llm serve --model "..." --auto > /tmp/mesh.log 2>&1 & disown; echo "PID=$!"'

• Do not use --headless — it disables the web UI but does not change process behavior. The name is misleading and does not help with backgrounding.
• The mesh process writes TUI-formatted output to stderr which looks like errors but is normal.
• Wait for models to appear via polling curl -s http://localhost:9337/v1/models before sending requests.
• Kill with pkill -f "target/debug/mesh-llm" or pkill -f mesh-llm.

Deploy Checklist — MANDATORY

Every deploy to test machines MUST follow this checklist.

Before starting nodes

1. Bump VERSION in the root Cargo.toml ([workspace.package] version; crates inherit it via version.workspace = true) so you can verify the running binary is new code.
2. just build && just bundle
3. Kill ALL processes on ALL nodes — pkill -9 -f mesh-llm
4. Verify clean — ps -eo pid,args | grep -E 'mesh-llm' | grep -v grep must be empty.
5. Deploy bundle — scp + tar + codesign on remote nodes.
6. On every macOS node, complete the deploy-macos skill's Local Network privacy preflight for the exact signed identity and launch context. Clear any blocking desktop alert before remote diagnosis.
7. Verify version — mesh-llm --version on every node.

After starting nodes

8. Verify exactly 1 mesh-llm process per node.
9. Verify no external llama serving child processes are required.
10. curl -s http://localhost:3131/api/status returns valid JSON on every node.
11. Check /api/status peers for new version string.
12. Verify expected peer count.
13. For same-LAN tests, require a direct iroh path to the intended LAN address from both nodes before attributing failures to candidate selection or collecting performance data.
14. Test inference through every model in /v1/models.
15. Test /v1/ passthrough on port 3131.

Debugging Embedded Runtime Startup

If the embedded runtime fails to load, check mesh-llm stderr/log output and ~/.mesh-llm/runtime/ for the active instance metadata. Embedded skippy/llama.cpp native logs are redirected away from the TUI into the active instance runtime directory:

<runtime-root>/<pid>/logs/skippy-native.log

To override the runtime root (e.g., for tests or systemd):

• MESH_LLM_RUNTIME_ROOT=/path/to/custom/root — highest priority
• XDG_RUNTIME_DIR — if set (typical on systemd: /run/user/{uid}/mesh-llm/runtime)
• $HOME/.mesh-llm/runtime — default fallback

For stale instances (crashed mesh-llm leaving behind a runtime dir):

• Other running mesh-llm instances GC dead-owner dirs older than 1 hour on startup
• Manual cleanup: rm -rf ~/.mesh-llm/runtime/<stale_pid>/

Common failures

• nohup over SSH doesn't stick — use bash -c "nohup ... & disown", verify process survives disconnect.
• Duplicate processes — always kill-verify-start.
• codesign changes the hash — don't compare local vs codesigned remote.

Releasing

See RELEASE.md for the full process.

Current release flow: kick off the Release workflow (.github/workflows/release.yml) from the GitHub Actions UI via workflow_dispatch with the version input (e.g. v0.X.Y), or run just release v0.X.Y to preflight and dispatch that same workflow.

The dispatched workflow handles everything: it bumps versions via scripts/release-version.sh, commits the tracked version surface to main before building, generates and patches the SwiftPM manifest, packages SDK console assets, creates and pushes the release tag at a release-prep commit, builds the full artifact matrix (macOS, Linux CPU/ARM64/CUDA/CUDA-Blackwell/ROCm/Vulkan, Windows CPU/CUDA/ROCm/Vulkan), and publishes the GitHub release with generated notes. Dispatch inputs include skip_gpu_bundles and canary (dry-run: build + smoke without mutating main or publishing).

Pushing a v* tag manually also triggers the workflow, but the tag must point to main history with the matching scripts/release-version.sh update already committed; it also requires preparing Package.swift and SDK console assets in the tag commit yourself. The workflow rejects a drifted tag. See RELEASE.md and prefer the dispatch path.

Installer checksum sidecars

Release/package scripts should keep generating .sha256 sidecars for new release archives. Do not rely on backfilling old release assets, because pinned versions and alternate repos may not have sidecars.

install.sh and install.ps1 must treat release-archive checksums as backward-compatible rollout metadata:

• If <archive>.sha256 exists, verify it and fail the install on malformed checksum data or checksum mismatch.
• If the sidecar is missing for a legacy/current release, warn and continue by default.
• If MESH_LLM_REQUIRE_CHECKSUM=1 is set, a missing sidecar is fatal.

Do not change installer behavior to hard-require sidecars by default unless the release policy also guarantees every supported/pinned release and alternate install repo has matching checksum assets.

What NOT to add

• No api_key_token feature; it was explicitly rejected and removed in v0.26.0.
• No domain logic in mesh/crates/mesh-llm/src/. Keep that crate to CLI dispatch wiring; put domain behavior in mesh-llm-host-runtime or a narrower peer crate.