Syncing & Indexing with Context Engine

Context Engine indexes private codebases via upload (not git clone): a client walks the repo, uploads file contents to object storage, and enqueues an index job. The @boltz.one/ce-cli CLI (npx @boltz.one/ce-cli, binary ce) automates this; the sync-* MCP tools are the low-level primitives behind it.

Check Freshness First

list-repos                       → repoId, status, lastCommitSha, chunks
sync-status({repoId})            → status (indexed|indexing|failed), chunkCount, lastIndexedAt, error

Index is stale / needs a push when any of:

• status ≠ indexed
• lastCommitSha ≠ local git rev-parse HEAD
• chunks/chunkCount is suspiciously low (e.g. 1) for a real codebase

ce-cli (preferred)

The uploader is the Rust CLI @boltz.one/ce-cli (binary ce), run via npx (zero install; per-platform prebuilt binary). It replaces the legacy ce-sync.mjs (removed). Source: apps/context-engine/tools/ce-cli.

The CLI resolves the CE endpoint + API key from .mcp.json (the context-engine MCP server entry) — or from CE_URL/CE_API_KEY env. The repoId is resolved from the key's scope via GET /api/sync/repos (list-repos); the last-synced commit comes from the server (sync-status.lastCommitSha).

# First-time setup of a NEW repo (needs CE_URL + CE_API_KEY, or a configured .mcp.json):
# create upload repo (auto-scoped to your key), upload all files, full index, install git hook.
CE_URL=https://mcp.boltz.one CE_API_KEY=sk-… npx @boltz.one/ce-cli init [name]

# Incremental: upload files changed since the server's lastCommitSha → incremental re-index.
npx @boltz.one/ce-cli push           # key from .mcp.json (or CE_API_KEY env)

# Show index status.
npx @boltz.one/ce-cli status

The CLI respects .gitignore + .dockerignore + .contextengineignore, recurses git submodules, and skips binary/lock/large (>1 MB) files and files containing NUL bytes.

Full re-index of an EXISTING repo

push is incremental (diffs the server's lastCommitSha..HEAD). To force a complete re-upload + re-index (e.g. when the index is empty/corrupt), use the built-in --full flag:

npx @boltz.one/ce-cli push --full
# preview only, no upload:
npx @boltz.one/ce-cli push --dry-run

Auto re-index

ce init installs a git post-commit hook that runs npx @boltz.one/ce-cli push in the background after every commit. No agent action needed once installed. The agent never runs init/push automatically on repo open — trigger it explicitly when the user asks.

Low-level sync-* MCP tools

Use these only when the CLI isn't available, or to drive a custom flow:

Flow (init): sync-init → sync-upload-url → PUT each file → sync-commit → poll sync-status. Flow (delete a single bad/stale path from storage): sync-push({repoId, deleted:[path], commitSha}).

Gotchas

• Meta-repo / git submodules: plain git ls-files at the root does NOT recurse into submodules (it only sees gitlinks) → submodule code is missed and the index ends up nearly empty (chunks:1). ce-cli enumerates each submodule (git submodule status --recursive, then ls-files inside each, prefixed by the submodule path). Verify with list-repos chunk count after a sync.
• NUL byte in source kills the whole index job: a text file with a \0 byte (e.g. used as a Map-key separator) makes Postgres reject the chunk (invalid byte sequence for encoding "UTF8": 0x00) and FAILS the entire job (all-or-nothing → status:failed). The uploader must scan the WHOLE file for NUL (not just the first 8 KB) and skip such files.
• Cold start: opening a repo does not auto-index. First-time init must be invoked explicitly via npx @boltz.one/ce-cli init (no local script needed — npx fetches the binary).
• After a failed job: read the error field from sync-status, fix the offending input, then re-push.

Checklist

- [ ] list-repos / sync-status → is it indexed and fresh?
- [ ] New repo → ce init (or sync-init + upload + sync-commit)
- [ ] Existing repo, normal change → ce push
- [ ] Existing repo, full rebuild → ce push --full
- [ ] Meta-repo → confirm submodule files are included (chunk count sane)
- [ ] Poll sync-status until status=indexed; if failed, read error and fix

Example: re-index boltz-one (meta-repo) end to end

1. list-repos → boltz-one repoId=be1730bd-…, chunks=1   (← too low, submodules missing)
2. npx @boltz.one/ce-cli push --full   → uploads ~956 files (root + all submodules), full re-index
3. sync-status({repoId}) → poll until status=indexed, chunkCount≈1400+
4. codebase-retrieval({repoId, …}) now returns code from core/, apps/*, ops/, tests/, tools/*