api-mirror

Context

The MirrorService owns the source-agnostic half of a local mirror — the embedded store, the sync-state machine, the runner — so a server supplies only the two parts that are irreducibly per-source: the ingester (a sync generator) and the schema. It targets the embedded-SQLite tier (~10⁴–10⁷ rows). Node/Bun only: bun:sqlite is built-in on Bun, better-sqlite3 is an optional peer dependency on Node; the store is unavailable on Workers (no SQLite, no persistent filesystem).

Import from @cyanheads/mcp-ts-core/mirror.

The shape

import { defineMirror, sqliteMirrorStore } from '@cyanheads/mcp-ts-core/mirror';

const papers = defineMirror({
  name: 'arxiv-papers',
  store: sqliteMirrorStore({
    path: config.mirrorPath,
    primaryKey: 'id',
    columns: { id: 'TEXT', title: 'TEXT', authors: 'TEXT', abstract: 'TEXT', updated: 'TEXT' },
    fts: ['title', 'authors', 'abstract'],          // opt-in FTS5 external-content index
    indexes: [{ columns: ['updated'] }],
  }),
  // The ingester — the one part that is always server-specific.
  async *sync({ mode, cursor, checkpoint, signal }) {
    for await (const page of harvestPages({ resumeFrom: cursor, since: checkpoint, signal })) {
      yield {
        records: page.rows,             // objects keyed by declared column
        tombstones: page.deletedIds,    // primary-key values to delete
        cursor: page.token,             // volatile resume position (see below)
        checkpoint: page.maxStamp,      // durable high-water mark (see below)
      };
    }
  },
});

await papers.runSync({ mode: 'init', signal: AbortSignal.timeout(3_600_000) }); // full; resumes on interrupt
await papers.runSync({ mode: 'refresh' });                                       // incremental
const { rows, total } = await papers.query({ match: 'transformers', limit: 10, offset: 0 });
const status = await papers.status();   // { status, ready, checkpoint, total, ... }

cursor vs. checkpoint — the core distinction

Two resume dimensions, deliberately separate. Conflating them silently corrupts resume for token-paged sources.

Why they can't merge: during a from-scratch init the records aren't ordered by the high-water field, so the max-so-far is not a valid resume position — only the cursor is. After a completed init the cursor is meaningless, but the high-water mark is the correct refresh seed. The framework persists both per page and threads the right one back into sync() per mode. The checkpoint must be lexicographically monotonic (ISO 8601 works); the runner advances the stored checkpoint only when a page's value compares greater.

What you own vs. what the framework owns

Querying

query({ match?, filters?, sort?, limit, offset }) covers the common case:

• match — an FTS5 MATCH expression (only when the store declares fts columns). Translate your own query grammar to FTS5 before calling.
• filters — [{ column, op, value }], AND-combined, over declared columns. op ∈ eq|ne|gt|gte|lt|lte|in (in takes an array).
• sort — { column, direction } or 'relevance' (FTS bm25; requires match). Defaults to insertion order.

For access paths the generic query can't express — junction tables for index-backed multi-value filtering, denormalized counters, bespoke bm25 weighting — use the raw handle: const db = await mirror.raw(); then run prepared statements against your own auxiliary tables (declare them via a migration). Add the auxiliary DDL in a migrations step; maintain it from your sync mapping or SQL triggers.

Readiness — key off the completion marker, not live status

status().ready is true once a full sync has ever completed (completedAt != null), not when status === 'complete'. The dataset stays transactionally queryable during a refresh, so a mirror mid-refresh — or one whose last refresh failed — is still ready and should keep serving. Gate the mirror read path on await mirror.ready(); fall back to the live API only when it is false (cold, never-completed init).

Scheduling and bootstrap (server-owned)

The service owns runSync + state; it does not schedule. Wire "self-refreshing" yourself:

• Refresh — register runSync({ mode: 'refresh' }) on a cron via schedulerService from @cyanheads/mcp-ts-core/utils, inside setup(). Gate on transport (HTTP) when stdio operators run it out-of-band.
• Init — run out-of-band (a CLI script / one-shot), never on startup: a full init can take hours and must not block the server. It is idempotent and resumable — re-running after an interrupt continues from the persisted cursor.

Checklist

• defineMirror({ name, store, sync }); the server holds the instance (one per mirror)
• sqliteMirrorStore spec declares primaryKey, columns, and (if searching) fts
• sync yields { records, tombstones?, cursor?, checkpoint? } per page; checkpoint is lexicographically monotonic
• Read path gated on await mirror.ready() with a live fallback when not ready
• better-sqlite3 added as a peer dependency for Node deployments; mirror disabled on Workers
• Refresh wired via schedulerService in setup(); init runs out-of-band
• bun run devcheck passes