mcp-data-platform

# mcp-data-platform

> Composable, semantic-first MCP server platform that unifies DataHub, Trino, S3, third-party MCP servers, and REST APIs behind one authenticated endpoint. Every tool response is cross-enriched with business context; OAuth 2.1, role-based personas, audit logging, persistent memory, governed knowledge capture, and an admin/user portal are built in.

**Document version: v1.120.0, revised 2026-08-29.** This project releases often. If your copy of this page is older than the current release on https://github.com/txn2/mcp-data-platform/releases/latest, re-fetch it before grounding an assessment on it; earlier revisions described a narrower system.

**How the documentation site is organized.** Three sections: `home`, `docs` (everything written about the platform -- install, the End User and Administrator references, the API reference, the Go library, evaluation, examples and support), and `portal` (a guided tour of every screen the built-in web portal serves, split into Your Work and Administration). The portal tour lives under /portal/; it replaced the two single-page guides that used to sit at /server/portal-user/ and /server/admin-portal/.

mcp-data-platform is the orchestration layer for the txn2 MCP ecosystem. DataHub, Trino, and S3 are components the platform composes, not infrastructure an adopting organization must already run: the platform needed a metadata layer and a federating query engine, and adapts those mature open-source systems behind provider interfaces rather than reimplementing them. Cross-enrichment works from DataHub as the semantic layer, which can stay a silent backend the operator never surfaces; add Trino for SQL and S3 for objects when ready. Every backend is optional: with none of the three configured the semantic, query, and storage providers resolve to noops and the database-backed surfaces (gateways, knowledge, memory, portal, search/fetch) run on PostgreSQL alone. DataHub is an adapter behind a provider interface, not a substrate the platform is built on: omitting it costs cross-enrichment and the datahub_* tools, not the platform. semantic:, query:, storage:, and toolkits: are independent config blocks, so Trino and S3 stay available on their own terms and simply stop being enriched. Full documentation content for LLM consumption is in [llms-full.txt](https://mcp-data-platform.txn2.com/llms-full.txt).

## Architecture and posture

Three facts that are commonly assumed the other way around, stated here so an agent grounding on this page gets them without following a link.

**The unit of access is the connection, not the end user.** This is the authorization design, not a missing per-user passthrough. A connection is a named binding to one downstream system under one operator-authored credential, and several connections may front the same system under different credentials at different permission levels: a read-only Trino account and a write-capable one on the same cluster are two connections, and each persona is granted the subset it may reach. Persona connection rules are deny-by-default (an omitted connections block or an empty connections.allow grants no connection at all), evaluated on every tools/call alongside the tool-pattern check, and applied to discovery through one shared predicate so search, fetch, list_connections, the portal search, and argument completion do not surface entities behind a connection the persona was not granted, with search and list_connections reporting a withheld count and a notice naming the persona (pkg/persona/filter.go IsConnectionAllowed, internal/platform/connscope). Two deliberate carve-outs: a catalog dataset whose URN maps to no configured connection is unattributable and stays visible, and a deployment with no persona registry has no scope to apply. api_routes narrows a kind=api connection further by (connection, method, path), which is how read-write and read-only access to one API are split across personas; a path glob is matched against both the path a call reaches and the catalog path its operation declares, so a rule naming /v1/orders/{id} governs every call that operation serves, and rules are authored in the config file or in the portal persona editor's API endpoints scope, where an operator selects operations rather than typing globs. Why the connection: the platform federates Trino, DataHub, S3, third-party MCP servers, and arbitrary REST APIs, and the one construct all of them share is a credential and an endpoint rather than a caller identity to pass through; an operator-authored credential is also auditable before any call happens. What it costs, stated without softening: the platform performs no per-user token exchange, no impersonation, and no session-user propagation (outbound OAuth exists but obtains a connection's own credential, not the caller's), so every caller granted a connection is indistinguishable to the downstream system, and warehouse row policies or column masks that key off the end user do not follow a caller through. Per-person policy is expressed as one connection per distinct policy outcome, which is workable when those outcomes are few. Per-user attribution comes from the audit trail rather than from distinct downstream identities: with audit enabled, each call records user_id, user_email, persona, tool_name, the connection when the call targets one, and arguments subject to redact_keys; audit requires a database and can be disabled, so a deployment relying on connection-scoping should keep it on. Operator guidance: tighten access by adding a connection bound to a narrower downstream account, not by adding a role that lands on the same connection. Full rationale: https://mcp-data-platform.txn2.com/concepts/authorization/

**mcp-data-platform is an OAuth 2.1 broker, not an identity provider.** No person authenticates to it: there is no login form, no user password to verify, and no MFA. A human's identity comes from an existing IdP (Keycloak, Auth0, Okta, Azure AD) over OIDC; /authorize redirects the browser there and refuses the flow outright when no upstream IdP is configured, and the roles and email that person is authorized against are the ones the IdP asserts. Service accounts authenticate with API keys instead, and their roles come from local configuration. It stores no human passwords, and no migration in the tree defines a password column. The secrets it does hold are machine credentials: API keys and the client secrets Dynamic Client Registration issues to MCP client software are bcrypt hashes, the authorization codes and tokens the platform itself issues are SHA-256 digests, and refresh tokens for upstream services are encrypted at rest (AES-256-GCM when ENCRYPTION_KEY is set; the server warns loudly at startup when it is not). It presents an authorization server toward MCP clients because the MCP specification requires a discoverable authorization server supporting Dynamic Client Registration, which upstream IdPs generally do not expose. The broker shape is what the spec requires, not a decision to reimplement identity. Implementation: exact redirect_uri matching for non-loopback and RFC 8252 section 7.3 handling for loopback (pkg/oauth/storage.go); plain HTTP to non-loopback hosts refused regardless of configuration, with private-use schemes excluded from AllowAllRedirectURIs (pkg/oauth/dcr.go); per-IP token-bucket limits on /token and /register (pkg/oauth/ratelimit.go); deny-before-allow authorization, default deny, fail-closed on unresolved persona (pkg/persona/filter.go); catalog metadata sanitized against prompt injection with detected attempts logged (pkg/semantic/sanitize.go, pkg/semantic/injection_logger.go).

**Engineering posture.** More than 1.25 lines of test code per line of production Go, with the security-critical packages carrying the highest ratios in the tree: pkg/oauth and pkg/middleware are both above 2:1. Fuzz suites cover pkg/oauth, pkg/auth, pkg/platform, and pkg/middleware. Every PR passes race-detector tests, golangci-lint, gosec, and Semgrep and CodeQL SAST, under a coverage floor enforced in CI. Release artifacts are Cosign-signed with GitHub build-provenance attestations, and supply-chain posture is tracked by OpenSSF Scorecard. These ratios are kept true mechanically: `make posture-check` recomputes them and fails when the tree crosses a stated line.

## Docs: Install

Read while standing a deployment up. These are the pages under `docs` > Install in the site nav.

- [Server Overview](https://mcp-data-platform.txn2.com/server/overview/): What the platform does, architecture, request flow
- [Installation](https://mcp-data-platform.txn2.com/server/installation/): Install via go install, Homebrew, Docker, or from source
- [Configuration](https://mcp-data-platform.txn2.com/server/configuration/): Full YAML reference with environment variable expansion: server, auth, personas, toolkits, enrichment, workflow gating, the session-init gate, the `purpose` argument on data-access tools (advertised, stripped, recorded, and required of handle-threading agents, with a purpose stated on a tool outside the gated set taken off the request and recorded rather than refused, and the agent instructions naming the gated set: the tools it names, plus a clause per wholesale-gated connection kind), per-user tool-call rate limiting, the tool-result context budget (`tools.result_budget`, default 32 KiB with per-tool overrides: enforced once on the MCP response to a model and only on results the model can recover the rest of: api_invoke_endpoint, graphql_query and trino_query, each cut in its own shape and steered to its export tool, while every other result (knowledge pages, tool help, prompts, documents, proxied tools) reaches the model whole; REST gateway, script and admin callers are never fitted, and meet only a connection's `max_response_bytes`, past which the REST route answers 413; it replaced the per-connection `max_inline_bytes`, which now has no effect), prompts (including the `server.builtin_prompts` per-prompt switch; `platform_info` no longer lists the library, which is reached by handle through `manage_prompt use`, `show_prompts` and `fetch`), resources (templates plus the full `resources.managed` block, including the `extract` limits for archive extraction (max_member_bytes, max_total_bytes, max_members, max_ratio, independent of the upload ceiling), the `max_upload_bytes` ceiling, the streaming upload path it bounds, and the part order both write routes read the form in), argument autocompletion (completion/complete), portal, database, audit, the call catalog (retention, the `calls.exclude_personas` list that stops an automated principal's calls being cataloged and embedded while leaving the audit log untouched, and the one caller that takes no knob: a managed script run, whose calls are audited and never cataloged because a run presents its author's persona and is by construction the re-run), session settings, and explicit session handles (platform_info-minted session_id, MCP 2026-07-28 sessionless readiness). Includes config versioning and hot-reloaded per-key database overrides. An S3 instance's `timeout` bounds a whole call, the request and the read of the response body, and now reaches the platform's own asset and managed-resource clients rather than only the `s3_*` tools; with none set those two take the deployment's object ceiling at a 1 MB/s floor instead of a flat 30 s, because a full-object read the platform itself sized is a throughput budget rather than a timeout (#1773) A `thumbnails` block (`enabled`, default on; `renderer_url`, default `http://127.0.0.1:9222`) points the platform at the headless Chrome it draws every asset, resource and collection tile in (#1787), and paces the worker (`concurrency`, `render_timeout`, `batch`, `lease`, `poll`, `max_attempts`, `retry_backoff`): a document whose attempts never finish is recorded as not drawable after `max_attempts`, held back longer after each (#1868).
- [Deployment](https://mcp-data-platform.txn2.com/server/deployment/): Docker Compose and Kubernetes deployment guides, including how connected agents pick up tool-contract changes across an upgrade. Kubernetes is plain manifests applied with kubectl (ServiceAccount, ConfigMap, Deployment, Service, Ingress, HPA, PDB, given in full); no Helm chart and no operator ship with the repository. Names PostgreSQL 16 and 17 as the supported majors, both covered by the migration gate on every change The Deployment manifest carries a second container, `chromedp/headless-shell` pinned by digest, that the platform draws thumbnails in: no Service or port, reached on loopback, never given an address to call; a document it draws reaches no network because the platform answers every request the page makes (#1787).
- [Deployment Shapes](https://mcp-data-platform.txn2.com/server/deployment-shapes/): Which backends a deployment needs, orthogonal to operating mode. The semantic stack (DataHub, optionally Trino and S3) for cross-enrichment; the API-and-knowledge shape on PostgreSQL alone, with no warehouse and no catalog, listing the twenty tools it registers and what it gives up (cross-enrichment, trino_*/s3_*/datahub_* tools, catalog search results, apply_knowledge with sink datahub); and the combined shape. Includes the minimal YAML for each, and running replicas over one database: sessions.store: database, the X-Platform-Instance response header naming the serving process, how the connection store rather than a replica's memory answers both what connections exist and a call that names one, and the local two-replica lane make dev runs behind a round-robin nginx that make acceptance connects to
- [The Data Stack](https://mcp-data-platform.txn2.com/concepts/components/): Why DataHub, Trino, and S3, what each component brings, and how cross-enrichment makes them greater than the sum of their parts. Components the platform composes, not a prerequisite list: a deployment can attach any subset, including none
- [Content Model](https://mcp-data-platform.txn2.com/concepts/content-model/): The four content layers and which one a given file belongs to. Resources are human-uploaded inputs used as-is, assets are AI-generated outputs, knowledge pages are curated facts to search and synthesize, and memory is per-user recall. Covers the canonical positioning statement held byte-identical across platform_info, the portal, and the docs; the resources section platform_info appends for the agent (search for a template before formatting a deliverable, resolve a named company file with search/fetch, treat prompt attachments as authoritative), tool-gated per persona; and the server.agent_instructions rule that makes an approved template mandatory; also names the built-in knowledge pages the platform ships (including asset references and the refresh loop, and writing a document that fits the reader's screen, each carrying a mermaid diagram of its mechanism) (embedded in the binary, reconciled at startup, badged Built-in and read-only, hidable and restorable per deployment, named by slug from the instruction baseline and manage_script help and fetchable by that slug)
- [Authorization Model](https://mcp-data-platform.txn2.com/concepts/authorization/): Why access is scoped at the connection rather than the end user, and what that boundary does and does not enforce. Several connections may front one downstream system under different credentials at different permission levels; personas are granted the subset they may reach, deny-by-default, on tool calls and on discovery alike; api_routes narrows an API connection by method and path. The trade stated plainly: no token exchange, no impersonation, no session-user propagation, so warehouse row policies and column masks that key off the end user do not follow a caller through, per-person policy is expressed as one connection per distinct outcome, and per-user attribution comes from the audit trail. Includes when to add a connection instead of a role, and how the model compares with a warehouse-native MCP server that inherits its warehouse's per-user policy inside that one warehouse

## Docs: End User

Using the platform. These are the pages under `docs` > End User in the site nav.

- [Tools](https://mcp-data-platform.txn2.com/server/tools/): Complete tool list for every toolkit (DataHub, Trino, S3, knowledge, memory, portal, gateways), plus platform_find_tools semantic tool discovery, manage_prompt with the use resolve-and-run verb (any handle: name, display name, mcp:prompt:<id>, free text), prompt resource attachments (a prompt carries its template, checklist, or brand asset as embedded resources and resource links, scope-checked at attach, promotion, and serve time), prompt versioning with approval provenance (content edits to approved shared prompts pend as draft versions until admin approval; served prompts carry version and approver; run counts from prompt-serve audit events), sharing an asset from the session (manage_asset share/list_shares/revoke_share: a recipient named by email or by a name resolved against the known-users directory, which refuses to guess between candidates; a restricted viewer or editor share that mails the recipient, or an authenticated link, or a public link that must carry an expiry), manage_table (register / list / unregister a stored CSV as a Trino external table, keyed by the reference a search hit carries so one action serves an uploaded resource and a saved asset alike), manage_resource (create / replace_content / get / list / delete / extract on the managed resource library; extract streams the members of a stored zip, gzip or gzipped tar into managed resources under a path, member folders kept as converted folder names, one member optionally filed under a fixed filename so if_exists=replace records the next version and followed tables move, zip-slip names, encrypted members, unsupported methods and any resources.managed.extract limit refused before anything is written, a mid-stream failure naming the members already written; the files a saved asset references: content as text or base64 bytes capped at the same portal.max_content_size save_asset uses, a create reporting the mcp:// URI save_asset's references argument takes, and a replacement keeping the resource's id, URI and filename so every asset referencing it serves the new bytes without being re-saved and every citation and prompt attachment keeps resolving -- recorded in the resource's version history with its author and change summary, restorable, and re-announced to connected clients; creating is scope authority defaulting to the caller's own user scope with the refusal naming the SCOPE and not the file, replacing is authority over that file, and a resource the caller cannot see is answered as absent; a managed-script run is judged as the person it acts for, since it authenticates as a principal that owns no file and is in nobody's library -- resource.Claims.OnBehalfOf, empty for every human, is read by VisibleScopes, CanWriteScope, CanAccessResource and CanModifyResource so a run reaches the file its author uploaded through the portal under their SUBJECT and its own creates land in the author's library rather than the principal's; and the lifecycle around those two writes -- get answering what is filed at an address (scope + path + filename) or what a reference names without the bytes, since search is relevance-ranked and a file it does not return is not a file that is not there, list reporting a folder and everything beneath it, create taking if_exists=replace so landing one rolling file per source is idempotent and the caller keeps no id, and delete removing a file and its version trail under the same authority a replacement takes -- refused while an asset references it, a prompt attaches it or a table is registered over it, since neither reference row is a foreign key and both deliberately outlive the file, counted rather than named because each carries an audience the caller is not necessarily in, with force=true deleting anyway and dropping any table over it; a knowledge page is NOT among them, since ParseCitableRef refuses an mcp:resource: citation on a shared page and no writer of knowledge_page_entity_refs bypasses it), the shared content-editing grammar on manage_asset, manage_prompt and manage_script (patch with anchored edits, locate, get_content, outline, stats, diff; text anchors never line numbers, all-or-nothing application, unified diff as output only, dry_run, base_version staleness checks, text-only refusal; the payload key required and the only one -- `replace` on op=replace, `text` on every other writing op, an omitted or undeclared key REFUSED by name rather than read as an empty replacement that deleted the anchor and reported success (#1804), so deleting is always `"replace": ""` written out; syntax-aware region naming so an HTML/JSX/SVG asset is addressed by CSS selector or heading with balanced element spans, while markdown is addressed by heading and structureless content by anchored edits only), the semantic index over catalog dataset descriptions that makes a fact applied to the catalog reachable from a topical query naming no entity, the governance search source that returns DataHub glossary terms, tags, and domains as entities in their own right (with their definitions in the hit, and fetchable by their URNs to the definition plus the datasets that carry it), the `purpose` argument the platform adds to every data-access tool and to the two tools that write an asset, save_asset and manage_asset (one sentence naming the wider task the call serves, stripped before the handler and recorded on the audit row, refused with PURPOSE_REQUIRED when a handle-threading agent omits it, and taken off the request and recorded rather than refused when stated on a tool outside that set; an asset is the output a person opens weeks later and the first question they ask is what it was for, so the sentence is recorded beside the write, while apply_knowledge stays outside the set because what it applies is itself the explanation), the sessions search source and `fetch mcp:session:<id>` that let an agent recall its own past work by what it was for — the session's summary, the assets and insights it produced as references to follow, and its call timeline with each call's purpose and, where the catalog recorded one, its kind, outcome and `mcp:call:` reference — scoped in the read so another caller's session id is answered exactly as an id that never ran, and the uniform structured error envelope, and the MCP tool annotations every registered tool advertises on tools/list (readOnlyHint on every tool, since the specification tells a client to assume a tool is NOT read-only when the hint is absent and clients that act on that confirm every unannotated call, which made the mandatory platform_info-then-search-then-fetch opening three write prompts; destructiveHint stated rather than omitted on every write, because the default for an absent one is true; and the hint describing the TOOL rather than the call, so a mixed read/write tool like manage_asset or s3_object advertises the most it can do, held against internal/toolwrite by a structural gate so the advertised hint and the draft write barrier cannot disagree)
- [Content Types and Viewers](https://mcp-data-platform.txn2.com/server/content-viewers/): Where an asset's or resource's media type comes from, and what renders it. Content-type detection at every write path (save_asset, manage_asset update, manage_resource create and replace_content, api_export, resource upload) with alias normalization, a bounded-prefix sniff that keeps streaming exports streaming, and a hard rule that detection may only reclassify into passive families, never into text/html, text/jsx or image/svg+xml. Where detection cannot help, a declaration is required rather than guessed: save_asset and manage_resource create refuse a write with no content_type, because SVG, HTML, JSX and Markdown all read as plain text to a byte sniffer and text/plain under nosniff is a file that silently does not render, while replace_content keeps the type the resource already carries so a refresh cannot reclassify a file under every reference to it; the types to choose between ship as a built-in knowledge page generated from the same tables the code decides on. One stored-type allowlist across the three doors that take a caller-declared type for string content (REST inline create, save_asset, manage_asset update), with application/xhtml+xml absent; the byte-carrying resource upload keeps a denylist so the reference library still takes the long tail of document formats. One shared renderer registry across the portal viewer, public/guest viewer, collection items, and the resource viewer: a searchable collapsible JSON tree with JSONPath copy, NDJSON, CSV/TSV tables, image zoom and pan, audio and video with seek, embedded PDF, a Parquet viewer that reads the file by byte range (schema with the Trino type each column registers as, file facts, one row group at a time) and a Table | Records toggle on JSON lines (#1833), CodeMirror for structured text and code, and a metadata card for anything else. Per-family inline size limits, read from the one registry by the asset page, the preview modal and the admin asset viewer alike, with a failed content read shown with its status, Retry and Download rather than a loading state (#1874), and raw-content serving with nosniff, sanitized types, attachment-only active types, byte-range support, and a private-by-default cache directive. What a public share page actually loads: its chrome and its stylesheet inline, and the renderer as a module reference to /portal/view/_assets/, where each family's viewer is a separate content-hashed chunk the browser fetches only if the asset needs it, so a markdown document does not ship CodeMirror, the JSX transformer, the CSV parser or the diagram engine, and a document with no mermaid fence does not ship the diagram engine either; the chunk route is outside both the share access gate and the viewer rate limiter, since there is no token in the path and the same bytes serve every viewer, while the limiter is sized for page loads and one cold view with a diagram in it fetches around thirty chunks at once; its immutable caching means the second share someone opens costs no JavaScript, and a chunk that does not arrive (a tab left open across a deploy) is caught by an error boundary rather than blanking the page. The stylesheet is compiled against the viewer's own bundle rather than copied from the portal SPA. The public viewer's Content-Security-Policy, where one policy has to serve both the viewer page and the untrusted artifacts that inherit it in blob: frames: inline script, 'self' for the bundle, and https sources stay, plaintext http and 'unsafe-eval' do not, and each client-rendered family (HTML, JSX, markdown, SVG) is verified against a live stack by `make frontend-e2e-public-viewer`, which is not part of make verify. A JSX artifact carries a second, tighter policy of its own in the document the renderer builds around it, whose img-src and connect-src name the asset-reference route on the viewer's origin with its path -- so a JSX dashboard shows a referenced logo and loads a referenced data file, while the frame still reaches no other path on the platform
- [Asset References](https://mcp-data-platform.txn2.com/server/asset-references/): How an asset names something instead of carrying its bytes: a managed resource by its mcp:// URI, or ANOTHER ASSET by its mcp:asset:<id> reference. A save declares what its content names (`references` on save_asset and on manage_asset update/patch, capped at 20 across both kinds, replacing whatever the asset referenced before -- absent leaves them alone, an empty list removes them all), the content itself writes the reference where it loads the file, and every viewing surface rewrites the DECLARED references into a serving URL as it serves: the portal asset and version reads, the public share and collection-item reads, and the admin console's reads. The rewrite is a whole-document replacement over textual content rather than an attribute rewrite, so a reference resolves in an img src, in a fetch() inside a <script> block, or in a markdown image alike. A reference string in the content that was never declared is served exactly as written and resolves to nothing, so the grant is the declaration and never a string that happens to appear in the body; save_asset and manage_asset update/patch report those as undeclared_references, judged against the references in effect after the call. A reference LOADS a file and is not a link: the rewrite yields a URL serving raw bytes and the sandboxed frame blocks navigation, so a write using one as a link target (an <a>/<area> href in HTML, JSX, SVG or markdown, a markdown [text](ref) link or <ref> autolink) is refused naming each one, with nothing written; a markdown image is accepted (#1875). The declaration is checked once against whether the AUTHOR can REACH the target -- for a resource, visible scopes OR write authority over its library OR the uploader arm, the same rule the resource detail/content/replace routes and knowledge fetch resolve a NAMED file through, so an administrator may reference a file in a persona library they do not belong to; for an asset, ownership and shares -- and refused, naming only the reference the author wrote, with nothing created (library membership alone remains the rule for ENUMERATION, a listing or a search, which hands a caller material they did not name); from then on the reference carries the REFERENCING ASSET's audience: anyone that asset is shared with can load the target through it, including an anonymous viewer of a public link, which the tool response states at the moment the reference is made (the same grant model a managed script uses, where a run acts as its version author). That rule is deliberately NOT the referenced asset's own shares: the reference is served to a reader with no session, so there is no identity to resolve them against; what protects the target is the author's own read at declaration time and the notice stating the consequence. An ASSET reference resolves to the referenced asset's CURRENT content on every read, which is what makes a refresh loop expressible -- a scheduled script rewrites a CSV or JSON asset hourly and a dashboard that names it renders the new numbers without being re-saved and without an agent spending output tokens carrying the data across. The URL is absolute and takes no session, because an HTML or JSX asset renders inside an iframe with an opaque origin and a public share is read by someone with no account: a 256-bit token minted per (asset, target) pair is the whole authorization, it reaches a reader only inside content they could already open, and it resolves to nothing on any other asset's path. A referenced asset is served through its OWN reference list, one level and never a walk -- the rewrite writes URLs and following one is the reader's next request -- so a CYCLE between two assets is answered with content each time rather than followed; the portal refuses a self-reference outright. A surviving reference keeps its token so a reader's open page does not break on every save. manage_asset get_content is deliberately NOT rewritten -- an agent handed a rewritten URL would patch a platform-internal path back into the asset and lose the reference. A deleted target (a soft-deleted asset included) answers 404 and leaves the asset rendering with one thing missing; an unreachable reference store serves the content as stored; a copy carries only the references its new owner can read for themselves, each under a fresh token, so a copy never inherits a grant it did not earn; and a version is rewritten against the asset's CURRENT references, since the references belong to the asset rather than to any one version. A referencing artifact's THUMBNAIL is drawn by the platform's renderer in a frame under the same policy, from the same definition, that the viewer's frame uses, with the platform answering the frame's reference requests from its own routes, so a reference resolves while the tile is drawn as it does for a reader; the frame reports what it could not load and a document with a failed reference is RECORDED as not drawable with that reason rather than stored, and an owner (and an administrator on any asset) can discard a stored tile or a recorded failure from the Thumbnail panel in the metadata sidebar to have it drawn again without the asset's version moving. The portal manages the same references a person can see: the asset viewer's sidebar carries a References panel where a resource row names the file, its scope and its type (with a thumbnail for an image, loaded through the reference's own URL rather than the target's route, so it renders for a reader who was only ever shown the asset) and an asset row is marked as one and names the asset, its type and its owner; every row carries the reference string with a copy control, because adding a reference does not change the asset's content and the markup has to name it for the target to load. An owner, a shared editor and an administrator add through a picker with a tab per kind -- the resources they can reach, the assets they can open -- which states what the reference gives away and names the asset's CURRENT audience (public link / shared with people / not shared yet) before it is confirmed, and remove, warning first with the lines the stored content still writes the reference on and proceeding on confirmation. The scan does not always run (binary, over 2 MB, or a storage fault) and the response says WHICH: "the content does not name this" and "we could not look" produce an identical empty list and mean opposite things, so a removal is confirmed whenever the check did not run. A portal add and remove log asset_reference.granted / .revoked with the target's kind as well as its id, the same record a save writes. A reader with no edit authority sees the list and is offered neither control; a reference whose target was deleted is flagged and named rather than dropped. The reverse view is on both ends: a Used by section on a resource AND on an asset names the assets referencing it, flags any carrying a public link, and COUNTS without naming the ones the reader cannot open; it is bounded at 50 and reads as "at least n" when the bound cut it, and an asset's own section is refused to a reader who cannot open that asset. A delete warns with the same list, holds its button disabled until the check answers, and says so when the check fails. No configuration: references are available wherever there is a database, a resource reference additionally needs a managed-resource layer (refused with that reason where there is none, while asset references still record). /portal/refs/ is mounted at the composition root as its own prefix beside /portal/view/, because the portal UI claims /portal/ and would otherwise answer every reference URL with the SPA index.html. The reference route's per-client limiter is sized at 20 times the portal.rate_limit values with their defaults applied first, and the thumbnail renderer's in-process requests to it are not counted (#1791).
- [Provenance](https://mcp-data-platform.txn2.com/server/provenance/): What an asset was built from, and how the platform knows. Every asset write (save_asset, a manage_asset content update or patch, trino_export, api_export) captures the calls that fed it by reading the audit log at write time: the default window is every data-access call the session made since its previous capture, and an agent that knows better names the calls itself with `sources`, citing the `call_id` (or `mcp:call:<id>` reference) each query and API invocation now returns in its own result. Being in the window is a record of the session's work, not a claim that the call produced the asset: only a NAMED call reads `satisfied` in the call catalog, where naming is either the caller's `sources` (the whole capture is cited) or a capturing export's own record of the statement it streamed (that one call is badged Source inside a windowed capture). Captures accumulate, one per write, so an asset's provenance reads as the history of what fed each of its versions. Each capture holds both the audit event ids and a snapshot of those calls taken at write time (kind sql/api/tool, tool, connection, the statement for a query or the request for an API call — the path it addressed with the values it passed substituted in from the connection's catalog, the query string it sent, and its request body, bounded, which is what tells two calls to one operation apart — the purpose the caller stated, outcome including a failed call, duration, timestamp), because audit rows are retained for a fixed window and assets are not. Sources resolve only among the caller's own calls, and reading the audit log rather than a per-process buffer is what makes a capture correct across replicas. The portal groups the panel by capture, marks a cited capture and a truncated one, and links each call to its reference and the whole session; it leads with the newest capture and puts every earlier one behind a single disclosure that opens them one at a time, since a scheduled refresh writes a capture per run. Reads are bounded because captures are not: a listing (`manage_asset list`/`search`, the assets and shared-with-me routes) carries a `provenance_summary` -- capture and call counts, the first and last capture times, the newest capture's tool and session -- and never the captures; a single asset read (`manage_asset get`, `GET /portal/assets/{id}`, the admin twin) carries the newest 20 captures plus `captures_total`; and the rest are paged newest-first through `manage_asset action=provenance` and `GET /{portal,admin}/assets/{id}/provenance` with `offset` and `limit` (default 20, max 100), authorized exactly as the asset is. A capture belongs to the version it produced, so a version prune removes its capture in the same transaction, keeping the origin capture and any capture that names no version; a startup pass trims the captures that outlived their versions on assets written before that rule existed
- [What Produced a File](https://mcp-data-platform.txn2.com/server/content-producers/): The record of what wrote a portal asset or a managed resource: each managed script, agent session and person that created or modified it, when they first and last wrote, how many times, and the last version they produced. It answers what provenance does not -- provenance records the data CALLS a report's content was built from and cannot be read backwards, a script's link to its declared outputs was one idempotency-key string nothing joins on, and a resource recorded an uploader that for a run was the script's NAME, severed by a rename. A producer is recorded by id, never by name, so a rename changes only the label and deleting a script leaves its rows standing. Exactly one producer per write: an agent's save_asset is filed under its session (which names the person on its own page), a portal write under the person, a run under its script id -- which the run stamps on its own session, since the middleware sees only the script:<name> principal. Recorded at the write funnels every write already passes through (the asset store's insert and CreateVersion, where version 1 is the content half of the create and is not counted twice; CreateResource and ReviseContent), best effort and never failing the write. Read from both ends: an asset's and a resource's Written by panel, and a script's Files written section listing everything it has written across every run, which no run history can answer. No foreign key to scripts, portal_assets or resources, so a deleted script renders as one that no longer exists and a deleted file stays listed by id. The migration derives rows only from an asset's script:<id>:<output> idempotency key and a resource uploaded by an unambiguously named script; a history that was never recorded is not reconstructed by guessing
- [Registered Tables](https://mcp-data-platform.txn2.com/server/registered-tables/): Registering a stored CSV, JSON-lines or Parquet file -- a managed resource or a portal asset -- as a Trino external table (a JSON-lines file, written by `trino_export` or a script as `format=jsonl`, returns every string exactly, line breaks included, and a null as NULL, and is refused by line number when a line is blank, holds anything but one object, repeats a key, or holds a key whose values cannot be one type, #1820; its columns are TYPED from every record -- all booleans BOOLEAN, all integers BIGINT, any fraction DOUBLE, anything else VARCHAR, objects ROW, lists ARRAY -- and a Parquet file registers from its footer alone, read by range, with the types the footer declares mapped to Trino types and a type nothing reads back exactly (TIME, UUID, FLOAT16, INTERVAL) refused by column, the catalog set to `hive.timestamp-precision=MICROSECONDS`, and a JSON-lines registration made before typing keeping its VARCHAR columns across a follow as `all_varchar`, #1833) over the directory the file already sits in, so it joins to warehouse tables without being copied or ingested. Covers the operator's `scratch: {catalog, schema}` target on a Trino connection and the Hive-over-object-store catalog behind it; the three surfaces (the portal's Query as a table panel on both kinds, the REST routes, and the `manage_table` tool, whose register / list / unregister actions are keyed by the `reference` a search hit carries -- mcp:resource:<id> or mcp:asset:<id> -- so one action serves either kind and an agent can register a file a person uploaded without a portal step); who may register (authority to CHANGE the file, not to read it: an asset by its owner or an administrator, a resource by its uploader or an administrator of its scope, one rule resolved in one place for both surfaces); and every refusal with its reason, including a leading UTF-8 byte-order mark before a QUOTED first header field, which failed the parse on line 1 and was answered as "the file has no header row" -- the shape every Facebook Insights, Excel "CSV UTF-8" and Google Ads export takes (#1774), and a comma in a heading, which Hive refuses as a column name whatever the quoting so that the same export then failed at the DDL after its rows had already been corrected -- each comma now becomes a space, the comma being the only character Trino 476 was found to refuse; a record that ends BEFORE the header does is not a defect and registers, its absent trailing columns coming back null, while a record carrying MORE fields than the header is still refused (#1779) -- a first line the reader cannot parse, whose refusal now carries the parse error, and a platform failure, whose 500 names the stage it stopped at while every failed registration, refused or failed, writes an audit event carrying the caller, the connection, the file and the whole error (#1775). A CSV whose cells carry line breaks cannot be read at all: the Hive reader splits records on newlines before the quotes are seen, so such a file registers as a torn table with no error anywhere, and registration therefore inspects the whole body it was already reading and refuses an embedded line break, bytes that are not UTF-8, and lines ending in a bare carriage return rather than a newline (#1445 -- Go's csv reader splits on \n exactly as the Hive reader does, so such a file is ONE record to both; every LONE \r is rewritten as \n before anything is counted, and the rewrite is kept only where it RECOVERS RECORDS, by two measures for two regimes -- a body with NO \n at all is one record to every reader here and nothing in it is ambiguous, so the PLAIN record count decides (which is what stops a classic-Mac file whose rows do not match its header from being merged into one row, since it scores no better on header width after the rewrite than before, and kept it is refused honestly by checkFieldCounts instead); a body that ALREADY has \n-delimited records is ambiguous, since splitting a cell adds a record exactly as a line ending would, so records OF THE HEADER'S WIDTH decide and a \r inside a cell stays the in-cell break it is. One stray line feed no longer disqualifies an otherwise CR-delimited file -- which is what keeps a refusal from naming columns the file does not have), naming the rows and columns; asked (`repair=true`, `"repair": true`, or the portal's control on the refusal) it saves a CORRECTED VERSION of the file through the version trail its kind already has -- the uploaded bytes stay as the version before it, the correction is revertible, and the sentence describing the repair is recorded on the new version for EITHER kind so the version panel says why the file changed (#1450 -- the resource reviser took the summary and dropped it, since `resource.Revision` had no field for it, so a corrected managed resource read as an upload while a corrected asset carried the reason) -- and registers that version's directory, while a record whose field count differs from the header is refused rather than padded or truncated -- and the correction is NOT OFFERED for such a file, nor for one the reader cannot parse through, since both are settled by the same read that found the defect (#1449 -- `Correctable` inspected the encoding and nothing else, so a caller told to register again asking for the correction got a second, different refusal naming a problem the offer had not mentioned; neither condition refuses a file on its own, so a ragged CSV with newline endings and no in-cell break still registers) -- and a wide encoding (a UTF-16/UTF-32 mark, or a NUL byte) is refused OUTRIGHT with no correction offered, because every byte of a UTF-16 file is also valid windows-1252 and "correcting" it would write mojibake back as the person's file (#1447 -- the NUL test runs BEFORE the UTF-8 test, since a NUL is valid UTF-8 and a markless UTF-16 export of ASCII content would otherwise pass it and register columns whose names carry the NULs), and a file carrying one of the five bytes windows-1252 leaves undefined (0x81, 0x8D, 0x8F, 0x90, 0x9D) is refused on the same ground (#1448 -- the decoder emits a replacement mark for each of them and returns NO error, so such a file converted "cleanly" into the person's new version with those marks in it and a summary naming windows-1252 as its source; the code page is now the answer only where every byte is one it defines, and the four defined bytes beside them -- 0x80, 0x8E, 0x9E, 0x9F -- still convert). The correction is written before the last checks and before the DDL, so a refusal, an unreachable coordinator, or a database that cannot record the registration can follow one: the answer leads with what changed and the audit event records it either way (#1446 -- all four failure paths carry the correction, and a registration that fails partway reconciles both ways: either store write failing after the CREATE drops the table, and a replacement whose CREATE failed after its DROP ran forgets the row whose table is now gone, while one whose DROP failed changed nothing and is left alone -- so no surviving row advertises the columns of the version before it and the "register it again" the answer ends on is true). Two consequences a reader has to know: a CSV table's columns are all VARCHAR because that is the Hive CSV storage format's rule and not a platform choice, so a join to a typed column needs a CAST (a JSON-lines or Parquet table's carry their types, reported as `column_types`); and a directory holding a sibling Trino would read is refused by name, because Trino reads every non-hidden object under an external location and parses it as CSV without erroring, which is why portal thumbnails take hidden filenames that the check skips as Trino does. A new revision or version moves the head key and the table keeps serving the one it was registered against -- reported as stale on the panel, on a search hit and in manage_table action=list -- while an overwrite at the same key needs no re-registration. A file can carry several registrations, and the discovery layer reports the set rather than its newest member (#1627): `fetch` returns `tables`, one entry per registration, newest first, carrying the columns, the registration_id, follow, repair and follow_error the listing carries, while a `search` hit keeps ONE `table` and it is the newest registration whose follow_error is empty, so a hit never points at a table a follow has already reported gone. THE KEY IS THE USE (#1666): `tables` is the query view a fetch document and a search hit carry, the document's rows carrying the columns so the query needs no second call while the hit's one `table` carries none (a hit is a pointer chosen from a ranked page, and a page each carrying a wide table's column list is a large answer to which record to read), and `table_registrations` is the maintenance view `manage_table action=list`, the per-file REST route and a refused resource delete answer with, carrying the registration_id an unregister takes; the listing answered under `registrations` until then while every other surface said `tables`, so a script checking for an existing registration read `tables`, found nothing and re-registered every run, changing the registration id each time. The lifecycle is register, register again, go stale, unregister, delete the file: a file is not limited to one registration, so re-registering the SAME name on the same connection replaces that one (the repair for staleness) while a different name or another connection adds a second table over the same file, and unregistering drops the table while leaving the file untouched. The scratch schema is a shared workspace: resource scopes and asset ownership are NOT carried into Trino, the persona prefix on a table name is collision avoidance rather than a boundary, and what keeps a registration off the warehouse is the Trino identity the connection authenticates as, never the platform's read_only flag. Because that schema is shared, what is registered on it is listed rather than found one file at a time: the portal's Scratch Tables section (`/scratch-tables`, one registration at `/scratch-tables/{id}`) and `GET /api/v1/tables` / `GET /api/v1/tables/{regID}` span both source kinds, with the qualified name, connection, source file, column count, who registered it and when, and whether the table has fallen behind that file or the file is gone entirely (#1472). Visibility follows the CONNECTION -- the registrations on connections the persona is granted, every one of them for an administrator -- which is the boundary Trino itself applies and deliberately wider than the register form's connection list, since that narrows to connections that can hold a NEW table and a connection turned read-only would otherwise hide a table from the person who made it. Staleness and the source's name come from a bulk per-kind source read (`tableregister.Sources`, over `portal.AssetStore.GetByIDs` and `resource.Store.GetByIDs`) whose `CanModify` is a field on the answer rather than the answer itself: seeing a table is decided by the connection, dropping one by authority over its file, and unregistering goes through the source's own DELETE route so that rule stays written once. Registering stays on the file's own page because it needs the file's header row. A registration FOLLOWS its file by default (#1536): a revision of a managed resource, an edit or revert of an asset, or a script's `platform.export` / `publish_data` moves every following table onto the version it wrote -- the same DROP/CREATE a re-registration runs, columns re-read from the new header, under the same registrant, before the write returns -- and the write's result (`table_changes` on `replace_content`, the `manage_asset` content edits, the portal and admin content routes, the export record, and the run log as `table_changes: <output>: ...`) names every table over the file as followed or as pinned and now behind -- a change report, named apart from the `tables` a caller queries (#1666). `follow=false` pins a table to the version it was registered over, which holds because every version -- a script output's included, at `scripts/<script>/<asset>/<run>/content.<ext>` -- has a directory of its own, and a table pinned over an output written before that, whose directory holds several versions, is reported by the next write as reading them all rather than as followed (#1851); the choice is stored on the registration, shown as Pinned / Follows the file, and settable again by re-registering the name. A follow never fails the write: a refused CREATE puts the table back and records the reason as `follow_error`, which the listing shows on a Behind-the-file table; a new version that cannot be read as a table is refused as a registration would refuse it. The `repair` choice is carried on the registration too (`table_registrations.repair`, #1577), because a registration made with it corrected the file on the day it was made and then stopped: a source repeating one correctable defect on a schedule -- a weekly spreadsheet export, a script writing a CSV whose text fields carry paragraph breaks -- stranded its table permanently, and a producer reading its own rows back through the stalled table then regressed its own file while reporting success. A following registration carrying the choice now saves the corrected bytes as the file's NEXT VERSION through the same reviser a register-with-repair uses, under the REGISTRANT rather than whoever made the triggering write, and moves the table onto that version; the defective version stays below it and is revertible, its change summary is the one `repairSummary` produces, and the write's result says the table followed AND that a corrected version was saved. The file is corrected ONCE for the version whatever the number of tables over it (the head is read once per write behind `*followHead`), and every following table lands on it. The correction is REPORTED once too, on the outcome of the registration it was made for -- the oldest one carrying the choice -- rather than on whichever follow read the new head first, which is the newest registration over the file whether or not it asked for anything (#1583); a table registered with `repair` off is told only that it moved. Nothing that was refused becomes accepted: an uncorrectable defect (a wide encoding, a NUL, records that do not match the header, bytes the reader cannot parse through) leaves the registration behind with its reason and writes no version, a registration made WITHOUT repair is left behind with the sentence offering the correction, and a pinned one is never moved onto a new version so the choice does nothing for it. The choice is the one made at the registration that is current, and is reported in `manage_table action=list`, on the REST views and as a *Corrects the file* badge on the portal panel. Every follow is a `table_follow` audit event under the registrant with the version and the columns before/after.
- [Scratch Catalog](https://mcp-data-platform.txn2.com/server/scratch-catalog/): One complete, copyable Trino setup that makes every scratch feature work -- registered tables (CSV, JSON lines, Parquet), archive extraction followed by registration, and webhook sources (#1888): two Trino identities (`mcp-server` read-only everywhere it queries, the scratch catalog included; `mcp-scratch` with `all` on the scratch catalog, read-only `system`, nothing else); the Hive file-metastore catalog with every required key (`hive.recursive-directories=false`, `hive.timestamp-precision=MICROSECONDS`, `hive.allow-register-partition-procedure=true`, one S3 client per catalog so one store per catalog) and the rule that the coordinator AND every worker restart on a catalog change; file-based access control with a `procedures` rule for the scratch identity on `sync_partition_metadata|register_partition|unregister_partition` (a catalog `all` rule does not grant procedures, and with no `procedures` section only `system.builtin` procedures run) and a `functions` section that keeps the table functions a deployment uses (OpenSearch `raw_query`); the platform connections; a checklist with the expected answer for each check; and every scratch refusal with the setting that fixes it. The Trino connection test (`POST /api/v1/admin/connection-instances/trino/{name}/test`) runs the procedure checks on a scratch connection that accepts writes, calling each procedure on a table that does not exist so "Table ... not found" is the passing answer, and names every missing rule or property in one 503.
- [Inbound Webhooks](https://mcp-data-platform.txn2.com/server/webhooks/): Administrator-managed sources at `POST /hooks/{source}` (#1870) authenticated by hmac (optional signed timestamp window), header token, basic or path token, with secret rotation and the CloudEvents handshake; a 202 only after the events are written to object storage as gzipped JSON-lines segments, 503 with Retry-After when a source's buffer is full or the write fails; each compaction window (`compact_every_minutes`, an hour by default, down to a minute) compacted after it ends into one deduplicated Parquet file stored as a managed resource in the source persona's library; the view `webhook_{source}` serving each window from the raw segments until its compacted partition is registered, so events are queryable on acknowledgement; per-source retention, the source's status page, the `webhooks:` config section and the webhook_* metrics.
- [API Operation Browser](https://mcp-data-platform.txn2.com/server/api-browser/): Two read-only surfaces over the indexed operations of every API connection: /admin/apis lists what has been loaded, catalog by catalog and spec by spec, including catalogs no connection references yet; /apis lists what one caller may call, narrowed by their persona's connection rules and by the route policy api_discover applies, so an operation a deny rule refuses is absent from the list and from the connection's count. An operation's parameters, request body and per-status responses come from the same resolution api_discover returns, and each one carries a copyable curl against POST /api/v1/gateway/{connection}/invoke with the method, path and required parameters filled in, which is what a non-MCP client (Apache NiFi, a cron job) needs to compose a call. Read-only: the page names operations and makes no upstream calls
- [Session-Start Notices](https://mcp-data-platform.txn2.com/server/session-notices/): The `notices` block platform_info attaches to the first call of every session, for the person who works through an agent and opens neither email nor the portal: unresolved feedback other people left on assets the caller owns (the caller's own threads and their own replies excluded, capped at ten with a total count, each carrying the asset's mcp:asset: reference for fetch and manage_feedback), and the assets, collections, and prompts newly shared with them by name (a public link nobody was named on is not a share with anyone; who shared it is the person who made the grant, not the artifact's owner), and the automations they own whose latest run failed (#1934: the run to open, its cause and retryable, the error's last line, consecutive failures, last success, whether it is scheduled; listed at every session start until a run succeeds, `new` marking a failure since the last briefing). Each list is capped and the watermark advances past what did not fit, so the note tells the agent to name the portal as the complete view. Delivery is single-shot: a per-user watermark advances as the digest is issued, so the next session hears only what is new, and the agent instructions in the same response tell the agent to relay it rather than act on it silently. A caller never briefed gets a 30-day window rather than their whole history, and a half that failed to load holds the watermark back rather than being swallowed. No configuration: present wherever the portal and a database are
- [Resources](https://mcp-data-platform.txn2.com/portal/resources/): Human-uploaded reference files surfaced to AI assistants via MCP resources/list and resources/read, with global, persona, and user scopes. SUPERSEDING the Library dropdown, the All view, the recently-updated strip and derived-only folders described below: the page is a FILE MANAGER (#1872) -- a folder tree (My Resources, Global, each readable persona, and an administrator-only People folder from GET /api/v1/resources/people), a path bar, one listing (folders then files, Name/Modified/Size sorting on the server via sort=name|name_desc|updated|updated_asc|size|size_desc), a preview pane, file-manager selection (click, Shift, Cmd/Ctrl, Cmd/Ctrl-A), drag-to-move with Undo, a context menu, F2 rename and the WAI-ARIA tree keys; folders are STORED (migration 000160 resource_folders, POST and DELETE /api/v1/resources/folders, a folder outliving its last file), and the page never says "library". PostgreSQL metadata, S3 blobs, REST API at /api/v1/resources. Upload many (#1862) loads files, a folder or a browser-unpacked .zip in one action with shared library, folder, tags and a {name} description template, four uploads at a time through POST /api/v1/resources with if_exists=skip_unchanged: identical bytes (SHA-256, recorded per version as content_sha256, migration 000158) answer outcome unchanged and write nothing, different bytes become the next version, and every create answers outcome created/revised/unchanged. A refused registration opens a dialog rather than wrapping into the 320px details column, and a tabular cell that does not fit is read by opening the row's record in a dialog rather than through a title tooltip (#1780, #1781). A text resource's source is editable in the portal (Preview/Source switch, the asset viewer's own editor, Save writing the next version through the replace-content route with a change_summary of "edited in the portal"); the button beside Download is Edit details and has always edited the record rather than the file (#1775). Also discoverable through the universal search: a background indexer embeds each resource's metadata plus a bounded text prefix extracted from the uploaded file (a PDF's and an Office document's as well as a text file's, so a deck is found by a phrase on one of its slides), search hits carry an mcp:resource:<id> reference and an MCP resource link, and fetch returns the file in whichever form it admits -- text inline, a PDF as its extracted text, an Office document as its XML parts, an image as an MCP image block, anything else as an MCP resource block carrying the bytes -- with metadata plus the canonical URI reserved for a file over the 1 MB inline limit (#1657). Content is revisable in place: a replacement upload keeps the resource id, URI, and filename so references and prompt attachments keep resolving, every revision is recorded in a bounded version history (default 10, oldest blob pruned) with download and restore, a revision the platform wrote on the uploader's behalf carries the reason it was written and the panel renders it beneath the row (#1450), matching what a corrected portal asset's version has always shown, reads through all three surfaces (MCP read, search fetch, REST download) produce resource_read audit events, and the portal shows per-surface read counts, last-read recency, and a never-read flag. One resource opens at a route of its own -- /resources/{id} for the reader, /admin/resources/{id} for the administrator -- taking the same chrome a portal asset takes: the content at the page's full width in the page's own scroll region, with the metadata, tags, canonical URI, usage rollup, version history, table registration panel and attaching prompts in a sidebar beside it, and Download, Edit and Delete in the header under the uploader-or-administrator rule. The library is browsed as a TREE of folders (#1530): its LOCATION -- which library, which folder -- is a route segment (`/resources/lib/global/data/media-manager`, always at least two segments so it can never be confused with `/resources/{id}`, which is exactly one) rather than a query parameter, because `readPath()` drops the query string; each level is a distinct address that a reload returns to, a link opens, and Back steps out of one folder at a time, with breadcrumbs from the library down that navigate to every level and that the resource viewer renders as the same component. A folder's count is everything beneath it at every depth, written `12+` while pages are still loading because it is how many have arrived rather than how many the folder holds. Search spans the WHOLE library rather than the open folder, each hit showing the path it was found at with a Reveal control that walks the tree to it. The library reads like the Assets gallery (#1553): ONE Library dropdown (All, Mine, one entry per persona, Global) replaces the tab strip and All is where both pages open, a persisted grid/table switch replaces the old images-only-folder heuristic and drives the folders, the recents and the files alike, so EVERY file gets a card. A resource's tile is a PNG stored beside its object (#1554, migration 000134), drawn by the platform's own headless renderer since #1787 (the browser capture queue and its `PUT` and `thumbnails/pending` routes are gone); `GET|DELETE /api/v1/resources/{id}/thumbnail` are its surface, and a tile is owed when it is missing or older than the row's `updated_at` — a TIMESTAMP rather than a version because a resource row has none (its revisions live in `resource_versions`), so recording a tile must not touch `updated_at` or every tile would make itself owed forever. The tile used to be the file itself, scaled by CSS and blank past a cutoff. A type nothing draws keeps its content-type icon; `ThumbCard` now falls back to that icon on a FAILED load, not only on a missing URL. That capture rule is one definition per language rather than four (#1568): `internal/thumbtypes` in Go, `CAPTURABLE_TYPES` in `ui/src/lib/thumbnailSupport.ts` for the browser gate AND the capturer's dispatch, held to each other by a Go test that reads the TypeScript table; since #1882 a type is matched WHOLE (canonical media type, parameters removed, aliases settled) or by a `+json`/`+xml` structured suffix, any other `text/` type drawn as plain text, rather than as a fragment found anywhere in the type -- every Office Open XML type contains "xml", so a workbook, Word document or presentation was drawn as the text of its zip bytes -- and migration 000162 cleared the tiles drawn that way; the merged list is html, jsx, svg, pdf, markdown, csv, json, plain text and the raster formats a browser DECODES (png, jpeg, gif, webp, avif, bmp, ico -- named one by one rather than as an `image/` prefix, since a TIFF or HEIC can never be captured and the pending query is a bounded window, so trying one is work that fails every time), with markdown, csv, json and plain text captured in both schemes; since #1794 a PDF is in that list and shows its FIRST PAGE, rasterized rather than borrowed from the viewer's frame -- the tile page rasterizes page one with pdf.js (a chunk of its own, loaded by a PDF tile and nothing else) onto a canvas sized in device pixels and scaled to cover, travelling by URL like a raster image rather than through the page's JSON payload, drawn once for both schemes, and held to a 32 MB source bound (`thumbtypes.LargeSourceLimit`, because only page one is decoded and one 300dpi scanned letter page already measures about 2 MB) with a password-protected or unreadable document recording its reason and keeping its icon; since #1802 csv and tab-separated share that bound, the platform handing the tile page the HEAD of a table -- 64 records or 256 KiB, cut at a record boundary so a cell holding a line break is never torn -- since the tile is the header row and ten rows and the rest of a multi-megabyte export was parsed and discarded, which is why a 5 MB CSV kept an icon beside a 7 KB one that tiled; since #1789 svg comes first in that list and html and jsx are captured in both schemes too, drawn with `prefers-color-scheme` emulated so a document's own dark stylesheet decides its dark tile, and every tile is stored at 800x600 (a page painted at full size and reduced with a Catmull-Rom filter, a tile-sized family painted at twice the density); a collection gets a dark mosaic from its members' dark tiles (`?variant=dark`), tile URLs carry the renderer generation (`&r=`, from `thumbnail_renderer`) so a redraw is not served from cache, and the portal root's `color-scheme` follows the theme toggle so a framed HTML or JSX document's `prefers-color-scheme` answers the reader's choice rather than the OS. The four copies had drifted, so a JSX resource and an image asset were never offered work the capturer can do, and plain text was in no list at all although the capturer has prose CSS that draws it. The tile builder now takes the reader's colour scheme for a resource as it always has for an asset -- the grid and the recently-updated strip drew the LIGHT capture in a dark portal, a white card in a dark grid, from a dark capture that was already stored -- and a resource's own page carries the same Thumbnail panel an asset has, with a Recapture control under `CanModifyResource`; `DELETE /api/v1/resources/{id}/thumbnail` takes BOTH variants and no longer reads a `variant` parameter, because two views of one file go together. Drawing an image tile is audited as its own `portal_preview` read surface, the one surface that does not stamp `last_read_at`, so browsing a library of photographs cannot clear the never-read flag on every image in it or reorder the recently-read sort. a RECENTLY UPDATED strip of the ten files that changed last heads a library at its ROOT only (left off inside a folder and under a search or tag filter, each of which already answers what is relevant), and the upload dialog's folder is a listbox of the folders that exist plus a `New folder...` entry that swaps in a text field, since a folder is created by filing into it and a path that does not exist yet must stay typeable. The All view names no library, so the upload dialog asks, offering the libraries `CanWriteScope` accepts with the caller's own first; folder RENAME is withheld there, naming one library being what a prefix rewrite needs. A library's FACETS come from the server in one request (#1555): `GET /api/v1/resources/facets` returns every folder with its EXACT count (a lateral expansion over each path's segments, so a folder counts everything beneath it at every depth) and the distinct tags its resources carry. Both were derived in the browser from the paged listing, so a folder read `25+` until the last page landed, the tag facet offered only what that page mentioned, and a library ROOT paged 138 rows to draw 4 folder names and displayed none of them — with a Load-more control and a `showing 100 of 138` line over content that was nowhere on screen. A root now runs NO listing at all, and a folder lists only its OWN level (`GET /api/v1/resources?path=<folder>&direct=true`, #1837): the subtree listing had paged 146 poster rows under `brand/posters` into a `brand` view that displayed none of them, beside `Showing 146 of 2541` and a Load more that changed nothing on screen; a search, and a tag chosen at a root, are the exceptions that replace the tree with a flat list of hits. Server side, `resource.ListScopes` is the listing predicate: a platform administrator's unnarrowed listing is unrestricted (`Filter.AllScopes`, a `TRUE` visibility clause, since user libraries are keyed by subject and address and there is no roster to enumerate) and a narrowed one accepts any library `CanSeeLibrary` allows — membership OR write authority, the rule a folder move already ran under — so an administrator can list back the persona material `CanWriteScope` let them upload and `CanAccessResource` let them open by id. Every other caller's listing is unchanged, and the agent-facing ranked SEARCH stays membership-scoped whoever asks -- enumeration hands a caller material they did not name -- while `fetch` on a stated `mcp:resource:` reference resolves on `CanAccessResource`, the by-id rule, so an administrator is not told a file does not exist while the same session can download and replace it (#1584); `resources/read` remains membership-scoped. A tag filter sits beside the search box and sets the `tag` parameter the list endpoint has always supported. Rows are multi-selectable, with one Move, Tag or Delete over the selection reporting what it did to each file; a file dragged onto a folder is refiled into it and a folder dragged onto another is nested inside it, both through the same confirmation. A page of records that POINT AT resources reads them in one query through `Store.GetByIDs`, keyed by id with a missing row simply absent, which is what the cross-source scratch-table listing resolves its source names and head keys with (#1472). A resource is filed under a FOLDER PATH inside its library (#1529, replacing the one flat `category` that could not nest): slash-separated, each segment keeping the rule the flat label carried, widened to a leading digit by #1886 (`^[a-z0-9][a-z0-9-]{0,30}$`) so every pre-tree row is already a legal one-segment path and no existing URI changes, at most 8 segments and 200 characters, with a refusal naming the rule it broke. Folders are DERIVED from the paths in use rather than stored as rows, so an empty folder cannot exist; renaming or nesting one is a prefix rewrite over every resource beneath it in a single transaction (`POST /api/v1/resources/folders/move`), each recording the address it vacated, refused whole on a collision or on a file the caller may not change, and capped at 500 resources with the true count stated rather than a partial move. Six first-level names are suggested in the portal -- `data`, `visual`, `samples`, `playbooks`, `templates`, `references`, `data` for records read as fact and `visual` for logos, photographs, diagrams and design elements meant to be displayed, the two that do not describe prose -- as completions and not a closed set, with the server left a shape check. A resource can also be MOVED to another library after upload (#1502), through a Library field on the Edit dialog carrying `scope`/`scope_id` on the existing PATCH route: the owner may move a file into a persona they merely BELONG to (looser than uploading, which still needs `persona-admin:{name}`), and an administrator to any persona, the global library, or a named person's library. Editing the FOLDER is the same rewrite on the other half of that address (#1528, which it did not do before: the breadcrumb came from the `path` column and the Details panel from the `uri` column, and a category edit changed only the first, so a resource's own page printed two different paths for one file); a library and a folder changed in one save produce one URI carrying both, one alias for the one address vacated, and one audit event. The row's scope, folder and canonical URI are rewritten while the blob, id, version trail, table registration and read history are not, every address the resource has vacated stays resolvable through a `resource_uri_aliases` table that a live URI always wins over, declaring assets and attaching prompts keep serving the file because both key on the resource id, a move onto an address another resource holds is refused naming that resource, and each move writes a `resource_move` audit event carrying the library and URI on both sides
- [Running Managed Scripts](https://mcp-data-platform.txn2.com/scripts/running/): How a managed script runs and what happens when it does. Includes tests and recorded runs (#1939, #1940, #1942): every draft and run records its host calls, a script carries test_* functions replaying a recording through testing.replay and asserting with assert, a save requires passing tests reaching 80% of statements and refuses tests that assert nothing or pass on altered data, and a new version whose replay of recent runs or whose reach differs needs change_summary and user_agreed; the built-in page platform-reference-script shows a whole script with its tests. Includes what a save checks (#1913): the canonical format every save stores (#1937), the main() entry point a new script keeps its work in (#1944), and the structural lint and limits a save is refused for, each finding naming its rule, line and fix, with scripts saved before the gates refused only for what an edit adds (#1938). A table a `register=` export made is read back with `params={"t": out["table"]}` and `FROM :t`, bound as its quoted name (#1948). Includes seeing every schedule at once: the portal's Schedules tab and GET /api/v1/portal/scripts/fires, which expands every fire server-side with the scheduler's own parse (#1891). Covers the central rule — a SAVED script runs: `run_script`, the portal's run action, and a cron schedule all execute the script's latest saved version, there is no approval step and no state in which a script exists but nothing may execute it, and `manage_script run_draft` remains the way to execute an edit as yourself before saving it, refusing every write it reaches through `platform.call` and previewing every `platform.export` unless `allow_writes` is sent so a landing pipeline is rehearsed without landing (#1664), in which case the exports are written for real too (#1822); `run_draft` also runs source that is not saved yet, under the name it will have. `platform.export(..., format="jsonl", register={...})` writes a file and registers it as a table in one call, the lossless path from script rows to a queryable table (#1820). `platform.export(name, {"sheets": [...]}, format="xlsx")` writes an Excel workbook described as data: named sheets of columns and rows, column types `string`/`integer`/`decimal`/`currency`/`date`/`datetime`/`percent`, a bold header row, widths, a frozen pane and an optional title row; a currency is the exact decimal text or integer cents written into the cell as it is (never through a float), and an invalid sheet name, a control character, more than 15 significant digits or an oversized workbook fails the export naming the sheet, row and column; the record and a draft report each sheet's rows and the size, and the portal offers the workbook for download (#1849). `platform.export(..., format="html", references=[...])` declares the `mcp://` URIs and `mcp:asset:` references a portal document names through `manage_asset update` on the asset it wrote, and every portal document export reports the ones its body names undeclared as `undeclared_references` and in the run log, with `validate` warning about literal ones (#1834). Covers the authority a run carries (the script's own principal presenting the roles its author held at the save, captured on the immutable version row and settable no other way, resolved to a persona by the middleware at every call so the persona filter decides which connections a run reaches at run time and a persona change takes effect on the next run), who may save (a script is one person's, so its owner and an administrator edit it, schedule it, and DELETE it from the script's own page in the portal (`DELETE /api/v1/portal/scripts/{id}`, #1575) or with `manage_script command=delete` — one store call behind both, taking the saved versions, the schedule, the run history and the carried state, leaving the assets and resources the script wrote and the producer records naming it as their writer, and answering with one account composed once (`script.DeleteMessage`, #1593) that names only what the script actually had, so a script that was never scheduled and carried no state is not reported as having lost either, and recorded from the portal as a `script_delete` audit event of kind `admin`, and an administrator can move it to another owner, chosen from the people who have signed in at least once because an address nobody has authenticated with cannot open the portal — a transfer that hands over everything at once and re-captures the run identity from the administrator making it, recorded in the audit log, and that states what happens to the assets and collections the script's runs have created: `outputs: move` hands them to the new owner in the same transaction, `outputs: keep` leaves them behind and the response lists the files the new owner cannot open, share or delete, and a script that has created any is not moved until the request says which (#1588)), and where output may go (`scripts.destinations` declares each bucket destination by name and complete address — connection, bucket, optional prefix — resolved at run time so repointing one takes effect on the next run, with the portal and the managed-resource library built in: `destination="resources"` with the path as its `key` writes the file at that path, create-or-replace, so an output other things read keeps one identity across runs (#1663)). Covers WHAT A RUN MAY CALL (`platform.call(tool, args)` invokes any platform tool by name and hands the script its structured result — writing a table with `trino_execute`, fetching an external API server-side with `api_invoke_endpoint`, reading an object, capturing a memory — with `platform.query`/`platform.export`/`platform.publish_data` kept as named helpers for the same mechanism; every one of them is one ordinary MCP tool call authorized by the persona filter at the moment it is made under the roles the version's author held at the save, so a script reaches exactly what its author reaches and a deployment that does not want scheduled writes withholds `trino_execute` from the persona rather than from the script layer; `validate` reads the literal tool names into `tools` and reports `dynamic_tools` for a computed one; a write made by tool call is NOT one of the run's outputs — the run's output list and the per-run output cap cover platform.export and platform.publish_data, plus the files trino_export, api_export and graphql_export write inside a run, listed with the tool that wrote them, a named one versioning the script's asset for that name across runs exactly as platform.export does (#1854), and everything else is in the audit log — and a query issued by tool call carries no row cap pushed into the statement, which is why the helpers remain the way to do those three things; a tool answering with plain text arrives as {"text": "..."}; `run_script` and `manage_script run_draft` are refused from inside a run because a run waiting on a run it started holds a worker slot while it waits, and runs waiting on each other can hold every slot a replica admits). Covers `run_script` (arguments checked against the script's parameter contract, a queued run executed by a worker on whichever replica claims it, a bounded wait that hands back a run id and pending status rather than holding the call open, and the run executing the version it was queued against so a save during the wait does not swap code underneath it), what a run hands back and reports (#1845/#1847: `platform.result(value)` returns one JSON value, capped at 1 MiB, carried by `run_script`, `get_run` and `POST /api/v1/portal/scripts/{id}/runs?wait=N`, which answers 200 with the finished run or 202 with its id; `platform.progress(message, done, total)` and the log so far are written to a running run every two seconds; `manage_script cancel_run` and `POST .../runs/{runID}/cancel` stop a run, a queued one never starting and a running one ending `canceled` within seconds with its outputs kept), stable output identity (one portal asset per script and output name, a new version per run, so a daily report accumulates versions instead of assets), WHO THE OUTPUT BELONGS TO (#1551: the person who owns the script — it is in their Assets page and their `manage_asset action=list` beside the ones they saved themselves, `search` finds it, `fetch` resolves its `mcp:asset:` reference, and they open, rename, retag, share, register a table over and delete it with no administrator; the row records the run's principal as `owner_id`, which is what keeps one asset per (script, output) and what the stored object key is built from, and the script owner's address as `owner_email`, and ownership is judged on EITHER identifier, so a transfer hands over what the script's runs have already produced along with everything else), the two content shapes an output takes (rows serialized in the declared format for csv/json/markdown/text, or a string body written verbatim so a script can compose a document — an HTML or JSX dashboard, a prose report — in markdown, text, html, or jsx) and external delivery for the other case (`platform.export` with a `destination` configuration declares as a bucket writes the same bytes out of the platform at a `key` beneath the configured prefix, so one computed result can refresh a dashboard AND hand a CSV to another system, once per destination per run), the DATA-REGION REFRESH of a semi-dynamic dashboard (`platform.publish_data(name, data)`: the presentation lives in the asset — an html, jsx, or markdown document marking exactly one element `id="data"`, conventionally a `<script type="application/json">` island — and the script refreshes only that element's interior, its dict-or-list payload serialized as JSON and structurally spliced through the same anchored-editing engine `manage_asset` patch uses, writing an ordinary new asset version so every refresh is a self-contained as-of snapshot; the name resolves through the same output identity an export uses, a document without the marked region fails the run, and the layout is edited in the asset like any document with no script change at all), a draft run that persists nothing and reports the size a real run would write, measured by serializing the rows in the declared format rather than estimating them and refused at the same output ceiling, reading run history and logs through `manage_script runs` / `get_run`, the failure model (the platform never re-executes a failed run; a failed run carries its cause and whether it is retryable, and a script failure names where the failure was raised, not that it repeats: a failure raised straight after an upstream answered the script's last call with a 5xx or 429 is recorded as upstream, fail(msg, retryable=True) records one as transient, and the owner's email says the script needs correcting only after three runs in a row fail the same way (#1935); a `rate_limited` refusal of a script's call is not a script failure, the host waits the envelope's `retry_after_seconds` against the run's deadline, re-issues the call, and logs each wait, so a run whose deadline arrives while pacing fails as a timeout; platform faults retry with backoff; a crashed worker's run is reclaimed by lease and cannot double-write its output), configurable run retention (`scripts.run_retention_days`, one year by default because run history is refresh history), where runs execute (`scripts.worker.enabled`, a `*bool` default on: every replica executes what it enqueues unless a deployment splits serving from execution, and a worker-off replica still registers `run_script`, validates, enqueues, and waits on the result a worker deployment produces), HOW MANY a replica executes at once (#1843: `scripts.worker.concurrency` is adaptive by default, claiming another run only while the process's memory is under `max_memory_percent` of the container limit or GOMEMLIMIT and its CPU under `max_cpu_percent` of the quota, between `min_concurrency` and `max_concurrency`, stopping and requeuing its newest run past `shed_memory_percent` on the platform's retry budget, never refusing or failing a run for want of room; a whole number fixes it and 1 is one at a time; `run_timeout` (default 15m, the lease derived as timeout plus five minutes), `max_steps` and `max_query_rows` are configurable and reported by `manage_script help`; `script_run_admission_refusals_total` by reason and `script_run_queue_wait_seconds` tell capacity from load), and the drain behavior of a stopping worker (claiming stops at once, a run in flight gets a short capped window out of the shutdown budget rather than the whole of it, anything unfinished is RELEASED rather than failed and is claimable immediately, and every write the stopping worker makes is itself bounded). Covers cron SCHEDULING (a `script_schedules` row of cadence, timezone, and bound parameters and nothing else; standard five-field expressions or descriptors, parsed by robfig/cron/v3 parse-only, read in an IANA zone so a report keeps its wall clock across a daylight-saving change; at most one schedule per script, replaced in place, never deleted because disabling keeps the row that explains its runs; a paused schedule reports no next fire, the stored due time being what it resumes on; set by the script's owner at any scope or by an administrator, from `manage_script` or from the portal's own cadence controls, which ask for a cadence in the terms a person has it in and DERIVE the cron expression rather than asking for it, keeping a Custom field for what the builder cannot express; the `${fire_date}` token expanded onto the run at materialization so a scheduled run is reproducible; single-fire across every replica by a unique index on (schedule, fire time) rather than a leader; skip-if-running overlap recorded as a visible `skipped_overlap` run; fire-once-latest misfire so recovery from downtime produces one run and a missed-fire count instead of a catch-up burst; a failed scheduled run mailed to the script's OWNER, while a `run_script` failure is not, being already in its caller's response; and the alert's rate-limit key being the script principal so one bad night does not silence every other automation's alerts). Covers editing from the portal (`PUT /api/v1/portal/scripts/{id}/source` through `script.ApplyEdit`, the one gate every mutation surface crosses: the edit lands on the live row, is captured as a version, and is the version that runs from then on, with the save saying so — or saying instead that the script is disabled or retired and nothing will execute it), documenting a script (`PUT /api/v1/portal/scripts/{id}/metadata`, or `manage_script update`: display name at 200 characters, the markdown DESCRIPTION rendered as the document it is, the lowercase-slug CATEGORY the listings filter on, and tags; a description refused only above 64 KiB, a structural limit because `script_fts` is built into a GIN index, with an advisory at about 16 KiB that the background might belong in a knowledge page; the category and tag axes narrowing `manage_script list` and the portal listing on the SERVER), CHECKING an edit before saving it (`validate` parses and reports what the edit would reach without executing or storing anything, and reports each destination it names that this deployment does not declare, so a script broken by a configuration change is found without running it; `dry-run` executes the source it is given — the saved version when none is sent — as the caller with the draft limits and persists nothing, one implementation shared with `manage_script run_draft`, leaving an account of the run keyed by the SHA-256 of the source that executed so it attaches to whichever version later carries that code — and a version with no account is code that first executes unattended, which the version detail states plainly), list and date_range parameters and form metadata (#1844: a `list` names `items` (string|int|float|date|enum) with optional min_items/max_items and binds a JSON array with every element checked, reaching the script as a list platform.query binds as IN (...); a `date_range` binds {from, to} with from on or before to; `label`, `order`, `group`, `min`/`max`, `pattern` and an opaque `ui` hint are stored with the contract, returned by GET /api/v1/portal/scripts/{id}, and drive the portal run form), the `connection` parameter type (the platform holds the whole set of values, so every surface that asks for one offers the connections the caller's persona reaches, narrowed to the connections a script can query since a connection is identified by kind and name together and a deployment may carry one name across kinds; an optional one must declare a default, since there is no meaningful empty connection), RUNNING one from the portal (`POST /api/v1/portal/scripts/{id}/runs` queues exactly what `run_script` queues under the same gate, worker and principal, recording `portal` as the trigger, and a script nothing would execute says so instead of offering a control that cannot work), RUN GRANTS (#1846: an owner or administrator grants a script's runs to a persona, a role or an API key by name from the Access card or `/api/v1/portal/scripts/{id}/grants`, audited as `script_grant`/`script_revoke`; a grantee runs it over HTTP and reads the runs it started and the definition everyone signed in reads (#1866), but never the state, gets 404 on a script it was not granted, and lists its catalog with `GET /api/v1/portal/scripts?scope=granted`; the run still executes as the script with its author's roles) FINDING OUTPUTS AGAIN (#1848: `platform.export(..., tags=[...], metadata={...})` adds tags to the asset and stores metadata on the version beside the recorded run_id, script, script_version and requested_by; `GET /api/v1/portal/assets` takes repeated `tag` and `metadata.<key>=<value>`, ANDed; `GET /api/v1/portal/scripts/runs/outputs` lists the outputs of the runs the caller requested across scripts with signed download links; `GET /api/v1/portal/assets/{id}/content-url?ttl=300` mints an expiring signed URL for one version that downloads without a session and answers 403 once expired, keyed off `auth.browser_session.signing_key`), and CALLER-BOUND parameters (`bind: "caller.<claim>"` takes the value from the caller's claim, which for an API key is one of its `attributes`; a request value for it is refused with 400, a caller without the claim with 403, and a script with one cannot be scheduled), reading what happened in the portal's Automations pages (the listing, one script's contract, its version history with each version's author and the roles a run of it presents (the same history `manage_script command=versions` returns over MCP, newest first and without the source, so an agent asked who WROTE a script answers from the version records rather than from `owner_email`, which an administrator's transfer moves), its run history with logs and output links, and — on a script the caller owns — the cadence, timezone, bound parameters, and pause/resume; a run is readable by the script's owner, an administrator, and whoever requested that run), that every run is measured (script_runs_total, script_run_duration_seconds, script_runs_running, script_missed_fires_total) with the admin portal's Runs tab drawing them beside the run rows themselves, and what a deployment needs for each capability. Covers STATE, what a run carries to the next (#1537): one JSON object per script, `run.state` in and `platform.save_state(obj)` out, 64 KiB, keys meaning whatever the script says (a watermark is `state["synced_through"]`); read once at creation with its revision and recorded on the run row beside params, applied when the run SUCCEEDS in the same transaction under a compare-and-set on the revision read, so a failed run leaves it alone and a second run that read the same revision fails at its write naming the first with its outputs standing; the misfire remedy that follows (a job written against its own watermark reads where the last successful run stopped and needs no backfill after downtime); the owner's and administrator's `manage_script command=state` with `state_action` get/set/clear and the portal page's State card, a reset moving the revision so a run in flight fails at its write; a draft reading the live state and reporting what it would have saved; `validate` reporting `reads_state`/`saves_state` and the contract's `state` block; state surviving a save, a disable and a transfer, and deleted with the script; and the worked example `example-incremental-sync`; failures classified by cause (script, upstream, memory, worker_lost, platform, state_conflict) with `retryable` true only for upstream and state_conflict, the platform never re-executing a run on its own, and the owner email worded by cause (#1859); an upstream 429 (and a 503 to a read) from `api_invoke_endpoint` or `api_export` waited on by the host at most three times before the script has the answer as data; the per-run memory budget `scripts.worker.max_run_memory` (a size, a share of the memory limit, or unlimited; half the limit by default) measured over the values a run can reach at every host call, walked whenever the process's heap growth since the last walk could cross the budget and once more when the script ends (#1867), `peak_memory_bytes` on the run and on `run_draft`, and the lone run past `shed_memory_percent` failed rather than left to the kernel (#1861); `platform.export(..., append=True)` paging csv or jsonl into one output written when the script finishes; take-overs of a run whose worker died capped by `scripts.worker.max_reclaims` (default 2) and then failed with cause worker_lost, each attempt recorded in `attempts`, `liveness`/holder/heartbeat on `get_run` and the portal, `live_runs` on `get`, `cancel_run` ending an orphaned run at once, and `script_run_reclaims_total` (#1860)
- [Reading XML in a Managed Script](https://mcp-data-platform.txn2.com/scripts/xml/): The `xml` module a managed script is given (#1735), for the upstreams that do not speak JSON: SOAP services, WebDAV PROPFIND, RSS and Atom feeds, sitemaps, and most ERP integration surfaces. `xml.decode` returns a tree whose elements carry `tag` (local name), `ns`, `attrs`, `text` (the element's own character data, trimmed) and `children` (ordered, so repeated siblings and their order survive); `json.encode` renders an element as those same five fields. `xml.find`/`xml.findall` search with a deliberately small XPath subset — child steps `a/b/c`, a descendant step `//c`, `*`, `[@name='value']` and a 1-based `[n]` — matching on local names so the sender's prefix choice (`soap:`, `soapenv:`, `S:`) never reaches the script, and refusing anything outside the subset where the path was written rather than answering it with no matches, because a path language that returns nothing for a construct it does not implement teaches the author that the data is missing. `xml.encode` writes a tree, or a dict of the same five fields, back to a document for a request body built from data; `ns` is per element, so a child without one is in no namespace and the encoder says so with `xmlns=""`. Parsing takes no instruction from the document: `<!DOCTYPE>` is refused outright, strict mode makes an unknown entity or a mismatched tag an error, a non-UTF-8 encoding is refused by name, and documents are capped at 8 MiB, 200 levels and 200,000 elements on the way in and the way out. The parser is shared with `api_invoke_endpoint`'s `decode` argument, so one document has one shape through either surface.
- [Overview](https://mcp-data-platform.txn2.com/knowledge/overview/): Tribal knowledge capture for data catalogs: memory_capture records insights during AI sessions and, through its sources argument, confirms the recorded call that answered the question so a refined query survives as something the next person can find, apply_knowledge promotes reviewed knowledge to one of three homes -- a DataHub entity, a canonical knowledge page, or the deployment's own byte-bounded customized agent instructions (section-addressed, so a promotion rewrites one rule and leaves the rest byte-identical, with a long body diverted to a page the section then indexes) -- all with changeset tracking and rollback, and the universal search/fetch tools are the discovery and read path across every knowledge source (including the business glossary, tags, and domains as first-class entities, and the caller's own sessions, found by what their calls said they were for and opened to the work they did), bounded by the caller's persona connection rules and reporting what those rules withheld. A delivered insight whose linked entity resolves to a queryable table carries a verifiable block naming that table and connection, on every delivery surface (search hits, fetch, and the memory_context enrichment block), so a claim arrives with the one query that would settle it
- [Governance Workflow](https://mcp-data-platform.txn2.com/knowledge/governance/): Human-in-the-loop curation: bulk review, approve/reject, synthesized change proposals, DataHub write-back, changeset tracking, and rollback (which returns the reverted changeset's source insights to the review queue as pending, keeping the application they lost, so undoing a promotion never strands the fact that motivated it), with review-queue staleness reported to whoever looks and emailed to operators who do not, and the observed warehouse state a pending claim is decided against (what the entity is queryable as, the rows it currently estimates, and an advisory marker when a stated count disagrees)
- [Overview](https://mcp-data-platform.txn2.com/memory/overview/): Persistent memory for agent and analyst sessions backed by PostgreSQL + pgvector, with hybrid semantic/lexical recall, cross-enrichment attachment, recall-first capture dedup with duplicate consolidation review, staleness detection, and embedding backfill
- [Configuration](https://mcp-data-platform.txn2.com/memory/configuration/): Memory configuration reference: embedding provider, input caps, the staleness watcher (what its two columns mean: `last_verified` applies to entity-linked records only and stays null on a record no catalog can be asked about, and `updated_at` is the last content change, which verification never moves), persona opt-in, and pgvector setup

## Docs: Administrator

Configuring and operating the platform. These are the pages under `docs` > Administrator in the site nav.

### Authentication

- [Auth Overview](https://mcp-data-platform.txn2.com/auth/overview/): Fail-closed security model, stdio vs HTTP authentication, and what an identity-provider outage looks like to a client (in-band retryable refusals, never a 401 that would trigger a futile re-auth)
- [OIDC](https://mcp-data-platform.txn2.com/auth/oidc/): Keycloak, Auth0, Okta, Azure AD setup with required claims; self-healing JWKS cache that refreshes on demand for key rotation
- [API Keys](https://mcp-data-platform.txn2.com/auth/api-keys/): Service-account authentication, and keys issued against a person's account so a client that can only send a bearer token authenticates as them with their roles; a key whose roles no persona carries is flagged No persona at creation and in the listing, since it authenticates and lists no tools. The self-service routes under /api/v1/portal/api-keys refuse an API-key request 403, since they act on the strength of the caller being signed in; the admin routes require the admin persona and accept any credential carrying it, an API key included, so a service key with an admin role issues, lists and revokes keys -- including one bound to a person, which then authenticates as them (#1762)
- [OAuth Server](https://mcp-data-platform.txn2.com/auth/oauth-server/): Built-in OAuth 2.1 authorization server for Claude Desktop and other MCP clients, with PKCE, Dynamic Client Registration, upstream IdP integration via OIDC discovery of the authorization/token endpoints (works with any OIDC-compliant provider; optional explicit-endpoint override), refresh tokens hashed at rest, HS256 access tokens with `kid`-based signing-key rotation (verify-only previous keys), and default-on rate limiting (trusted-proxy-aware per-IP + global backstop) on the `/token` and `/register` endpoints
- [OAuth to Upstream MCPs](https://mcp-data-platform.txn2.com/auth/oauth-gateway/): Outbound OAuth to gateway upstreams: client_credentials and authorization_code + PKCE grants, encrypted refresh tokens that survive restarts, background refresh, endpoint URL validation, and a full auth-event history

### Personas

- [Overview](https://mcp-data-platform.txn2.com/personas/overview/): Role-based tool access control with connection-level filtering, where the connection is the authorization boundary rather than the end user (rationale and trade-offs on the Authorization Model page). Personas are the access boundary: a caller whose roles match no persona reaches nothing (tools refused, portal 403), there is no fallback persona, and anonymous/no-auth callers carry the "anonymous" role a persona must list to grant them access. Deny-by-default connection rules bound discovery as well as action: search, fetch, list_connections, and the portal search show only granted connections, and report what they withheld. Some tools are a unit and must be granted together (search with fetch; memory_capture and apply_knowledge with search): the platform checks each persona against its registered tool set at startup and on every persona write, and warns with the persona name, the missing tool, and the fix, because the loss is otherwise silent. api_routes narrows an api-kind connection by (connection, method, path), authored in the config file or in the portal's API endpoints scope where an operator selects operations rather than typing globs; a path glob is matched against both the path a call reaches and the catalog path its operation declares, and wildcards do not cross a "/"
- [Tool Filtering](https://mcp-data-platform.txn2.com/personas/tool-filtering/): Allow/deny patterns with wildcards; persona-level filtering (security boundary) vs global tool visibility (token optimization); prefer allow ["*"] with a targeted deny, since an enumerated allow-list silently loses each tool a later upgrade adds
- [Role Mapping](https://mcp-data-platform.txn2.com/personas/role-mapping/): Map OIDC roles to personas; roles matching nothing resolve to the deny-all persona rather than a configured default

### Connections, portal and API

- [Multi-Provider](https://mcp-data-platform.txn2.com/server/multi-provider/): Connect multiple instances of each service
- [Admin API](https://mcp-data-platform.txn2.com/server/admin-api/): REST endpoints backing the admin portal: system info, config, personas, keys (each with the persona its roles reach, or no_persona and a creation warning when they reach none; every replica reads and authenticates them from the one key store, so a create or delete takes effect everywhere when it returns), tools (schemas and detail carrying the annotations tools/list advertises), users, audit, sessions (derived from audit history: the list with its filters, and one session with its outputs and paged call timeline), knowledge, connections, and index-jobs health. Connection instances also carry a test action (POST /connection-instances/{kind}/{name}/test opens the connection and asks its upstream one harmless question -- SELECT 1, a bucket listing, an introspection, a GET at the base URL, a tools/list -- answering 200 with what answered, 503 with the upstream's own error, or 409 when no toolkit of that kind runs in this process; a connection saved through another replica is taken on from the store first, so it is testable the moment its save returns, and a trino scratch connection that accepts writes is also checked for the partition procedures webhook sources call, #1888) and a config-schema discovery pair (GET /connection-kinds and /connection-kinds/{kind}, the JSON Schema of the config object the PUT takes, with a note saying an unnamed key is stored and ignored and that a config the schema admits is not thereby a connection that works); a config is stored literally, so an unexpanded ${VAR} is refused at the PUT naming every key it sits on rather than creating a connection that fails every call (#1805). Interactive Swagger UI at /api/v1/admin/docs/, with a ReDoc rendering of the same spec from the admin navigation. The reference covers the whole REST surface, not only the admin control plane: the API gateway data plane under the Gateway tag, portal, resources, the DataHub catalog and table registrations, grouped into User API and Admin API. Its landing section is an introduction authored in internal/apidocs/introduction.md and injected beside the tag and security-scheme descriptions, carrying Getting started, Authentication, Calling a connection through the gateway, Conventions and where the MCP surface is, each a navigation entry of its own; the served copy names the origin it was served from as its host rather than the generator's localhost. A route registered without annotations and a tag missing from the tag-group script are both hard test failures, and a contrast sweep over the rendered ReDoc page fails on any text below WCAG AA on either theme
- [Admin API](https://mcp-data-platform.txn2.com/knowledge/admin-api/): REST endpoints for managing insights and changesets, and listing every user's memory records

### Operations

- [Audit Logging](https://mcp-data-platform.txn2.com/server/audit/): PostgreSQL-backed audit logging for tool calls: schema and field reference including the `purpose` column that records WHY a call was made (the agent's one-sentence statement of the wider task, taken off the request before the tool saw it and outside the parameter redaction policy), the sessions read back OUT of that log (derived rather than stored, since session rows expire and audit rows do not: kind from the id prefix, the caller and persona of the first event with the live handle's minted persona outranking it, the tools and connections touched, and the assets and knowledge-dimension memory records the session produced), the principal each row is attributed to (an OIDC subject for a person, `apikey:<name>` for a key, `script:<name>` for a managed-script run) beside the address that principal acts for, which is the script owner's on a run and therefore identifies nobody on its own, parameter sanitization with configurable redact_keys and log_parameters opt-out, async vs sync delivery semantics and the audit_events_dropped_total metric, caller-class separation, monthly partition rotation, and retention
- [Observability (Metrics)](https://mcp-data-platform.txn2.com/server/observability/): OpenTelemetry Prometheus metrics covering tool calls, gateway HTTP calls, toolkit/provider internals, background indexing (indexjob_jobs_total by kind/trigger/outcome, indexjob_duration_seconds on buckets to an hour, indexjob_running per replica, enqueues, embedded vs reused items, embed calls by ok/timeout/error, reaper lease releases, parked-unit deferrals, and scrape-time database gauges for queue depth by state, open failures, oldest runnable wait and vector coverage, read with max; the admin Indexing dashboard's Throughput and Embed latency panels are drawn from them), and managed-script execution (script_runs_total by script/trigger/status, script_run_duration_seconds, the script_runs_running gauge bracketed around execution so a wedged worker is visible, script_missed_fires_total — the one thing the run table cannot show, because a missed fire is a run that does not exist — and the worker's admission: script_run_admission_refusals_total by reason (ceiling, memory, cpu) and script_run_queue_wait_seconds, #1843), plus optional OTLP distributed tracing and an authenticated PromQL proxy for the portal; apigateway_outbound_total carries the calling persona, so an automated principal's upstream traffic is separable from an analyst's on a connection they share
- [Notifications](https://mcp-data-platform.txn2.com/server/notifications/): Branded email notifications for shares, feedback, and @-mentions (thread events reach the target owner, the thread author, and the people it is shared with, never the person who wrote the event; anyone the comment named is notified in the separate mention category, queued first): admin-configured SMTP with encrypted password and a send-test action that surfaces the target's opt-out state, per-user preferences (off, immediate, daily digest) that go inert with an explanation when no SMTP delivery path is configured, a durable database-backed queue with a retrying send worker, a no-login confirm-then-unsubscribe footer link (no mutation on GET, so mail-scanner prefetch cannot opt recipients out) plus RFC 8058 one-click List-Unsubscribe headers for recipients without an account, an opt-back-in action on the share landing page, Message-ID stamped with the From-address domain, and implementor-configured branding: terms/privacy footer links, help/about footer text, and a Reply-To address, with direct transactional delivery of one-time guest view links; per-share sharer control (a notify flag defaulting to on, and an optional plain-text note that is delivered only in the email and never persisted, with markup and links refused rather than escaped), display-name address entry reduced to the bare address at every door, and two retention-bounded delivery-history views: an admin monitoring tab carrying attempt counts and the verbatim mail-server error, and a self-scoped per-user screen that omits it; plus the operator review-queue alert (#803): an hourly check of the pending knowledge insight queue that emails when it crosses its admin-configured pending-count or age threshold, deep-linking to the queue itself, de-duplicated by a cooldown claim keyed per queue that also makes each a cluster-wide singleton; and the connection revocation alert (#1694): when an upstream rejects a connection's refresh and the platform discards the credential, the identity that authorized the connection is emailed once per revocation with the upstream, what it returned, and a link that reconnects it, and a revocation nobody has acted on after the admin-configured window is escalated to the addresses the operator named; a refused `oauth_grant: jwt_bearer` assertion (#1734) opens the same alert, mailed to those addresses at once and never escalated, quoting the upstream's error_description and naming the fixes at the upstream instead of a reconnect, and cleared by the next exchange the upstream accepts; and notification channels (#1720, #1723): operator-configured destinations that are not one person - a Mattermost channel, an incoming webhook, or a named list of addresses - configured in the portal admin settings and stored in the database, each of the two HTTP kinds naming an existing api connection rather than storing a credential, so the bot token stays encrypted where every other upstream credential lives and who may send to a channel is exactly who may reach its connection; a send is an ordinary queue row claimed by the same worker on the same backoff and retention, with the claim narrowed by transport so a deployment with channels and no mail server still posts to chat and one with no gateway still delivers email; an email channel fans out to one row per recipient so each keeps their own delivery mode and unsubscribe link; a refusal retrying cannot fix (a deleted channel, a revoked token, a chat channel the bot was never invited to) fails the row at once in the upstream's own words; reached through the `notify` tool (list, send, publish an asset) and through `platform.notify` and `platform.publish` in a managed script, whose post links back to the run that produced it and is refused in a draft run without allow_writes
- [Operating Modes](https://mcp-data-platform.txn2.com/server/operating-modes/): Two deployment modes, standalone (no database) and database-backed. Feature availability by mode, example configurations, decision guide
- [Data Retention](https://mcp-data-platform.txn2.com/server/data-retention/): What the platform removes on age (deleted portal items with their objects after 30 days, archived memory, orphaned producer records, unresolved failed index jobs, expired webhook window records), what is bounded by count, and what is kept on purpose (script, prompt and knowledge-page version history; configuration and knowledge change logs), with each `*_retention_days` setting (#1904).
- [Session Externalization](https://mcp-data-platform.txn2.com/server/session-externalization/): Externalize session state to PostgreSQL for zero-downtime restarts and horizontal scaling, including live tools/list_changed, prompts/list_changed, and resources/list_changed notifications in multi-replica deployments
- [Self-Configuration](https://mcp-data-platform.txn2.com/server/self-configuration/): A built-in loopback gateway connection exposes the platform's own admin REST API to admin MCP sessions, so admins manage personas, connections, and prompts by asking the agent; a raw path outside /api/v1 is refused naming the prefixed path and the operation_id

### Gateways

- [Gateway Toolkit](https://mcp-data-platform.txn2.com/server/gateway/): Re-expose third-party MCP servers through the platform's auth, persona, and audit pipeline. Connections are portal-authored with encrypted credentials, OAuth 2.1 grants, and optional declarative cross-enrichment rules. The platform and each upstream negotiate protocol revisions separately, so neither side's revision crosses the proxy boundary
- [API Gateway Toolkit](https://mcp-data-platform.txn2.com/server/api-gateway/): Proxy REST/HTTP APIs through the same pipeline with four tools instead of one per endpoint. Auth modes span bearer, API key, basic, signed JWT, OAuth 2.1, and mTLS, with a REST shim for non-MCP clients that returns the whole body and answers 413 past the connection's `max_response_bytes` (never the MCP context budget, #1878), and bounded-memory streaming exports. Request bodies are encoded from the catalog's declared media type, including multipart/form-data file parts. A response is read by the `decode` argument (`auto`, `xml`, `json`, `text`): `auto` parses JSON when the response declares it and returns an XML tree when the catalog declares an XML media type on the operation's success response, so a SOAP or WebDAV answer arrives as nested {tag, ns, attrs, text, children} objects rather than an opaque string (#1735), while a connection with no catalog reads as it always has until a caller passes `decode`; a body that will not parse under a `decode` the caller passed comes back as text with the reason in `hint`, `json` and `xml` alike (#1763), while `auto` falls back to text silently; and `decode` is refused with `paginate` because a walk merges the pages it collects. A paginate block walks every page of a collection inside one call, pinned to the connection's host and paced by the upstream's Retry-After, over the REST shim's invoke route as well as the tool (refused on invoke-raw). An export's destination can be a MANAGED RESOURCE at a path rather than a new asset (#1663): create-or-replace by path, so the same call next time records the next version of one file, keeping its id, its mcp:// URI, the assets referencing it and the tables registered over it, with the response streaming from the upstream into the file's storage; a non-2xx response is refused rather than landed, and trino_export, graphql_export and a managed script's platform.export reach the same destination. `api_discover` with a query returns a relevance BOUNDARY rather than a page: lexical is an AND filter, hybrid and semantic keep the operations containing every token and then at most five neighbors by intent scoring at or above the floor, every ranked row carrying `score` and `lexical_match` and the result carrying `matched_lexical` and `shown_semantic` (#1626). `oauth_grant: jwt_bearer` (RFC 7523, #1734) signs a short-lived assertion with the `jwt_*` keys (RS256 by default, `jti` on every assertion, audience defaulting to `oauth_token_url`, `jwt_issuer` and `jwt_subject` both required) and exchanges it at `oauth_token_url` for an access token cached in memory and re-obtained near expiry, 15 minutes when the response carries no `expires_in`; client id and secret are optional client authentication; `invalid_grant`, `invalid_client` and `unauthorized_client` fail the call with the upstream's code and description and raise the connection alert; available on `api` and `graphql`, refused on `mcp`. A saved `api` connection is served by every replica from the moment the save returns, not when its reload announcement arrives: a replica asked about a connection it does not hold reads the connection store before answering "not found", shares that read across the requests waiting on it, and withdraws a connection deleted while it was read (#1746, the same resolution the graphql kind got in #1714, in one shared implementation). A catalog spec entry carries `spec_format` (`openapi`, `wsdl` or `graphql`) beside `source_kind`, so a SOAP upstream is described by the WSDL it already publishes (#1736): the document is converted to OpenAPI at save time and the WSDL kept beside it, each portType operation becomes a discoverable operation carrying its XSD as a schema, an object `body` is assembled into the envelope the version requires (media type, quoted `SOAPAction` for 1.1 or a Content-Type `action` parameter for 1.2, element order from `xsd:sequence`, attributes and namespace qualification from the schema) while a string `body` is still sent verbatim, and a `soap:Fault` is reported as the upstream's own error rather than a 500 carrying XML. Every SOAP operation is a POST to one address, so the rendered document keys each under `<servicePath>/<OperationName>` and a persona `api_routes` path glob names one operation exactly; document/literal SOAP 1.1 and 1.2 only, RPC and encoded refused by name; `upstream_retryable` and `retry_after_seconds` on a 429, and on a 503 to a GET or HEAD, and a transport failure or timeout reported with error code `upstream_unavailable`, category `upstream` (#1859); a resource-destination `api_export` answers a non-2xx upstream with `upstream_status`, `upstream_headers` and `resource_unchanged: true` as data instead of failing
- [Signed JWT Upstreams](https://mcp-data-platform.txn2.com/server/signed-jwt-auth/): `auth_mode: signed_jwt`, for upstreams that issue an identifier and a signing key and expect the client to mint its own short-lived assertion — Sage X3 connected applications, Snowflake key-pair authentication, Apple App Store Connect and APNs, and internal services built the same way. There is no token endpoint and nothing is exchanged, which is why neither `bearer` (a fixed string cannot carry a 300-second expiry) nor `oauth` (there is no endpoint to exchange against) can reach one. HS256 over `jwt_client_secret`, RS256 or ES256 over `jwt_private_key_pem`, with `jwt_key_id` emitted as the token's `kid`; `jwt_issuer` and `jwt_subject` each omitted when empty and at least one required, because a Sage X3 connected application checks both, an App Store Connect team key checks `iss` alone and an individual key checks `sub` alone; `jwt_audience` defaults to the connection's endpoint URL and is matched byte for byte. One assertion is minted and reused until it is within `jwt_issued_at_skew` of expiry. The upstream's own 401 body reaches the model unchanged, because that body is the only place it says which claim was wrong. Available to every HTTP-based kind (`api`, `graphql`) from the shared seam; the secret and the private key are encrypted at rest and returned as `[REDACTED]`. The same keys sign the assertion `oauth_grant: jwt_bearer` exchanges at a token endpoint; the page says when to pick `signed_jwt` (the upstream validates the client-minted JWT) and when `jwt_bearer` (the upstream exchanges it for a token it issued)
- [API Catalogs](https://mcp-data-platform.txn2.com/server/api-catalogs/): Versioned, globally-owned spec bundles shared by many connections, ingested by paste, upload, or URL with SSRF guards. Per-operation embeddings power semantic endpoint ranking, and each connection resolves the spec's base path against its own base_url. A spec entry's `spec_format` says what its content is (`openapi`, `wsdl`, `graphql`) independently of how it arrived: a WSDL is rendered to OpenAPI at save time, and a `graphql` entry is SDL served to `kind: graphql` connections referencing the catalog rather than to the HTTP gateway, which skips it by format (#1745). A catalogued schema refreshes from a URL on its etag, is browsed operation by operation through the same routes an OpenAPI spec is, and is embedded once for every connection sharing it; a catalog carries at most one, an introspection result is refused in favour of SDL, and a connection that names a catalog never introspects and refuses an upload, naming the catalog to edit
- [GraphQL Toolkit](https://mcp-data-platform.txn2.com/server/graphql-gateway/): The third gateway kind (`kind: graphql`): one endpoint URL, a schema the platform reads by introspection and keeps compressed with its hash (and the refusal its last read met, reported by every replica and after a restart; every replica serves a saved connection and its stored schema from the moment the save returns), and three tools (`graphql_discover`, `graphql_query`, `graphql_export`) serving every connection. Its auth modes are the shared set, `signed_jwt` included, which is what a Sage X3 connected application needs. A kind of its own because a GraphQL endpoint has no route to authorize, no path to rank, and reports failure as an `errors` array inside an HTTP 200 — which is what `upstream_error` classifies on, so a failed GraphQL call is not recorded as a success. Namespace descent walks a namespaced schema (package, entity, verb) into dotted operation ids and a flat one into its root fields; `graphql_discover` returns a skeleton document that already validates, including the `edges { node { ... } }` shape of a paged connection. A document is parsed, depth-capped and validated against the stored schema before it is sent (strict refuses and names when the schema was read; warn passes it through with the violations). Persona `api_routes` govern it unchanged, with the operation kind as the method and the dotted id as the path, reduced from the operations a document actually invokes rather than its root fields; a read-only persona is one ALLOW naming QUERY, since a rule set that names a connection requires a matching allow and a deny on its own would close it entirely. `paginate` walks a Relay connection by default and any other upstream through a named `next_cursor_path`. An endpoint that disables introspection is a named error with an operator upload path, never a silently empty index. A connection may instead name a catalog in `catalog_id` and take its schema from the SDL that catalog holds (#1745): it then never introspects, reports `source: catalog` beside the hash and read time, reads the catalog on a re-read, ranks on the catalog's operation embeddings so two connections against one endpoint embed it once, and refuses an upload by naming the catalog to edit

### Cross-Enrichment

- [Overview](https://mcp-data-platform.txn2.com/cross-enrichment/overview/): How automatic context enrichment works between services
- [Trino to DataHub](https://mcp-data-platform.txn2.com/cross-enrichment/trino-datahub/): Trino results include DataHub metadata
- [DataHub to Trino](https://mcp-data-platform.txn2.com/cross-enrichment/datahub-trino/): DataHub results show query availability
- [S3 Enrichment](https://mcp-data-platform.txn2.com/cross-enrichment/s3/): S3 operations include semantic context
- [Knowledge Pages](https://mcp-data-platform.txn2.com/cross-enrichment/overview/#knowledge-pages-entity-to-knowledge): Entity tool results carry the canonical knowledge pages that document the named entities and connections
- [Lineage Inheritance](https://mcp-data-platform.txn2.com/cross-enrichment/lineage/): Automatic column metadata inheritance from upstream datasets via DataHub lineage
- [Session Dedup](https://mcp-data-platform.txn2.com/cross-enrichment/overview/#session-metadata-deduplication): Avoids repeating semantic metadata for previously-enriched tables within a session, saving LLM context tokens
- [Column Context Filtering](https://mcp-data-platform.txn2.com/cross-enrichment/trino-datahub/#column-level-enrichment): Limits column-level enrichment to columns referenced in the SQL query (default: enabled)
- [Schema Preview](https://mcp-data-platform.txn2.com/cross-enrichment/datahub-trino/#schema-preview): Adds a bounded column preview to datahub_search results, eliminating intermediate schema calls (default: enabled)

### MCP Apps

- [Overview](https://mcp-data-platform.txn2.com/mcpapps/overview/): Interactive UI panels rendered inline in the MCP host. Built-in platform-info and prompt-browser apps (the prompt-browser binds to the presentation-only show_prompts tool so the agent's own prompt work renders no UI), branding via config, custom apps via assets_path
- [Configuration](https://mcp-data-platform.txn2.com/mcpapps/configuration/): MCP Apps enabled by default; branding overrides, custom app registration, CSP settings
- [Development](https://mcp-data-platform.txn2.com/mcpapps/development/): Docker-based development with a test harness and live-edit asset overrides; apps calling tools must handshake with platform_info and thread the session_id
- [Tutorial](https://mcp-data-platform.txn2.com/mcpapps/tutorial/): Step-by-step guide building a platform-info app

### Security

- [Threat Model](https://mcp-data-platform.txn2.com/security/threat-model/): The security model as a whole: a trust-boundary diagram (inbound surfaces, identity mechanisms, outbound dependencies, at-rest stores), STRIDE-style attacker analysis across six personas (unauthenticated network, low-privilege persona, malicious upstream, malicious query data, database reader, compromised downstream credential), the recorded identity-provider-outage decision (edge passes an unvalidatable credential through, protocol layer refuses as retryable, pinned by an end-to-end test), a threat-to-mechanism mitigations table with package/config citations, and explicit non-goals (stdio local-process trust, no defense against a malicious admin, best-effort async audit loss model, per-connection rather than per-user downstream identity stated as a design boundary with its rationale and its cost, no content sanitization, deployment-owned TLS/segmentation)
- [Managed Scripts: Security Model](https://mcp-data-platform.txn2.com/scripts/security/): The threat model for managed scripts, the agent-authored Starlark programs the platform stores, versions, and governs. States the authority claim structurally — a script can never do what the person who WROTE it could not do, because a draft runs as the caller and a platform run runs as the principal `script:<name>` carrying the roles its author held, captured on the immutable version row (`script_versions.author_roles`) at the save and presented by the runner; no surface anywhere accepts roles as input. Covers the run gate (`script.RefuseRun`: a SAVED script runs, and the only refusals are disabled, deprecated, and superseded — re-read at enqueue and again at claim, so a script taken out of service refuses a run already on the queue; a run executes the version it was queued against, the latest saved at the moment of the request or the fire, loaded by its immutable id, so a save landing during a queue wait cannot swap code underneath it). A run ACTS ON WHAT ITS AUTHOR OWNS: it authenticates as `script:<name>` (what audit records and what it stamps as the owner id of the assets it writes, whose OWNER is the person who owns the script -- #1551, ownership judged on the address recorded beside the principal) and carries the address of the VERSION AUTHOR — the same person whose roles it presents, so a run never pairs one person's authority with another's ownership — which ownership checks accept alongside a user id for a person, and INSTEAD of one for a run (`ownsResource`), because a principal that owns nothing a person owns would otherwise be refused the very assets its author can edit, by something that is not the persona filter (#1419). The principal is not an arm of its own: `script:<name>` is unique only within an owner (`idx_scripts_name_owner`), so two people who each keep a `daily-sales` present one subject and matching it gave a run of one person's script the outputs of the other person's, readable and rewritable wherever `ownsResource` is the gate while the person that run acts for could reach none of it (#1579); a run's own writes record the address beside the principal, so reading only the address loses nothing. For a managed RESOURCE the modify rule follows the person rather than the library the file is in (#1576): a move rewrites a resource's library, folder and `mcp://` address and never its uploader columns, so whoever may replace a file's content before a move may after it, and their scripts may too -- holding a run to the library meant moving a CSV into a persona you merely belong to (which the platform deliberately permits) silently broke the schedule refreshing it, and whether it broke turned on whether the author was a platform administrator. It is ONE rule for the person and for what acts for them (`uploadedByCaller`), reading the recorded ADDRESS for an unattended caller and the recorded SUBJECT only for a caller acting as themselves, since `script:<name>` is unique only within an owner; a stand-in written for the run alone diverged the other way, leaving a person's other scripts able to rewrite a row a run filed once it left that person's library. `CanAccessResource` keeps the scope-bound arm, so a run SEES nothing it did not see before, though a surface authorizing on modify alone (registering a resource as a Trino table) grants the run what it grants the person. It grants nothing new: the address is captured from an authenticated context at the save exactly as the roles are and is never an argument, both sides of the match must be non-empty so an unrecorded author never matches an unowned resource, shares are NOT inherited (the share lookup carries no address for a run, so a grant to a person is not a grant to everything they automate), enumeration stays the script's own outputs (`manage_asset list`, `search` over assets and the collection listing scope an unattended caller by the PRODUCER recorded for its own writes, `content_producers`, which names the script by id -- neither identifier on the row does, since the owner id is shared by every same-named script and the owner_email is the script owner's address as of the insert, which a transfer does not rewrite -- a separate judgment in the code from the one that lets it act on a named asset, and the one `fetch` reads too so a run can dereference everything its search returned), and a draft carries no second identity because it already authenticates as a person. Author and owner are frequently DIFFERENT people — a transfer writes the new version authored by the transferring ADMINISTRATOR while the owner becomes somebody else, so from then on a run presents that administrator's roles and acts for them while the new owner is who may trigger it, which is the save's widening (already in residual risks) rather than this binding's. A run may READ the script surface but never author, edit, delete or schedule a script: a run that could would schedule unbounded work, and a run that could edit itself would capture the roles it is executing with as a new version's authority under the owner's address. A script CALLS THE TOOLS ITS AUTHOR CAN CALL: `platform.call(tool, args)` invokes any platform tool by name, with `platform.query`/`platform.export`/`platform.publish_data` kept as named helpers for the same mechanism with a constant, and there is no script-side allowlist in front of any of them (#1419 retired the three-capability list, which prevented a script from doing what its author could already do interactively and bought only the appearance of a sandbox). What replaces it as the reviewer's material is the source: `validate` reports the literal tool names as `tools` and sets `dynamic_tools` when a call computes one, a connection named literally inside a literal argument dict feeds the same connection list, and a computed argument dict sets `dynamic_connections` since the connection is the only claim the report makes about what is inside those arguments. `run_script` and `manage_script run_draft` are refused from inside a run on `PlatformContext.Source`, as a runaway-work guard rather than an authorization rule: a worker executes one run at a time per replica, so a script waiting on a run it started would wait on the worker running it. The persona filter is the ENTIRE authorization boundary at run time: every host call is one MCP tool call over a per-run in-memory session against the assembled server, so authentication, persona and connection authorization, rate limiting and audit apply exactly as they do to an agent's call, none of it re-implemented (the limiter reads the script principal's auth type and holds a platform run's over-limit call until the sustained rate admits it rather than refusing it, which is #1534), and the roles are resolved to a persona fresh at every call — narrowing a persona takes effect on the next run with no script-side action, and there is no stored per-script allowlist to drift out of step with the persona configuration it would duplicate. Destinations are CONFIGURATION rather than a per-version record: `scripts.destinations` declares each bucket destination as a complete address (the platform S3 connection, the bucket, an optional key prefix), a run resolves the name a script writes against that list at run time so repointing one takes effect on the next run, the portal is built in with its name reserved and configuration cannot redeclare it, an undeclared name is refused inside the interpreter naming the configured set, a draft resolves through the same list so a destination a real run would refuse fails while the author is iterating, and the write is still authorized by the middleware, so a destination whose connection the run's persona cannot reach is refused however configuration names it. Covers external DELIVERY as one ordinary audited tool call rather than a private route to object storage, with the explicit statement that arbitrary egress does not exist — a script supplies no endpoint, credential, bucket or host name, and there is no binding that opens a socket, so the only network it reaches is the operator-configured connection set — plus the prefix as a boundary a key cannot climb out of (an absolute key, a `..` segment or an empty segment is refused rather than normalized away), exactly-once per run per destination and one object per key, `destination` and `key` required as NAMED arguments because a positional one would be invisible to the static read that reports where a script writes, and audited argument values bounded at 16KB so a delivered report does not put a second copy of itself in the audit table. Covers the data-region refresh (`platform.publish_data`, which adds no authority — the author can already rewrite the whole document — and whose region confinement is a behavioral contract: the target is pinned by the export identity rule so the call reaches only this script's own portal outputs and creates nothing, the splice is structural through the one element matching `#data` with the payload's `<` `>` `&` written as \u escapes so it cannot corrupt the document, and the validator reports the refresh target names), the run queue (lease-based claiming with fencing on every write, crashed-worker recovery folded into the claim predicate so there is no reaper and no leader election, and no double-written output because each output is recorded as it lands), retry classified by WHERE a failure happened rather than by matching error text, audit under the script principal joined to a `script_run` lifecycle event by the run id, the sandbox (Starlark has no ambient clock, randomness, filesystem, network, or module system; `while` and recursion off; the predeclared set is exactly platform/json/date/run/sum), the resource limits with the honest gap (no hard MEMORY cap in any embedded interpreter of this class) and the control that bounds what that gap COSTS rather than preventing it (`scripts.worker.enabled: false` on serving replicas plus a worker deployment of the same binary, so heap pressure lands on a pod that accepts no request and the worst case is a restarted worker whose run another replica reclaims), typed SQL parameter binding with a state-aware scanner instead of string concatenation, a write statement passed to `platform.query` refused by `trino_query` itself in the tool's own words now that its advice leads somewhere, the destination set stated as a bound on `platform.export` rather than a perimeter around the run (a persona holding an S3 connection reaches `s3_put_object` from a script exactly as its author does at a prompt, and the control is which tools and connections that persona holds), a truncated query result failing the run because silently wrong is the one outcome the determinism contract exists to exclude, the credential-literal scan (error on a credential FORMAT, warning on a naming convention, and a tripwire rather than a proof), unparseable source never stored, the three `SourceScript` middleware behaviors (exempt from the session and search-first gates because there is no model in a script run, an isolated per-run session identity so a run never advances the gate or provenance state of the person it runs for, and enrichment skipped), and the determinism contract stated exactly: same script version + same parameters + same underlying data produce the same output, which is reproducibility rather than identical forever. The scheduling posture: a schedule carries cadence, timezone, and parameters only, is set by the script's OWNER at every scope or by an administrator — deliberately a weaker rule than the edit rule, because the run gate and the persona filter are re-read at every fire, so re-timing reaches nothing new — and fires nothing on a script the gate refuses; the one-fire-a-minute floor and the one-open-run-per-schedule overlap policy are what bound unattended repetition, single-fire across replicas is a unique index on (schedule, fire time) rather than a leader, and a failed scheduled run mails the script's OWNER. Covers DISCOVERABILITY as a security-relevant widening: a script is addressable as `mcp:script:<id>` and reachable from `search`, `fetch`, and a prompt that references it, each applying the script's ownership rule as a store predicate, returning the contract (name, parameters, whether a run would be admitted, cadence, last run) and never the source, and granting nothing; the semantic index embeds the description card and never the Starlark, and both ranking arms apply the same ownership predicate so the index widens nothing. Reading and writing in the portal grants nothing either: the script pages write five things — a cadence, the SOURCE through the same `ApplyEdit` funnel every mutation surface crosses, a run of the latest saved version under `RefuseRun`, a DRAFT run executed as the caller with the draft limits that persists nothing it produced, and what the script SAYS about itself (display name, markdown description, category, tags), which is not an input to any decision the platform makes — and apply the rules every surface shares: a script's definition (contract, source and version history, the authors' roles withheld) to everyone signed in (#1866); the run history, state and every action to the script's owner and administrators; one particular run additionally to whoever requested it; and the cadence controls to the owner and administrators, refusing a caller who does not own the script with the same answer as one who may not see it. Residual risks are named rather than minimized: no hard memory cap; a save is unattended execution with no second reader, which since #1419 covers the author's whole tool surface including the tools that write (bounded by the roles being the author's own and never more, by the persona filter enforcing them at every call and re-resolving them at every run, by editing a shared script being an administrator's action, and by disable/deprecate/supersede stopping it at execution — a person can, through a script, arrange for their OWN access to be exercised on a schedule, which is the feature, and the audit trail under the script principal is its record); a version authored by an admin captures admin roles; standing authority outlives the author; a schedule multiplies what a save permitted; delivery is standing egress on a schedule once configuration declares a destination; a draft run has no per-request rate limit of its own; and a dry run's stored log is free text the script printed under its CALLER's access. Covers script STATE (#1537): one JSON object per script, read as `run.state` (pinned on the run row at creation, an input of the run beside its parameters so the determinism contract reads same version + same parameters + same state read + same data) and written with `platform.save_state`, bounded at 64 KiB and checked for JSON-representability at the call; applied by `RunStore.Finish` for a succeeded run only, in the transaction that records the status, as an upsert predicated on the revision the run read, so a failed run leaves the state alone, the loser of two runs that read one revision fails naming the winner with its outputs standing, and a stale worker's lease is refused before the state row is touched; a person's reset moves the revision and is recorded, and the tool's `state set`/`clear` are refused from inside a run; state is not an asset, not shared, not delivered, and deleted with the script; a failure recorded with its cause, read from the error's type rather than its text, with only an upstream_retryable call re-issued by the host and no run ever re-queued on a cause (#1859); memory measured at every host call against `scripts.worker.max_run_memory` with a lone run past the shed threshold failed rather than killed, stated as a measure and not an allocation ceiling (#1861); take-overs of a dead worker's run capped so one oversized run cannot crash-loop every replica (#1860)

## Docs: API Reference

- [Tools API](https://mcp-data-platform.txn2.com/reference/tools-api/): Complete tool specifications with parameters and responses
- [Configuration Reference](https://mcp-data-platform.txn2.com/reference/configuration/): Redirect page; the full YAML schema lives at [Configuration](https://mcp-data-platform.txn2.com/server/configuration/)
- [Providers](https://mcp-data-platform.txn2.com/reference/providers/): Semantic, query, and storage provider interfaces
- [Middleware](https://mcp-data-platform.txn2.com/reference/middleware/): Request processing chain: the result-type stamp every client revision requires, tool visibility, description overrides, the open advertised output schemas, the search-first gate, icon enrichment, client logging, progress notifications, and the rule every middleware that appends a block follows: a block is merged into the structured result the tool's own handler set and never synthesized into one the handler left empty, so an export's asset metadata is not replaced by the platform's own additions
- [Tuning and Scaling](https://mcp-data-platform.txn2.com/reference/tuning-and-scaling/): Resource limits, Go runtime tuning, horizontal scaling characteristics, connection pool sizing, autoscaling guidance, and measured single-replica throughput/latency limits from the load harness (test/load); the sibling agent-effectiveness benchmark harness (bench/) measures arm-ablated accuracy and efficiency with audit-derived metrics

## Docs: Evaluation

- [Knowledge-Layer Benchmark](https://mcp-data-platform.txn2.com/reference/benchmarks/): Product-framed introduction to the agent-effectiveness benchmark (bench/, #930). Names what is measured -- the knowledge layer specifically (cross-enrichment, search, and the memory/apply_knowledge lifecycle), not the whole platform (it says nothing about OAuth 2.1, personas, audit, the gateways, or the portal). Headline, arm-vs-arm on a pinned model: on knowledge-trap questions the layer lifts accuracy 42.7% -> 98.7% (+56 pts, 95% CI +44 to +67), and is statistically tied on plain lookups and arithmetic where no business context is needed, so the gain is specific to knowledge-gated tasks, not a blanket boost. Explains the two-halves framing (capture/curation populates the sinks; surfacing via cross-enrichment and search delivers them). Points to the Benchmark Report for all numbers, and to bench/ as the index to the whole report series (each study's published page, DOI, protocol document, recompute toolchain, and archived run data); the method behind its numbers is bench/docs/knowledge-layer-protocol.md
- [Benchmark Report](https://mcp-data-platform.txn2.com/reference/benchmark-report/): The canonical, citable, neutral evaluation report of the knowledge layer, written from committed run data under bench/results/. Carries author (Craig Johnston), publication date, a citable Report v2.0 version tied to the platform build/commit and release tag, a How-to-cite block with a ready-to-paste citation and BibTeX, and a Zenodo concept DOI 10.5281/zenodo.21438044 (published, resolves to the latest version; v2.0.1 is version DOI 10.5281/zenodo.21751635 and the v1.0 snapshot is 10.5281/zenodo.21438045; also in CITATION.cff and .zenodo.json). Two complementary sub-studies (never mixed, different client paths): a single-shot S1-S3 ablation showing the layer lifts trap accuracy 42.7% -> 98.7% (+56 pts, 95% CI +44 to +67) and is neutral where no business context is needed; and a cold-start learning curve (claude-cli, sonnet) climbing from a reproducible ~47% empty-layer floor (five replicates: 44/44/47.8/48/52) to 90.7% (K=3) / 100% (K=1) as six facts are taught and promoted, each trap class unlocking at its own promotion checkpoint. Includes an S5 lifecycle scorecard, re-run at k=5 over 30 protocols on v1.118.0 for report v2.0 (transfer 98.9% CI [96.8-100.0], capture 91.9%, personal recall 95.3%, duplicate rate 22.0% CI [9.8-34.1] as a point estimate rather than the v1.1 range; the transfer change is across-code, driven by #1129, not a scale effect), a mandatory threats-to-validity section, and full data-availability table. Every number is recomputed from raw JSON by bench/reports/knowledge-layer/report.ipynb; figures under bench/reports/knowledge-layer/figures/. A 2026-08-02 erratum records that every run executed with fetch (and list_connections) absent from the arm personas' allow-lists (19 fetch attempts across all 4,173 archived transcripts, 19 denials, #1176): the arms measured search-only, single-hop delivery, uniformly, so the contrasts stand and no statistic changed; the bench configs now grant both tools
- [Benchmark Report: Knowledge Use](https://mcp-data-platform.txn2.com/reference/benchmark-report-knowledge-use/): Knowledge-use report, version 1.0 (2026-07-26, DOI 10.5281/zenodo.21614059), second in the benchmark report series, headline tables rerun and replicated on a v1.116.0 tag build: when do agents use stored knowledge? Two-factor result over the perishable-knowledge fixture: reliance is governed by derivability (Sonnet 5 re-derives checkable delivered state at any recheck cost 1-11 calls, zero effort delta vs no-knowledge controls, but uses a non-derivable reporting convention 8/8 and fabricates one 6/8 without it) and by capability (Haiku 4.5 inverts the derivable result, trusting delivered state 29/32). Sonnet headlines replicate exactly on the raw Messages API (no agent client), and the tier flip occurs within one client, so neither is a client artifact. Companion lifecycle probe: supersede at ceiling conditional on capture; strict capture 33%, decomposed into deterministic entity mis-filing and silent non-capture. All exploratory (the pre-registered H1a was falsified by the power pre-run and the confirmatory matrix never ran); every table recomputes offline via bench/reports/knowledge-use/pk_tables.py and every figure via figures.py beside it (both runnable from the notebook bench/reports/knowledge-use/report.ipynb) from bench/results/knowledge-use/. A 2026-08-02 threats-to-validity entry records that every run executed with fetch denied by the arm persona, so every rate is measured under search-only, single-hop delivery; the denial was uniform across cells and the contrasts stand (#1176)
- [Benchmark Report: Knowledge Pollution](https://mcp-data-platform.txn2.com/reference/benchmark-report-knowledge-pollution/): Knowledge-pollution report, version 1.0 (2026-08-07, DOI 10.5281/zenodo.21834813), third in the benchmark report series, the confirmatory outcome of the pre-registered protocol bench/docs/knowledge-pollution-study-design.md (#1166): when a wrong claim passes the platform's own capture-approve-apply gate into the shared applied tier beside a co-present correct source, what governs whether other identities adopt it? Result inverts the pre-registered hypothesis: the only claim adopted anywhere is the checkable one (an order count one query settles) - 16/24 on Haiku 4.5, 0/24 on Sonnet 5 and Opus 5 - while the non-derivable convention is adopted nowhere on the agent client. The mechanism is exact across 120 episodes with no exception: every episode that observed the refuting count answered correctly, every episode that did not adopted, and unplanted controls query 24/24 - the claim displaces verification rather than winning an argument. Robust across directive strength (bare == imperative, so adoption not compliance), storage sink (knowledge page at least as contagious as the catalog entity; search delivery alone carries it), fixture (24/24 vs 0/24 controls on the API fixture), and client (raw Messages API replication). Client-bound: the convention's immunity (raw-API haiku adopted it 4/8). Confounded on the strongest tier: the plant's reviewer note disclosed the plant through fetch of the insight, provenance inspection is capability-graded (9/24 -> 18/24 -> 24/24 on the contested convention cells), and the weak tier adopts straight through the disclosure. Retraction (RQ3) never ran. Every table recomputes offline via bench/reports/knowledge-pollution/pollution_tables.py and every figure via figures.py beside it (both runnable from bench/reports/knowledge-pollution/report.ipynb) from bench/results/knowledge-pollution/, and make bench-report-check (part of make verify and CI's harness job) pins the published headline numbers to the archives
- [Benchmark Report: Graph Completion](https://mcp-data-platform.txn2.com/reference/benchmark-report-graph-completion/): Graph-completion report, version 1.0 (2026-08-10, DOI 10.5281/zenodo.21881798), fourth in the benchmark report series, the boundary-condition outcome of the pre-registered protocol bench/docs/graph-completion-study-design.md (#1250, run under #1251): when a completion task's constraints live across a knowledge-page graph, do authored edges deliver what retrieval structurally cannot - constraints search cannot rank, and a decidable notion of done? Design: a deterministic generated corpus (fixed 27-page core, Seed 1250, EdgeDensity 3) at 50/500/5000 pages, graph vs stripped-to-prose arms holding page meaning constant, two discontinuity constraints per cell certified unreachable twice per scale (offline embedding rank outside the horizon for every task phrasing AND a live sweep gate requiring absence), 99 episodes on one agent configuration. Headline: the pre-registered instrument kill fired and is the finding - stripped-arm agents grounded the certified-unreachable constraints at 1.00 (500) and 0.93 (5000) by reading the authored edge's meaning-preserving prose fallback and re-querying in the named institution's own vocabulary, so unreachable-by-search must hold against read-derived queries, which meaning-constant authored prose cannot provide (the certifications sampled task-derived phrasings only; the agent runs the classical pseudo-relevance-feedback loop spontaneously). What the archives affirmatively show: discovery is not enumeration (grounded coverage at ceiling in every cell, ~11 fetches grounding every constraint against 5000 pages; scale moved cost, not coverage); authored edges buy cost and robustness, not coverage (searches per grounded constraint 0.59/0.67/0.63 graph vs 0.73/1.02/1.44 stripped across two orders of magnitude - flat vs roughly doubling - with the matrix's only failure and only sub-ceiling cell in stripped/5000, and graph-arm page-provenance fetches rising 0.09/0.28/0.34 with scale); with search removed the graph arm walked every closure at full depth (1.00 grounded, zero searches, scale-invariant with the pilot's 0.96/0.42 floors vs stripped 0.00); and the elicited completeness claim was uniformly conservative (0 of 98 episodes claimed complete), leaving closure awareness unmeasured, not null. Every table recomputes offline via bench/reports/graph-completion/graph_tables.py and every figure via figures.py beside it (both runnable from bench/reports/graph-completion/report.ipynb) from the committed archives, and make bench-report-check (part of make verify and CI's harness job) pins the published headline numbers to the archives - including the instrument kill's presence and the archived analyzer's deliberate non-zero exit

## Docs: Examples

- [Examples Gallery](https://mcp-data-platform.txn2.com/examples/): Real-world configurations for enterprise governance, data democratization, and AI/ML workflows, plus the read-only warehouse beside a writable scratch schema: two Trino connections to ONE cluster where the difference that matters is the Trino IDENTITY, not the platform flag. States plainly what `read_only` does and does not guard -- a statement-prefix denylist evaluated per connection name, with no catalog or schema restriction anywhere in the toolkit and `catalog`/`schema` being session defaults rather than bounds -- so a write-capable connection authenticating as the same Trino user as the read-only one can write `INSERT INTO warehouse...`; the scratch connection needs its own Trino account whose access-control rules allow DDL only on the scratch catalog. Adding a catalog allow-list to the toolkit is deliberately NOT done: parsing SQL is the wrong layer for a boundary the engine already enforces on its identities. Also carries the inline join a short external list actually needs -- `JOIN (VALUES (...)) AS t(id)` through `trino_query` on the read-only connection, which the instruction baseline tells agents about so a handful of pasted keys is joined rather than refused or turned into a request for a table.

## Docs: Ecosystem

- [Ecosystem](https://mcp-data-platform.txn2.com/ecosystem/): Sister MCP projects (mcp-datahub, mcp-s3, mcp-trino) and how they compose

## Docs: Support

- [Release Highlights](https://mcp-data-platform.txn2.com/release-highlights/): Reverse-chronological timeline of notable features; GitHub Releases remain the authoritative per-release changelog
- [Troubleshooting](https://mcp-data-platform.txn2.com/support/troubleshooting/): Common issues, error codes, debugging guide

## Docs: Go Library

- [Library Overview](https://mcp-data-platform.txn2.com/library/overview/): Build custom MCP servers using the Go library
- [Quick Start](https://mcp-data-platform.txn2.com/library/quickstart/): Code examples for common patterns
- [Architecture](https://mcp-data-platform.txn2.com/library/architecture/): Package structure, MCP protocol middleware, provider interfaces, and connection identity — the three names a connection carries (the instance it is configured and stored under, the name a call binds it by, the toolkit serving it), which of them each surface keys on, and connid.Resolver as the one translation between them
- [Extensibility](https://mcp-data-platform.txn2.com/library/extensibility/): Custom toolkits, providers, middleware
- [API Stability](https://mcp-data-platform.txn2.com/library/stability/): Supported import surface, the TestPublicSurfacePolicy gate bounding what else may live under pkg/, change policy for other pkg/ packages, config-file compatibility policy

## Portal

The `portal` rail section: a guided tour of every screen the built-in web portal serves. Its two halves are the sidebar's own, Your Work and Administration.

- [The Portal](https://mcp-data-platform.txn2.com/portal/): What the portal is, how to enable it, how branding resolves (brand name, theme-aware logos, the version link, the public viewer's own branding and CSP), the map of the tour, and the statement each section opens with of what it holds
- [Activity](https://mcp-data-platform.txn2.com/portal/activity/): Your own sessions and the calls they made, each with the purpose the agent stated for it
- [Assets](https://mcp-data-platform.txn2.com/portal/assets/): Your own asset library — what you saved and what your managed scripts produced, on the same terms — plus the asset viewer, its provenance panel, references, version history and retention, sharing, and presenting a slide deck (an HTML asset on the reveal.js runtime the platform serves at `/portal/vendor/reveal/`, with Present and Export PDF on every HTML asset, a print copy always rendered light whatever the deck's theme, Overview on a deck, a frame that fills the page, and the built-in page `platform-presentations` an agent is pointed at). The knowledge pages that cite an asset are a Referenced by button beside its version picker, and a thumbnail that could not be drawn says which part failed (#1791, #1792). Source has Wrap (on when a line is wider than the editor) and Format for HTML, JSON, JSX, YAML and Markdown: an unsaved edit that only Save stores, refused when the HTML would display different text or does not parse (#1839).
- [Collections](https://mcp-data-platform.txn2.com/portal/collections/): Grouping assets, and sharing a group as one. The knowledge pages that cite a collection are a Referenced by button beside Feedback (#1792).
- [Resources](https://mcp-data-platform.txn2.com/portal/resources/): The file manager, stored folders, uploads, revisions, moving a file between libraries, registering a CSV as a queryable table, the export destination that keeps a file current without its bytes passing through anybody, and files delivered in an archive: a worked example of api_export, manage_resource extract to one rolling CSV, a table registered once with follow, and the warehouse load
- [Scratch Tables](https://mcp-data-platform.txn2.com/portal/scratch-tables/): What is registered, whether it is still current, and what a failed follow looks like
- [Shared With Me](https://mcp-data-platform.txn2.com/portal/shared/): Work other people shared with you, and the feedback thread on it
- [Knowledge and Memory](https://mcp-data-platform.txn2.com/portal/knowledge/): Promoted knowledge pages (including a page citing a managed script, resolved per reader: the owner and administrators see the citation, other readers read the page without it), the catalog and graph views, the insight review queue, and captured memory
- [Prompts](https://mcp-data-platform.txn2.com/portal/prompts/): The prompt library, collections, authoring, version history and diffs
- [Automations](https://mcp-data-platform.txn2.com/portal/scripts/): The portal section named Automations (#1912): work that runs on a schedule or on request, every one of which is a managed script today; the pages live at /portal/automations and /portal/admin/automations, and the old /portal/scripts paths (including run links mailed before the rename) redirect with the rest of the path intact. The listing has a Kind column reading Script, and a grid view whose tiles are the scripts' flow diagrams, drawn light and dark by the thumbnail worker and served at GET /api/v1/portal/scripts/{id}/thumbnail (#1909). The Automations, Schedules and Runs tabs (the Schedules tab draws when every scheduled script fires, on today, this week or three months by how often it fires, #1891). A managed script's page: its schedule, its code as a Flow tab (a diagram derived from the saved version's source: every platform call a card colored input/reads/writes/output, edges following values with implied ones removed, functions as boxes, one-call helpers as chips, computed names dashed as {their source}, parameters listed with the steps they reach, run.state and its save; double-click opens the lines in Source and lines selected in Source light their cards; GET /api/v1/portal/scripts/{id}/versions/{version}/graph and the admin counterpart, REST only, #1906; the Flow tab opens on the latest run drawn on the diagram for the owner or an administrator, each card with its calls, time and rows, unreached cards dimmed and a failed run's card in the error color, calls attributed by the call site each audited script call records in audit_logs.call_site, GET /api/v1/portal/scripts/{id}/runs/{runID}/flow, #1907; each older version compares with the running one on the diagram, cards marked added, changed or removed, ?compare=<version> on the graph route, #1908) and a Source tab, dry run, versions, run history (a running run's progress, live log and Cancel/Stop control, and the result a run returned, #1845/#1847; a failure saying whether a retry should succeed, the peak memory a run held, a run whose worker stopped reporting badged worker not responding or worker gone with its holder and attempts, and a Running now panel, #1859/#1860/#1861), state and delete
- [Settings](https://mcp-data-platform.txn2.com/portal/settings/): Your notification delivery mode, category toggles, the API keys you have issued for your own account, and what the platform has sent you
- [Admin Dashboard](https://mcp-data-platform.txn2.com/portal/admin-dashboard/): The operator's summary, indexing health, and the MCP, API Gateway (including outbound calls split by the persona that caused them), Health, Events and Notifications tabs
- [Admin Tools](https://mcp-data-platform.txn2.com/portal/admin-tools/): Every registered tool with its schema, a live runner, per-tool activity, enrichment rules, and the visibility kill-switch
- [Admin Sessions and Calls](https://mcp-data-platform.txn2.com/portal/admin-sessions/): Who was working, what they ran, and what came of it
- [Admin Knowledge Review](https://mcp-data-platform.txn2.com/portal/admin-knowledge/): Reviewing and promoting captured insights, with the warehouse observation beside each claim
- [Admin Content](https://mcp-data-platform.txn2.com/portal/admin-content/): Assets, collections, resources, prompts and scripts across every owner
- [Admin Connections and Catalogs](https://mcp-data-platform.txn2.com/portal/admin-connections/): The deployment's downstream systems and the OpenAPI catalogs its API connections resolve against
- [Admin Personas](https://mcp-data-platform.txn2.com/portal/admin-personas/): The persona editor: roles, priority, tool and connection patterns, API endpoint rules, and the access test
- [Admin Keys and Users](https://mcp-data-platform.txn2.com/portal/admin-access/): The API keys programmatic callers present, and the directory of people the platform knows
- [Admin Settings and Change Log](https://mcp-data-platform.txn2.com/portal/admin-settings/): Deployment settings, agent instructions, the deployment description, and the record of every change

## Key Capabilities

- Semantic first: every data response carries DataHub business context (owners, tags, glossary terms, quality scores, deprecation warnings) added automatically at the protocol layer.
- Composable toolkits: DataHub for cross-enrichment, Trino and S3 optional, all three omittable, plus a Toolkit interface for custom integrations. Multi-instance per service with isolated failure domains.
- Bidirectional cross-enrichment: Trino results include DataHub metadata; DataHub searches include query availability and sample SQL; S3 operations include semantic context; lineage inheritance fills column metadata from upstream datasets.
- OAuth 2.1 inbound: built-in authorization server with PKCE and Dynamic Client Registration so Claude Desktop and any MCP client can sign in directly to the platform.
- OAuth 2.1 outbound to upstream MCPs and APIs: client_credentials (M2M) and authorization_code+PKCE (browser sign-in) grants, encrypted refresh tokens persist across restarts.
- OIDC discovery for Keycloak, Auth0, Okta, Azure AD; API keys for service accounts; fail-closed by default.
- Role-based personas: map OIDC roles to personas with allow/deny tool patterns and connection-level filtering.
- Audit logging to PostgreSQL with user, persona, tool, sanitized parameters, duration, and enrichment metrics. Searchable from the admin portal.
- Admin and user portal: connections, personas, API keys, configuration entries, knowledge governance, audit, gateway, plus user-facing assets, collections, prompts, resources, activity.
- Gateway toolkits: re-expose any third-party MCP server or REST/HTTP API through the platform's auth, persona, audit pipeline, with optional declarative cross-enrichment rules.
- Knowledge capture: tribal knowledge from AI sessions written back to DataHub through human-in-the-loop governance with changeset rollback.
- Memory layer (PostgreSQL + pgvector): persistent memory across sessions with semantic recall, entity recall, graph traversal, and staleness detection.
- Managed resources: human-uploaded reference material with scope-based visibility, surfaced to AI assistants via MCP resources/list and through the universal search (indexed over file contents, fetched by mcp:resource:<id>).
- MCP Apps: interactive UI panels rendered inline in the MCP host.
- Two transports: stdio for desktop clients, http with OAuth 2.1 for hosted clients.
- Two operating modes: standalone (no database) for stateless deployments, file + database for the full feature set.

## Quick Start

```bash
# Install
go install github.com/txn2/mcp-data-platform/cmd/mcp-data-platform@latest

# Minimal config (DataHub only) - DATAHUB_URL and DATAHUB_TOKEN are expanded
# from the process environment via ${VAR} syntax; the binary does not read
# them directly, so they must be referenced somewhere in the YAML
cat > platform.yaml <<EOF
server:
  name: mcp-data-platform
  transport: stdio
semantic:
  provider: datahub
  instance: primary
toolkits:
  datahub:
    enabled: true
    instances:
      primary:
        url: "\${DATAHUB_URL}"
        token: "\${DATAHUB_TOKEN}"
    default: primary
EOF

# Wire to Claude Code
claude mcp add data-platform \
  -e DATAHUB_URL=https://datahub.example.com/api/graphql \
  -e DATAHUB_TOKEN=$TOKEN \
  -- mcp-data-platform --config platform.yaml
```

## Other links

- [GitHub Repository](https://github.com/txn2/mcp-data-platform): Source code, issues, and releases
- [Security Article](https://imti.co/mcp-defense/): MCP Defense, a case study in AI security
- [Working paper: MCP Gateway OSS Landscape Survey (2026-04-24)](https://mcp-data-platform.txn2.com/research/mcp-gateway-landscape-2026/): Dated engineering working paper, not product documentation. Surveys nine open-source MCP gateways against the platform's gateway design; conclusions may have been superseded
- [Working paper: MCP Gateway Live End-to-End Test (2026-04-24)](https://mcp-data-platform.txn2.com/research/mcp-gateway-livetest-2026/): Dated engineering working paper, not product documentation. Records one verification run of the gateway path against a then-unmerged branch; conclusions may have been superseded