Releases

Changelog indexer and registry for AI agents and developers. The user-facing CLI lives in this monorepo at apps/cli/ (published as @buildinternet/releases); alongside it are the backend (API worker + D1), the remote MCP server, and the web frontend.

Stack

• Runtime: Bun
• Language: TypeScript (strict mode)
• Database: Cloudflare D1 + Drizzle ORM. No local SQLite path in this repo — bun:sqlite is only used for test fixtures (tests/db-helper.ts).
• API: Cloudflare Worker with Hono (apps/api/)
• MCP: Remote MCP server (apps/mcp/)
• AI: Anthropic SDK (@anthropic-ai/sdk); managed agents via @anthropic-ai/claude-agent-sdk

Layout

apps/
  web/            Next.js frontend (Vercel)
  cli/            the `releases` CLI (published as @buildinternet/releases)
  api/            REST API (Cloudflare Worker, Hono + D1)
  mcp/            remote MCP server (Cloudflare Worker)
  webhooks/       webhook delivery queue consumer (Cloudflare Worker)
packages/         shared code: published @buildinternet/releases-* (incl. releases-lib) and private @releases/*
actions/          GitHub Actions (publish-changelog)
skills/           published agent skills
scripts/          operational + maintenance scripts
tests/            shared test helpers, fixtures, evals, cross-app tests (app tests live in apps/*/test)
docs/             architecture docs, runbooks, plans

Per-directory details: apps/README.md and packages/README.md.

Commands

• Lint + format + type-check: bun run check (oxlint with typeCheck via oxlint-tsgolint, then oxfmt --check)
• Lint only: bun run lint (bun run typecheck is an alias)
• Format: bun run format:check
• Tests: bun run test (not part of check; runs the root script below — a bare bun test puts every suite in one process; bun test apps/cli is the last process in it)
• Targeted tsc: apps/mcp still uses npx tsc --noEmit in CI (carved-out workspace; excluded from root oxlint). apps/web and tests/ have their own tsconfigs for local runs.
• Evals (tests/evals/) are manual and on-demand only. They call AI APIs, cost money, and take minutes. bun run eval:evaluation is the only in-repo suite (URL evaluation, ~30s). Parsing + discovery evals live with the CLI (apps/cli/).

Local development

Running the four dev:* services behind portless, fresh-worktree bootstrap, worktree teardown, and local D1 schema parity (db:reset:local / db:pull) and auth rate-limit / signing-secret notes: local-development.md.

Workspaces and carved-out packages

Root package.json declares apps/api, apps/web, apps/webhooks, and packages/* as workspaces. apps/mcp/ is intentionally excluded — it pins zod to the MCP SDK's nested copy, so it carries its own bun.lock and needs its own bun install.

The root test script is bun test packages/ && bun test tests/ apps/web/ apps/mcp apps/webhooks && bun test apps/api, so apps/api runs in its own bun test process, after the rest. This is deliberate isolation, not a style choice: bun's mock.module() is process-global, keyed by resolved module, and not restorable per-specifier (mock.restore() doesn't undo it, and re-mocking with the real impl is impossible — every import of the path resolves to the mock). The scrape-fetch tests that mock.module("@releases/adapters/cloudflare", …) at module scope (now in tests/adapters/) stub the same path apps/api's render-check tests import for real — an ordering-dependent flake if they ran in the same process. Splitting apps/api into a separate process makes the leak structurally impossible. Keep tests/adapters/ (and mcp/webhooks) in the root-cwd multi-dir invocation — that's where their module mocks resolve against the single root-workspace copy of @releases/adapters the source uses. Isolating apps/api (rather than tests/adapters/) is what keeps both sides resolving correctly.

Reach for interface injection before process isolation. Process isolation is the fix of last resort — it only works when the leak crosses a boundary the test script can split on. When a mock stubs a third-party module that sibling suites import, prefer passing a fake through the seam the code already has. packages/ai/src/aisdk-text-model.test.ts used to mock.module("ai", () => ({ generateText })), which dropped every other export and made overview-content.ts's import { Output, parsePartialJson } from "ai" fail with an unhandled SyntaxError between tests, ordering-dependent. Spreading the real module back into the factory does not rescue it: mocking ai at all forces a second evaluation of the module, so ai/test's MockLanguageModelV3 and the generateText it runs through resolve to different instances and the run hangs. The fix was to delete the module mock and drive the real generateText with a MockLanguageModelV3 — a plain argument, scoped to one test, invisible to every other file. Most SDKs ship such a seam (ai/test here); use it.

Shared code is split between published npm packages (@buildinternet/releases-*) and private in-tree packages (packages/):

• packages/core/ → published as @buildinternet/releases-core from this monorepo. Pure, runtime-neutral helpers shared with the CLI: DB schema (source of truth), categories, dates, changelog-range, changelog-slice, overview, id, slug, tokens, cli-contracts, lookup-coordinate, fts (FTS5 query sanitizer — wraps tokens in phrase quotes so org/repo and similar punctuation don't error). Consumed here and by the in-tree CLI via workspace:*, so schema changes and their CLI updates land in one PR.
• packages/core-internal/ → imported as @releases/core-internal. Private, workspace-only. DB-coupled / worker-only helpers the thin client doesn't need: release-upsert (drizzle upsert config), hash (node crypto), webhook surface (webhook-sign, webhook-resilience, webhook-url-safety, webhook-delivery, webhook-alert-format, plus release-event wire types), schema-coverage (the release_coverage table, part of the drizzle composite schema).
• packages/api-types/ → published as @buildinternet/releases-api-types from this monorepo. Wire protocol — request/response shapes served by the API worker, consumed by the MCP worker, web frontend, and the OSS CLI. Consumed here and by the in-tree CLI via workspace:*; wire changes and the CLI that adopts them ship together. Additive by default — renames/removals go through a one-minor-version deprecation alias before removal.
• Publishing core / api-types / sdk: in-repo exports point at src/*.ts; npm pack runs scripts/pack-dist.ts, which builds dist/, adds .js to relative imports, pins workspace: deps, and swaps exports to it, so npm consumers get JS + .d.ts. Keep relative imports in packages/core extensionless (Turbopack won't map ./x.js to ./x.ts there); CI dry-run packs the packages.
• packages/adapters/ — adapter primitives (types, source-meta, content-hash), the github, cloudflare, crawl, and feed adapters, plus the shared scrape/agent fetch orchestration (scrape-fetch / extract-deps-worker / deterministic-update, persisting via an injected API fetcher — used by the API worker's update workflow, #1946). All pure / worker-safe.
• packages/ai/ → imported as @releases/ai-internal. evaluate (URL recommendation + buildMetadataFromEvaluation), playbook (deterministic markdown generation), providers (provider-detection table), release-content (Haiku 5.5 summarization for title_generated / title_short / summary — shared by scripts/generate-release-content.ts and the ingest-time hook), marketing-classifier (Haiku 5.5 binary verdict on whether a feed item is real product news vs. marketing — used by fetchOne when metadata.marketingFilter is set). Worker-safe; caller passes the Anthropic client.
• packages/rendering/ → imported as @releases/rendering/*. Atom feed helpers, markdown/JSON formatters, and media URL helpers.
• packages/search/ → imported as @releases/search/*. Embedding providers/cache, Vectorize hybrid search, and release/entity/changelog embedding pipelines.
• packages/lib/ — slim private utilities (config, errors, source-edit, Anthropic client/error helpers, managed-agent rate limits, anthropic-pricing for list-price cost estimates on managed-agent sessions, spend-cap daily-spend KV gate, session-error-classify). logger is published as @buildinternet/releases-lib/logger.
• packages/releases-lib/ → published as @buildinternet/releases-lib (CLI helpers: config, legacy-env, logger), part of the CLI's fixed release group. A workspace here; the CLI and @releases/* packages consume it via workspace:*.

Onboarding is agent-driven, not a remote harness: a local Claude Code session runs the local-ingest skill, self-serve listing, or on-demand /v1/lookups to add orgs/products/sources by hand. See agents.md.

The release_coverage schema lives with the rest of the DB-coupled internals in packages/core-internal/src/schema-coverage.ts (imported as @releases/core-internal/schema-coverage), not at the repo root.

Conventions

Keep entries to one line: the rule + a pointer to the doc that owns the detail. When a feature needs a paragraph, that paragraph belongs in docs/architecture/, not here. This section has bloated twice from append-on-ship; resist it.

• This repo is public — keep PII out of committed content. No absolute home-dir paths (/Users/<name>/… — write ~/… or repo-relative), no personal email addresses (use @example.com in fixtures/docs), no customer data or real tokens, in any committed file — including generated plans and specs under docs/plans/ and docs/superpowers/.
• Logging splits by runtime: worker code (apps/api, apps/mcp, apps/webhooks) MUST log via logEvent() from @releases/lib/log-event (worker-safe structured JSON); CLI + runtime-neutral packages use @buildinternet/releases-lib/logger (stderr + ~/.releases/logs/). Never import the fs-backed @buildinternet/releases-lib/logger into a worker. Payload conventions, severity, and Error unwrapping: logging.md.
• Source types (fetch adapters): github, scrape, feed, agent, appstore. Adapter behavior + appstore materialization: ingest.md.
• Ingest pipeline (fetch → parse → insert) — dedup (UNIQUE(source_id,url) + RELEASE_URL_UPSERT + same-day body hash), smart-fetch backoff, URL exclusion (ignored_urls org-scoped / blocked_urls global, via isUrlExcluded()), release suppression (suppressed=1), and the ingest-time Haiku 5.5 passes (content summarization, the per-source marketing classifier via metadata.marketingFilter, feed-content enrichment via FEED_ENRICH_ENABLED): ingest.md. Cron/Workflow orchestration: remote-mode.md.
• GitHub Action ingest (#2291): actions/publish-changelog diffs changelog markdown on push and POSTs upsert-content to the existing /releases/batch route (no parallel write path). Planning is @buildinternet/releases-core/changelog-publish so the Action and releases publish share one parser (#2377). Product docs: Publish from GitHub Actions, Publish from any CI. See ingest.md → GitHub Action ingest.
• Push-fed sources (#2374, #2381): metadata.ingestMode = "push" marks a source fed directly by its publisher (e.g. actions/publish-changelog) — mirrors firecrawl.enabled as "externally driven, no local cadence": excluded from the poll cron, the SourceActor alarm, the scrape/agent OrgActor drain, and the first-party staleness scan; POST .../releases/batch stamps lastFetchedAt on a successful write since there's no poll to do it. A separate, quieter staleness scan (cron/push-staleness.ts) instead flags a push-fed source whose publisher appears to have stopped pushing. Set via the freeform source metadata PATCH routes — no schema change — or automatically on the first batch write made with an owner's publish token (#2390; only when the key is absent, so a curator's explicit "poll" opt-out sticks). See ingest.md → GitHub Action ingest.
• Release importance (1–5, AI-scored, releases.importance): scored inside the same ingest-time release-content summarize call, fail-open to null; filterable via ?minImportance= on GET /v1/releases/latest (forces a cache bypass); web shows it as a flame glyph at 4-5 only. See ingest.md → Content summarization.
• source.url is for humans; fetch routing lives in metadata. source.url is the canonical human-readable URL; machine fetch endpoints (RSS via metadata.feedUrl, GitHub-CHANGELOG override via metadata.githubUrl, …) live in metadata. Test: would a human ever want to land on this URL? If no, it's metadata. See remote-mode.md → Display URL vs. fetch routing.
• Crawl mode uses Cloudflare's /crawl endpoint for multi-page changelogs, stored in source.metadata.crawlEnabled. See packages/adapters/src/crawl.ts.
• Firecrawl monitoring: external fetch backend for scrape sources behind an anti-bot challenge our Browser Rendering can't clear; toggled per source via source.metadata.firecrawl (not a new type), excluded from the poll-fetch cron, prod-only secrets. Gotcha: the webhook carries a hunkless whole-document diff (no @@ headers) — parse only via addedContentFromDiff. See firecrawl-monitoring.md.
• Full-history backfill for windowed scrape sources: POST /v1/workflows/backfill-source { sourceId, markdown?, maxWindows?, dryRun? } loops extraction over every window and upserts idempotently; deep Firecrawl path routes to durable BackfillSourceWorkflow (R2 snapshot + per-window steps, resumable) behind BACKFILL_WORKFLOW_ENABLED (default off); returns 202 { instanceId, statusUrl } async. See firecrawl-monitoring.md.
• Source staleness signal + render dry-run (#1528, #2381): daily scans on the 0 4 * * * tick flag overdue first-party (cron/source-staleness.ts, source-staleness), Firecrawl (cron/firecrawl-staleness.ts, firecrawl-staleness), and push-fed (cron/push-staleness.ts, push-staleness) sources; when any are actionable, sendStalenessDigest emails operators via SEND_EMAIL (logs-only when the rollup is empty). Render dry-run: POST /v1/sources/:id/fetch?dryRun=true on a client-rendered scrape source (crawlEnabled/renderRequired) renders the index once and returns { renderCheck, candidateCount, sampleUrls } with no extraction/MA loop (CLI: releases source fetch <src> --dry-run). No feature flag. See firecrawl-monitoring.md.
• Transactional email — every message renders through one shell (renderEmail from @releases/rendering/email-shell): lane + severity rule, inline-styled table HTML plus a matched plain-text part, a copyable URL under every button, a footer that always states why it arrived, markdown-rendered release content, and Gmail Go-To/One-Click annotations (dormant until the domain is registered with Google). See emails.md.
• Admin test emails: /admin/emails (admin session) previews every outbound template via fabricated [test] sends (GET/POST /v1/admin/emails/{samples,test}); live follow digests stay on POST /v1/admin/digest/test, real collection digests on POST /v1/admin/digest/collection-test. See web.md → Admin hub.
• AI model lanes: /admin/models (admin session) overlays OpenRouter model ids per cheap-call lane (summarize / extract / feed-enrich / marketing) from OpenRouter's live catalog; stored in site_settings, wrangler vars are the fallback. See web.md → Admin hub.
• LLM model pins live in @releases/lib/models (modelId / resolveModel); a bump changes the pin plus an ANTHROPIC_PRICING row. Request builders use samplingParams / thinkingParams. Wrangler vars and /admin/models still override the OpenRouter lanes.
• Marketing classifier threshold: /admin/classifier overlays the marketing-classifier suppression threshold (GET/PUT /v1/admin/marketing-classifier, site_settings key marketing_classifier, default 0.65) and lists/toggles per-source metadata.marketingFilter via the existing source PATCH route. Stored rows are re-classified with POST /v1/admin/marketing-classifier/reclassify (dry-run by default, 25 per call). See ingest.md → Marketing classifier.
• Raw capture + re-extract: the steady-state scrape path captures the scraped markdown to released-raw behind raw-snapshot-capture-enabled (default off; the update path POSTs to POST /v1/orgs/:orgSlug/sources/:sourceSlug/raw-snapshot, scrape-only, #1283); POST /v1/workflows/reextract-source { sourceId, snapshotId?, maxWindows?, dryRun? } re-runs extraction from a stored snapshot with no live scrape, reusing the backfill machinery (#1284). See firecrawl-monitoring.md.
• In-place release re-fetch: POST /v1/workflows/refetch-release { releaseId, url?, dryRun?, force? } re-fetches ONE release's live page and updates the row in place — same rel_ id, replaced title/content/publishedAt (media only on extractor hit), AI fields nulled for regen; a stored #fragment URL requires an explicit same-host url, which rewrites the stored URL to the canonical permalink; placeholder extraction content is rejected and a >50% shrink needs force: true (#2077). See firecrawl-monitoring.md.
• Local ingest (local Claude Code, no remote dispatch): the local-ingest skill (.claude/skills/local-ingest/) has the agent fetch + extract releases itself and write via the idempotent /batch upsert — skipping the update workflow's server-side extraction and its inference bill. Mandatory robots.txt/Content-Signal opt-out preflight (preflight.ts) refuses ai-input=no (ai-train=no alone proceeds; conductor.build is refused via its ai-input=no). See local-ingest.md.
• Local backfill workflow (local Claude Code, no remote dispatch): the backfill-source / backfill-sweep dynamic Workflows (.claude/workflows/) wrap the local-ingest primitives in a deterministic harness — fail-closed preflight, explicit window cap, budget-gated extract waves, known-URL dedup, /batch upsert by typed src_ id — to backfill a source's history locally without the remote extraction bill. Front-door: the backfilling-sources skill; dry-run is the default. See local-ingest.md.
• Opt-out / safety gates fail closed. On an ambiguous or unparseable response (e.g. an HTML anti-bot challenge), a gate returns the safe verdict (unknown/refuse, never proceed); when merging multi-value directives the strictest reading wins (a no is never overwritten by a later yes). Reference gate: local-ingest/preflight.ts (0 proceed / 1 refuse / 2 unknown). See local-ingest.md → Preflight.
• Prod D1 local backups: bun run db:backup dumps every real table (auth included, FTS excluded — D1 export can't handle FTS5 virtual tables) to ~/Code/.backups/releases/ with retention rotation; dumps hold real user data — never commit/upload them. See local-development.md → Prod D1 backups.
• daysAgoIso() from @buildinternet/releases-core/dates for date cutoffs.
• D1's hard limit is 100 bound parameters per prepared statement. Dynamic OR/IN lists and batch INSERTs go through maxChunkLength / chunkForBindBudget (@buildinternet/releases-core/d1-limits); releases inserts use RELEASES_BATCH_CHUNK_SIZE, sized for one bind per column. inArray(...) lookups chunk at 90 IDs. An unchunked statement surfaces as SQLITE_ERROR: too many SQL variables on poll/ingest. Details: storage-portability.md.
• Local wrangler D1 one-shots are wall-clock bounded via node scripts/run-timed.mjs <sec> -- … (kills the whole process group on deadline) so a timed-out agent can't orphan a multi-GB miniflare boot; db:migrate:local/db:reset:local/db:query already route through it, long-lived wrangler dev/preview:* deliberately do not, and manual local D1 calls should be wrapped the same way. See local-development.md → Local D1 schema parity.
• Entity IDs are single typed nanoids (rel_/src_/org_/…): the same string is PK and public ID. No dual UUID/public_id prep. New SQL goes through apps/api/src/queries/*; no new releases_fts MATCH sites. Seams + capability map: storage-portability.md.
• Shared read queries: reads needed by both the API and MCP workers live in packages/queries (@releases/queries/*, API defaults for deleted/hidden rows; MCP differences are explicit options). Migration map + open decisions: shared-queries.md.
• Workflows-based ingest: every per-source fetch runs as a POLL_AND_FETCH_WORKFLOW instance, create()d from that source's SourceActor DO alarm (#1776/#1819 — the actor is the sole fetch driver; per-source tier intervals normal=4h / low=24h + smart backoff pace the alarm). The hourly poll cron is a re-seed heartbeat only — no fan-out, no inline fetch fallback. See remote-mode.md.
• OrgActor scrape/agent drain (#1777/#1946): SourceActor self-flags stranded scrape/agent sources and arms a per-org OrgActor DO that dispatches one deterministic update run per org via the shared startDeterministicUpdate gate (kill switch + spend cap + #1815 scrape lock; no budget/lock logic in the actor). Update runs execute entirely via the API worker's DeterministicUpdateWorkflow. See remote-mode.md.
• Feature flags via Cloudflare Flagship (Tier 1). Boolean kill switches / rollout gates evaluate at runtime through the FLAGS binding; registry is @releases/lib/flags (flag(binding, varValue, def)); order is Flagship → wrangler var → default, failing open to the var. Adding a flag: add a FLAGS entry, convert the read to await flag(...), and create the same kebab-case key in BOTH Flagship apps (releases-platform{,-staging}). Numeric tunables and secrets are intentionally NOT in Flagship. See feature-flags.md.
• Be judicious with feature flags — do NOT add one by default. Every flag is permanent maintenance surface: a registry entry, a runtime branch on every read, a key that must be created and kept in sync across BOTH Flagship apps, and a dead code path to eventually retire. A flag per feature is how a flag registry rots into dozens nobody dares delete. Default to shipping a feature enabled, with no flag. Reach for one ONLY when there's a concrete reason runtime toggling earns its keep: a genuine kill switch for something risky/expensive/external-facing, a staged rollout you actually intend to ramp, or an operational lever you'd pull without a redeploy. If you can't name which of those applies, you don't need a flag. When unsure, ship it on and add the flag later if a real need appears — adding is cheap, a sprawl of stale flags is not. Prefer one well-scoped kill switch over several fine-grained per-feature toggles.
• Extract tier: extractFromBody() branches on body token count — ≤50K one-shot /v1/messages, >50K a multi-round tool-use loop via AI SDK (extract-with-tools-aisdk.ts; legacy Anthropic SDK loop in extract-with-tools.ts only when no aiSdkModel), gated behind EXTRACT_TOOLLOOP_ENABLED (per-source metadata.extractStrategy = "toolloop"); falls back to one-shot on any error. See extract.md.
• CLI (apps/cli/, #2445): releases is built and published from this monorepo as a 7-package changesets fixed group (@buildinternet/releases, 5 platform binaries, @buildinternet/releases-lib); publish-cli.yml ships npm + GitHub release + Homebrew tap. Public API calls go through @buildinternet/releases-sdk (apps/cli/src/api/sdk.ts); internal routes stay on apiFetch, and both share cliFetch. Ship flow and trusted-publisher setup: cli-distribution.md.
• Classification taxonomy — source kind enum, products, release type (feature/rollup), tags, categories, collections, and how these axes differ (kind vs type vs category): taxonomy.md.
• REST route surface — route-naming buckets (#494), org-scoped routes + dual-registration + bare_slug_rejected (#690/#698), the /v1/lookups family (coordinate POST + on-demand materialization, by-domain, slug resolvers), org catalog, entity resolution (IDs over slugs), pagination shape, and the OpenAPI coverage gate (#894): routing.md.
• Public API surface: apps/api/src/api-surface.ts lists the public operations (production /v1/openapi.json, SDKs); every other route is internal. New routes start internal; promoting one needs a tag, 2xx response schema and request body, enforced by scripts/check-openapi-coverage.ts. See routing.md → Public API surface.
• Inbound provider webhooks live at /v1/inbound/<provider> (firecrawl, github): a namespace with no shared auth middleware, where each handler checks its own secret. Never put a receiver under a gated namespace (/v1/integrations, /v1/webhooks). A router's .use("/prefix/*") gates other routers' routes too, which apps/api/test/route-gate-isolation.test.ts guards (#2428). See routing.md.
• Product and source URLs share one namespace: every product and source lives at /{org}/{slug}, and a product slug may never equal a source slug in the same org; product create/adopt and source rename return 409, generated source slugs take -releases. /sources/:id is a 308 alias. Any new product/source write path must check isSlugHeldByProduct / isSlugHeldBySource (@releases/queries/entities). See web.md → Product and source URLs.
• Friendly release URLs — /release/rel_<id>-<slug>: the rel_ ID (positional, rel_ + 21 chars — nanoid may contain -/_) is the only routing key; the slug derives from title_short at request time (no stored column, no backfill) and stale/bare forms redirect to canonical with a 308 (permanentRedirect). Helpers in @buildinternet/releases-core/release-slug. See routing.md.
• Error responses — every non-2xx response is the nested envelope { error: { code, type, message, details? } }; throw a ReleasesError subclass and let respondError serialize it (never hand-roll c.json({ error })). Taxonomy source of truth is packages/core/src/errors.ts; the three-layer split, producers/consumers, and how to add a code: errors.md.
• Search-query log: /v1/search and the MCP search tool write each query (≤200 chars) + mode/counts/duration to search_queries (web carries X-Releases-Surface: web). Read via GET /v1/admin/search-queries{,/top}. Kill switch SEARCH_QUERY_LOG_DISABLED. Distinct from telemetry_events, which carries only command names and stays PII-clean for the OSS CLI contract.
• Release coverage: multiple releases can cover one launch (marketing post + changelog + app note); canonical + coverage items tracked in release_coverage, read paths hide coverage-side rows by default. Ingest also links release candidates, backports, and sibling-package releases (system:version-line). See coverage.md.
• Org overviews: AI-generated knowledge_pages (scope org) summarize recent activity; display staleness warning OVERVIEW_STALE_DAYS = 30 from @buildinternet/releases-core/overview. Automated regen runs on a daily cron with a per-org cadence (7d default, 2d velocity fast tier, overview_cadence_days manual override; #1895). See web.md → Org overviews.
• Collection daily summaries: one collection_daily_summaries row per (collection, ET day) — title + one-line summary + bullet takeaways generated nightly over closed ET days via the shared summarization lane (reuses SUMMARIZE_MODEL + its Haiku fail-open, distinct only by generationName; no per-feature model var), gated per-collection by collections.daily_summary_enabled; read via GET /v1/collections/:slug/daily-summaries, rendered as timeline date headers. See web.md → Daily summaries.
• GitHub CHANGELOG files are fetched alongside tagged releases, stored in source_changelog_files (refresh piggybacks on every GitHub fetch); web surfaces them via GET /v1/sources/:slug/changelog. See web.md.
• ISR revalidation is ingest-driven, not clock-driven. Ingest pings web's POST /api/revalidate (notifyWebRevalidate); the pages' revalidate = 86400 is only a backstop. A fetch revalidate anywhere in the ROOT LAYOUT's import graph caps EVERY route in the app — guarded by apps/web/src/lib/isr.test.ts. See web.md → ISR revalidation.
• API-worker → web internal calls share ONE channel credential (RELEASES_SERVICE_KEY / WEB_SERVICE_KEY, verified by verifyServiceKey), mirroring RELEASES_PROXY_KEY inbound. Reuse it for the next internal endpoint — do NOT mint a per-feature secret; it is deliberately not the root API key. See web.md → ISR revalidation.
• Media handling: at ingest, normalizeMediaUrl() (packages/rendering/src/media-url.ts) strips Next.js/Vercel image-optimizer wrappers so the underlying CDN URL is stored. Ingest-time R2 mirroring runs whenever the MEDIA bucket binding is bound (always in prod); an unbound bucket stores third-party URLs verbatim. See web.md → Media handling.
• Inline hosted-video cards (#1549): the cron poll-fetch media pre-pass detects Wistia/Loom/Vimeo/YouTube links in a new release's body (detectInlineVideos, packages/rendering/src/video-embed.ts), resolves a poster via oEmbed, and appends a { type:"video", url:<poster>, alt, linkUrl:<watchUrl> } media[] item that rides the existing processMediaForR2 mirror; web renders a read-only play-thumbnail card (InlineVideoCard) linking out. Special-cased to video (first inline-body asset promoted to mirrored media); fail-open, no flag; iframe embed deferred. See web.md → Inline hosted-video cards.
• Manual release media edit: PATCH /v1/releases/:id { media: [...] } REPLACES a release's stored media[] wholesale (curator fix without re-sync); not-yet-mirrored items (no r2Key, not on MEDIA_ORIGIN) run through the ingest processMediaForR2 mirror, gated on env.MEDIA != null. Cron re-fetch never clobbers this (onConflictDoNothing / RELEASE_URL_UPSERT only backfills stored-empty media). See [web.md → Media handling].
• Inline-video retrofit (#1549 backfill): POST /v1/workflows/backfill-video { releaseId?|sourceId?|all?, limit?, dryRun? } (admin, idempotent, no flag) re-runs detectInlineVideos over a stored release body, mirrors the poster, and APPENDS the type:"video" item to the existing media[] (dedup by linkUrl) — built on runVideoBackfill in lib/media/media-backfill.ts, sibling of backfill-media. See web.md → Inline hosted-video cards.
• Org avatars: stored at orgs/{slug}.{ext} in released-media, served from https://media.releases.sh/orgs/{slug}.{ext}; pointer on organizations.avatar_url, writable via PATCH /v1/orgs/:slug { avatarUrl }. New orgs land null with the OG/web fallback to github.com/{handle}.png (#982 open). Reuse the orgs/{slug}.{ext} key for any new avatar-write path. See web.md.
• Scoped API tokens: opaque relk_<lookupId>_<secret> Bearer tokens (api_tokens table), scope ladder read ⊂ write ⊂ admin; static RELEASES_API_KEY is implicit root; kill switch API_TOKENS_DISABLED. MCP gates the AI tools + the on-demand /v1/lookups fallback on write and forwards the caller's own token. See remote-mode.md → Auth model and mcp.md → scope enforcement. User-owned API keys use the Better Auth @better-auth/api-key plugin (prefix relu_, apikey table, per-key rate-limit + metering), capped read-only (USER_API_KEY_MAX_SCOPE = "read" — write/admin refused at the mint route and clamped at auth time, so a user key can never satisfy write), gated by user-api-keys-enabled; the relk_ lane stays for machine principals. Browser releases login mints one of these read-only relu_ keys with no copy-paste via device authorization (RFC 8628; deviceAuthorization() + bearer(), always registered) — verificationUri must be an absolute web-origin URL, not a relative /device. The CLI never stores the device session: it signs out after each step, account commands run their own one-shot approval with a named purpose (the device-code scope), and logout self-revokes via DELETE /v1/tokens/me (see remote-mode.md → Auth model).
• OAuth role provisioning: a user's OAuth scope ceiling is the Better Auth user.role column (user→read, curator→read+write, admin→read+write+admin; NULL → read-only, fail-closed); manage it via the root-key-gated PATCH /v1/admin/users/role route / releases admin user set-role (audited role-changed logEvent), NOT an env var (the old OAUTH_ADMIN_USER_IDS bootstrap is gone, #1484). First admin seeded by a one-time D1 write. See remote-mode.md → Role provisioning.
• OAuth client provisioning: register/manage "Sign in with Releases" OAuth clients via the root-key-gated admin/oauth route family (POST/GET/PATCH/DELETE /v1/admin/oauth/clients[...], rotate-secret); reloc_ secrets shown once, trusted ⇒ skip_consent, tokenEndpointAuthMethod:"none" ⇒ public/PKCE client. RFC 7591 dynamic client registration is ON (allowDynamicClientRegistration: true) so agent-run MCP clients self-register at the public /oauth2/register endpoint (untrusted ⇒ consent-required, forced public/PKCE, stored scopes capped at identity+read / DCR_SCOPES — never write/admin/skip_consent; rate-limited 5/min/IP in prod); the admin route stays for first-party/trusted clients, and the plugin's session self-service write endpoints remain admin-only (#1482). The consent page never trusts a client's self-reported name/client_uri: it shows the signed redirect_uri destination and a verified badge only for operator-set metadata.official (admin route official, default true on create). CIMD (#2409) is on too: an HTTPS-URL client_id (Claude's "published identity") is resolved by @better-auth/cimd on first authorize into a row with the same DCR ceiling; fetch transport is auth/oauth-cimd.ts. See remote-mode.md → OAuth client provisioning.
• OAuth resource-server JWT lane: the REST API + MCP workers verify the AS's JWT access tokens with jose + the JWKS endpoint (NEVER by importing better-auth into apps/mcp — zod-pin) via the shared @releases/lib/oauth-jwt (isJwtShaped/verifyOAuthJwt); checks sig + iss + aud + exp, maps the scope claim (already role-clamped at issuance) onto the read ⊂ write ⊂ admin ladder. Additive + fail-consistent: a verify failure is treated like an invalid relk_ (REST 401 on a write/admin route, public read stays open; MCP → anonymous read, never gates the unauthenticated path). MCP audience via OAUTH_JWT_{ISSUER,AUDIENCE} wrangler vars (#1483). Generic MCP clients (MCPJam-class): DCR extra grant_types are intersected, DCR application_type is defaulted/coerced to native for loopback/private-use-scheme redirect clients (opencode/Cursor-class; live on Better Auth 1.7 — host-bearing private-use redirects are accepted upstream since 1.7.8, so the dist patch is gone; an explicit web is still rewritten here because oauth-provider never overrides a sent application_type), kitchen-sink scope= is downscoped, origin and /mcp are both valid RFC 8707 resources, and same-environment MCP aud is accepted on REST JWT gates so follows tools can forward the Bearer. See mcp-cimd-interop.md and remote-mode.md → Auth model.
• Rate-limit tiers: three rungs on CF native limiters — anonymous-IP 120 / account-userId 300 / machine-token 600 — selected by the shared @releases/lib/rate-limit-tiers; account-tier relu_ verify is CREDENTIAL_CACHE-backed; consumption is the rate-limit/decision Axiom event. These read-surface tier counters never move to KV/D1 (the separate /api/auth/* brute-force limiter is the one limiter that's KV-backed — #1728, see the auth note above). See remote-mode.md.
• Entity notices: a small curator-set note on an org/product/source, stored under metadata.notice ({ message, linkText?, coordinate?, href? }); set via the entity PATCH route, surfaced as a typed notice field + a web banner + an MCP pointer. See web.md → Entity notices.
• Owner-declared manifest (releases.json v2): declaration manifest at domain /.well-known/releases.json (org + optional products[] + top-level releases[] locators) and repo-root releases.json (product binding + repo releases[]); fill-if-empty reconciliation + cost-tiered source materialization; fail-closed, never clobbers curator/editorial fields. github + path + publish: "push" materializes a push-fed source only for a verified owner. See well-known-config.md.
• Self-serve listing lane (#1947 phase 2): anonymous POST /v1/listing/{validate,activate} — live manifest validation + instant-stub activation with a tracking_requested_at demand signal; invalid-manifest activation is a 400 ValidationError (no 422 in the taxonomy); rate limited per-IP (10/min) and per-domain on activate (3/min), both CF-native; routes live in a new publicWriteRoutes namespace bucket (integrity enforced by handler-level guards, not auth middleware); kill switch listing-self-serve-enabled. Curator surface is GET /v1/orgs?trackingRequested=1 (admin-gated inside the handler), not a separate /admin/* route. See well-known-config.md → Self-serve listing.
• Ownership claims (#1947 phase 3a): signed-in-only POST /v1/listing/claim{,/verify} + GET /v1/listing/claims proves domain control via a relv_ token, checked either as a .well-known/releases-verify.txt file or a _releases-challenge.<domain> DNS TXT record (either passes, fail-closed on any ambiguous response); reuses listing-self-serve-enabled; unlocks self-serve Tier-1 promotion (follow-up PR), not org edit rights. See well-known-config.md → Ownership claims.
• Publish tokens (#2373): a verified claim holder mints a relk_ token bound to ONE source (api_tokens.source_id, scope publish, off the ladder) via session-only /v1/me/publish-tokens (cookie + exact web origin, or a one-shot device-approval session as Bearer, #2388 — never a key or JWT; web UI on /account/webhooks, #2387); it passes auth only on POST to that source's two releases/batch routes (middleware path allowlist + handler source check), and the claim/source is re-checked every request, fail closed. Ending a claim (DELETE /v1/listing/claims/:id for the owner, DELETE /v1/orgs/:slug/claims/:id for admins, kept as revoked, #2389) stops its tokens on the next request. See remote-mode.md → Publish tokens.
• Mobile-app discovery (#1907): separate from the manifest — discovers a domain's undeclared native apps from /.well-known/apple-app-site-association + /.well-known/assetlinks.json. iOS bundle IDs resolve via iTunes bundleId= lookup and land as paused, hidden appstore candidates (curator unpauses); Android package names become an internal org.metadata.discoveredApps hint (no playstore type). Own 06:30 UTC cron + POST /v1/orgs/:slug/discover-apps; gated by well-known-materialization-enabled. Requires hidden-source containment (public read/search/embed paths filter is_hidden=0 via sources_visible). See well-known-config.md → Mobile-app discovery.
• User follows + feed + webhooks: signed-in users follow orgs/products (user_follows); /v1/me/follows + /v1/me/feed (org follow = its products too). Self-serve /v1/me/webhooks (org-scoped max 10 + one scope: "follows" URL) fans out real-time release.created using the follow graph; account UI on /account/notifications, CLI releases webhook …, API docs at /docs/api/webhooks. Gate: Better Auth session OR Bearer user principal (relu_ / OAuth JWT) — same as follows, NOT relk_/root/anonymous. MCP covers both surfaces via list_webhooks / manage_webhook (#1678, #2326). Workspace-owned equivalent: /v1/workspaces/:workspaceId/webhooks (org-scoped only, owner/admin manage, #2324). See routing.md, web.md, mcp.md, docs/webhooks.md.
• Semantic alerts (#2304): signed-in /v1/me/semantic-alerts CRUD (max 5, threshold default 0.80, follows-only) behind semantic-alerts-enabled (default off). When on, release.created scores enabled alerts with one JEV noul batch per release and sends webhook and/or email. Admin preview inserts synthetic releases through batch publish (POST /v1/admin/semantic-alerts/preview). Match quality is GET /v1/admin/semantic-alerts/summary (AE blob4 = semantic-alert); JEV spend stays on ai_usage lane semantic-alert-match. Query text stays out of Analytics Engine. See semantic-alerts.md.
• Stripe customer registration (@better-auth/stripe, billing groundwork — customer management only, no subscriptions yet): createCustomerOnSignUp creates a Stripe Customer per user, linked via user.stripeCustomerId. GATED on BOTH STRIPE_SECRET_KEY + STRIPE_WEBHOOK_SECRET resolving (buildStripePlugin, same inert-until-provisioned seam as dash/sentinel/social) — secrets bound via Secrets Store (prod bound; staging deliberately unbound until its sandbox Stripe secrets exist — the gate keeps it inert). Worker-compat: Stripe client uses Stripe.createFetchHttpClient(). Webhook served at /api/auth/stripe/webhook (existing /api/auth/* catch-all).
• Passkeys (@better-auth/passkey, WebAuthn/FIDO2): always-on (no flag, like magic link) — adds the passkey table and signIn.passkey / passkey.{add,list,update,delete} endpoints. rpID/origin are pinned to the WEB origin via derivePasskeyRp(env) (from WEB_BASE_URL), NOT the plugin's baseURL-derived default — the ceremony runs in the browser on releases.sh, not the API worker on api.releases.sh, so the default would fail every origin/rpID check. The challenge cookie rides the .releases.sh cross-subdomain cookie domain. Web: passkeyClient() in auth-client.ts, a manage panel on /account (passkeys-panel.tsx), and a passkey button + conditional-UI autofill on the sign-in form. The auth UI itself is no longer behind NEXT_PUBLIC_AUTH_UI_ENABLED (retired) — it follows AUTH_CONFIGURED (= NEXT_PUBLIC_BETTER_AUTH_URL set). See remote-mode.md → Auth model.
• Account settings — email change + social connections: /account also surfaces an email-panel.tsx (Better Auth changeEmail, server-enabled via user.changeEmail — a confirmation link goes to the user's CURRENT address, updateEmailWithoutVerification left off) and a social-connections-panel.tsx (listAccounts / linkSocial / unlinkAccount; lists/links/unlinks the providers in NEXT_PUBLIC_AUTH_SOCIAL_PROVIDERS, self-renders nothing when none are configured, and relies on Better Auth refusing to unlink the last sign-in method). Provider chrome (SOCIAL_PROVIDERS, PROVIDER_META) is shared with the sign-in form via apps/web/src/lib/social-providers.tsx.
• Workspaces (Better Auth org plugin): user-tenancy "Workspaces" — every user gets a personal one (lazy session.create.before provisioning that backfills existing users; deterministic ws-<userId> slug for race-dedupe), DISTINCT from the registry organizations (indexed vendors). auth*-prefixed tables (organization/member/invitation) in schema-auth.ts; always-on, no flag; minimal /account/workspaces UI. Inert org-keyed Stripe subscription seam (plans: [], referenceId = workspace id, authorizeReference owner/admin gate). member.role ≠ user.role. See workspaces.md.
• uploads.sh OAuth client (workspace connect): Releases Index is a public PKCE client of uploads.sh (releases-sh, files:read + offline_access). Owner/admin connect and disconnect from /account/integrations; tokens are encrypted at rest per workspace. Connect plumbing only — no screenshot proxy yet. Redirect URIs, env, and API: uploads-oauth.md.

Further reading

Deep dives live in docs/architecture/. A reader's guide with task-based entry points is at docs/README.md.

• local-development.md — portless dev URLs, worktree bootstrap/teardown, local D1 schema parity, auth rate limiting and the local signing secret.
• staging.md — the staging read-surface: hosts, per-environment managed agents, disabled crons, access gate, deploy and prod-sync commands.
• deploy-coupling.md — account-scoped wrangler bindings and open-core boundary for forks/self-hosters.
• remote-mode.md — D1, auth model (scoped API tokens), rate limiting, migrations, sessions, cron polling + retier, workflows-based ingest, discovery guardrails.
• storage-portability.md — SQLite/D1 seams, createDb, entity-ID invariant (single typed nanoid), FTS ownership, backend capability map, and what an optional future Postgres backend would cost (aspirational, not in progress).
• ingest.md — ingest pipeline: source-type adapters, dedup + D1 batching, smart-fetch backoff, URL exclusion / suppression, and the ingest-time AI passes (summarization, marketing classifier, feed enrichment).
• emails.md — transactional email: the shared shell, the three lanes, the email-client constraints it encodes, Gmail annotations, and the admin preview catalog.
• logging.md — per-runtime logging: logEvent() for workers vs. the fs-backed logger for CLI/neutral packages, payload conventions, severity, Error unwrapping.
• routing.md — REST route surface: naming buckets, org-scoped routes + dual-registration, the /v1/lookups resolver family + on-demand GitHub materialization, org catalog, entity resolution, pagination shape, public API surface + coverage gate.
• errors.md — standardized error envelope: taxonomy source of truth, the three-layer package split (core/api-types/lib), producers (respondError) + consumers, how to add a code.
• idempotency.md — optional replay-safe POST contract, encryption secret provisioning, supported routes, cleanup, and failure boundary.
• taxonomy.md — classification axes: source kind, products, release type, tags, categories, collections, and how they differ.
• semantic-search.md — Vectorize indexes, hybrid RRF, query cache, related-entity rails.
• mcp.md — remote MCP server, scope enforcement, WebMCP parity, MCP Registry listing.
• mcp-cimd-interop.md — generic MCP client OAuth (DCR grant_types, kitchen-sink scope, origin vs /mcp resource, forwarded Bearer).
• agents.md — how local Claude Code agents onboard/maintain orgs: skills, local workflows, evals, and the onboarding API surface.
• coverage.md — release coverage + ingest-time grouping, cron observability.
• web.md — changelog range/slicing API, GitHub CHANGELOG ingestion, Open Graph images, org overviews, category overlay, collections, media pipeline, org avatars.
• events.md — release event bus: ReleaseHub Durable Object, GET /v1/releases/stream WebSocket, fire-and-forget publish from batch + cron ingest.
• cli-distribution.md — OSS repo, npm, Homebrew tap.
• ai-gateway.md — optional Cloudflare AI Gateway passthrough for Anthropic SDK calls; covers direct worker calls, leaves Voyage embeddings + managed-agent internal loops on the direct path.
• classification-analytics.md — marketing-classifier decisions in Analytics Engine: positional schema, sampling, retention, admin summary/recent routes, and the Status Classifications tab.
• extract.md — two-tier extraction path: one-shot inline for small bodies, multi-round tool-use loop for large ones, hard fallback to one-shot on any failure. Feature-gated behind EXTRACT_TOOLLOOP_ENABLED.
• local-ingest.md — local Claude Code onboarding path: the local-ingest skill has the agent extract releases itself and write via /batch (no remote MA, no extraction billing), gated by a mandatory robots.txt/Content-Signal opt-out preflight.
• feature-flags.md — Cloudflare Flagship Tier-1 boolean flags: registry, evaluation order, per-flag reference, dashboard setup.
• content-pipelines.md — map of every routine/scheduled AI-content pipeline (overview regen, batch summarize, collection summaries, digests, ingest-time passes): schedule, gate, model lane, manual trigger.
• firecrawl-monitoring.md — external Firecrawl fetch backend for challenge-blocked scrape sources: monitor lifecycle, the hunkless monitor.page diff wire format, the diff-delta vs. full re-scrape ingest paths, cost gate, poll-fetch exclusion, and staleness resilience.
• maintenance-workspace.md — per-user ~/.releases/work/ (tasks / runs / reports) convention for agent-driven admin maintenance; durable, cost-aware trail for the seeding/maintaining/managing/overview skills, reachable across the monorepo and CLI checkouts.
• well-known-config.md — owner-declared releases.json v2 manifest: host-scoped authority (domain = org + products + release locators, repo = product binding), locator-first matching, cost-tiered source materialization, reconciliation precedence, the sync route + daily sweep.
• workspaces.md — Better Auth organization module: personal-workspace provisioning (lazy, on session create), the schema/naming split from the registry organizations, role separation (member.role vs user.role), and the inert org-keyed Stripe billing seam.
• uploads-oauth.md — Releases Index as an uploads.sh OAuth client: workspace connect/disconnect, redirect URIs, encrypted tokens, env.

Environment

Do not edit .env directly. Required vars are documented in .env.example. App env vars use the RELEASES_ prefix (RELEASES_API_URL, RELEASES_API_KEY, RELEASES_DATA_DIR, …).

Staging

Staging is a read-surface for UI/API iteration plus an agent-iteration sandbox, not a full replica — no crons, no webhooks, no Vectorize, access-key gated. Hosts, deploy commands, and the prod-sync script: staging.md.

Legacy naming

The project was originally called "Released"; the rename to "Releases" leaves the Cloudflare resource names deliberately unchanged:

• Cloudflare resources keep the released- prefix: D1 database released-db, and R2 buckets released-media (permanent, public media) and released-raw (ephemeral raw-page snapshots for backfill — content-hash keyed, 90-day lifecycle, public domain raw.releases.sh; see firecrawl-monitoring.md). The first two predate the rename, so renaming them is a live migration, not a text change; new resources (released-raw) keep the prefix for consistency.

Everything else — env vars (RELEASES_*), copy, prompts, display names, webhook headers (X-Releases-*), package/workspace names, the ~/.releases data dir, localStorage keys — uses the new name.