unblink
CLAUDE.md8 starsChanged 5 months agoMITView on GitHub
unblink is a pure-Go (no cgo, no Chromium) MCP server that exposes the web to AI agents. It fetches a page, parses HTML5, optionally runs the page's JavaScript, strips visual-only junk, and returns clean, token-budgeted Markdown over the Model Context Protocol (stdio). The job is semantic reduction, not rendering. The single binary is cmd/unblink. The authoritative design doc is docs/architecture.md — read it before any non-trivial change. The Makefile is the canonical task runner. CI (.github/workflows/ci.yml) runs the gofmt check,…
- Installs packages
The file
We split this CLAUDE.md into the rules it gives an agent
4 rules. About 0 are common, 1 is distinctive, and 3 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈1
Distinctive · estimate
This author's own choices. Worth a look.
3
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 39
Run a single test (not in the Makefile — standard Go):
- Very specific: line 78
Hybrid statefulness: Every page tool takes an optional session.
- Very specific: line 168
stdout is reserved for MCP JSON-RPC: All logging goes to stderr via slog (--log-level).
- Very specific: line 186
SSRF guard: every page fetch — the primary, per-session, and one-shot clients *and* page-JS subrequests — is blocked from reaching …
≈ “Common” is an estimate: at least 1 in 5 of the 27,641 CLAUDE.md files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
CLAUDE.md
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
What this is
unblink is a pure-Go (no cgo, no Chromium) MCP server that exposes the web to AI agents. It fetches a page, parses HTML5, optionally runs the page's JavaScript, strips visual-only junk, and returns clean, token-budgeted Markdown over the Model Context Protocol (stdio). The job is semantic reduction, not rendering.
The single binary is cmd/unblink. The authoritative design doc is
docs/architecture.md — read it before any non-trivial change.
Commands
The Makefile is the canonical task runner. CI (.github/workflows/ci.yml)
runs the gofmt check, make vet, golangci-lint (.golangci.yml — default
linters, errcheck relaxed for idiomatic Close/test handlers, third_party/
exempt), then make test eval on every push/PR. make lint runs the same
lint locally (needs the golangci-lint binary).
make build # go build (version stamped from the nearest v* tag)
make test # go test ./... (includes every fuzz target's seed corpus)
make eval # offline in-process MCP eval gate (build tag `eval`; scorecard on stderr)
make wpt # offline web-platform-tests conformance report (build tag `wpt`; on-demand, not a gate)
make wpt-sync # refresh the vendored WPT subset in testdata/wpt/ (needs network; local-only)
make fuzz # coverage-guided fuzzing of the untrusted-input parsers (FUZZTIME=15s each)
make bench # perf benchmarks over the hot-path packages (narrow: BENCH=regexp, repeat: BENCHCOUNT=N)
make membench # unblink vs. headless Chromium footprint/latency (scripts/membench, separate module)
make crossbench # membench across every tool in docs/comparison.md incl. token cost (TOOLS=, ARGS=)
make vet # go vet ./...
make fmt # gofmt -w .
make tidy # go mod tidy
make run # build, then serve MCP over stdioRun a single test (not in the Makefile — standard Go):
GOTOOLCHAIN=auto go test ./internal/js -run TestName -v
GOTOOLCHAIN=auto go test ./internal/dom -run TestExtract/subtest -vProving a performance change: make bench > /tmp/base.txt on the baseline
commit, apply the change, make bench > /tmp/new.txt, then
go run golang.org/x/perf/cmd/benchstat@latest /tmp/base.txt /tmp/new.txt.
Include the benchstat table in the PR. JS renders settle on two tiers (ADR
0004): provably-idle pages (no in-flight network, no live timers) close in
~1-3ms; anything armed pays the ~60ms quiet window after its last activity.
Go-version gotcha: the MCP SDK needs Go ≥ 1.25. The Makefile exports
GOTOOLCHAIN=auto so go downloads the toolchain pinned in go.mod
automatically. If you invoke go directly instead of via make, prefix it with
GOTOOLCHAIN=auto.
Drive the server by hand (MCP over stdio) — see the JSON-RPC snippet in
README.md. JavaScript rendering is on by default (reads render unless the
caller passes render=false); ./bin/unblink --disable-js turns it off for the
static-only path. --version prints the version.
Architecture
One page.Page (internal/page/page.go) flows through the pipeline and is
enriched in place — stages never re-fetch or re-parse:
fetch ──▶ dom.Parse ──▶ [js render] ──▶ dom.Extract ──▶ reduce ──▶ emitThree load-bearing decisions hold the design together:
- The
*html.Nodetree (Page.Doc) is the single source of truth. Every stage reads or mutates that one tree; goquery/cascadia are views over it, not a second model. When JS runs it mutates the same tree, so reduce/emit never need to know whether JS ran (Page.Renderedrecords that it did). - Hybrid statefulness. Every page tool takes an optional
session. With one: cookies + history + current page persist, andinteractkeeps a live JS runtime alive for the page (internal/session). Without: a throwaway one-shot fetch. Keepread(url)trivial while supportingclick/submit_form/interact. - goja is quarantined behind the
browser.Rendererinterface, so the JS engine stays optional and swappable. A*goja.Runtimeis single-goroutine and never shared: a one-shotRenderbuilds and discards one per call (bounded by a semaphore + prewarm pool); a persistentjs.Context(Open/Dispatch/Snapshot/Close, for true sessions) keeps one alive on its own event-loop goroutine — all access marshalled viaRunOnLoop, reads via on-loop snapshots. It is the only single-threaded island.
Package boundaries (enforced dependency direction)
page → capability packages → browser → mcpserver. Respect these:
internal/page— core data model (Page,Article,Link,Form…). Pure types.internal/fetch— HTTP-client-as-browser: cookies, redirects, brotli/gzip/deflate, charset→UTF-8, opt-in per-host rate limit (--rate-limit, default off), retries, optional utls TLS mimic.internal/ratelimit— process-global, per-host token-bucket limiter behind--rate-limit(off by default);fetchclients share it.internal/dom— the only importer ofx/net/html/cascadia: parse; extraction (regions/landmarks, interactive ARIA state, stable content-hash ids, heading outline); caller-directed schema extraction (dom.Records); repeating-collection discovery (dom.detectCollections); find.internal/content— converts non-HTML bodies to Markdown: JSON and plain text inline, plus thefeed/image/pdfsubpackages (RSS/Atom/JSON feeds, image manifests, PDF text viathird_party/pdf).internal/reduce— readability extraction + bluemonday sanitize (ArticlevsFull);Fullalso suppresses repeated link-dense blocks (desktop nav + mobile-drawer twin) viadom.StripDuplicateBlocks.internal/emit— serialize reduced content to Markdown + heading outline.internal/tokens— token estimation + Markdown cursor pagination.internal/session— per-session cookie jar + navigation history + a livejs.LiveContext(persistent JS runtime) bound to the current page; navigation/eviction/close tears it down (manageronEvicthook).internal/js— quarantined goja + eventloop DOM bridge (scripts, ESM, fetch/XHR, events, cookies). One-shotRenderand persistentContext/LiveContext(true sessions). Opt-in. Defines its ownTransport/CookieJarinterfaces and never importsinternal/fetch; the browser supplies adapters (internal/browser/transport.go).internal/robots— exposure-grade robots.txt (REP) parser: groups,*, Allow/Disallow (*/$), Crawl-delay, Sitemaps. Pure stdlib, no enforcement — unblink surfaces rules as context, never gates a fetch.internal/sitemap— pure-stdlib sitemaps.org XML decoder (<urlset>/<sitemapindex>, gzip-aware) behind themaptool. No fetch, nox/net/html.internal/search— optional web search behind aProviderinterface (SearXNG + Brave adapters); off unless--search-provideris set, injected intobrowser.internal/webext— models MV2/MV3 WebExtensions (manifest, match patterns, a tokenized no-ReDoS AdblockurlFiltermatcher, dir/archive loaders with traversal + zip-bomb guards) so unblink can runtime-load one (e.g. uBlock Origin Lite). Pure Go; imports neitherjsnorbrowser— the direction isbrowser → js → webext. The JS-side surface lives ininternal/js/ext*.go.internal/browser— the orchestrator that wires the pipeline + sessions. The only packagemcpservercalls. Owns the host-scoped robots.txt/llms.txt cache (sitecache.go,site.go), themap/searchdiscovery surface, and the engine-lifetimeExtensionHost(shared read-only across renders/sessions).internal/mcpserver— thin MCP adapter: tool registration + handlers + transport.
No MCP types appear below internal/mcpserver, and no browser logic lives in
it — that keeps the engine transport-agnostic and the SDK swappable.
Security posture
unblink executes untrusted page JavaScript, so it treats the page's own code as
hostile and applies the browser's security model as defaults over it. The
guiding principle mirrors a browser + its devtools: gate the page, not the
operator — a blocked cross-origin request is still sent and logged (visible to
the agent via the requests tool); only the page-JS read is denied. The layers
(each detailed in an ADR; escape hatches are opt-in-to-less-secure, default off):
- SSRF guard (default on): no fetch — primary, session, one-shot, or page-JS
subrequest — reaches a private/loopback/metadata IP (checked on the resolved
IP). Hatches
--allow-private,--js-allow-private. (Detailed gotcha below.) - Same-Origin Policy (ADR 0015): SOP is the default. Cross-origin network
reads go through CORS; cookies are origin/domain-scoped (public-suffix jar,
HttpOnly hidden from
document.cookie); Web Storage is origin-partitioned per session (internal/session). The cross-document half (iframes/contentWindow,window.open/opener/frames,postMessage,document.domain,window.name) is moot by architecture — single-document, one runtime per render, no reachable foreign document — a deliberate non-goal. - CORS (ADR 0011, default on): cross-origin
fetch/XHR over page JS obeysAccess-Control-*+ preflight; non-credentialed cross-origin drops cookies;no-corsyields an opaque response. A same-origin request that redirects cross-origin is re-validated against the final response's CORS (so a redirect can't launder a cross-origin read).internal/js/cors.go(doFetch); hatch--js-allow-cross-origin. - CSP (ADR 0014, default on): the document's
Content-Security-Policy(header +<meta>) is enforced over page JS —script-srcnonce/hash/host +strict-dynamic,connect-src,unsafe-eval(viaEvalErrorshims); report-only surfaces without blocking. Nonces are hidden — captured before page JS runs and blanked in the DOM (hideNonces), so page code can't scrape and reuse one.internal/js/csp.go; hatch--no-csp. - SRI (ADR 0012, default on):
integrity-pinned scripts/modules are hashed over the raw pre-transcode bytes and blocked on mismatch.internal/js/sri.go; hatch--no-sri. - Injected-credential origin scoping (Phase 10): bearer/basic/custom headers go only to their configured origin, stripped on cross-origin redirect. (Gotcha below.)
- Truthful request/isolation signals (ADR 0013): per-context
Sec-Fetch-*on every request; truthfulwindow.crossOriginIsolated/isSecureContextfrom COOP/COEP (fidelity, not enforcement). - Resource bounds on untrusted JS: process heap watchdog +
SetMemoryLimit(ADR 0003,--js-memory-limit), per-render/dispatch download-bytes budget (ADR 0009,--js-max-bytes), wall-clock render budget + timer clamp, live-runtime LRU cap.
Shared implementation pattern for the JS-side policies: an Env field
(AllowCrossOrigin/DisableSRI/DisableCSP/ResponseHeaders) → a bridge field →
the enforcement hook, with violations recorded via recordError → render
diagnostics. Gotcha: recordError appends without a lock — only call it on the
loop goroutine (inside RunOnLoop or an on-loop native handler), never from an
off-loop fetch goroutine (this is why connect-src is checked in fetchPromise,
not doFetch).
Conventions & gotchas
- stdout is reserved for MCP JSON-RPC. All logging goes to stderr via
slog(--log-level). Never print to stdout outside the MCP transport. - Never return an unbounded blob.
readdefaults to article-mode extraction and paginates via an opaque cursor; whole-page Markdown can be huge. - reduce/emit run on a copy of the page so shared (cached or session) pages
are never mutated — see
Browser.Read. - New capability code goes in the relevant
internal/<pkg>; route everything to the MCP layer throughinternal/browser, not directly. - Tool surface (all routed through
internal/browser, registered ininternal/mcpserver): 18 tools. Beyondread/browse/links/forms/find/site/click/submit_form/controls/interact/data/session/map/search, the recent additions areextract(caller-directed CSS-schema extraction,dom.Records),browse'scollections(auto-proposedextractschemas,dom.detectCollections), and the inspection toolsrequests/console/cookies. Operators narrow the advertised set with--tools/--disable-tools(presetscore/read-only/full), and capability-unusable tools auto-hide (searchwithout a provider;interact/requests/consoleunder--disable-js). - SSRF guard: every page fetch — the primary, per-session, and one-shot
clients and page-JS subrequests — is blocked from reaching
private/loopback/metadata IPs (checked against the resolved IP). On by default
(
ssrfControlininternal/browser/transport.go, wired intobaseinbrowser.New+ the JS transports); escape hatches--allow-private(page fetches) and--js-allow-private(JS subrequests, also has a per-render budget). - Injected credentials are origin-scoped (Phase 10):
fetch'sWithBearer/WithBasicAuth/WithHeadersapply only to requests matchingWithCredentialScope's origin and are stripped on any cross-origin redirect (Client.checkRedirect) — a bearer/API-key never leaks to another host. Session auth lives insession.Config; secrets resolve viatoken_env/password_envininternal/mcpserver/auth.goand are never logged or echoed in session state. - Tests use external
_testpackages with fixtures intestdata/.internal/jscarries the densest suite;internal/browser/browser_test.gois the integration-level test of the whole pipeline. Every parser that eats untrusted bytes (dom, reduce, pdf, robots, token cursors) has a fuzz target —make fuzz(FUZZTIME per target); seed corpora run inmake test. docs/decisions/holds the ADRs (numberedNNNN-slug.md): 0001 keepsdslipak/pdfbut vendors it atthird_party/pdf(imported by its in-repo path…/unblink/third_party/pdf— never via areplacedirective, which would breakgo install <module>@version) with three loop-bounding patches after fuzzing found an infinite-loop DoS that recover+ctx couldn't contain — sync upstream manually, re-fuzz before adopting; 0002 is the dependency pinning policy (goja/goja_nodejs pseudo-version pins are deliberate — bumping goja is its own reviewed change); 0003 is the JS memory guard (process-level heap watchdog +debug.SetMemoryLimit, because goja has no per-runtime accounting); 0004 is the provable-idle settle invariant (every new async primitive must route through the timer audit /pendingbracket); 0005 is the composed Shadow DOM (flattened tree- cross-boundary events); 0006 is the broad Go-backed
crypto.subtle(reversing the earlier "leave undefined", since real pages call it on first paint); 0007 is the semantic-only structured representation (computed from the node tree, never a layout — spatial/bounding-box geometry is a permanent non-goal); 0008 is the stable content-hash element ids (a reference/cache key, not aninteracttarget — interact still uses CSS selectors); 0009 is the move to resource-based JS budgets — a per-render/per-dispatch byte budget (--js-max-bytes, default 64 MiB) replaces the fixed request-count cap (now an off-by-default backstop) and the removed 60s live-session rate window, plus concurrent<link rel=modulepreload>warming so code-split SPAs' serialimport()graph loads as cache hits (adds no async primitive, so ADR 0004 holds); 0010 is WebExtensions runtime loading (the operator supplies the extension; MIT/GPL separation, no vendored extension bytes — see the WebExtensions gotcha above); 0011 is CORS enforcement over untrusted page JS (default on, enforced ininternal/js/cors.godoFetch; the request is still sent + logged inrequests, only the page-JS read is gated — history preservation;--js-allow-cross-originopts out); 0012 is Subresource Integrity (default on,internal/js/sri.go; hashes the raw pre-transcode bytes — hence theRawfield onfetch.Result/js.Response;--no-sriopts out); 0013 is Fetch-Metadata + cross-origin isolation fidelity (truthful per-contextSec-Fetch-*on primary + every subrequest viainternal/js/secfetch.go, decoupled from--tls-mimic; truthfulwindow.crossOriginIsolated/isSecureContextfrom COOP/COEP ininternal/js/isolation.go, no COEP enforcement — the document response headers now reach the engine viaEnv.ResponseHeaders); 0014 is Content-Security-Policy enforcement over untrusted page JS (default on,internal/js/csp.go: header +<meta>parsing, script-src nonce/hash/host incl.strict-dynamic, connect-src on fetch/XHR,unsafe-evalviaEvalErrorshims; report-only surfaces without blocking; a resource must satisfy every enforced policy;--no-cspopts out); 0015 is the Same-Origin-Policy posture (SOP is the default: network reads via CORS, cookies, and now origin-partitioned Web Storage are isolated; the DOM/window/frame half is moot-by-architecture — single-document, one runtime per render — so it's a deliberate non-goal. The one behavior change was origin-keying localStorage/ sessionStorage ininternal/session); 0016 is web-platform-tests as a conformance measurement tool (thewpt/package +make wpt, build tagwpt, mirroringeval/: runs the in-scope WPT testharness.js corpus directly against the JS engine and buckets every subtest into pass / (a) in-scope-gap / (b) by-design / (c) goja-ceiling / (d) runner-limit so conformance = pass/(pass+a) is honest; a pinned vendored corpus undertestdata/wpt/; on-demand analysis, not a CI gate. The ranked roadmap it feeds isdocs/wpt-compatibility.md— headline findings: HTML tree-construction is already strong (html/syntax 0.944), the top content-fidelity gaps are the URL parser and TextDecoder/TextEncoder); 0017 is the[\s-…]regex-class rewrite (goja rejects a valid-ECMAScript-after a\s/\d/\wshorthand — the camelCase/slugify idiom — with "invalid character class range", aborting the whole script/module;fixClassRangeHyphenininternal/js/regexfix.goescapes the hyphen before compile at compileCached + classicProgram — value-preserving in every JS context, odd-backslash guard for the real[\\s-z]range; a targeted goja workaround, not a regex-compat layer); 0018 is the per-render module-instance registry (internal/js/moduleregistry.go) giving dynamicimport()browser module-map semantics — each resolved URL fetched + esbuild-CJS-transformed + evaluated once, its live namespace shared across importers, so a dep shared by twoimport()s (React across an Astro island's component + renderer chunks) isn't duplicated into two singleton copies (the "useMemoCache of null" failure); two-phase (off-loop instantiate/dedup, on-loop post-order evaluate), replaces bundleDynamicChunk, preserves the ADR 0004 settle bracket; scope = the dynamic-import path only (static<script type=module>still bundles per entry). Add a new ADR when a decision would otherwise live only in a PR description. (reference/andinternal/config/were empty scaffolding, deleted in Phase 20.)
- cross-boundary events); 0006 is the broad Go-backed
- WebExtensions (opt-in, ADR 0010):
--extension <dir|.xpi|.crx|.zip>/--extensions-dirruntime-load a real, unmodified extension — unblink ships no extension bytes, keeping GPL tooling (uBlock Origin is GPL-3) out of the MIT tree the way a browser loads a user add-on. The model lives ininternal/webext; the engine surface (thechrome/browserAPI, content scripts, cosmetic filtering as physical node removal, background worker + runtime messaging, static/dynamicdeclarativeNetRequest, and MV2webRequestblocking) ininternal/js/ext*.go. Extensions are rejected under--disable-jsand run in the same heap/byte/SSRF sandbox as page JS. uBlock Origin Lite (MV3/DNR) is verified working (18,249 host-evaluated rules, blocks real trackers, ~0.1s, no service worker); full uBO (MV2) is not viable in-process (compiling its filter lists in goja is too slow). - Framework rendering (flat-DOM model): with its JavaScript engine (on by
default;
--disable-jsopts out) the engine renders mainstream SPA frameworks (React/Vue/Preact/Svelte/Lit) — a real Node/Element prototype chain, MutationObserver, custom-element upgrade, and an encapsulating, composed Shadow DOM (Phase 23, ADR 0005): each shadow root is a detached subtree (page-JSquerySelectorrespects the boundary), and a compose pass flattens it — resolving<slot>distribution — into the light tree for extraction. Events cross the boundary correctly (composed path +targetretargeting +composedPath()); declarative Shadow DOM (<template shadowrootmode>) is flattened on the static no-JS path. Still permanent non-goals (no layout engine): real element layout/geometry and CSSOM (constant-stubbed to zeros/empty, never computed), canvas/WebGL, Workers/WebSocket/IndexedDB, and Shadow-DOM style scoping (:host/::slotted/::part) / slot reprojection / closed-mode privacy from extraction. The viewport environment is the one exception (Phase 21):innerWidth/screen/devicePixelRatioare a truthful constant 1280×720@1x andmatchMediaevaluates against it, so responsive code takes its real branch. Untrusted page JS is also bounded on heap (--js-memory-limit, ADR 0003), time, network bytes (--js-max-bytes, ADR 0009), and live-runtime count. Seedocs/architecture.md(Phases 8, 21, and 23).
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

