web-access — robust web page fetcher

Built-in WebFetch / WebSearch fail on a meaningful fraction of real-world URLs: Cloudflare challenges, bot walls, JS-rendered SPAs, sites with geo / TLS fingerprinting, paywalls, login gates. This skill provides a layered fallback chain so Claude can still retrieve usable content.

When to trigger

Use this skill when WebFetch returns any of:

• 403 / 429 / 503 status
• empty body
• a "please enable JavaScript" / "checking your browser" stub
• a CAPTCHA or challenge screen
• obviously-truncated content (far shorter than the page should be)
• unrendered SPA HTML (React / Vue / Next.js / etc.)

Or when the user explicitly says WebFetch didn't work, or asks to "properly" / "actually" read a URL.

Do NOT use this skill when:

• WebFetch already returned the content — it is faster and cheaper.
• The URL is a PDF — hand it to a dedicated PDF-extraction tool (pdftotext, pdfplumber, or an LLM-based extractor). scripts/fetch.sh detects PDFs and exits 2 on purpose.
• The task is performance testing, Core Web Vitals, network tracing, or accessibility audits — use the dedicated chrome-devtools skill.
• The task is academic paper search — use a paper-lookup skill.

Coordination with other skills (important)

scripts/fetch.sh refuses PDF URLs on purpose and exits with code 2 so that routing to a PDF extractor is explicit, not implicit. Detection uses (a) a case-insensitive URL-pattern match, then (b) a 6-second HEAD probe for Content-Type: application/pdf. These cover the common cases; if both miss (e.g. a server that blocks HEAD and serves a PDF under an HTML-looking path), Layer 1 (Jina Reader) may still extract PDF content before you notice. When you suspect a remote PDF and want certainty, run curl -sI <url> manually first, or download the file and extract it directly.

Primary entry point — scripts/fetch.sh

One command handles 90% of cases. It wraps the full fallback chain.

bash ~/.claude/skills/web-access/scripts/fetch.sh "https://target.site/page"

Exit codes:

Stdout: cleaned page content — markdown when Layer 1 (Jina) succeeds, else HTML passed through pandoc or html2text.py. Stderr: per-layer progress logs ([web-access] Layer 1: Jina Reader etc.) so you can see which layer produced the output.

Typical invocation (captures output, inspects exit):

OUT=$(bash ~/.claude/skills/web-access/scripts/fetch.sh "$URL")
rc=$?
case $rc in
  0) echo "$OUT" | head -c 20000 ;;             # feed to Claude
  2) echo "PDF — use a dedicated extractor instead" ;;
  3) echo "Escalate to chrome-devtools MCP" ;;
esac

Parallel fanout — scripts/fetch-parallel.sh

For research fanout (N independent URLs), use the parallel wrapper instead of looping fetch.sh serially:

bash ~/.claude/skills/web-access/scripts/fetch-parallel.sh -P 4 \
  "https://site-a/page" \
  "https://site-b/page" \
  "https://site-c/page"

# Or pipe URLs in from stdin (one per line):
cat urls.txt | bash ~/.claude/skills/web-access/scripts/fetch-parallel.sh -P 4 -

Options:

• -P N — max concurrent fetches (default 4). Keep modest: Jina Reader is rate-limited, curl-impersonate is CPU-heavy.
• -o DIR — output directory (defaults to a fresh mktemp dir).

Output (stdout, TSV): idx<TAB>exit_code<TAB>url<TAB>content_file, sorted by input order. Per-URL exit codes are preserved verbatim from fetch.sh — route rc=2 rows to a PDF extractor, rc=3 rows to chrome- devtools MCP. The wrapper itself exits 0 if every URL was dispatched.

This is cheaper than spawning parallel subagents for pure URL fanout: shared-nothing processes that only touch the filesystem.

Per-domain hints — references/site-patterns/

Every time you discover that a specific site needs a particular header / CSS selector / layer order to fetch cleanly, capture it in references/site-patterns/<host>.conf so future fetches benefit automatically.

Lookup order: exact host, then host with leading www. stripped. Filenames are sanitized (^[a-z0-9.-]+$); files are shell-sourced, so keep them to simple KEY=value assignments.

Recognized variables:

See references/site-patterns/README.md for the full spec and _template.conf for a copy-ready starting point.

When to add a pattern: after a tricky fetch succeeds, ask what made it work. If the answer is "a specific selector / referer / layer", capture it. If "nothing special", do not — avoid clutter.

The fallback chain

Layer 1 — Jina Reader (primary)

Jina Reader prefixes the target URL with https://r.jina.ai/ and runs it through a real headless browser server-side, returning clean Markdown. Bypasses most Cloudflare / bot walls and handles JS rendering. No API key required for basic use.

curl -sL --max-time 45 \
  -H "X-Return-Format: markdown" \
  "https://r.jina.ai/https://target.site/page"

Useful request headers (optional, add only when needed):

Search alternative to WebSearch — same idea, search endpoint:

curl -sL --max-time 45 "https://s.jina.ai/your+search+query"

Returns the top results as a single clean markdown document.

Rate limits: the free anonymous tier is generous but rate-limited; exact quotas are documented at jina.ai/reader. Treat HTTP 429 as "back off 10–30 s, then retry once; if still limited, switch layers".

Layer 2 — Stealth curl

For sites that only check HTTP headers (no real JS / TLS fingerprinting). Fast, cheap, sometimes works when Jina is rate-limited.

scripts/fetch.sh auto-detects curl-impersonate (brew install curl-impersonate) and uses it when present, which also defeats TLS / JA3 fingerprinting used by Cloudflare. If not installed, falls back to plain curl with a full Chrome 131 header set.

For non-English sites you can add to the header set:

-H "Referer: https://www.google.com/"
-H "Accept-Language: en-US,en;q=0.9,<your-locale>"

Layer 3 — Wayback Machine

Last-resort archive lookup via the public Wayback availability API. Particularly useful for paywalled articles, deleted posts, rate-limited pages. The script strips the Wayback toolbar by rewriting /web/TS/ → /web/TSid_/.

curl -sL "https://archive.org/wayback/available?url=$URL" \
  | jq -r '.archived_snapshots.closest.url // empty'

Layer 4 — Chrome DevTools MCP (heaviest fallback)

When all three scripted layers fail (exit code 3), or when the page clearly needs auth cookies / multi-step interaction, use the chrome-devtools MCP tools directly.

Workflow:

1. mcp__chrome-devtools__list_pages — check if a browser is already running.
2. mcp__chrome-devtools__new_page with the target URL if not.
3. mcp__chrome-devtools__wait_for to wait for a content selector.
4. mcp__chrome-devtools__take_snapshot — returns a structured DOM snapshot (token-efficient).
5. Or mcp__chrome-devtools__evaluate_script with:

(() => {
  const main = document.querySelector('article, main, [role="main"]') || document.body;
  return main.innerText.slice(0, 50000);
})()

Use Layer 4 when:

• Page requires a logged-in session (leverages your real browser profile).
• Multi-step flow required (click "I agree", then fetch content).
• Heavy client-side rendering that Jina Reader mis-handles.
• Need to interact (form fill, scroll to trigger lazy-load).

Do NOT use Layer 4 when Layer 1 would work — a real browser is ~10× slower and consumes a browser process.

Scope overlap with the chrome-devtools skill: the dedicated chrome-devtools skill is for performance / audit workflows (Core Web Vitals, Lighthouse, network tracing). This skill only borrows the same MCP tools for content retrieval. If the user's goal is measurement, use the other skill.

Output sanity checks (already done by fetch.sh)

The script applies these automatically, but if you invoke a layer directly by hand, replicate:

1. Length check: discard results < 150 chars (likely an error stub).
2. Challenge markers: grep case-insensitively for cf-challenge, just a moment, checking your browser, please enable javascript, captcha, attention required, access denied, 403 forbidden, error 1020, cloudflare ray id. Presence → fall through to the next layer.
3. Token budget: truncate to ~20 KB before feeding back to Claude (head -c 20000).

Common traps

• Scheme is mandatory. Jina Reader needs https://... in the suffix — do not drop it.
• Quote the URL. Always wrap URLs with ? or & in double-quotes so the shell does not split them.
• Do not percent-encode the URL before prefixing https://r.jina.ai/ — Jina accepts raw URLs as path segment.
• Do not log in on the user's behalf. For auth-gated pages, ask first; at most reuse a pre-existing browser profile via Layer 4.
• Do not retry in a tight loop. HTTP 429 means back off. One retry is plenty; switch layers after that.
• Legal / ToS. This skill is for reading content the user is entitled to access but is blocked by over-aggressive bot walls. Do not use for credential stuffing, mass scraping, or clear ToS violations.

Quick reference

# The 95% case — one command, exit code tells you what happened
bash ~/.claude/skills/web-access/scripts/fetch.sh "$URL"

# Manual Jina Reader
curl -sL --max-time 45 -H "X-Return-Format: markdown" "https://r.jina.ai/$URL"

# Manual Jina Search (WebSearch alternative)
curl -sL --max-time 45 "https://s.jina.ai/$QUERY"

# Manual Wayback latest snapshot
curl -sL "https://archive.org/wayback/available?url=$URL" \
  | jq -r '.archived_snapshots.closest.url // empty'

# DuckDuckGo HTML interface (no JS, extra search fallback)
curl -sL --max-time 20 \
  -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 Chrome/131" \
  "https://html.duckduckgo.com/html/?q=$QUERY"