Web Scrape to Markdown

Fetch a URL and convert it to clean, LLM-ready Markdown. Supports multiple backends with automatic fallback.

Usage

/devkit:scrape https://example.com
/devkit:scrape https://example.com --json
/devkit:scrape https://example.com https://other.com
/devkit:scrape https://example.com --backend playwright

Arguments

• One or more URLs to scrape
• --json — wrap output as Markdown-in-JSON: { "url": "...", "title": "...", "markdown": "..." }
• --backend <name> — force a specific backend (see below). Default: auto-detect.

Backends (in priority order)

1. Jina Reader — prepend https://r.jina.ai/ to the URL. Returns Markdown by default. Use if JINA_API_KEY is set (higher rate limits) or anonymously (~20 RPM). Best for articles and docs.
2. Playwright — use if npx playwright --version succeeds (optional dep, install with npx playwright install chromium). Best for JS-heavy SPAs and sites that block headless scrapers. Free and local — no API keys.
3. Camoufox — patched Firefox with leak-fixed JA3/TLS fingerprints. Use if python -m camoufox version succeeds (install: pip install -U camoufox[geoip] && python -m camoufox fetch). Free and local. Use when Playwright returns a challenge page (Cloudflare "Just a moment...", DataDome, PerimeterX). Auto-triggered when Playwright response body matches challenge signatures (cf-chl-bypass, __cf_chl_, _Incapsula_Resource).
4. Scweet (X/Twitter only) — specialized scraper for x.com / twitter.com URLs. Use if SCWEET_COOKIES_PATH is set (points at a JSON cookie file exported from a logged-in browser). Auto-routed when host is x.com or twitter.com — other backends don't work reliably on X post-API lockdown.
5. Firecrawl — use if FIRECRAWL_API_KEY is set. Paid API. Last-resort anti-bot bypass when stealth browsers still get blocked.
6. WebFetch fallback — use Claude's built-in WebFetch tool. No API key needed, but returns raw content (less clean).

Execution

Single URL

For each URL, try the highest-priority available backend:

Jina Reader (default):

Use WebFetch to fetch: https://r.jina.ai/{url}
Set header: Accept: text/markdown

If --json flag is set, instead fetch with:
  Accept: application/json
This returns: { "url": "...", "title": "...", "content": "..." }
where "content" is the Markdown.

Playwright:

Validate the URL first (see Rules: http/https only, no private IPs, no @).

Check availability: npx playwright --version. If not installed, tell the user once per session: Playwright not installed. Install with: npx playwright install chromium, then fall through to the next backend. The winning backend is always reported in Step 3 (Error Handling), so the user sees which one served the result.

Never build scripts via inline -e "..." strings. Write the script to a file that reads the URL from process.argv[2], then invoke it:

DRIVER=$(mktemp -t devkit-scrape.XXXXXX.mjs)
trap 'rm -f "$DRIVER"' EXIT

cat > "$DRIVER" <<'EOF'
import { chromium } from 'playwright';

const url = process.argv[2];
if (!url) { console.error('usage: node devkit-scrape.mjs <url>'); process.exit(2); }

const browser = await chromium.launch();
try {
  const page = await browser.newPage();
  await page.goto(url, { waitUntil: 'domcontentloaded', timeout: 30000 });
  const title = await page.title();
  const html = await page.content();
  process.stdout.write(JSON.stringify({ title, html }));
} catch (err) {
  console.error('playwright failed:', err.message);
  process.exit(1);
} finally {
  try { await browser.close(); } catch (e) { console.error('playwright close error (browser may have crashed mid-scrape):', e.message); }
}
EOF

node "$DRIVER" "$URL"

Pass the URL as a shell variable (URL=https://example.com), never inline into the command string. The .mjs reads it from process.argv[2] so quotes, backticks, and $() in the URL can't escape.

Then convert HTML → Markdown. Prefer a local converter if available (pandoc, turndown). If none available, strip script/style/nav/footer tags and extract text from article/main/body.

Camoufox (when Playwright returns a challenge page or --backend camoufox):

Check availability: python -m camoufox version. If missing, print once per session: Camoufox not installed. Install with: pip install -U camoufox[geoip] && python -m camoufox fetch, then fall through.

Write the driver to a file (never inline -c "..." — same reasoning as Playwright). Pass the URL as an env var, never interpolated into the command string. Use mktemp rather than a hardcoded /tmp/ path (portability + predictable-path attack surface):

DRIVER=$(mktemp -t devkit-camoufox.XXXXXX.py)
trap 'rm -f "$DRIVER"' EXIT

cat > "$DRIVER" <<'EOF'
import os, json, sys, traceback
from camoufox.sync_api import Camoufox

url = os.environ["DEVKIT_SCRAPE_URL"]
stage = "init"
try:
    with Camoufox(headless=True, humanize=True) as browser:
        page = browser.new_page()
        stage = "goto"
        page.goto(url, wait_until="domcontentloaded", timeout=30000)
        stage = "title"
        title = page.title()
        stage = "content"
        html = page.content()
        print(json.dumps({"title": title, "html": html}))
except Exception as err:
    print(json.dumps({"error": str(err), "stage": stage, "traceback": traceback.format_exc()[-2000:]}))
    sys.exit(1)
EOF

DEVKIT_SCRAPE_URL="$URL" python "$DRIVER"

Then convert HTML → Markdown (same converter as Playwright). If the JSON has an error field, treat the backend as failed and fall through.

Challenge-page detection (triggers auto-fallback Playwright → Camoufox): treat the Playwright result as blocked and retry with Camoufox when ANY of the following holds:

• Positive anti-bot signature present: cf-chl-bypass, __cf_chl_, <title>Just a moment...</title>, _Incapsula_Resource, dd_cookie_test_, datadome, px-captcha, perimeterx (case-insensitive match against the HTML)
• Body text under 200 chars AND no <article>/<main>/<section>/<h1> tag (skeleton/challenge shell, not a real small page)

Narrower than a size heuristic alone — valid small pages (404s, hand-rolled minimal HTML, API docs) keep their original backend instead of burning a Camoufox cycle.

Scweet (X/Twitter — host is x.com or twitter.com):

Check availability: python -c "import Scweet". If missing: pip install Scweet. Requires SCWEET_COOKIES_PATH pointing at a JSON cookies file exported from a logged-in browser session (devkit convention: export with a cookie extension, store outside the repo, set SCWEET_COOKIES_PATH=~/.config/devkit/x-cookies.json). Do NOT hardcode auth tokens in env.

Scweet's Python API varies by version — the scrape_tweets / scrape_profile / scrape_tweet entry points differ across 2.x releases. Inspect the installed version with python -c "import Scweet; print(Scweet.__version__)" before wiring. Reference template (adapt to your installed Scweet version):

DRIVER=$(mktemp -t devkit-scweet.XXXXXX.py)
trap 'rm -f "$DRIVER"' EXIT

cat > "$DRIVER" <<'EOF'
import os, json, sys, traceback, re
from urllib.parse import urlparse

url = os.environ["DEVKIT_SCRAPE_URL"]
cookies_path = os.environ["SCWEET_COOKIES_PATH"]
stage = "init"
try:
    # Scweet 2.x: choose entry point by URL shape
    path = urlparse(url).path
    if re.match(r"^/[^/]+/status/\d+", path):
        stage = "import_tweet"
        from Scweet.user import get_tweet
        data = get_tweet(tweet_url=url, cookies_path=cookies_path, headless=True)
    elif re.match(r"^/[^/]+/?$", path):
        stage = "import_profile"
        from Scweet.user import get_user_information
        handle = path.strip("/")
        data = get_user_information(handles=[handle], cookies_path=cookies_path, headless=True)
    else:
        raise ValueError(f"unsupported x.com path: {path}")
    print(json.dumps({"data": data}, default=str))
except Exception as err:
    print(json.dumps({"error": str(err), "stage": stage, "traceback": traceback.format_exc()[-2000:]}))
    sys.exit(1)
EOF

DEVKIT_SCRAPE_URL="$URL" python "$DRIVER"

Output is structured tweet/profile data — render to Markdown (username + timestamp + text + engagement stats for tweets; bio + follower counts for profiles). If the JSON has an error field, fall back to Jina.

Firecrawl (if FIRECRAWL_API_KEY is set and --backend firecrawl):

Use Bash to call (use jq to safely construct JSON — never interpolate URLs directly):
jq -n --arg url "{url}" '{"url": $url, "formats": ["markdown"]}' | \
  curl -s -X POST https://api.firecrawl.dev/v1/scrape \
    -H "Authorization: Bearer $FIRECRAWL_API_KEY" \
    -H "Content-Type: application/json" \
    -d @-

Response contains: { "data": { "markdown": "...", "metadata": { "title": "..." } } }

WebFetch fallback:

Use WebFetch on the raw URL.
Note: output will be noisier than Jina/Firecrawl — no boilerplate removal.

Multiple URLs

When given multiple URLs, scrape them in parallel:

• Launch one researcher agent per URL (max 5 concurrent)
• Each agent fetches its URL using the backend logic above
• Collect all results before outputting

Error Handling

Fallback chain (single linear order): Jina Reader → Playwright → Camoufox → Firecrawl → WebFetch. Each backend is tried in order. Move to the next only on error, empty content, challenge-page detection, or missing dependency. Exception: URLs on x.com / twitter.com route directly to Scweet (skip the chain) when SCWEET_COOKIES_PATH is set; if Scweet is unavailable or fails, fall back to Jina.

• Report which backend actually ran for each URL, and (if fallbacks happened) why the earlier ones failed. Never silently substitute content — the user must know a paywall/cookie-wall/SPA-skeleton from one backend wasn't the real page fetched by another.
• If a URL is unreachable by every backend, report the error and continue with remaining URLs.
• Never silently drop or substitute a URL — always report what happened, with the winning backend named.

Output

Default (plain Markdown)

Output the extracted Markdown directly, prefixed with the source URL:

## Source: {url}

{extracted markdown content}

With --json flag

Output structured JSON:

{
  "url": "https://example.com",
  "title": "Page Title",
  "markdown": "# Heading\n\nContent here...",
  "backend": "jina",
  "timestamp": "2026-04-02T12:00:00Z"
}

For multiple URLs, output a JSON array.

Rules

• URL validation — only accept http:// and https:// URLs. Reject file://, ftp://, data:, and all other schemes. Reject URLs targeting private/reserved IPs: localhost, 127.0.0.1, 0.0.0.0, 169.254.x.x (cloud metadata — AWS/GCP/Azure), 10.x.x.x, 172.16-31.x.x, 192.168.x.x, [::1]. Reject URLs containing @ (credential-in-URL attacks). When rejecting, report the exact reason and stop — never silently skip a URL in a batch.
• No shell injection — never interpolate user URLs directly into shell command strings. Use jq to construct JSON payloads for curl. For Playwright, write scripts to files that read URLs from process.argv, and pass the URL as a shell variable — never inline into -e strings.
• API keys from env only — never hardcode API keys. Always reference $JINA_API_KEY, $FIRECRAWL_API_KEY from environment variables.
• Always try Jina Reader first — it's free and produces the cleanest output
• Respect rate limits — if scraping many URLs, add a brief pause between requests
• Don't scrape login-gated or paywall content — report it as inaccessible
• Strip cookie banners, nav bars, and footers when possible (Jina/Firecrawl do this automatically)
• If the user just wants to read a page, output Markdown. If they want to process it, use --json.