web_extract — Web Content Extraction

TRIGGER: User gives a URL and asks to read, extract, fetch, summarize, or look up its content. Also triggers when a task needs page content (e.g. research, verification, data gathering).

HARD RULE — Extraction Priority (in this order, no exceptions):

Step 1: Jina Reader (r.jina.ai)   ← ALWAYS try first — EXCEPT x.com URLs
Step 2: trafilatura (Python)      ← when Jina fails/blocked; ALSO always used for x.com
Step 3: Firecrawl keyless         ← when Jina + trafilatura both fail, OR for X comments/engagement, OR grok/JS-heavy; skip if daily credits spent or reddit.com
Step 4: Headless browser          ← final catch-all when all above fail

When to use which (the golden rule)

The complementarity rule: Jina is best for JS-heavy pages; trafilatura is best for X.com and anything Jina can't open — and things trafilatura can't open, Jina usually can. They cover each other's blind spots. Firecrawl adds two things neither has: X thread context (comments/engagement) and reliable grok.com/share reads. Only when ALL of these fail do you go to the browser.

Method 1 — Jina Reader (default first choice)

curl -sL "https://r.jina.ai/https://TARGET_URL"

Run via terminal. Replace TARGET_URL with the full URL.

• Output is clean markdown: Title:, URL Source:, Markdown Content:
• Free, no API key (anonymous)
• Strips prompt-injection from scraped content
• Do NOT retry Jina on YouTube — always 403 (bot detection). Go straight to browser.
• X.com — skip Jina entirely (see Method 2). Jina's x.com block is usually temporary (< 24h on first access), but don't wait for it.
• Rate limits: if 429/451/403, wait a few seconds OR fall through to trafilatura immediately.

Detecting failure: Jina returns a JSON error like {"code":403,"name":"AbuseAlleviationError",...} or an HTTP error, or an empty/garbage body. If so → go to Method 2.

Method 2 — Trafilatura

Trafilatura is installed in the Hermes venv (pip install trafilatura) — importable from both terminal and execute_code. No path resolution needed.

Usage — Method A: execute_code (PRIORITY — preferred)

✅ Cleanest way — no shell quoting, no path resolution, output comes back as a clean tool result:

# execute_code — trafilatura is importable directly
import trafilatura
d = trafilatura.fetch_url('TARGET_URL')
print(trafilatura.extract(d))

Why it's priority: no shell quoting, no path resolution, output comes back as clean tool result. The sandbox already uses the Hermes venv Python.

Usage — Method B: terminal (fallback)

Use when execute_code is unavailable or the extraction is very long (avoid 50KB stdout cap in execute_code).

python -c "
import trafilatura
url = 'TARGET_URL'
downloaded = trafilatura.fetch_url(url)
print('Downloaded:', len(downloaded) if downloaded else 0, 'chars')
if downloaded:
    text = trafilatura.extract(downloaded)
    print(text)
"

One-liner form (works from any cwd):

python -c "import trafilatura; print(trafilatura.extract(trafilatura.fetch_url('TARGET_URL')))"

Practical tips

• X.com: trafilatura works great — the post text is in the raw HTML. This is the FIRST choice for x.com.
• JS-heavy pages: trafilatura often returns only the <title> (e.g. grok.com gave 71 chars = title only while the page was 468K). If extracted text is suspiciously short or title-only → the page is JS-rendered → use Jina (Method 1), Firecrawl (Method 3), or browser (Method 4).
• When Jina is blocked: trafilatura is the immediate fallback.
• trafilatura.exe CLI is a broken trampoline — ALWAYS use the Python API above, never the .exe.

Method 3 — Firecrawl Keyless (when Jina + trafilatura both fail, or for X comments / grok)

What it is: Firecrawl's keyless tier — Search / Scrape / Parse against the production Firecrawl Cloud API with no API key, no signup. Works unauthenticated. Free but metered by credits: normal page scrapes ≈ 1 credit each; X.com scrapes = 30 credits (heavy postprocessor); keyless daily pool ≈ ~100 credits (unpublished). Once spent → ~24h lockout of ALL endpoints (scrape + search).

When to use Firecrawl (and when to skip):

Scrape a URL (keyless)

curl -s -X POST "https://api.firecrawl.dev/v2/scrape" \
  -H "Content-Type: application/json" \
  -d '{"url":"TARGET_URL"}'

Returns { success, data: { markdown, metadata, ... } } with creditsUsed in the response.

Site-specific options

• grok.com/share (and slow/JS-heavy pages): add "waitFor": 5000 to the JSON body (5s) — without it, Firecrawl returns only the page shell.
• X.com: Firecrawl auto-detects X and applies its x-twitter postprocessor — you get post text + author + likes/RTs + top comments. Costs 30 credits.
• JSON output: add "formats": ["markdown"] (default) or "formats": ["json"] for structured data.

Search (keyless)

curl -s -X POST "https://api.firecrawl.dev/v2/search" \
  -H "Content-Type: application/json" \
  -d '{"query":"YOUR_QUERY"}'

Returns LLM-friendly, excerpt-ranked results (their 94.7% SimpleQA claim, self-run). Shares the same credit pool as scrape — don't spend the daily pool on search when TinyFish/Tavily are free/cheaper.

Credit budget guard (HARD RULE)

• Track Firecrawl usage this session. The keyless daily pool is roughly ~100 credits (unpublished). X scrapes at 30 credits each mean ~3 X scrapes/day max before the ~24h lockout.
• Once the pool is likely spent (2-3 X scrapes, or a 429/retry_after error), STOP using Firecrawl for the rest of the session → fall through to browser (Step 4). Never hammer a 429 — the lockout is ~24h.
• Check the creditsUsed field in each response to track burn.

Detecting failure

• HTTP 429 with retry_after_seconds (e.g. 86358 ≈ 24h) → daily pool exhausted → browser, and don't retry Firecrawl today.
• "success": false with an error message (e.g. Reddit: "we do not support this site") → unsupported site → browser.
• "success": true but EMPTY content → Firecrawl silently returns success on dead X posts → verify output is non-empty; if empty → browser.
• Redirects/403 → browser.

Method 4 — Headless Browser (final catch-all)

If Jina, trafilatura, AND Firecrawl all fail (login-gated, captcha, exotic JS, empty, credits exhausted):

browser_navigate(url)   # then read the returned snapshot
browser_snapshot()      # full accessibility tree if needed

• Always works for rendered content (it executes JS)
• Captures the accessibility tree — good enough for reading post text, comments, etc.
• Slower than the other three; use only when they fail.

Verify extraction quality

After extracting, sanity-check the result:

• If output is only a title / nav text / "No content" → the real content is JS-rendered → try the OTHER tool (complementarity rule).
• If output contains the substantive content (article body, post text, guide steps) → done.

First-Time Run & Install

The skill is ready to use once its folder exists — no build step and no API keys required. Jina Reader is free and anonymous, trafilatura ships in the Hermes venv, and the headless browser is built in. Two optional setup items make the agent prefer this skill automatically.

1. Optional: make sure trafilatura is installed

Trafilatura is already installed in the Hermes venv. If it's ever missing from a fresh environment:

python -m pip install trafilatura

2. SOUL.md Injection — make the agent prefer this skill

For the agent to automatically use this skill instead of the built-in web_extract tool, add a few lines to the profile's SOUL.md (e.g. <profile>/SOUL.md) under a suitable section (e.g. ## General Agent System prompt or a new ## Web Extract section). Reference injection (live in the gf-helen profile):

## Web Extract
When performing a web extract, use the `web_extract` skill `skill_view(name='web_extract')` instead of the built-in web_extract tool, unless the user explicitly requests the built-in tool. Native `web_extract` requires an API-key backend (firecrawl/tavily/exa/parallel) unless the web-native plugin is present. When uncertain, load the extraction skill and follow its 4-tier fallback:

1. **Jina Reader first** (except X.com): `curl -sL "https://r.jina.ai/https://<URL>"`
   - Best for JS-heavy pages (grok.com, SPA apps, forums).
2. **trafilatura** when Jina fails/blocked — and ALWAYS first for X.com:
   `python -c "import trafilatura; print(trafilatura.extract(trafilatura.fetch_url('<URL>')))"`
   - Best for X.com and raw-HTML pages; bundled in the internal venv, no install.
3. **Firecrawl keyless** when both above fail, or for X comments/engagement, or grok/JS-heavy:
   `curl -s -X POST "https://api.firecrawl.dev/v2/scrape" -H "Content-Type: application/json" -d '{"url":"<URL>"}'`
   - Skip if daily credits spent or reddit.com (hard-blocked).
4. **Headless browser** as last resort: `browser_navigate("<URL>")` then read the snapshot.

**Complementarity rule:** Jina best for JS-heavy; trafilatura best for X.com; Firecrawl adds X thread context + grok reads. Only when ALL fail do you go to the browser.

💡 On a fresh install, you can simply tell your agent: "Set up the web_extract skill: add the SOUL.md rule for me." The agent will handle the rest (backups included).

3. Verify

curl -sL "https://r.jina.ai/https://example.com"

Expect clean markdown back (Title: / URL Source: / Markdown Content:). Or give the agent a URL and ask it to read / extract / summarize the page — it should load the skill and follow the 4-tier fallback.

Known limitations

Related

• Search: use search-this skill (Tavily → TinyFish → Composio tiered priority)
• Browser interaction: web-interaction skill (built-in browser best practices)
• If this skill is missing a dependency, install via python -m pip install trafilatura