tavily-extract
Skill6 starsChanged 22 days agoMITView on GitHub
Use when a web page won't come through a normal fetch — WebFetch (or curl) returns an empty shell, a "Loading…" skeleton, or a 403/Forbidden. That happens on JavaScript-rendered single-page apps (React/Next.js/Vue dashboards, leaderboards and docs such as Artificial Analysis, LMArena, developers.openai.com, platform.claude.com) and on bot-protected hosts (Cloudflare/WAF). This skill calls the Tavily Extract API to run the page's JavaScript and return clean markdown, and can batch up to 20 URLs in one call; it also does Tavily web search. Reach for it when WebFetch comes back empty, truncated, a skeleton, or blocked, or when a task says "the SPA/client-rendered page won't fetch", "I got a 403", "scrape the data behind the JS", "the charts/tables won't come through", or "extract these N URLs". Try WebFetch first — it's free and handles server-rendered sites; use this when it can't.
- Reads credentials
7 other projects use a version of this.
Adopted by
Most recently taken up by
- Neetx/ai-research-radartheir own version, first seen Oct 31, 2026
- Neetx/ai-security-research-radartheir own version, first seen Oct 22, 2026
- PracticalSwan/agent-skillstheir own version, first seen Oct 19, 2026
- wkylin/pro-react-admintheir own version, first seen Oct 8, 2026
- AlexisZ12/tavily-skillstheir own version, first seen Jun 22, 2026
- jpoindexter/theft-kittheir own version, first seen May 20, 2026
Other versions7 projects use a version of this.
Word for word · 0
None found.
Edited · 0
None found.
Wrote their own version · 7
- wkylin/pro-react-admin★ 948same name
- PracticalSwan/agent-skills★ 22same name
- Neetx/ai-security-research-radar★ 8same name
- Neetx/ai-research-radar★ 5same name
- AlexisZ12/tavily-skills★ 1same name
- jpoindexter/theft-kit★ 1same name
- baofeng-tech/agent-skills-so★ 0same name
How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.
In this repository29 other files it gives its agents.
- AGENTS.mdAGENTS.md
- CLAUDE.mdCLAUDE.md
- skills/prompt-engineer/references/claude.mdCLAUDE.md
- brand-assetsSkill
- brand-systemSkill
- codex-reviewSkill
- deck-to-videoSkill
- elevenlabs-sttSkill
- elevenlabs-ttsSkill
- excalidraw-diagramsSkill
- gdoc-syncSkill
- generating-video-clipsSkill
- gsheetsSkill
- gslides-syncSkill
- image-generationSkill
- namecheap-domainsSkill
- narrating-pptxSkill
- office-renderSkill
- pair-intelligenceSkill
- presentation-generatorSkill
- producing-animated-episodesSkill
- prompt-engineerSkill
- proton-passSkill
- reviving-sessionsSkill
- self-presenting-decksSkill
- tech-design-docSkill
- viewing-videosSkill
- writing-plainlySkill
- writing-project-instructionsSkill
The file
We split this skill into the rules it gives an agent
5 rules. 0 are common, 3 are distinctive, and 2 are very specific to this project.
0
Common
Most of the 6 edited versions we read keep these.
3
Distinctive
This author's own choices. Worth a look.
2
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 14
Use the cheapest tool that works, in this order:
- Distinctive: line 20
Do not reach for this on pages WebFetch already handles — it costs credits and adds a hop.
- Very specific: line 32
Never hardcode the key in a command you commit or in this repo.
- Very specific: line 36
Run the bundled script (it's executable.
- Distinctive: line 65
Structured data > scrape: If the site has an API (see step 3 above), one authenticated call beats extracting rendered markdown and …
“Common” is measured: kept by at least 6 in 10 of the 6 edited versions of this file we could read. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Tavily Extract
Core principle
A normal fetch returns the raw HTML. For a JavaScript-rendered SPA that HTML is an empty shell (content is hydrated in the browser), and bot-protected hosts answer 403. Tavily runs the page's JavaScript and fetches through browser-like infrastructure, so it returns the rendered content as markdown — the one thing WebFetch structurally can't do (it executes no JS and is more easily blocked).
When to use / when NOT
Use the cheapest tool that works, in this order:
- WebFetch first — free, built in, and it handles most sites, including many JS-framework doc pages that server-render their content. Same domain can differ:
developers.openai.com/api/docs/...server-renders and WebFetch reads it;developers.openai.com/cookbook/...is client-rendered and comes back a 404 shell. - This skill (Tavily Extract) — when WebFetch returns a shell/skeleton, truncated content, or a 403. This is the JS-rendering + anti-block tool.
- The site's own API / embedded JSON — for structured data (leaderboards, metrics), a real API beats any scrape. Artificial Analysis, for example, exposes
GET https://artificialanalysis.ai/api/v2/data/llms/modelswith anx-api-key; its rendered charts are JS data-viz that serialize to axis labels, not numbers. When you need exact figures, prefer the API.
Do not reach for this on pages WebFetch already handles — it costs credits and adds a hop.
Setup (one-time)
Needs a Tavily API key (get one at https://app.tavily.com). The script reads it in this order: $TAVILY_API_KEY, then ~/.config/tavily/api-key. Set either:
export TAVILY_API_KEY=tvly-...
# or, persistent:
mkdir -p ~/.config/tavily && printf 'tvly-...' > ~/.config/tavily/api-key && chmod 600 ~/.config/tavily/api-keyNever hardcode the key in a command you commit or in this repo. (Shahar's key already lives in api-keys.md under "Tavily" and in ~/.config/tavily/api-key.)
Quick reference
Run the bundled script (it's executable; python3 also works). The first argument may be a bare URL — it defaults to extract.
SC=scripts/tavily_extract.py # relative to this skill's directory
# One page -> clean markdown on stdout
"$SC" https://lmarena.ai/leaderboard/agent
# Focus the extraction (reranks/trims toward your intent — cuts nav boilerplate)
"$SC" https://artificialanalysis.ai/models/gemini-3-6-flash --query "intelligence index and price"
# Batch (up to 20 URLs) and save one <slug>.md per URL instead of printing
"$SC" extract URL1 URL2 URL3 --save /tmp/out
# Cheaper/faster (less content): basic depth
"$SC" URL --depth basic
# Web search (finds URLs; returns an answer + ranked results)
"$SC" search "Gemini 3.6 Flash release date" --max-results 5Options: --depth {basic,advanced} (default advanced — more content incl. tables; basic is cheaper), --format {markdown,text}, --query TEXT, --chunks 1-5, --save DIR.
Gotchas
- Soft 404s. A rendered SPA can return HTTP 200 with a "Page not found" body (often buried under nav chrome, so the response is long, not empty). The script scans for that and prints
[warn] … likely a soft 404— heed it; don't treat the nav shell as real content. Verify the URL if you see it. - Nav boilerplate. Extracts of doc sites often lead with hundreds of lines of navigation before the real content. Use
--queryto rerank toward your intent, or justgrepthe output for the section you need — don't add an HTML parser (trafilatura etc.): Tavily already content-extracts, and there's no rendered HTML to re-parse for an SPA anyway. - 20-URL cap per extract call; the script warns and takes the first 20 if you pass more. Split larger batches.
- Cost.
advanceddepth ≈ 2 credits / 5 URLs,basic≈ 1 / 5. Pass--depth basicwhen you don't need tables/embedded content. Usage is printed to stderr as[usage]. - Structured data > scrape. If the site has an API (see step 3 above), one authenticated call beats extracting rendered markdown and re-parsing it.
Reference
Full API docs: https://docs.tavily.com/documentation/api-reference/endpoint/extract — the script covers the common cases; add fields to scripts/tavily_extract.py if you need include_images, topic, etc.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

