agentleFS
jiahao-shao1jiahao-shao1/sjh-skills/skills/web-fetcher/SKILL.md

web-fetcher

Skill43 starsChanged 7 months agoMITView on GitHub

Fetch any URL as clean markdown. Use instead of WebFetch for JS-rendered pages, login-required platforms (Twitter/X, zhihu, reddit, weibo, xiaohongshu, bilibili, etc.), and complex pages. Routes known platforms through OpenCLI (browser login state), others through Jina Reader / defuddle.md / markdown.new. Invoke when the user provides a URL to read, extract, summarize, or convert to markdown.

  • Installs packages

2 other projects use a version of this.

Adopted by

Most recently taken up by

The file

We split this skill into the rules it gives an agent

1 rule. About 0 are common, 0 are distinctive, and 1 is very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈0

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 1

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    If opencli is not installed:

    line 63

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Web Fetcher

Fetch any URL as clean markdown. Two paths: known platforms go through OpenCLI (uses browser login state), everything else falls back through a chain of free markdown services.

Strategy Overview

  1. Known platforms (Twitter/X, zhihu, reddit, weibo, xiaohongshu, bilibili, etc.) → OpenCLI first. It uses the user's browser login state, is deterministic, and costs zero LLM tokens.
  2. Generic URLs (blogs, docs, articles) → Jina Reader > defuddle.md > markdown.new > WebFetch.

These are soft priorities — skip obviously wrong strategies (e.g., don't try OpenCLI for a random blog post, don't try Jina for a login-walled zhihu page).

Known Platform → OpenCLI Commands

URL PatternCommandNotes
x.com/.../status/<id>opencli twitter thread <id>Tweet threads. If the result is just a t.co link (very short text), the tweet likely links to an X Article — retry with opencli twitter article <same-id>
x.com/.../article/<id> or x.com/i/article/<id>opencli twitter article <id>X Article long-form
zhihu.com/question/<id>opencli zhihu question <id>
zhuanlan.zhihu.com/p/<id>opencli zhihu download <full-url>
reddit.com/r/.../comments/...opencli reddit read <full-url>
weibo.com/...opencli weibo search <query>
xiaohongshu.com/...opencli xiaohongshu download <id>
bilibili.com/video/BV...opencli bilibili download <bvid>

Dynamic Discovery

The table above covers high-frequency platforms. OpenCLI supports 80+ platforms. For anything not listed:

# See all supported platforms
opencli --help

# See subcommands for a specific platform
opencli <platform> --help

Generic Fallback Tools

For URLs that don't match a known platform, try these in order:

ToolHow to UseStrengthsWeaknesses
Jina Readercurl -s -H "Accept: text/markdown" "https://r.jina.ai/<url>"Best markdown quality, JS support20 req/min free limit
defuddle.mdcurl -s "https://defuddle.md/<url>"Good quality, by Obsidian creatorUndocumented limits
markdown.newcurl -s "https://markdown.new/<url>"Browser rendering fallback500 req/day
WebFetchBuilt-in tool, no curl neededNo setup neededNo JS rendering, poor on complex pages

Decision Guide

  • Known platform with login → OpenCLI first
  • Static article/blog → Jina Reader first
  • If a tool returns too-short content (<500 chars) → likely an error page, try next tool in chain
  • If Jina fails or is rate-limited → defuddle → markdown.new → WebFetch
  • If OpenCLI fails → fall back to the generic chain above

OpenCLI Setup

If opencli is not installed:

  1. npm i -g @jackwener/opencli
  2. Download browser extension from https://github.com/jackwener/opencli/releases
  3. Chrome → chrome://extensions → Developer mode → Load unpacked → select unzipped folder
  4. opencli doctor to verify

One-time setup. After installation, known platform URLs automatically use browser login state.

Limitations

  • WeChat articles partially supported via opencli weixin; proxy may be needed for some URLs behind GFW
  • OpenCLI requires browser extension setup (one-time)
  • Jina Reader free tier: 20 req/min
  • markdown.new: 500 req/day/IP

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.