agentleFS
Sign inStart free
MHD-Nour-KJMHD-Nour-KJ/Web_Scrape_Dude/Skill/SKILL.md

web-scrape

Skill0 starsChanged 5 months agoMITView on GitHub

Use this skill to scrape and download websites/documentation into organized markdown files. Crucial for researching technology stacks, APIs, and frameworks before implementing them. Supports Planning Research and User Research modes.

  • Installs packages

4 other projects use a version of this.

Adopted by

Most recently taken up by

  • Other versions4 projects use a version of this.

    Word for word · 0

    None found.

    Edited · 0

    None found.

    Wrote their own version · 4

    1. Barrios88/barrios-skills★ 63same name
    2. huanchong-99/PocketAide★ 9same name
    3. zedxter/utility-belt★ 4same name
    4. stimulir/managed-skills★ 1same name

    How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.

The file

We split this skill into the rules it gives an agent

12 rules. 0 are common, 8 are distinctive, and 4 are very specific to this project.

  • 0

    Common

    Most of the 3 edited versions we read keep these.

  • 8

    Distinctive

    This author's own choices. Worth a look.

  • 4

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    Before running the scraper, verify if it is already installed.

    line 12
  2. Distinctive:

    Trigger: When the user asks you to plan a program, implement a new feature, or explicitly requests "planning research".

    line 49
  3. Distinctive:

    Use your web search tools to find the exact official documentation links for every item in the stack.

    line 53
  4. Distinctive:

    If your search tools are unavailable or insufficient, prompt the user to provide the links (listing exactly what documentation you need).

    line 54
  5. Distinctive:

    If the user does not provide the links, use your own internal knowledge to deduce and construct the exact documentation URLs.

    line 55
  6. Very specific:

    Verify the downloaded markdown files by reading them.

    line 57
  7. Distinctive:

    Trigger: When the user explicitly requests "user own research" or asks for a lightweight, specific scrape.

    line 60
  8. Distinctive:

    Only download exactly what the user absolutely, definitely needs.

    line 64
  9. Distinctive:

    If you think additional documentation *might* be useful, ask the user first.

    line 65
  10. Distinctive:

    Do NOT run the scraper on additional resources until the user gives an explicit greenlight to download them.

    line 66
  11. Very specific:

    Always review the downloaded index.md manifest to understand what pages were captured.

    line 69
  12. Very specific:

    If a scrape fails due to robots.txt but the user insists, use --ignore-robots.

    line 70

“Common” is measured: kept by at least 6 in 10 of the 3 edited versions of this file we could read. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Web Scraper Skill

You are equipped with a powerful local Python web scraper designed to fetch websites, download their images, fix links, and convert their HTML into clean Markdown for AI consumption.

Bootstrap & Setup

Before running the scraper, verify if it is already installed. If scrape.py is not found in the current workspace, you must clone the repository and install its dependencies:

  1. Clone the repository:
    git clone https://github.com/MHD-Nour-KJ/Web_Scrap_Dude.git
cd Web_Scrap_Dude
  2. Install dependencies:
    pip install -r requirements.txt

Executable

The scraper is run via the scrape.py CLI:

python scrape.py <url> [options]

To see all available options, run:

python scrape.py --help

Key Capabilities & Options

  • Deep crawling: --depth N (default: 2), --max-pages N (default: 100)
  • Image downloading: --download-images locally caches images and updates markdown links.
  • Resuming: --resume lets you continue an interrupted crawl using the .scrape_state.json.
  • Boilerplate stripping: Automatic navigation/footer removal (disable with --no-clean).
  • Robots.txt: Respected by default. --ignore-robots to bypass.
  • Non-interactive: You MUST pass --no-prompt when running automated crawls to avoid blocking on user input for folder names!

Operating Modes

if the user didn't specify a mode, then ask the user about the mode he/she wants to use the tool for before running the tool:

1. Planning Research Mode (Ambitious & Comprehensive)

Trigger: When the user asks you to plan a program, implement a new feature, or explicitly requests "planning research".

Behavior:

  1. Understand and list out every technology, tool, and third-party library in the planned tech stack.
  2. Use your web search tools to find the exact official documentation links for every item in the stack.
  3. If your search tools are unavailable or insufficient, prompt the user to provide the links (listing exactly what documentation you need).
  4. If the user does not provide the links, use your own internal knowledge to deduce and construct the exact documentation URLs.
  5. Once URLs are gathered, use python scrape.py <url> --no-prompt --download-images to rigorously download the complete documentation for every technology.
  6. Verify the downloaded markdown files by reading them. If the data is insufficient, increase the --depth or --max-pages and run the scraper again until you are fully satisfied with the amount of data gathered for every single technology.

2. User Own Research Mode (Conservative & User-Directed)

Trigger: When the user explicitly requests "user own research" or asks for a lightweight, specific scrape.

Behavior:

  1. You must be way less eager and ambitious.
  2. Only download exactly what the user absolutely, definitely needs.
  3. If you think additional documentation might be useful, ask the user first.
  4. Do NOT run the scraper on additional resources until the user gives an explicit greenlight to download them.

Best Practices

  • Always review the downloaded index.md manifest to understand what pages were captured.
  • If a scrape fails due to robots.txt but the user insists, use --ignore-robots.
  • For large documentation sites, start with --depth 1 or --depth 2 to verify the output structure before committing to a massive scrape.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

Check your setup, free