agentleFS
slilbudgetslilbudget/knowledge-delta-skills/skills/engineering/web-scraping/SKILL.md

web-scraping

Skill3 starsChanged 49 days agoMITView on GitHub

Mass crawl/scrape sites incl. anti-bot (Cloudflare, Akamai, DataDome, PerimeterX). Crawl4AI = scale (arun_many, dispatchers, deep crawl, LLM-ready markdown); Camoufox = fingerprint evasion. For: scrape N pages, crawl a site, extract structured data, 403/challenge/bot-detection, captcha.

25 other projects use a version of this.

Adopted by

Most recently taken up by

Used with

Projects with it also use community-led-growth, ph-community-outreach and 10 more.

Other files people use alongside this one.

Files found in the same public projects as a version of this one, more often than they turn up anywhere else. Each project counts once, large collections count for less, and this file's own repository is left out.
  • nicepkgSkill

    community-led-growth

    Expert in community-led growth (CLG) - leveraging user communities to drive acquisition, retention, and expansion. Covers building developer communities, user groups, ambassador programs, and turning customers into advocates. Knows the difference between community as a feature vs community as a growth engine, and how to measure community ROI. Use when "community-led, CLG, ambassador program, champion program, user community, developer community, word of mouth, user group, " mentioned.

    Also in 5 projects that use this

  • nicepkgSkill

    ph-community-outreach

    Engage Reddit, Indie Hackers, Hacker News and other communities for Product Hunt launches. Use this skill to identify relevant communities, craft appropriate posts, and drive cross-platform engagement without appearing spammy.

    Also in 5 projects that use this

  • xfstudioSkill

    micro-saas-launcher

    Expert in launching small, focused SaaS products fast - the indie hacker approach to building profitable software. Covers idea validation, MVP development, pricing, launch strategies, and growing to sustainable revenue. Ship in weeks, not months. Use when: micro saas, indie hacker, small saas, side project, saas mvp.

    Also in 5 projects that use this

  • mastepanoskiSkill

    ux-audit-rethink

    Comprehensive UX audit using IxDF's 7 factors, 5 usability characteristics, and 5 interaction dimensions. Holistic evaluation with redesign proposals based on user-centered design principles.

    Also in 5 projects that use this

  • ncklrsSkill

    seo-content-strategist

    Expert SEO content strategy for SaaS and B2B sites. Use when doing keyword research, planning content clusters, creating pillar pages, optimizing on-page SEO, fixing technical SEO issues, building links, refreshing content, implementing programmatic SEO, mapping search intent, optimizing for SERP features, or measuring content performance. Use for SEO audits, content gap analysis, and organic growth strategy.

    Also in 5 projects that use this

  • hieutrtrSkill

    code-review-security

    Security-focused code review checklist and automated scanning patterns. Use when reviewing pull requests for security issues, auditing authentication/authorization code, checking for OWASP Top 10 vulnerabilities, or validating input sanitization. Covers SQL injection prevention, XSS protection, CSRF tokens, authentication flow review, secrets detection, dependency vulnerability scanning, and secure coding patterns for Python (FastAPI) and React. Does NOT cover deployment security (use docker-best-practices) or incident handling (use incident-response).

    Also in 5 projects that use this

  • +6 more from nicepkg

The file

We split this skill into the rules it gives an agent

1 rule. 0 are common, 0 are distinctive, and 1 is very specific to this project.

  • 0

    Common

    Most of the 18 edited versions we read keep these.

  • 0

    Distinctive

    This author's own choices. Worth a look.

  • 1

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    Check result.success per item — arun_many returns failures, never raises.

    line 82

“Common” is measured: kept by at least 6 in 10 of the 18 edited versions of this file we could read. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Mass Scraping — Crawl4AI + Camoufox

Verified 2026-08-27 (macOS arm64). Both engines and all three templates smoke-tested.

Install (if crawl4ai-python is not on PATH)

  • crawl4ai — github.com/unclecode/crawl4ai · PyPI crawl4ai
  • camoufox — github.com/daijro/camoufox · PyPI camoufox[geoip]
  • curl_cffi — github.com/lexiforest/curl_cffi · PyPI curl_cffi

Dedicated venv, then crawl4ai-setup and python -m camoufox fetch. Four traps, each verified here:

  1. Python 3.10–3.13 only. 3.14 is not supported — if python3 is 3.14, install 3.13 explicitly.
  2. crawl4ai, not crawl4ai[all]. The extra pulls torch + transformers (GBs) that scraping never uses.
  3. crawl4ai-setup, not playwright install chromium. Setup also installs Patchright, which rung 3 needs; without it the undetected adapter dies.
  4. Shim the venv python with a wrapper, never a symlink. A symlinked venv python resolves sys.prefix to the system interpreter and cannot import crawl4ai:
    printf '#!/bin/sh\nexec "$VENV/bin/python" "$@"\n' > ~/.local/bin/crawl4ai-python

Entry points

crwl https://example.com -o md -bc   # one page → markdown
crawl4ai-python script.py            # anything bigger

Both are shims into the dedicated venv. System python3 cannot import crawl4ai/camoufox — always crawl4ai-python or the venv binary directly. Don't pip-install these into another environment.

crwl: -o md · -o all (JSON) · -bc (bypass cache) · --deep-crawl bfs --max-pages N. -o json fails without an extraction schema — use -o all.

Escalation ladder — climb only on failure

#ApproachCostWhen
0curl_cffi (TLS/JA3 impersonation, no browser)~10ms/pageTry first. Clears more than expected — see 03
1crwl / plain AsyncWebCrawler~1s/pagePage needs JS
2BrowserConfig(enable_stealth=True)~0Rung 1 → challenge/403
3UndetectedAdapter() (= Patchright)+~1s/pageRung 2 still blocked
4Camoufox + raw:// bridge200MB/instance, slowChromium fingerprint-blocked
5Residential proxies$Blocked by IP, not fingerprint

Rungs 2 and 3 are mutually exclusive — browser_manager.py:763: if self.config.enable_stealth and not self.use_undetected:. The undetected adapter silently drops enable_stealth. "Combine both for max evasion" advice is false; verified in installed source.

Captcha

reCAPTCHA v3 and Turnstile are not solvable offline — behavioral/TLS scoring, no image. The only play is not triggering them (rungs 2-4). Repos named *-turnstile-solver = token harvesters (minutes-long TTL) or paid-API wrappers.

Simple image captchas are offline-solvable → ddddocr, not installed. Ask first.

Still blocked at rung 5? Say so. Never return partial data silently.

References

NeedFile
arun_many, dispatchers, deep crawl, extraction, tuningreferences/01-crawl4ai-scale.md
Fingerprints, proxy+geoip, humanize, raw:// bridgereferences/02-camoufox.md
Benchmarks, failure signatures, what beats whatreferences/03-anti-bot-reality.md

templates/ — copy, don't retype: mass_crawl.py (rungs 1-3) · camoufox_bridge.py (rung 4) · deep_crawl.py (whole site).

Non-negotiables

  • cache_mode=CacheMode.BYPASS / -bc — default ENABLED serves stale pages and still reports success.
  • One AsyncWebCrawler context for all URLs. Per-URL async with relaunches the browser (~50x slower).
  • Check result.success per item — arun_many returns failures, never raises.
  • Respect robots.txt + rate limits unless the target is the user's. Use RateLimiter, not a bare loop.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.