Crawl4AI

Overview

Use this skill to crawl web pages into inspectable markdown or structured JSON. In this repository, crawl4ai is a required peer for deep-research: search discovers sources, Crawl4AI captures raw/source content, and the research skill synthesizes evidence.

The skill is harness-neutral. Use the local CLI, Python SDK, Docker API, MCP bridge, or an equivalent harness-provided Crawl4AI tool.

Required Setup

Install Crawl4AI in the Python runtime used by the current agent:

python -m pip install -U crawl4ai
crawl4ai-setup
crawl4ai-doctor

Official project: https://github.com/unclecode/crawl4ai

If the crwl command is available:

crwl crawl https://example.com -o markdown --bypass-cache

If the current harness uses a dedicated virtual environment, run the same commands with that environment's Python and crwl executable. Do not hardcode another machine's venv path in reusable skills.

If browser setup fails:

python -m playwright install chromium

On Linux servers that need browser system dependencies:

python -m playwright install --with-deps chromium

Core Workflow

1. Start from URLs selected by advanced-web-search, browser inspection, or user input.
2. Choose the lightest adequate capture mode:
   • simple markdown for normal pages
   • scan_full_page, wait_for, or longer timeout for dynamic pages
   • CSS/JSON extraction for repeated structured pages
   • screenshot/PDF only when visual evidence matters
3. Capture raw markdown/JSON into a durable path such as research/raw/YYYY-MM-DD/<topic>/.
4. Inspect extraction quality before using the content:
   • non-trivial length
   • expected title/headings/entities
   • no login wall, CAPTCHA, cookie-only shell, or empty JS shell
   • relevant section present, not just boilerplate
5. Mark each source as fetched, partial, blocked, empty, or needs_review.
6. Pass raw paths and quality notes back to deep-research for source cards and evidence matrices.

CLI Patterns

# Markdown to a file; create the parent directory first
mkdir -p research/raw/2026-07-05/example
crwl crawl https://example.com -o markdown --bypass-cache -O research/raw/2026-07-05/example/example.md

# Dynamic page
crwl crawl https://example.com -c "wait_for=css:.content,page_timeout=60000" -o markdown

More CLI forms (JSON output, full-page scroll, browser flags, structured extraction) are in references/cli-guide.md.

On Windows consoles, prefer UTF-8 output and -O <file> if rich terminal output causes encoding issues.

Python SDK

For scripted capture, prefer the bundled scripts below. For custom SDK code — browser/run configs, arun_many batches, CSS extraction schemas — read references/sdk-guide.md.

Bundled Scripts

• scripts/crawl_url.py - single URL markdown capture with quality checks.
• scripts/batch_crawler.py - batch URL capture into research/raw/....
• scripts/extraction_pipeline.py - CSS/JSON structured extraction helper.
• scripts/google_search.py - Google result-page scraping fallback when normal search tools are unavailable.

Run scripts with the Python environment that has crawl4ai installed.

References

• references/cli-guide.md for command-line usage.
• references/sdk-guide.md for Python SDK usage.
• references/quality-checks.md for extraction validation.
• references/windows-notes.md for Windows-specific encoding/path notes.

Pitfalls

• Do not treat result.success or HTTP 200 as enough; inspect the extracted content.
• Create output directories before using -O.
• result.markdown may be an object; use .raw_markdown when present.
• markdown-fit can be empty if filters are too aggressive or the site blocks crawling.
• For bot-protected/authenticated sites, expect partial results unless a profile/cookies/session is configured.