Firecrawl — Web Scraping & Extraction

Always use the firecrawl CLI first. Fall back to the firecrawl-mcp MCP server only when the CLI genuinely cannot do the job. The CLI runs token-free via Bash and is faster; the MCP costs context on every call.

Step 0 — Verify prerequisites

Before running anything, confirm the CLI is installed and the API key is set:

firecrawl --version          # should print a version number
echo $FIRECRAWL_API_KEY      # should print fc-...

If the CLI is missing:

npm install -g firecrawl-cli

If the API key is empty, add it to .env and export it:

export FIRECRAWL_API_KEY=fc-your-key-here
# Get your key at: https://www.firecrawl.dev/app/api-keys

Step 1 — Choose the right command (CLI first)

Work through this decision in order. Stop at the first match.

Is the task a single URL scrape? → firecrawl <url>

firecrawl https://example.com                          # Markdown (default)
firecrawl https://example.com --format html,links      # specific formats
firecrawl https://example.com -o .tmp/page.md          # save to file
firecrawl https://example.com --json --pretty > .tmp/page.json

Is the task a web search? → firecrawl search

firecrawl search "your query"
firecrawl search "AI pricing" --limit 10               # cap results
firecrawl search "topic" --scrape --scrape-formats markdown  # fetch full pages

Is the task a multi-page site crawl? → firecrawl crawl

firecrawl crawl https://example.com --wait --progress
firecrawl crawl https://example.com --limit 50 --max-depth 3 --wait > .tmp/crawl.json

Is the task URL discovery across a domain? → firecrawl map

firecrawl map https://example.com
firecrawl map https://example.com --search "blog" --json > .tmp/urls.json

Is the task AI-driven open-ended extraction? → firecrawl agent

firecrawl agent "Extract all pricing plans from this site" --wait
firecrawl agent "Extract contact info" \
  --schema '{"type":"object","properties":{"email":{"type":"string"}}}' \
  --wait

Save all output to .tmp/ — never to tracked paths.

Step 2 — Escalate to MCP only when CLI cannot do the job

The CLI covers ~90% of use cases. The MCP is the right choice only for:

If none of those apply, use the CLI. The MCP has the same underlying API — it just adds protocol overhead.

Available MCP tools (firecrawl-mcp): firecrawl_scrape, firecrawl_batch_scrape, firecrawl_search, firecrawl_crawl, firecrawl_map, firecrawl_agent, firecrawl_extract, firecrawl_check_crawl_status, firecrawl_check_batch_status, firecrawl_agent_status

Step 3 — Handle errors and escalate when stuck

Output handling

• Always write to .tmp/ (scratch space, not committed)
• Read .tmp/page.md inline to inspect content in-session
• Pass .tmp/page.json to a Python script or the next pipeline step
• Final deliverables go to cloud services: Google Drive, Supabase, Sheets, email, etc.