Firecrawl

Web scraping CLI. Turns any URL into clean LLM-ready markdown. 8 commands, all via the firecrawl CLI.

Environment

The CLI reads its API key from the FIRECRAWL_API_KEY environment variable. Set your FIRECRAWL_API_KEY (via env or your secrets manager, e.g. Doppler) before running any command. If your secrets are injected by a manager, run the CLI inside that manager's shell (for example, doppler run -- firecrawl ...).

Command Reference

Escalation Pattern

Follow this order — start simple, escalate only if needed:

1. No URL yet → firecrawl search "query" --scrape --limit 3
2. Have a URL → firecrawl scrape "<url>"
3. Large site, need a specific subpage → firecrawl map "<url>" --search "keyword" then scrape the match
4. Need bulk content from a section → firecrawl crawl "<url>" --limit 20
5. Scrape failed (JS gating, modals, login, pagination) → firecrawl browser "<url>"

Common Patterns

Research a topic (no URL)

firecrawl search "AI agency pricing models 2025" --scrape --limit 5 -o .firecrawl/search-results.json

Scrape a specific page

firecrawl scrape "https://competitor.com/pricing" -o .firecrawl/competitor-pricing.md

Pull all docs from a site

firecrawl crawl "https://docs.example.com" --limit 50 -o .firecrawl/docs/

Extract structured data with AI

firecrawl agent "https://example.com" --prompt "Extract: company name, pricing tiers, features per tier" -o .firecrawl/extracted.json

Find a specific subpage on a large site

firecrawl map "https://example.com" --search "pricing"
# Then scrape the matching URL

Parallel scrapes (faster)

firecrawl scrape "<url-1>" -o .firecrawl/1.md &
firecrawl scrape "<url-2>" -o .firecrawl/2.md &
firecrawl scrape "<url-3>" -o .firecrawl/3.md &
wait

Output Rules

• Always write to a temp folder with -o — never dump large content inline into the conversation
• Use .firecrawl/ as a scratch folder for session work
• For content worth keeping, move it somewhere permanent when done
• Never read entire large files at once — use head -100 or search for specific sections first
• search --scrape already fetches full page content — don't re-scrape those same URLs

# Extract URLs from search output
jq -r '.data.web[].url' .firecrawl/search.json

# Check file size before reading
wc -l .firecrawl/file.md && head -100 .firecrawl/file.md

# Clean up temp files when done
rm -rf .firecrawl/*

Important: Always Quote URLs

URLs with ? and & characters must be quoted in bash — they're shell special characters:

# Correct
firecrawl scrape "https://example.com/page?id=123&ref=abc"

# Wrong — shell will break the URL
firecrawl scrape https://example.com/page?id=123&ref=abc

Credits

Each scrape/search/crawl costs credits. Check balance:

firecrawl --status
firecrawl credit-usage

Top up at firecrawl.dev if you run low.