site2cli

# site2cli

Turn any website into a CLI/API for AI agents.

## Architecture

Progressive Formalization: 3-tier system that auto-graduates from browser automation (Tier 1) → cached workflows (Tier 2) → direct API calls (Tier 3).

## Project Structure

```
src/site2cli/
├── cli.py              # Typer CLI entry point
├── config.py           # Configuration management
├── models.py           # Pydantic v2 data models
├── registry.py         # SQLite site registry
├── router.py           # Tier router (picks best execution method)
├── discovery/
│   ├── capture.py      # CDP-based network traffic capture
│   ├── analyzer.py     # LLM-assisted pattern analysis
│   ├── spec_generator.py  # OpenAPI 3.1 spec (JSON + YAML)
│   ├── client_generator.py # Python client code generation
│   ├── js_client_generator.py # Zero-dep ES-module JS client (.mjs)
│   ├── coverage_report.py  # Dark-theme HTML coverage report with gap detection
│   └── trace.py        # Portable Trace JSON (save_trace / load_trace) for offline replay
├── browser/
│   ├── retry.py        # Async retry with delay for browser actions
│   ├── wait.py         # Rich wait conditions (network-idle, selector, stable)
│   ├── cookie_banner.py # Cookie consent auto-dismissal (3 strategies)
│   ├── detectors.py    # Auth/SSO/CAPTCHA page detection
│   ├── a11y.py         # Accessibility tree extraction + indexed [@N] notation
│   ├── context.py      # Unified browser context factory (profiles + sessions)
│   └── session.py      # Named browser session persistence and reuse
├── output_filter.py    # Output filtering (grep, limit, keys-only)
├── generators/
│   ├── cli_gen.py      # Dynamic CLI command generation
│   ├── mcp_gen.py      # MCP server generation
│   └── agent_config.py # Agent config generation (Claude MCP, generic)
├── content/
│   ├── converter.py    # HTML to markdown/text conversion, main content extraction
│   ├── chunker.py      # RAG chunking (fixed / sentence / heading strategies)
│   └── pdf.py          # PDF parsing via pdfplumber (text / markdown / page count)
├── search/
│   └── engine.py       # DuckDuckGo search + chained scrape / extract
├── crawl/
│   ├── crawler.py      # Async BFS site crawler with resume and streaming
│   ├── links.py        # Link extraction and normalization from HTML
│   └── robots.py       # robots.txt parser and URL filtering
├── extract/
│   └── extractor.py    # LLM-powered structured extraction with schema validation
├── monitor/
│   ├── watcher.py      # Change detection with snapshot comparison and webhooks
│   └── differ.py       # Line-level diff engine (stdlib difflib)
├── screenshot/
│   └── capture.py      # Full-page and element screenshots via Playwright
├── auth/
│   ├── manager.py      # Auth flow management (Playwright cookie format)
│   ├── cookies.py      # Cookie CRUD, import/export (Playwright-compatible)
│   ├── profiles.py     # Chrome/Firefox profile detection & import
│   ├── device_flow.py  # OAuth device flow (RFC 8628)
│   └── providers.py    # Pre-configured OAuth providers (GitHub, Google, Microsoft)
├── tiers/
│   ├── browser_explorer.py  # Tier 1: LLM-driven browser
│   ├── cached_workflow.py   # Tier 2: Recorded workflow replay
│   └── direct_api.py        # Tier 3: Direct API calls
├── daemon/
│   ├── server.py       # Background browser daemon (JSON-RPC over Unix socket)
│   └── client.py       # Daemon client for CLI commands
├── mcp/
│   └── server.py       # Unified MCP server for ALL discovered sites
├── orchestration/
│   ├── data_flow.py    # JSONPath-like data extraction between pipeline steps
│   ├── loader.py       # YAML/JSON pipeline loading
│   └── orchestrator.py # Sequential pipeline executor with error policies
├── health/
│   ├── monitor.py      # API health checking
│   └── self_heal.py    # LLM-powered breakage repair
└── community/
    └── registry.py     # Community spec sharing

experiments/
├── experiment_8_live_validation.py   # Live validation against 5 real APIs
├── experiment_9_api_breadth.py       # Breadth test across 10 diverse APIs
├── experiment_10_unofficial_api_benchmark.py  # Coverage vs known unofficial APIs
├── experiment_11_speed_cost_benchmark.py      # Speed, cost, throughput benchmarks
├── experiment_12_mcp_validation.py   # Deep MCP server validation
├── experiment_13_spec_accuracy.py    # Spec accuracy vs ground truth
├── experiment_14_resilience.py       # Health monitoring & resilience
├── experiment_15_live_browser_validation.py  # Real Playwright browser → CDP capture pipeline
└── run_all_experiments.py            # Master runner for all experiments
```

## Conventions

- Python >=3.10, type hints everywhere
- Pydantic v2 for all data models
- async/await for I/O-bound operations
- Typer for CLI, Rich for output formatting
- SQLite for local storage (no server deps)
- ruff for linting

## Testing

```bash
pytest                    # 569 unit/integration tests (no network)
pytest -m live            # 6 live tests (hits jsonplaceholder + httpbin)
pytest -v                 # Verbose output
```

**Total: 575 tests (569 offline + 6 live; live tests require network access and are reported separately).**

**Test files:**
- `test_analyzer.py` — Traffic analysis & grouping (23 tests)
- `test_cli.py` — CLI commands via CliRunner (16 tests)
- `test_models.py` — Pydantic model validation (15 tests)
- `test_router.py` — Router execution, fallback, promotion (15 tests)
- `test_cookie_banner.py` — Cookie banner detection & dismissal (12 tests)
- `test_auth.py` — Keyring store/get, auth headers (11 tests)
- `test_integration_pipeline.py` — Full pipeline with mock data (11 tests)
- `test_registry.py` — SQLite registry CRUD (10 tests)
- `test_wait_conditions.py` — Rich wait conditions (10 tests)
- `test_detectors.py` — Auth/CAPTCHA page detection (10 tests)
- `test_tier_promotion.py` — Tier fallback & auto-promotion (9 tests)
- `test_config.py` — Config singleton, dirs, YAML save/load (8 tests)
- `test_health.py` — Health check with mock httpx (8 tests)
- `test_generated_code.py` — compile() validation (8 tests)
- `test_retry.py` — Async retry utility (8 tests)
- `test_a11y.py` — Accessibility tree extraction (8 tests)
- `test_output_filter.py` — Output filtering (grep, limit, keys-only) (8 tests)
- `test_agent_config.py` — Agent config generation (8 tests)
- `test_spec_generator.py` — OpenAPI spec generation (6 tests)
- `test_community.py` — Export/import roundtrip (6 tests)
- `test_integration_live.py` — Live API tests, marked `@pytest.mark.live` (6 tests)
- `test_client_generator.py` — Python client code gen (4 tests)
- `test_cookies.py` — Cookie CRUD, import/export, migration (23 tests)
- `test_workflow_recorder.py` — Workflow recording, parameterization, CRUD (15 tests)
- `test_mcp_server.py` — Unified MCP server, tool schemas, registry (14 tests)
- `test_profiles.py` — Chrome/Firefox profile detection, import (12 tests)
- `test_daemon.py` — Daemon server lifecycle, JSON-RPC protocol (12 tests)
- `test_session.py` — Named session persistence and reuse (10 tests)
- `test_content_converter.py` — HTML-to-markdown/text conversion, main content extraction (21 tests)
- `test_extract.py` — Schema loading, validation, extraction prompt building (26 tests)
- `test_proxy.py` — ProxyConfig: URL building, Playwright/httpx formats, auth (13 tests)
- `test_data_flow.py` — JSONPath extraction, data flow between pipeline steps (17 tests)
- `test_device_flow.py` — OAuth device code request, polling, token refresh (14 tests)
- `test_orchestrator.py` — Pipeline execution, error policies, step result tracking (12 tests)
- `test_providers.py` — OAuth provider configs (GitHub, Google, Microsoft) (8 tests)
- `test_crawl.py` — Link extraction, BFS crawler, dedup, resume, formats (35 tests)
- `test_crawl_robots.py` — robots.txt parsing, allow/disallow, sitemaps (12 tests)
- `test_monitor.py` — Diff computation, watcher, webhook, registry CRUD (41 tests)
- `test_screenshot.py` — Screenshot model, CLI help, formats (8 tests)


## Live Validation (8 Experiments)

Full pre-launch validation suite: `python experiments/run_all_experiments.py`

| # | Experiment | What It Proves |
|---|-----------|----------------|
| 8 | Live Validation | 5 APIs, full pipeline end-to-end |
| 9 | API Breadth | 10 diverse APIs (33 endpoints), 7 categories |
| 10 | Unofficial API Benchmark | 62% coverage of known APIs, 2M x faster than manual |
| 11 | Speed & Cost | 74% cheaper than browser-use, 80% time in HTTP capture |
| 12 | MCP Validation | 20 tools, 14/14 quality checks, schema-spec match |
| 13 | Spec Accuracy | 80% accuracy vs ground truth (5 APIs) |
| 14 | Resilience | 100% health check accuracy, drift detection, bundle integrity |
| 15 | Live Browser Discovery | Real Playwright browser → CDP capture → full pipeline (5 sites) |

Experiments 8-14 pass in ~74 seconds. Experiment 15 requires `site2cli[browser]` + Chromium.

## Backward Compatibility (webcli → site2cli)

The project was renamed from `webcli` to `site2cli`. These migration paths exist:
- **Data dir**: `config.py` auto-migrates `~/.webcli/` → `~/.site2cli/` on first run
- **Keyring**: `auth/manager.py` falls back to old `"webcli"` keyring service when credentials aren't found under `"site2cli"`
- **Community bundles**: `community/registry.py` accepts both `.site2cli.json` and `.webcli.json` bundle formats

## Optional Dependencies

Heavy deps are optional to keep base install lightweight:
- `site2cli[browser]` — Playwright, browser-cookie3
- `site2cli[llm]` — Anthropic SDK
- `site2cli[mcp]` — MCP Python SDK
- `site2cli[content]` — markdownify for HTML conversion
- `site2cli[rag]` — pdfplumber for PDF parsing
- `site2cli[search]` — duckduckgo-search for web search
- `site2cli[all]` — Everything (browser, llm, mcp, content, rag, search)
- `site2cli[dev]` — All + pytest, ruff, mypy

## Bug Fixes

- **client_generator.py**: Fixed Python syntax error where required params could follow optional params in generated methods. Required params are now sorted before optional ones.

## Key Docs

- `PLAN.md` — Full architecture plan, research bible, implementation phases
- `RESEARCH-EXPERIMENT.md` — Experiment records, findings, learnings & mistakes
- `RESEARCH-DEEP-DIVE.md` — Market analysis (Perplexity, WebMCP, competitive landscape)
- `CLAUDE.md` — This file; conventions and project structure

## Running

```bash
pip install -e ".[dev]"
pytest
site2cli --help
```