Web Doc Resolver Skill

Resolve URLs and queries into compact, LLM-ready markdown using a progressive, free-first cascade with 2026 production patterns.

When to Use

• Fetch and parse documentation from a URL
• Search for technical information across the web
• Build context from web sources for AI agents
• Validate content against agent-docs-spec v0.3.0
• Trace resolution trajectories for debugging

Prerequisites

• Python 3.10+ in an isolated virtual environment.
• From the skill directory, run python -m pip install . to install the base package and dependencies declared in pyproject.toml before the quick start.
• Install provider, ML, or database extras only for the features that need them; text resolution does not require the optional visual/ML stack.

Quick Start

python -m scripts.cli "https://docs.rust-lang.org/book/"
python -m scripts.cli "Rust async programming" --profile quality --json

Cascade Strategy

URL: Cache → llms.txt → Jina → Direct Fetch → Firecrawl → Mistral Browser → DuckDuckGo

Query: Cache → Exa MCP → Exa SDK → Tavily → Serper → DuckDuckGo → Mistral Web Search

Providers (12 total)

Execution Profiles

Key Features

Structured Tool Contracts: Every provider returns ProviderResult(ok, content, error, meta) with ProviderMeta(tool, duration_ms, cache_hit, quality_score, error_type).

Layered Routing Memory: TTL-based decay (30 days), metadata filtering, recency weighting. See scripts/routing_memory.py.

Structured Resolution Evidence: CLI --json reports available source, quality, and provider metrics. Trace model types live in scripts/models.py; the CLI has no --trace flag.

Agent-Friendly Docs Validation: Validates against agent-docs-spec v0.3.0 — 10 checks across 7 categories.

Quality Scoring: Content scored 0.0-1.0. Penalties: too_short(-0.35), missing_links(-0.15), duplicate_heavy(-0.25), noisy(-0.20).

Circuit Breakers: 3 failures → 300s cooldown per provider.

Configuration

export EXA_API_KEY="" TAVILY_API_KEY="" FIRECRAWL_API_KEY="" MISTRAL_API_KEY=""
export WEB_RESOLVER_MAX_CHARS=8000 WEB_RESOLVER_MIN_CHARS=200

Output

{"source": "jina", "content": "...", "score": 0.87,
 "metrics": {"total_latency_ms": 1234, "provider_metrics": [...], "paid_usage": False},
 "trace": {"trace_id": "...", "steps": [...], "success": True}}  # when --trace

File Structure

scripts/
├── resolve.py           # Main orchestrator & CLI
├── models.py            # Data models, enums, trace types
├── providers_impl.py    # 12 provider implementations
├── quality.py           # Content quality scoring
├── docs_validation.py   # Agent-docs-spec v0.3.0 checks
├── routing_memory.py    # Layered memory with TTL decay
├── routing.py           # Budget-aware routing
├── circuit_breaker.py   # Circuit breaker patterns
├── cache_negative.py    # Negative cache
├── synthesis.py         # LLM synthesis gate
└── utils.py             # Utilities

See Also

• web-search-researcher — Web search and research
• agent-browser — Browser automation

Rationalizations

Red Flags

• Ignoring quality scores and accepting low-quality content without validation
• Disabling circuit breakers or skipping cooldown periods during provider failures
• Not setting max character limits — fetching unbounded content wastes tokens
• A host-validation change landing in utils/http.py with no matching utils/async_http.py change and no async test case
• Adding another IP encoding to _normalize_host instead of rejecting non-canonical hosts and checking the resolved address
• A resolution failure reaching the allow return, or an IPv6 literal embedding a v4 address (::/96, 64:ff9b::/96, 2002::/16) judged only against IPv4 networks

References

Voice & Context

• Default: professional + blog
• Reference: voice-profiles skill for definitions and auto-detection.