Advanced Web Search

Overview

Use this skill to discover high-quality candidate sources. Use the bundled OpenCLI Google backend by default for repeatable Google-operator discovery and saved JSON output. Other search, browser, HTTP, API, terminal, or MCP tools are fallbacks or verification aids. Search results are discovery leads, not evidence.

Core Workflow

1. Clarify the search target, constraints, freshness needs, and source classes.
2. Build several targeted query classes instead of one broad query.
3. Use search operators where supported: site:, filetype:, after:, before:, quoted phrases, boolean OR, exclusions, intitle:, inurl:, and intext:.
4. Run broad, primary-source, technical/paper/code, community, risk/counter-evidence, and recent-update queries as relevant.
5. Triage candidate URLs by source type, credibility, relevance, date, and extraction feasibility.
6. Hand off the selected source list to direct fetching, crawling, browser inspection, or the deep-research skill if synthesis is needed.

Required Search Tool

For repeatable source discovery, use the OpenCLI Google backend by default. Do not use a harness built-in web search as the primary discovery step for deep research when this script is available. Do not rely on terminal JSON redirection; always pass --output <path> so the result file is UTF-8 JSON and can be handed to deep-research.

python scripts/opencli_google_search.py \
  --query "Hermes Agent NousResearch site:github.com" \
  --max-results 20 \
  --pages 6 \
  --output outputs/hermes_agent_sources.json

Files:

• scripts/opencli_google_search.py - default Advanced Web Search backend. It uses OpenCLI's browser bridge to render Google in Chrome, supports Google operators and start= pagination, deduplicates canonical URLs, and writes clean JSON for any website with best-effort site metadata for common social, news, code, official, and research domains.
• scripts/opencli_network_probe.py - optional: scores likely JSON/list endpoints behind a dynamic page. Usage in references/dynamic-page-loading.md.
• scripts/opencli_api_paginate.py - optional: paginates a probe-found JSON endpoint in Chrome with cookies. Usage in references/dynamic-page-loading.md.
• scripts/google_opencli_x_search.py - Google result pagination through OpenCLI's logged-in Chrome bridge for X/Twitter result discovery. Use when raw Google HTML is blocked or when you need clean X post JSON with canonical URL, title, approximate likes, and X snowflake timestamp.
• scripts/operator_registry.py - operator parsing, validation, and query construction.
• scripts/test_opencli_google_search.py - offline tests for the default OpenCLI Google backend.

The OpenCLI scripts require Python, Node.js, @jackwener/opencli, the OpenCLI Chrome extension, and a passing opencli doctor.

Use Google operators directly in the query:

python scripts/opencli_google_search.py \
  --query "hermes agent after:2026-07-04 site:x.com" \
  --max-results 20 \
  --pages 6 \
  --after 2026-07-04 \
  --strict-dates \
  --output outputs/hermes_agent_x.json

For multi-query discovery, pass repeated --query values or a --query-file; --max-results is a global unique-link cap and the script stops as soon as that cap is reached. The OpenCLI Google helpers request 20 Google results per page by default (--num 20) while advancing Google offsets by 10 by default (--page-stride 10: start=0, start=10, start=20). They stop the current query immediately if Google renders a no-results page. When Google operators are too loose, use local URL filters such as --require-url-regex "/(?:status|article)/\\d+" for X post-only result sets. Exit codes: 0 all queries succeeded, 2 Google served a block/challenge page, 3 some queries failed on tool errors. On 2 and 3 the --output JSON is still written with results collected so far; check stderr and record the shortfall.

If OpenCLI is unavailable, say so explicitly and fix/install OpenCLI when possible. Use other harness search tools only as a fallback or for verification, and label that deviation.

Query Classes

• Official/primary: site:<official-domain> <topic> docs, <product> release notes, <standard> official guidance
• Code/project: site:github.com <topic> README OR docs, <project> issues limitations
• Academic/technical: <topic> paper benchmark evaluation, <topic> arxiv, <topic> methodology dataset
• Community/user evidence: site:reddit.com <topic> experience, site:news.ycombinator.com <topic>, <topic> forum workaround
• Risks/counter-evidence: <topic> limitations, <topic> criticism, <topic> failure cases, <topic> not working
• Recent updates: <topic> after:YYYY-MM-DD, <topic> changelog <year>, <topic> release notes <year>
• Competitive/market: <product> alternatives, <vendor> pricing, <market> vendors comparison

Source Triage

For each useful result, record:

• URL and title
• Source type: official docs, primary data, code repository, paper, standard/regulator, filing, community, news, secondary analysis, SEO/low-quality
• Credibility: high, medium, or low
• Relevance: high, medium, or low
• Date or version signal if freshness matters
• Extraction status: not fetched, fetched, partial, blocked, or low-content
• Reason for inclusion or exclusion

Prefer official, primary, and direct sources. Use SEO pages and AI-generated summaries only as leads.

Direct Verification Handoff

After discovery, verify material facts from direct sources whenever possible:

• GitHub raw files: https://raw.githubusercontent.com/<org>/<repo>/<branch>/<path>
• GitHub releases, issues, commits, and API endpoints for software behavior and dates
• Hugging Face model/dataset APIs for model metadata
• arXiv abstract, HTML, and PDF pages for papers
• Regulator, standards, filing, license, pricing, and official documentation pages

If an important source is blocked or JavaScript-only, try an alternate endpoint, browser view, official API, or reader mirror. Label blocked or partial sources clearly.

Deep Research Handoff

For deep research, the normal pipeline is:

1. Run scripts/opencli_google_search.py with precise Google operators and --output outputs/<topic>.json.
2. Confirm the JSON file exists and contains the requested number of unique candidate URLs, or record the shortfall.
3. Pass that JSON to deep-research/scripts/batch_capture.py --search-json <json>.
4. Read the captured raw markdown files.
5. Write the final Hermes brief from the raw evidence.

Do not skip the saved JSON handoff when the user asks for deep research or bulk links.

References

Read these only when needed:

• references/operator-guide.md for operator syntax and examples.
• references/opencli-google-backend.md for Advanced Web Search 2.0 setup, JSON schema, pagination, site metadata, and limitations.
• references/dynamic-page-loading.md for the fastest safe way to collect full result lists from APIs, paginated pages, or infinite-scroll pages.

Output

For standalone use, return a compact search packet:

## Search strategy
- ...

## Candidate sources
| Source | URL | Type | Credibility | Relevance | Status | Notes |
|---|---|---|---|---|---|---|

## Recommended verification targets
- ...

## Queries run
- ...