Deep Research

Overview

Use this skill for research where claims must be traceable to sources. It is harness-neutral, but the default discovery-to-capture path is explicit: use sibling advanced-web-search/scripts/opencli_google_search.py to save candidate URLs as JSON, then use this skill's scripts/batch_capture.py --search-json to crawl them with Crawl4AI, then read the raw markdown and write the final brief.

Runtime requirement: Crawl4AI must be installed for raw capture. Install with python -m pip install -U crawl4ai, then run crawl4ai-setup and crawl4ai-doctor. See ../crawl4ai/SKILL.md.

Operating Contract

1. Prefer official, primary, and direct sources.
2. Use multiple targeted query classes, not one broad query.
3. Treat search snippets as leads, not evidence.
4. Inspect extraction quality; do not treat HTTP success as content success.
5. Separate facts, interpretation, recommendations, counter-evidence, open questions, and confidence.
6. Cite material claims with URLs or local raw/source artifact paths.
7. Label blocked, partial, old, borderline, and inferred evidence.
8. For durable research, write artifacts in the Hermes Obsidian layout, not a generic flat folder.

Workflow

Phase 0: Scope

Restate the question, decision target, constraints, time window, output format, and source classes needed. Ask a concise clarifying question only if ambiguity changes the source universe or tool calls materially.

Completion: scope is clear enough to define at least four query classes.

Phase 1: Discovery

Use advanced-web-search/scripts/opencli_google_search.py when it is available. Do not use a harness built-in web search as the primary discovery step when OpenCLI is available. Read references/query-patterns.md when constructing the search plan.

Always save discovery output to a JSON file with --output:

python ../advanced-web-search/scripts/opencli_google_search.py \
  --query "Hybrid XBRL tagging after:2026-03-01" \
  --query "site:github.com XBRL AI tagging after:2026-03-01" \
  --max-results 30 \
  --pages 8 \
  --output outputs/hybrid_xbrl_sources.json

If the user asks for a required count, set --max-results to that count or higher and verify the JSON contains the number of unique URLs requested. If fewer are available, record the shortfall and continue with the available sources.

Collect candidate URLs across:

• official and primary sources
• code repositories, issues, releases, and docs
• academic or technical papers
• community/user evidence
• risks, limitations, and counter-evidence
• recent updates
• competitive or market sources

Completion: outputs/<topic>.json exists, contains deduplicated candidate URLs from opencli_google_search.py, and candidates include primary/direct evidence and at least one counter-evidence path where relevant.

Phase 2: Source Selection

Triage each candidate by credibility, relevance, source type, date/version, and extraction status. Read references/source-quality-rubric.md when confidence matters or sources conflict.

Completion: selected sources are sufficient to answer the question, or gaps are explicit.

Phase 3: Direct Capture

Capture selected sources directly with crawl4ai whenever pages are reachable. Use browser/fetch/API fallbacks for pages Crawl4AI cannot access cleanly. Save raw excerpts or full captures when the task is large, audit-sensitive, or the user asks for artifacts.

Use the Hermes Obsidian date-root layout:

Research/
  Index/
    Research_Index.md
  YYYY-MM-DD/
    Briefs/
      Topic-Slug.md
    Raw/
      Topic-Slug/
        01_Source_Site_Heading.md
    Sources/
      Topic-Slug_Source_Index.md

Raw files must be named with stable readable IDs, for example 01_FinTagging_arxiv.md, KDA_Agentic-CUDA-Opt.md, Reddit_LocalLLaMA_Speed-Bench.md, or X_CodeZakh_GPU-LLM.md.

Phase 3 normally starts from the JSON saved in Phase 1. Feed that file directly to the bundled capture helper:

python scripts/batch_capture.py \
  --search-json outputs/hybrid_xbrl_sources.json \
  --topic Hybrid-XBRL-Tagging \
  --vault /path/to/Obsidian-vault/Hermes

The helper extracts unique url fields, preserves search metadata where available, crawls each URL with Crawl4AI, and writes raw captures, source index, capture JSON, and the research index entry — everything except the brief, which is agent-authored (Phase 5). It is cross-platform (pathlib); use python3 where that is the installed name.

Use official APIs, raw repository files, arXiv HTML/PDF, regulator pages, standards, filings, Crawl4AI captures, or browser views as needed. If a source is blocked, partial, empty, or JavaScript-only, label it and try a stronger fallback before downgrading to secondary summaries.

Completion: each key claim has direct evidence or is explicitly marked unverified/inferred.

Phase 4: Evidence Matrix

Create a compact matrix before writing conclusions:

Evidence types: primary source, official documentation, paper/preprint, code/repository evidence, regulator/filing/standard, secondary analysis, community/anecdotal, inferred.

Completion: every recommendation maps to one or more claim IDs.

Phase 5: Synthesis

Write the final artifacts before returning a summary:

1. Research index entry in Research/Index/Research_Index.md.
2. Source index in Research/YYYY-MM-DD/Sources/<Topic-Slug>_Source_Index.md.
3. Agent-authored final brief in Research/YYYY-MM-DD/Briefs/<Topic-Slug>.md.
4. Optional benchmark/process report in Research/Reports/ when the task compares methods, agents, or runtimes.
5. Optional standalone paper notes in Research/Papers/ for important papers that deserve their own note.

The brief is agent-authored: written only after reading the raw markdown captures, never assembled from capture-helper output, search snippets, source titles, or a scaffold. It should follow the Hermes narrative format: H1 title, **Research Brief | YYYY-MM-DD**, ## Source Index: [[...]], executive summary, key findings or key source briefs, claims table with confidence, consensus/dissent, gaps, and research log. Avoid link spam; put the full URL list in the source index and use raw-file links only where they support a finding.

Every key finding sourced from crawled material must include one or more - **Raw File:** [[YYYY-MM-DD/Raw/Topic-Slug/File-Stem]] lines. If a source was selected but not crawled, write - **Raw File:** (not crawled) and explain why.

The source index must group sources by type and include URL, source type, raw file wiki-link, and credibility. Include a Failed / Blocked Sources table when any source could not be captured.

Use templates/research-brief.md, templates/source-note.md, and templates/source-index.md when creating files. The research brief template is for agent-written synthesis, not for automatic capture generation.

If this skill is copied without the sibling crawl4ai skill, use scripts/batch_capture.py for a small standalone Crawl4AI-based capture helper.

Time-Bounded Research

If the user specifies a cutoff, split sources into:

• strict post-cutoff
• borderline or unclear date
• older but updated after cutoff
• older context only

Do not silently mix these categories. For ambiguous requests such as "after March 2026", state whether strict means April 1, 2026 onward or March 1, 2026 onward.

References

• references/query-patterns.md for query class examples.
• references/source-quality-rubric.md for credibility, relevance, and confidence.
• references/obsidian-layout.md for Hermes-style vault paths, filenames, and wiki-links.
• ../crawl4ai/SKILL.md for crawling, raw markdown capture, and extraction quality checks.

Quality Gate

Before finalizing, verify:

• advanced-web-search/scripts/opencli_google_search.py was used for primary discovery when available
• the search command used --output <json-file>; do not rely on stdout or shell redirection for the handoff
• the saved JSON file was passed to scripts/batch_capture.py --search-json <json-file>
• more than one query class was used unless the user explicitly requested one query
• important facts were checked from direct sources
• Crawl4AI or an equivalent crawler captured raw content for key reachable sources
• raw captures are under Research/YYYY-MM-DD/Raw/<Topic-Slug>/
• source index is under Research/YYYY-MM-DD/Sources/ and linked near the top of the brief
• the brief is agent-authored: written after reading raw markdown files, not copied from titles/snippets or generated by batch_capture.py
• every major finding has a Raw File wiki-link or explicit (not crawled) note
• primary sources were prioritized
• extraction quality was inspected
• counter-evidence or limitations were included
• confidence and evidence type were labeled
• material claims have citations
• gaps are stated instead of guessed