Deep Research — parallel multi-source orchestrator

🟢 What this is. A single Python script that fans out across 8 data sources in parallel and dumps quotable evidence into a research folder. Use it whenever you need cross-source convergence on a topic before writing anything substantive.

🔴 What this is NOT. It does not synthesize a brief, an opinion, or a post. The output is raw evidence for the writer to read and synthesize. Voice and structure happen downstream — never inside the orchestrator.

When to use

🟡 Not the right tool for: single-event news posts where the source is one tweet (just fetch that tweet directly), or re-research of topics already covered.

How to invoke

cd skills/deep-research/scripts
python3 research.py "your topic in quotes" [flags]

Common flag combinations:

# Standard pre-flight (default depth, 30 days, ~$0.10-0.20/run)
python3 research.py "agent harness engineering"

# Quick scan (~30s, smaller limits)
python3 research.py "Stripe Sessions 2026 announcements" --depth=quick

# Deep dive (~3-4 min, deeper per-source limits, more YT transcripts)
python3 research.py "openai vs anthropic enterprise 2026" --depth=deep --yt-transcripts=4

# Specific sources only (skip the ones you know won't have signal)
python3 research.py "Mitchell Hashimoto" --sources=x,github_repos,github_issues,hn

# Short window for news-shape topics
python3 research.py "GPT-5.5 launch" --days=7

# Custom output directory
python3 research.py "claude code skills" --out=/path/to/research/claude-code-skills/

Output

🟢 Default output: ./research/<slug>/research-<date>.{json,md} (relative to current working directory).

🟢 Override with --out=<dir> for project-specific paths.

The markdown groups results by source (X, Reddit, HN, GitHub repos, GitHub issues, Polymarket, YouTube, Exa) and within each section sorts by engagement signal. Each entry has the URL, engagement counts, author, date, and a quoted excerpt.

The 8 sources, what they give you

🟢 Total per-run cost: ~$0.10-0.20 at default depth. No monthly subscription.

🟢 EVERY source routes through monid — one auth (MONID_API_KEY), one balance, zero per-vendor keys. Paid sources (X, Reddit, YouTube, Exa) use native monid endpoints; sources with no native monid endpoint (HN Algolia, GitHub REST, Polymarket Gamma) are proxied through blockrun.ai/api/v1/exa/contents, which returns the upstream JSON body verbatim (raw bytes, no markdown cleaning) for ~$0.0022/call.

🔴 Do NOT proxy JSON through blockrun.ai/api/v1/surf/web/fetch — it runs a markdown cleaner that collapses whitespace inside JSON strings and mangles embedded URLs, producing invalid JSON on large/prose-heavy responses (GitHub issue bodies, Polymarket descriptions). Use exa/contents (raw passthrough) instead — that's what _monid.fetch_json does.

🟡 Per-source budget rules:

• Reddit costs scale with results × 0.0057 + a base scrape fee. A "deep" run can hit $0.25.
• YouTube transcripts cost $0.0075 per video — start with --yt-transcripts=2, raise to 4+ only if the topic is YouTube-heavy.
• X (tikhub) is $0.0015/page; default depth pulls 2 pages ($0.003), deep pulls 3 + a Latest sweep.
• HN/GitHub/Polymarket proxy calls are ~$0.0022 each. Polymarket Gamma is capped at limit_per_type=8 — exa livecrawl 504s on the ~1MB payload above that.

Example workflow

1. User: "do a deep dive on Greg Isenberg's agent agency podcast"
2. Read the source URL (the tweet, the YouTube link)
3. Run: python3 research.py "Greg Isenberg agent agency Hermes Orgo"
4. Wait ~60-90s for all 8 sources
5. Open research-<date>.md, scan the X + Reddit + Exa sections
6. For each high-signal post, pull the verbatim text from research-<date>.json
7. Draft your artifact with verbatim quotes lifted directly from the JSON

🔴 Critical rule: this skill produces research input, not draft output. Never copy/paste sentences from research-<date>.md into the artifact you're writing. Always pull the underlying primary source (tweet text, blog excerpt, GitHub description) from the .json and synthesize in your own voice downstream.

Voice contract

This skill must not synthesize editorial prose or take positions. Its only job is to retrieve, normalize, and rank.

If the script ever starts emitting things like "the takeaway is...", "this represents a shift...", or "the bottom line:", that's a bug — remove it. The synthesis is what humans (or downstream editorial workflows) do.

Files

Required env

🟢 One key for everything. Set MONID_API_KEY in your environment and ensure your monid balance is funded.

🟡 Optional env var: MONID_BIN — path to the monid CLI binary. Defaults to a monid lookup on $PATH. Set this if you've installed monid in a non-standard location.

🟡 Missing-auth behavior: if MONID_API_KEY is unset, sources return empty and the orchestrator logs warnings but still completes (with no data). Ensure the key + a funded monid balance before running.

Installing monid

This skill depends on the monid CLI. Install it with:

npm install -g monid
# or, if you prefer a local install:
npm install monid
export MONID_BIN="$(pwd)/node_modules/.bin/monid"

Get an API key at monid.dev and export it:

export MONID_API_KEY=your_key_here

Known gotchas

Updating this skill

When a source's input schema changes, GitHub adds a new search filter, or a monid endpoint changes — update the relevant sources/*.py and bump the Known gotchas section. Everything goes through sources/_monid.py:

• Native monid endpoint (X, Reddit, YouTube, Exa): call run_monid(provider, endpoint, body=/query=/path=, retries=) and normalize result["output"].
• No native endpoint (HN, GitHub, Polymarket — arbitrary JSON REST): call fetch_json(url) which proxies through exa/contents and returns parsed JSON. NEVER use surf/web/fetch for JSON (markdown cleaner corrupts it).

To add a source: monid discover → monid inspect the schema → write a thin sources/<name>.py using whichever helper fits. Treat this SKILL.md as a living spec.

Credits

Originally inspired by mvanhorn/last30days-skill. The 30-day default window came from that upstream; everything else (the 8-source fan-out, the monid routing, the _monid.py helpers, the per-source modules) was rebuilt from scratch.