OSINT — Passive Only, Recon Feeds

Full methodology: references/methodology.md. Hard passive boundary: no requests to the target. OSINT first, recon second.

When to Use

• User asks for passive intelligence, dork searches, leak hunting, or target research before scanning.
• Subdomains, exposed IPs, technologies, emails, or leaked credentials need to feed recon.

When NOT to Use

• Active scanning or host probing — that is the recon skill's job.
• Testing a found credential — that is active testing and needs separate authorization.
• Reporting or scoring — that is the reporting skill's job.

Rationalizations to Reject

• "OSINT is unnecessary, just nmap it." → OSINT understands the target (tech, leaks, subdomains) and steers nmap; leaks are invisible to nmap.
• "Google dorks are old school." → GitHub dorks + shodan CVE matching are still the highest-yield passive vectors.
• "Shodan shows the port, no nmap needed." → Shodan is history, not current state; active verification is mandatory.
• "I'll test the credential I found right away." → Testing a credential is an active action; it needs separate authorization.

Methodology

Step 0 — Ethics Boundary

CASE=$(date +%Y%m%d)-<slug>          # e.g. 20260802-e01
mkdir -p work/$CASE/{evidence,scans,findings,report}
# MANDATORY output locations (fixed, do not change):
#   findings.json → work/$CASE/findings/findings.json
#   evidence files → work/$CASE/evidence/
#   scan outputs → work/$CASE/scans/
#   scope.md → work/$CASE/scope.md with:
#     target: <DOMAIN_OR_IP>
#     auth: status: granted | basis: <signed scope / bug bounty / own lab> | evidence_of_auth: <file>
#     date: YYYYMMDD
# When the run ends, also copy findings.json to the run root (benchmark archive).

• OSINT is passive but still governed by target policy, privacy laws (GDPR/KVKK), and scope. Note "public info" in the scope file.
• Google dorks and shodan leave your own network — no request reaches the target.

Step 0.5 — Rate Limits and Service Quotas

1. Know the limit of every external service before you use it:

2. On 429/403 → wait 60s → continue; second limit hit → SKIP the step and note "rate-limited, skipped" in the finding.
3. Backoff strategy: 429 → wait 60s → retry once → second block → wait 300s or skip the step. Never hammer through a quota — account/IP bans follow.
4. Rule: a rate-limited skipped step is REPORTED (scope confession), never silently dropped.

Step 0.6 — Tool Check and Fallbacks (MANDATORY)

1. Check every tool you plan to use first: command -v <tool> || echo MISSING.
2. Missing tool → use its fallback from the common table below. Fallback also missing → do the same job with python3 stdlib (socket, urllib, http.server).
3. Note the fallback you used in findings/evidence (tool: <fallback>).
4. Add the missing-tool list to the report (Appendix C: tool versions — "missing: X, used: Y").

A missing tool NEVER stops the skill — warn, use the fallback, continue.

Step 1 — Domain Basics

whois example.com | grep -iE "registrant|creation|expiry|name server" | head -10
dig example.com ANY +noall +answer 2>/dev/null | head -20
dig MX example.com +short && dig TXT example.com +short   # SPF/DMARC = mail infra clues

Step 2 — Subdomains + Certificate Logs (passive, primary)

curl -s "https://crt.sh/?q=%25.example.com&output=json" | python3 -c "
import json,sys
seen=set()
for c in json.load(sys.stdin):
    for n in c['name_value'].split('\n'):
        n=n.strip().lstrip('*.')
        if n not in seen: seen.add(n); print(n)" | sort > work/$(date +%Y%m%d)-case/osint-subdomains.txt
subfinder -d example.com -silent >> work/$(date +%Y%m%d)-case/osint-subdomains.txt
sort -u work/$(date +%Y%m%d)-case/osint-subdomains.txt -o work/$(date +%Y%m%d)-case/osint-subdomains.txt

• crt.sh first (no API key, JSON); subfinder for passive depth. One request + wait — crt.sh rate limits with 429.

Step 3 — Google/GitHub Dorks (highest yield)

# Dirs: site:example.com intitle:"index of"
# Files: site:example.com ext:env OR ext:sql OR ext:bak OR ext:log
# Login:  site:example.com inurl:admin OR inurl:login OR inurl:dashboard
# Leaks:  site:example.com "password" OR "api_key" OR "BEGIN RSA PRIVATE KEY"
# GitHub: "example.com" filename:.env  /  "example.com" password

• Run dorks in a browser; API automation trips Google rate limits and gets the IP banned.
• A real GitHub leak (public repo) is a critical finding — report it, do NOT test it (separate authorization).

# Extra dork categories:
# Cloud: site:s3.amazonaws.com "example.com" | site:blob.core.windows.net "example.com" | site:storage.googleapis.com "example.com"
# Paste: site:pastebin.com "example.com" | site:gist.github.com "example.com" password
# GitLab: site:gitlab.com "example.com" | site:github.com "example.com" ".env"
# Container registry: hub.docker.com org search for "example.com" (pullable image → config.json env)
# Packages: pypi.org/project/ candidates — "example.com"-like SDK packages (typosquatting detection)
# Internal tool leaks: site:jira.example.com | site:confluence.example.com (public pre-SSO)
# Finding: each leak category = its own OSINT finding (CWE-200) — evidence: URL + date + dump/screenshot.

# GitHub dork rules:
# - Code search requires a token (401 anonymous): curl -s -H "Authorization: token <TOKEN>" \
#     "https://api.github.com/search/code?q=<query>"
# - No token → manual github.com search only.

Step 3.5 — Leak Triage

1. Collect leak evidence: URL, repo name + owner, file path, commit SHA, date (created/updated), visibility.
2. Validity check: a. Repo still public? (deleted → verify via Wayback) b. Secret format: api_key (length/prefix), private key (BEGIN ... PRIVATE KEY), token (ghp_, xoxb-, sk-) c. Age: 2+ years old + archived repo → "probably rotated" note (cannot be verified — do NOT test)
3. Finding: OSINT finding + CWE-522/CWE-798; severity High if it looks live, Low if old.
4. RULE: never TEST the leak (credential use = active test = separate authorization). Report "verification requires separate authorization".

Step 4 — IP/Infrastructure OSINT

for d in $(cat work/$(date +%Y%m%d)-case/osint-subdomains.txt); do
  host "$d" 2>/dev/null | grep "has address" | awk '{print $NF}'
done | sort -u > work/$(date +%Y%m%d)-case/osint-ips.txt
shodan init <API_KEY>
while read ip; do shodan host "$ip" --fields ip_str,port,org,product,cpe --separator " " 2>/dev/null; done \
  < work/$(date +%Y%m%d)-case/osint-ips.txt > work/$(date +%Y%m%d)-case/osint-shodan.txt
curl -s "https://web.archive.org/cdx/search/cdx?url=example.com/*&output=json&fl=original&collapse=urlkey" \
  | python3 -c "
import json,sys
for row in json.load(sys.stdin)[1:]: print(row[0])" | grep -iE "admin|api|backup|config|test|dev|\.env|\.sql" | head -50

• Free shodan tier = 100 queries. Spend on the most critical IPs first.
• Archived endpoints on the same domain stay in scope — report them, do not silently drop them.

# Archive content fetch (page no longer live):
curl -s "http://archive.org/wayback/available?url=example.com/admin"   # find the snapshot URL
# Parameter mining (feeds web-app):
curl -s "https://web.archive.org/cdx/search/cdx?url=example.com/*&output=json&fl=original&collapse=urlkey" \
  | python3 -c "
import json,sys,re
for row in json.load(sys.stdin)[1:]:
    for m in re.findall(r'[?&]([a-zA-Z_][a-zA-Z0-9_]*)=', row[0]):
        print(m)" | sort -u > work/$CASE/params.txt
# Rule: params.txt passes to the web-app skill (state it in next_steps).

Step 5 — Email/Username Discovery

theHarvester -d example.com -b crtsh,baidu -l 200 -f work/$(date +%Y%m%d)-case/osint-emails.html
sherlock "username" --timeout 5 --print-found

• Key-less sources only (crtsh,baidu); the rest error out with missing API keys.
• Do NOT test found credentials — that is active testing with its own permission gate.

Step 6 — Normalize to Findings

python3 skills/osint/scripts/normalize_osint.py work/$(date +%Y%m%d)-case/osint-subdomains.txt \
  --target example.com > work/$(date +%Y%m%d)-case/findings.json

• The normalize script parses subfinder JSONL, crt.sh JSON, or a plain subdomain list → one asset finding per subdomain.

Output Schema

CONTRACT (mandatory): findings.json is a BARE JSON ARRAY at the root — never wrap it in an object like {"findings": [...]}. Wrong: {"findings": [{...}]} → validator rejects (exit 1) Right: [{...}, {...}] → validator accepts Each finding is ONE finding object from schemas/findings.schema.json.

Severity rules:

• "Unscored" → vector "" (EMPTY) + baseScore 0.0. Filled vector + Unscored = validator error.
• Scored finding → vector "CVSS:4.0/..." + baseScore + severity band (Low/Medium/High/Critical).
• Unscored when: asset inventory (open port, subdomain, banner), decoy/false-positive, impact not yet determined.

Scoring contract:

1. If the impact is clear at discovery (e.g. verified SQLi → auth bypass): score immediately with CVSS 4.0 (vector + baseScore + severity).
2. If the impact is unclear or it is asset inventory (open port, subdomain, banner, decoy): Unscored (empty vector + 0.0).
3. The reporting skill scores the remaining Unscored findings at merge; already scored findings are left untouched (highest severity kept).
4. Either way the validator rule holds: Unscored = empty vector + 0.0.

After writing, MANDATORY: python3 skills/reporting/scripts/validate_findings.py work/$CASE/findings/findings.json (exit 0 required).

Every finding follows schemas/findings.schema.json (draft-07). Key fields:

{
  "id": "OSINT-001",
  "skill": "osint",
  "target": "dev.example.com",
  "title": "Subdomain discovered: dev.example.com",
  "description": "Passive OSINT asset discovery (sources: crt.sh). Active verification belongs to recon.",
  "severity": {"cvss40": {"vector": "", "baseScore": 0.0, "severity": "Unscored"}},
  "evidence": [{"file": "osint-subdomains.txt", "hash": "<sha256>", "note": "subdomain listed in passive OSINT output"}],
  "remediation": "Not a vulnerability — asset inventory. Verify exposure with recon; remove orphaned subdomains from DNS.",
  "detected_at": "2026-08-02T10:00:00+00:00",
  "status": "confirmed"
}

• Findings stay Unscored (passive asset inventory) unless the impact is verified — see the scoring contract above.

Pitfalls

• Active tools (nmap etc.) during OSINT → passive boundary broken, trace left. OSINT = passive only.
• Automating dorks via API → Google rate limit, IP ban. Browser, manual.
• Ignoring the shodan free limit → 100 queries gone, work half-done. Critical IPs first.
• Enabling all theHarvester sources → API key errors, empty results. Key-less sources only.
• Exploiting a GitHub leak immediately → unapproved active testing. Report it; verify separately.
• No privacy boundary on personal data → GDPR/KVKK violation. "Public info" note; collect the minimum.
• Ignoring crt.sh rate limits → 429, empty list. One request + wait.
• Treating archived endpoints as out of scope → attack surface lost. Same domain = in scope, report it.

Verification

• MANDATORY: python3 skills/reporting/scripts/validate_findings.py work/$CASE/findings/findings.json → exit 0. Without exit 0 the task is NOT complete.
• findings.json validates: python3 skills/reporting/scripts/validate_findings.py work/<case>/findings.json → exit 0.
• No request was sent to the target (passive-only check of the case folder).
• Subdomain list is deduplicated and wildcards stripped.
• Output includes next steps for downstream skills (recon, web-app for leak verification).