Recon — Network and Host Discovery

Full methodology: references/methodology.md. Follow this order; every scan is written with -oA because evidence is mandatory for the reporting skill.

When to Use

• User asks for port scan, subdomain discovery, service/version fingerprinting, or tech stack identification.
• User provides a target IP/domain with no further instruction.
• A later skill (web-app, exploitation) needs a verified surface: live hosts, open ports, service versions.

When NOT to Use

• No written authorization exists for the target. Stop at the scope gate.
• User asks to exploit a found service — that is the exploitation skill's job.
• User asks for reporting or CVSS scoring — that is the reporting skill's job.
• Passive OSINT only (emails, credentials, org structure) — that is the osint skill's job.

Rationalizations to Reject

• "Nmap finds everything, skip DNS." → Subdomains are the attack surface outside the main domain; skip them and you miss half the target.
• "Port scan is enough, version later." → Version is the input to exploit selection; deferring it makes the exploitation skill guess.
• "Target is too big, skip it." → masscan + naabu exist for this; split scope, run parallel.
• "XML output is unnecessary." → The reporting skill requires XML/grepable as evidence.
• "Quick peek, no full scan." → No. Run the full methodology or none.

Methodology

Step 0 — Scope and Authorization Gate (mandatory)

CASE=$(date +%Y%m%d)-<slug>          # e.g. 20260802-e01
mkdir -p work/$CASE/{evidence,scans,findings,report}
# MANDATORY output locations (fixed, do not change):
#   findings.json → work/$CASE/findings/findings.json
#   evidence files → work/$CASE/evidence/
#   scan outputs → work/$CASE/scans/
#   scope.md → work/$CASE/scope.md with:
#     target: <DOMAIN_OR_IP>
#     auth: status: granted | basis: <signed scope / bug bounty / own lab> | evidence_of_auth: <file>
#     date: YYYYMMDD
# When the run ends, also copy findings.json to the run root (benchmark archive).

• Follow RULES.md: no written authorization (signed scope, bug bounty program, own lab) → do NOT run any scan. Report: "No authorization, no action."

Step 0.5 — Tool Check and Fallbacks (MANDATORY)

1. Check every tool you plan to use first: command -v <tool> || echo MISSING.
2. Missing tool → use its fallback from the common table below. Fallback also missing → do the same job with python3 stdlib (socket, urllib, http.server).
3. Note the fallback you used in findings/evidence (tool: <fallback>).
4. Add the missing-tool list to the report (Appendix C: tool versions — "missing: X, used: Y").

A missing tool NEVER stops the skill — warn, use the fallback, continue.

Step 1 — DNS Resolution and Subdomain Collection

subfinder -d example.com -all -silent -o work/$(date +%Y%m%d)-case/subdomains.txt
curl -s "https://crt.sh/?q=%25.example.com&output=json" | python3 -c "
import json,sys
for c in json.load(sys.stdin):
    for n in c['name_value'].split('\n'):
        print(n.strip().lstrip('*.'))" | sort -u >> work/$(date +%Y%m%d)-case/subdomains.txt
dnsrecon -d example.com -t axfr
cat work/$(date +%Y%m%d)-case/subdomains.txt | sort -u | while read d; do host "$d" | grep "has address" | awk '{print $NF" "$1}'; done > work/$(date +%Y%m%d)-case/live-hosts.txt

• subfinder -all -silent: passive subdomain discovery across sources, clean output.
• crt.sh: certificate transparency logs, no API key needed.
• dnsrecon -t axfr: zone transfer attempt — rarely succeeds, costs 5 seconds.
• Expected output: subdomains.txt (unique names), live-hosts.txt (IP + hostname pairs).

# Wildcard detection (prevents false positives): if a random name resolves, wildcard DNS exists
dig +short random-$(date +%s).example.com A | head -3
# If wildcard → drop the wildcard IPs from the subdomain list, keep only unique IPs.

# Zone transfer fallback (when dnsrecon is missing):
for ns in $(dig NS example.com +short); do dig axfr @$ns example.com +noall +answer; done

# Subdomain brute force (only if passive collection ran dry; rate limit: 10 req/s):
ffuf -w /usr/share/seclists/Discovery/DNS/subdomains-top1million-5000.txt \
  -u http://example.com -H "Host: FUZZ.example.com" -fs <default-size> -rate 10 \
  -o work/$CASE/scans/ffuf-subdomains.json
# ffuf missing → python3 socket.getaddrinfo loop (same wordlist, sleep 0.1)

# Reverse DNS (IP → hostname, for the service map):
nmap -sL <subnet> 2>/dev/null | grep -i "example.com"

• Zone transfer almost never succeeds but costs seconds — always try every NS.
• Subdomain brute is active DNS traffic: keep -rate 10 and only after passive sources are exhausted.

Step 2 — Wide-Range Port Scan (masscan, broad targets only)

masscan -p1-65535 --rate=1000 -iL work/$(date +%Y%m%d)-case/live-hosts.txt \
  --exclude 10.0.0.0/8,172.16.0.0/12,192.168.0.0/16 \
  -oJ work/$(date +%Y%m%d)-case/masscan.json

• --rate=1000: 1000 packets/sec, network-respectful; 10k+ is overly aggressive (IDS alarm, client loss).
• --exclude: cut out-of-scope private ranges — legal requirement.
• Expected output: masscan.json with IP → open port list. Single host instead? Go straight to Step 3.

Step 2.5 — Rate Limits and Backoff

Active scans generate traffic the target/IDS/WAF can throttle — plan limits BEFORE each tool:

Backoff strategy: 429/reset → wait 60s → halve the rate → retry once; second block → STOP that vector and note "rate-limited, skipped" in findings (never barrel through — IP ban / WAF block mode).

Step 3 — Deep nmap Scan (primary discovery)

nmap -sV -sC -Pn -T4 -p $(cat work/$(date +%Y%m%d)-case/open-ports.txt | tr -d '\n') \
  -iL work/$(date +%Y%m%d)-case/live-hosts.txt -oA work/$(date +%Y%m%d)-case/nmap-full

• -sV: service version detection — exploit selection input.
• -sC: default NSE scripts. Do NOT add --script=vuln here; vuln scripts belong to the exploitation phase (false-positive storm otherwise).
• -Pn: skip host discovery — ping-blocking is common; without it hosts look "down".
• -T4: timing 1-5, 4 = aggressive but stable.
• Large target (100+ hosts) → -T3, --max-retries 1, --host-timeout 10m (IDS/rate-limit safety).
• -oA: nmap + grepable + XML output — evidence is MANDATORY.
• Expected output: nmap-full.xml/.nmap/.gnmap with open ports, services, versions.

Step 4 — Web Service and Technology Detection

whatweb -iL work/$(date +%Y%m%d)-case/live-hosts.txt --log-json=work/$(date +%Y%m%d)-case/whatweb.json
cat work/$(date +%Y%m%d)-case/live-hosts.txt | httpx -silent -title -tech-detect -status-code -json -o work/$(date +%Y%m%d)-case/httpx.json

• whatweb: CMS/framework/tech fingerprinting over HTTP.
• httpx -json: probe live web services, JSON output for agent pipelines.
• Expected output: tech stack per web host (e.g. nginx 1.24.0, PHP 8.2).

Edge Cases (active fingerprinting)

1. CDN/WAF behind real IP:
   • shodan search "hostname:<domain>" --fields ip_str,port,org — historical exposure
   • crt.sh CT IP hunt: resolve the name_value subdomains and look for origin IPs
   • DNS history / SecurityTrails-style passive sources (if API available)
2. Vhost discovery (multiple sites on one IP):

   ffuf -w /usr/share/seclists/Discovery/DNS/subdomains-top1million-5000.txt \
     -u http://<IP> -H "Host: FUZZ.<domain>" -fs <default-size> -rate 10

3. IPv6: nmap -6 -sV -Pn <target> (skip only if IPv4 surface is clean and IPv6 is out of scope).
4. Port 0 / high ports: nmap -p 0,1,65535-65536 <target> (liveness check).
5. Ambiguous service version: nmap -sV --version-intensity 9 on the single port.

Step 5 — UDP Scan (optional, after TCP is done)

nmap -sU -sV -Pn --top-ports 100 -iL work/$(date +%Y%m%d)-case/live-hosts.txt -oA work/$(date +%Y%m%d)-case/nmap-udp

• DNS (53), SNMP (161), NTP (123) live on UDP only; skipping UDP misses them.
• Expected output: nmap-udp.* with open UDP services.

# Verify each open UDP service (every check goes to an evidence file):
# SNMP (161) open:
snmpwalk -v2c -c public <IP> 2>&1 | head -20        # try public/private community strings
nmap -sU -p 161 --script snmp-brute <IP>            # script the service found in top-100
# NTP (123) open:
nmap -sU -p 123 --script ntp-monlist <IP>           # mode 6 monlist = info leak
# DNS (53) open recursion:
dig +short @<IP> example.com A                      # resolves an external name → open resolver

• UDP open is not a vulnerability by itself: the verification above decides (weak SNMP community, NTP monlist, open recursion are exploitable).

Step 6 — Normalize to Findings

python3 skills/recon/scripts/normalize_recon.py work/$(date +%Y%m%d)-case/nmap-full.xml --target example.com > work/$(date +%Y%m%d)-case/findings.json

• Every open port becomes a finding; version info goes into description; CVE IDs in output become references.
• Expected output: findings.json conforming to schemas/findings.schema.json.

Output Schema

CONTRACT (mandatory): findings.json is a BARE JSON ARRAY at the root — never wrap it in an object like {"findings": [...]}. Wrong: {"findings": [{...}]} → validator rejects (exit 1) Right: [{...}, {...}] → validator accepts Each finding is ONE finding object from schemas/findings.schema.json.

Severity rules:

• "Unscored" → vector "" (EMPTY) + baseScore 0.0. Filled vector + Unscored = validator error.
• Scored finding → vector "CVSS:4.0/..." + baseScore + severity band (Low/Medium/High/Critical).
• Unscored when: asset inventory (open port, subdomain, banner), decoy/false-positive, impact not yet determined.

Scoring contract:

1. If the impact is clear at discovery (e.g. verified SQLi → auth bypass): score immediately with CVSS 4.0 (vector + baseScore + severity).
2. If the impact is unclear or it is asset inventory (open port, subdomain, banner, decoy): Unscored (empty vector + 0.0).
3. The reporting skill scores the remaining Unscored findings at merge; already scored findings are left untouched (highest severity kept).
4. Either way the validator rule holds: Unscored = empty vector + 0.0.

After writing, MANDATORY: python3 skills/reporting/scripts/validate_findings.py work/$CASE/findings/findings.json (exit 0 required).

Every finding follows schemas/findings.schema.json (draft-07). Key fields:

{
  "id": "RECON-001",
  "skill": "recon",
  "target": "93.184.216.34",
  "title": "Open port: 80/tcp (http)",
  "description": "Open tcp port 80 on 93.184.216.34. Service: http — version: nginx 1.24.0",
  "severity": {"cvss40": {"vector": "", "baseScore": 0.0, "severity": "Unscored"}},
  "evidence": [{"file": "nmap-full.xml", "hash": "<sha256>", "note": "open port reported in raw scan output"}],
  "remediation": "Restrict exposure to this port or disable the service if not required.",
  "detected_at": "2026-08-02T10:00:00+00:00",
  "status": "confirmed"
}

• Recon findings are Unscored by default (asset inventory); score immediately only when the impact is verified at discovery — see the scoring contract above.

Pitfalls

• Skipping -Pn → hosts appear down, empty results. Always -Pn.
• masscan at 100k pps → network outage, IDS alarm, lost client. Start --rate=1000.
• Scanning private/out-of-scope IPs → legal violation. --exclude + scope.md check.
• No UDP pass → SNMP/NTP/DNS missed.
• No -oA → no evidence, no report possible.
• -sC --script=vuln on the first pass → hours of runtime and false positives. Keep vuln scripts for exploitation.
• Reporting raw nmap output as the result → unreadable. Normalize to the schema (Step 6).
• -T4 on every target → IDS alarm + rate-limit bans on big/IDS-heavy targets. -T3 + --max-rate 500; -T4 is for small lab targets.
• Rate-limit detection: 3 consecutive requests returning 429/reset → wait 60s, drop --rate/-T, then continue.
• masscan above --rate=1000 → network outage + IDS alarm + client loss. Never exceed it.
• Ignoring 429/reset during scans → IP ban and lost coverage. Step 2.5 backoff before every active tool.

Verification

• MANDATORY: python3 skills/reporting/scripts/validate_findings.py work/$CASE/findings/findings.json → exit 0. Without exit 0 the task is NOT complete.
• findings.json exists and validates: python3 skills/reporting/scripts/validate_findings.py work/<case>/findings.json → exit 0.
• nmap-full.xml (or equivalent -oA artifact) exists as evidence for every finding.
• scope.md exists and authorization was confirmed before any scan ran.
• Rate-limited/skipped steps are noted in findings (Step 2.5) — no silent drops.
• Output includes next steps for downstream skills (web-app, exploitation, osint).