Discover agent notes

Conventions agreed with the operator for Discover development. Read and follow this file. When we discuss and agree on a durable rule (layout, install order, Update UX, host-scan gating, etc.), add or update it here in the same change set when practical—do not leave it only in chat history.

Writing (operator-facing text)

• Never use three periods in a row (...) at the end of a sentence. End with a single period (.).
• Same for Unicode ellipsis (…) in operator-facing text (logs, UI status lines, scan output.txt, help copy): use a single . when finishing a sentence. Do not write “Running…”, “Updating…”, or “pre-check (15s)…”.
• Mid-list “and more” truncation in docs/comments is fine when not sentence-final (e.g. 2xx,301,302,... as a list tail). Prefer plain ASCII over fancy punctuation in plain-text scan logs.

Shell (if)

• Blank line before and after an if / fi block (if / elif / else / fi is one block).
• Skip the blank line before if when a comment sits immediately above it. Keep a blank line after fi.
• Nested if: keep it tight against the parent then / else / elif and the parent fi. No blank line between then and the inner if, and none between the inner fi and the outer fi / else / elif:

    if ! apt install -y arp-scan; then
        if grep -qi '^ID=ubuntu' /etc/os-release; then
            apt install -y arp-scan/questing
        fi
    fi

• Sibling if blocks (an inner fi followed by another if at the same level) still get a blank line between them.

Git commits

• Only commit when the operator explicitly asks (e.g. “do a commit”). Do not commit proactively after finishing a feature or when they only ask whether a commit is needed.

old/ folder

• old/ is not part of the Discover framework. It holds personal/legacy scripts from earlier work that are unrelated to Discover’s recon/scan/report flow.
• Do not put new Discover features, helpers, or data files under old/.
• Live subdomain categorizer + rules live under recon/ (recon/subdomain-categorize.py, recon/subdomain-categories.tsv). Do not reintroduce Discover paths that point at old/.
• Exact patterns match a whole hostname label or one hyphen-separated piece of a label. dns matches dns.example.com and aj-dns-02.example.com. First match still wins, so Dev is checked before Infrastructure. The same matcher is inlined in recon/import-subdomains.sh.
• A star suffix ignores trailing digits. *dev matches almdev.example.com and almdev2.example.com. development.example.com stays on the exact development rule. device and devops do not match.

Operator name (audit log)

• First name only, max 10 letters only, stored at ~/.discover/operator-name.
• Prompted once when Discover starts if missing/invalid (f_ensure_operator_name in discover.sh).
• Audit lines: mm/dd/yyyy - hh:mm Z | <name> | <egress IP> | <action> (f_audit_log / host-scan f_audit). Dash-date stamps (mm-dd-yyyy - hh:mm Z), legacy (mm-dd-yyyy Z - hh:mm), and 3-field lines still parse on the Audit page.
• Imported subdomains / Imported CSV list subdomains, Imported names (incl. titles/emails), and Imported operator package: Operator IP is always a dash (- in the log, — on the Audit page). Do not record egress IP for those events. Ran Shodan enrichment, Updated Shodan data for an IP, and Updated software CVE data are not written to the audit log.

Timestamps (UTC write, view timezone)

• Always record timestamps in UTC when a scan runs or any engagement event is written (audit log, host-scan meta started/finished / *_utc, status stamps, export exported_at_utc, etc.). Human stamps are mm/dd/yyyy - hh:mm Z; ISO fields stay YYYY-MM-DDTHH:MM:SSZ.
• Operators may choose a timezone for viewing (e.g. Audit Config → Time zone). That preference is display only for Audit/metrics.
• View timezone must not change how data is written — no local-time stamps in tools/audit/log.txt, meta.json, or other on-disk artifacts. Convert only when rendering the report UI (and label the column, e.g. Time (UTC) vs local).
• Export filename is the exception: YYYYMMDD-HHMM in domain-kind-YYYYMMDD-HHMM.zip (or .csv) uses the operator view timezone (~/.discover/timezone). Ledger exported_at_utc and the audit line stay UTC.
• Preference store: ~/.discover/timezone (not inside a single engagement report), so it applies across engagements for that operator machine. UI: Audit Config → Time zone.

Passive tools

• Passive does not run Sublist3r. misc/update.sh does not install or update it.

Passive Names section

• Passive Names dump is name + title, aligned (column -t). Emails stay in the Emails section; phone is not shown there.
• import-names.sh and import-names-titles-emails.sh both write that format when they refresh pages/passive.htm.

Optional nav pages (General / Files)

• Nav items are omitted when the matching pages/*.htm file is missing. Standard reports that include those pages keep the menu items.
• Covered: General → Emails (emails.htm); Files → Excel (xls.htm), PowerPoint (ppt.htm), Text (txt.htm), Word (doc.htm), PDF (pdf.htm).
• Rebuilds of Audit / Active / Subdomains / Hosts copy Discover templates (full menus); writers strip items whose pages are absent.

Report homepage date (index.htm)

• Home hero date (#DATE# / first inc-home-meta value) is format Month DD, YYYY (same as Discover DATESTAMP).
• Update to today whenever the engagement changes: Passive finish, Active, host-scan finish, Shodan enrich/Update, software CVE refresh, KEV catalog refresh, subdomain/names imports, operator package import.
• Helper: recon/touch-report-date.py <report_dir> (also callable from Python via touch_report_index_date).

Import hub (Audit page)

• Import button on Audit only when Discover-hosted (http://127.0.0.1:17322/…). Label stays Import. Always targets the current engagement (statusd report_root); no report path in the UI.
• Hub choice rows: Operator scans · Names · Names, titles, and emails · Subdomains.
• Operator scans: path to unpacked other-operator report + their first name (1–10 letters). Backend: POST /import-operator-package → recon/import-operator-package.py --dest <live> --source <path> --operator <Name> --json. Merges host-scans, gowitness, httpx/whatweb, new subdomains, audit lines for that name. Never copies their pages/*.htm over the live tree.
• Names: optional manual TSV path (default tools/names-manual.tsv in the live report). POST /import-names → recon/import-names.sh --report <live> [--manual …] --json.
• Names, titles, and emails: source file path. POST /import-names-titles-emails → import-names-titles-emails.sh --report <live> --source … --json.
• Subdomains: mode existing | team-csv, import path (firefox allowed), optional run_active for CSV new public hosts. POST /import-subdomains → import-subdomains.sh --report … --mode … --import … [--run-active] --json.
• Domain menu no longer lists import items 6–8; menu is 6 Active · 7 Open report · 8 Previous. Scripts remain for CLI.
• After a successful Import, the Audit page reloads so KPI totals, charts, Target scans, and the log match the rebuilt audit.htm.
• Assets: inc-audit-import.js; bust ?v= / modern.css on Audit after UI changes; import-report injects the script on audit.htm. Restart statusd after endpoint changes.

Audit log operator filter

• Client-side dropdown next to Audit log heading: All operators + each unique name from the table.
• Rows use data-audit-operator (from audit-build.py). Optional URL ?operator=Name (case-insensitive; updates via history.replaceState).
• Works on statusd and file://. Asset: inc-audit-log-filter.js; bust ?v= / modern.css on Audit after UI changes.

Audit log table (hidden rows)

• Still written to tools/audit/log.txt. The Audit log table (and defender CSV) omit: Opened report in Discover, Exported …, nuclei pass-2, Imported operator package (imported lines stay), Finished robots (Finished robots in 2 sec. / Finished robots on …). Started robots stays (command + TXT/WEB). Metrics still count Finished robots.

Audit log line delete

• Delete on each Audit log row only when Discover-hosted (http://127.0.0.1:17322/…). Manual file:// never shows it.
• Confirm modal: <Operator>, are you sure you want to delete this line? (operator from ~/.discover/operator-name / GET /config) plus a short line preview. Cancel / Delete.
• Backend: statusd POST /audit-line-delete → audit-build.py --delete-line <sha256> <report> --json. Removes one matching line from tools/audit/log.txt (hash of stripped raw line), rebuilds pages/audit.htm (metrics + log).
• Row markup: data-audit-hash / data-audit-preview on each log <tr> from audit-build.py. Assets: inc-audit-line-delete.js; bust ?v= / modern.css on Audit; import-report injects the script. Restart statusd after endpoint changes.

Audit page refresh

• Discover-hosted Audit (http://127.0.0.1:17322/) reloads when pages/audit.htm is rebuilt, so a new log line shows up, then scrolls to that newest row. The first status poll only records the revision.
• An open Config, Import, Export, or delete dialog waits. The page reloads after it closes.
• The metrics range (Today, Yesterday, Last 7 days, Last week, All) is kept across that reload. A normal open still starts on All and stays at the top of the page.
• Manual file:// does not reload. GET /status adds audit_rev (page mtime and size). Asset: inc-audit-status.js. Bust ?v= on Audit after the script changes. Restart statusd after the /status change.

Audit last-7-days metrics (Option A strip)

• Shown above the Audit log on pages/audit.htm (all report modes; built by audit-build.py).
• Range dropdown (aligned with By CVE / Targets scanned): Today · Yesterday · Last 7 days · Last week · All (default). Asset: inc-audit-metrics-range.js. Live reports, Operator export, and Client export open on All.
  • Calendar windows use the operator view timezone (~/.discover/timezone). Audit stamps stay UTC on disk.
  • Today: current calendar day in the view timezone (start = today 00:00).
  • Yesterday: previous calendar day in the view timezone (yesterday 00:00 through today 00:00).
  • Last 7 days: last 7 calendar days inclusive of today in the view timezone (start = today−6 00:00).
  • Last week: previous full calendar week in the view timezone (Monday 00:00 through this Monday 00:00).
  • All: no date filter (Scans per day chart capped at newest 60 calendar days of activity). That chart is one bar per day. With more than 12 days, the axis prints every 7th date plus the last day. Weekly dates are left-aligned on that bar. The last date is pinned to the end of the bars so it stays inside the pane. Hover a bar for that date and count.
• KPI cards (compact row, centered): Targets scanned, Scans completed; Incomplete scans only when count > 0 (then three cards; otherwise two cards centered).
• Charts layout (top → bottom): KPI row; By CVE | By software (2 equal boxes); By tool | By category (2 equal boxes); Scans per day (full width of two-box row); By operator (full width of two-box row). Horizontal bar charts show top 10 entries each. Client export omits Scans per day and By operator. The live report and Operator export keep both.
• Data: host-scan Finished / Started lines in tools/audit/log.txt; software from (software: …) on Finished lines; CVEs from nuclei pass-2 meta.pass2.ids (canonical CVE-YYYY-NNNN only) for runs finished in the window; category from tools/subdomains / tools/private-subs host→category lookup (empty → (none)).
• By operator counts each host-scan Started line in the window (the scan that operator launched). A Finished line with no matching Started line for that operator, tool, and host still counts. Scans completed, By tool, By software, and By category stay on Finished lines.
• Incomplete: Started in the selected window with no Finished for the same tool+host at/after that start (Finished may be outside the window — e.g. started last week, finished this week is complete for Last week).
• Rebuild with Audit (host-scan finish, Open report, Config name rewrite, etc.). Bust modern.css?v= on Audit after CSS changes.

Config (Audit page only)

• Config button on Audit only when Discover-hosted (http://127.0.0.1:17322/…). Header order: Config · Import · Export.
• Modal hub with five choice rows: Operator name, Time zone, APIs, Google Sheet, X.
• Backend: recon/discover-config.py via statusd GET /config, POST /config/api-keys, POST /config/operator-name, POST /config/timezone, POST /config/op-notes, POST /config/op-notes-authorize, POST /config/x.
• Operator name: read/write ~/.discover/operator-name (1–10 letters). On change, rewrite only audit lines whose Operator field matches the previous name (other operators’ lines untouched; do not rewrite names inside Action text). Rebuild Audit after rewrite.
• Time zone: display preference (~/.discover/timezone). UTC + US zones. Does not change written stamps (always UTC). Audit Time column converts in the browser. Metrics Today / Yesterday / Last 7 days / Last week use this timezone when Audit is built. Saving it rebuilds Audit.
• APIs: show/edit/save NVD_API_KEY, SHODAN_API_KEY, WPSCAN_API_TOKEN in ~/.discover/api-keys (chmod 600). The NVD row shows https://nvd.nist.gov to the right of the label. The Shodan row shows https://www.shodan.io/ to the right of the label. The WPScan row shows https://wpscan.com/register/ to the right of the label. Localhost only. Saving APIs leaves X_API_KEY in that file unchanged. theHarvester is a link (not a field) that opens ~/.theHarvester/api-keys.yaml via statusd POST /config/open-theharvester or discover-theharvester: → misc/open-theharvester-keys.sh (open, else xdg-open). If that file is missing, copy ~/theHarvester/theHarvester/data/api-keys.yaml into place (do not overwrite an existing file). Do not store theHarvester keys in Discover api-keys.
• After a successful Save, that panel’s Back (or Cancel) and Save buttons are removed. Authorize on Google Sheet stays until Save. A failed save leaves the buttons. Opening the panel again shows them. The header close control stays.
• X: fifth Config row. Opens a second modal (the hub stays open behind it). Bearer token reads and writes X_API_KEY in ~/.discover/api-keys (chmod 600). Accounts are added and deleted in that modal, then Save writes ~/.discover/x-accounts (one handle per line, chmod 600, at most 40). The list is alphabetical (A–Z, ignoring case) each time the modal opens, and a newly added handle is inserted in that order. A handle is 1–15 letters, numbers, or underscores. @name and an x.com or twitter.com profile URL are accepted and stored as the bare handle. Empty key clears X_API_KEY. This is per user (~/.discover), not per engagement. POST /config/x. The APIs panel does not show this key.
• theHarvester sources: recon/passive.sh sources_no_api / sources_api must stay in lockstep with old/test-theHarvester.sh. Cover every name in ~/theHarvester SOURCE_SPECS. Split by theHarvester README API key column: No → sources_no_api; Required or Optional → sources_api. Domain menu 98 = free list, 99 = API list. After a theHarvester update, run option 99 leftover check (old/newModules.sh) and add any new engines to the matching list.
• Google Sheet: per-engagement spreadsheet URL in $REPORT_ROOT/tools/op-notes-url (chmod 600). Empty URL disables sync. Not stored in ~/.discover/api-keys (machine-global). Do not read a report-root .env. OAuth client_secret.json + google-token.json stay in ~/.discover/ (chmod 600). Authorize in this panel (browser OAuth); host-scan and Active appends never open a browser. Client/defender Export omit tools/op-notes-url (and a stray report-root .env if present). Operator Export may keep the URL file. Host-scan: uv run misc/op_notes.py append in the foreground before the tool starts (40s cap; Discover install provides uv). Active checks the sheet before probes and appends one row when the scan finishes (same 40s cap). Target is the engagement domain. Command is active recon, or active recon on imported hosts for an import batch. Result is the probed and alive counts. Empty URL stays off and does not stop the scan. A configured sheet that is not authorized, is missing uv or op_notes.py, or fails to open or append prints the error and stops the scan. Columns A–E: date, operator, IP, target, command. Date is the scan time in the spreadsheet timezone, mm/dd/yyyy - hh:mm. Target is plain text. A hostname is not a link. When the Action cell wraps past one line, that row grows to fit the text. An nmap PORT table of more than one line grows that same row to fit column F (Courier New). If the command also wraps, the row uses the taller of the two. Each scan is a new row under the last filled date. A scan does not replace that row. Every cell in that row is top aligned. nmap checks that the sheet opens before the scan (no row yet), then waits until the scan finishes and writes the PORT table in column F (Result; Courier New, using the append updatedRange so concurrent scans do not format the wrong cell). No open ports → No open ports. in the default font. nuclei writes the row before the scan. When output.txt ends on No vulnerabilities discovered., that same row’s column F becomes None in the default font and the row height stays as it is. A skipped pass 2, or a finding on the last line, leaves column F empty. transom writes the row before the scan. When transom.htm contains a software version or a disclosed internal address, that same row’s column F lists them in report order, one per line, in the default font. Several of those lines grow the row only when that is taller than the command. A result does not shrink the row. A software version is the Product:version label used on Active (for example Microsoft ASP.NET:4.0.30319). An internal address is the address from Location discloses internal address. A report with neither leaves column F empty. Redact --api-token in logged commands. Bust inc-audit-config.js?v= / modern.css on Audit after UI changes.
• Assets: inc-audit-config.js; bust ?v= / modern.css on Audit after UI changes; import-report injects the script on audit.htm. Restart statusd after endpoint changes.

Report UI layout (CSS)

When designing or tweaking report page layouts (especially Audit and other modern.css tables/containers):

• Prefer px for widths, min/max-widths, padding, and gaps you are dialing in with the operator.
• Do not use rem for layout sizing. Bootstrap sets html { font-size: 10px }, so 1rem = 10px (not 16px). That made earlier “rem” floors look far too narrow.
• % / width: 1% + nowrap is fine for shrink-to-content columns; once a column needs a fixed size, use px.
• Names table: lift the generic 1200px container cap (width: max-content; max-width: none) so Phone is not clipped. Do not assign column widths; Name / Title / Email stay auto/nowrap. Bust modern.css?v= on names.htm after CSS changes.
• Bust modern.css?v=… on the affected page after CSS changes and deploy to the live engagement report when applicable.
• Subdomains private table (top) uses the same columns as the public table beneath it: Subdomain, Category, IP Address. It is sized to that table’s width and column edges (inc-data-table.js). The gap between the two tables is 56px: the 32px gap plus one text line.

misc/update.sh tool install order

Tool install/update blocks in misc/update.sh must stay in case-insensitive alphabetical order by tool/display name (e.g. DomainPasswordSpray → droopescan → Egress-Assess → … → WhatWeb → Windows Exploit Suggester → wpscan → xdotool).

• Do not group by feature (e.g. “CMS tools together”). Place new tools where the alphabet says, not next to a related tool.
• Comments already mark some blocks this way (e.g. CISA KEV after chromium, before curl); follow that convention.
• Same idea for README Update bullet lists when they mirror install order.
• nuclei-templates: Update is sudo misc/update.sh. If ~/nuclei-templates exists, delete it (a symlink is removed without following it). Keep /opt/nuclei-templates as a git clone of projectdiscovery/nuclei-templates. Each Update clones that path when it is missing or has no .git, otherwise git pull --ff-only. Do not use nuclei -ut here: that only tracks release tags and would reset git HEAD. Point the operator’s ~/.config/nuclei/.templates-config.json at /opt/nuclei-templates. Set disable-update-check: true in ~/.config/nuclei/config.yaml so a later nuclei -v does not zip-install the last release over the clone. Print Already up to date. (no commit hash or subject).

Host-scan reachability pre-check (expand dropdown only)

• Applies only to tools launched from the Subdomains expand panel: nuclei, robots, nikto, transom, ffuf, feroxbuster, plus droopescan / wpscan when shown. nmap does not use this HTTP gate.
• Before the tool runs, run-host-scan.sh does a curl HTTP/1.1 GET (15s max, same UA as the tool). One request records status and time (not two sequential GETs). Retry once on 000.
• robots probes {origin}/robots.txt, not the site root (a slow/403 homepage must not skip a working robots.txt).
• If no HTTP response (000 / timeout): do not launch the tool. Write skip note in that run’s output.txt, set meta.skip_reason=host_unreachable and the same on status.json / latest.json, exit 1.
• Expand panel shows Unreachable (red) under that tool box (plus txt link when present).
• Nikto HTM, Transom HTM, and robots WEB are omitted when skip_reason=host_unreachable (no report / no Disallow list).
• Nikto has no second internal pre-check (shared gate only).
• Help modals (ⓘ) and README host-scan section document this gate.

Host-scan expand (Subdomains)

• Host-scan chevrons appear only when the report is opened via Discover statusd HTTP (http://127.0.0.1:17322/…, Open report / Active). Manual file:// open never shows chevrons, even if statusd is still running. Client export is the exception: the staged package shows the tool menu without Run and without the ⓘ button. TXT, HTM, and WEB in that package are ordinary links (web.htm for WEB). The live operator report is not rewritten.
• Chevrons on the full public Subdomains table (rows with HTTP status), not only ?software= / ?cve= filtered views.
• Title column: the default IIS start page (IIS Windows Server) is omitted, along with other generic status titles (403 Forbidden, It works!). An empty title shows as -.
• Software for expand: ?software= query wins; else fingerprint the row (Technologies tokens with version when present, title words so Welcome to nginx! matches nginx, web server, hostname labels — not only the first, so uatapi.iis.example.com matches IIS). Priority products include CMS, Kibana, Grafana, Prometheus, Elasticsearch, Jenkins, Kafka UI / Kafbat UI / AKHQ, Filebrowser, SonarQube, RabbitMQ, Redis Commander, Kiali, Kubecost, Superset, GitLab / Gitea / Gogs, Keycloak, Rancher, Argo CD (hostname argocd / argo-*), Eureka, Harbor, MinIO, Nexus, JFrog / Artifactory, Strapi, CloudBeaver, DBeaver, pgAdmin, UiPath (hostname accel360), Control-M (title / hostname control-m), Oracle (hostname oci-adb-control / realm OCI ADB Control; not the bare oci cluster label), Tomcat, WildFly / JBoss, SharePoint, IIS, nginx, Apache, ASP.NET, PHP, Node.js, CrafterCMS, Java (not JavaScript). Named apps beat generic Java (SonarQube + Java → SonarQube). Citrix / NetScaler AAA (title NetScaler AAA, Unified Access, tech Citrix) is fingerprinted for nuclei (-tags citrix,netscaler). Not Cisco ASA VPN. Technologies display drops bare Java when a named Java app is already listed, drops Windows Server, and aliases Wappalyzer SonarQubes to SonarQube.
• nuclei only when a product is known (filter or fingerprint). Do not offer nuclei for blind -tags tech with empty software. Backend run-host-scan.sh refuses nuclei when SOFTWARE is empty. Nuclei output.txt has no Output: filesystem path (Pass 1 and Pass 2 both use Findings:). Prefix real match lines with [*]; leave No vulnerabilities discovered. unprefixed. Blank line before === Pass 2. When that sentence is the last non-empty line of output.txt, the Google Sheet Result cell for that scan is None.
• robots / nikto / transom / feroxbuster / ffuf always on expand when the UI is shown. nmap after robots when that IP has Shodan ports, or the row URL is http/https (then include 80 or 443 even if Shodan is empty / not_found). Split TCP/UDP from Shodan host JSON data[].transport; if transport is missing, ports on the SCANNING UDP= list (scan/nmap.sh) are UDP, else TCP. Port 53 is always both TCP and UDP when it is in the Shodan set. TCP-only: nmap -Pn -n --open -sTV -p <ports> --stats-every 20 <host> (no sudo). Any UDP: sudo nmap --privileged -Pn -n --open -sTV -sUV -p T:<tcp>,U:<udp> --stats-every 20 <host> (sudo -v in the host-scan terminal if no cached ticket; abort if sudo fails — do not fall back to TCP-only). Already root skips sudo. No HTTP pre-check. If nmap raw logs sendto Operation not permitted (typical on VPN/TUN), TXT warns and rewrites UDP rows (tcpwrapped is not a fingerprint; VERSION UDP probe blocked locally). TCP rows stay. web (Firefox tabs) when SERVICE is http/https (ssl/http and ssl/unknown → https). Order: robots → nmap (if Shodan ports or http/https) → nuclei (if product) → CMS tools → nikto → transom → feroxbuster → ffuf.
• transom: box sits immediately to the right of nikto. Run scans the row URL only (--timeout 10s, hard stop 11m). txt is output.txt; htm is transom.htm when the scan ran. A software version in that report is written to the Google Sheet Result cell as the same Product:version label Software Versions uses. A disclosed internal address is written there as the address. Each is one line, in report order. Other findings stay out of that cell. Update installs the git tree at /opt/transom.
• robots: expand box label is robots.txt (not uppercased ROBOTS). Fetch /robots.txt (same idea as multiTabs → Directories in robots.txt). txt = raw robots.txt body; web = open Disallow directories in Firefox via discover-robots: → misc/open-robots-tabs.sh (only when disallow_count > 0). Skip Disallow: / (site root is not a directory to open). Keep query-string Disallow paths (Drupal /?q=admin/ is not site root). Run does not open Firefox. Same web label on Audit.
• ffuf / feroxbuster web: green web (Firefox tabs) only when the run has at least one finding URL. Auto-filter-only / empty JSON → txt only. Count stored as meta.url_count. Same web label on Audit log and Target scans.
• web on Windows / exported reports: Edge on Windows opens web as a normal target="_blank" link to web.htm in that run directory (same kind of link as Nikto htm). web.htm lists the same HTTP(S) URLs. The Discover host (statusd, browser not Windows) still opens Firefox through discover-robots: / discover-ffuf: / discover-ferox: / discover-nmap-web:. Audit build writes web.htm. A package exported before that file exists needs a new export.
• droopescan / wpscan gate on CMS software (query or fingerprint). WordPress → wpscan only (no droopescan for WP). Drupal / Joomla / Moodle / Silverstripe still use droopescan when matched. Droopescan runs --hide-progressbar; report TXT drops percent-bar lines ((0%) / 12/4000) and leading [+], and writes URLs found.
• Each tool box has a Unicode ⓘ help modal on the operator report. Client export omits it. Finished stamps use the Config Time zone (same display conversion as Audit; on-disk stamps stay UTC). Bust inc-host-scan.js?v=… (and modern.css?v=… on Subdomains) after changes and sync via Import when testing live reports.

Shodan panel Update (Subdomains)

• Shodan ▸ expand panel shows an Update button only on Discover-hosted pages (http://127.0.0.1:17322/…). Manual file:// never shows it.
• Backend: statusd POST /shodan-refresh → recon/shodan-enrich.py <report> --ip <IP> --json (force host API lookup; rewrites that host cache + index; audit log line).
• Requires SHODAN_API_KEY from shell or ~/.discover/api-keys. Bust inc-shodan.js?v=… / modern.css?v=… after UI changes; restart statusd (Open report) so the new endpoint is live.
• Opening the panel on statusd calls GET /shodan-status (api_key true/false only). If false, show where to add the key (~/.discover/api-keys) and disable Update.

API keys

• Preferred file: ~/.discover/api-keys (KEY=value lines; chmod 600). Template: resource/api-keys.example.
• Keys: NVD_API_KEY, SHODAN_API_KEY, WPSCAN_API_TOKEN (optional free token for wpscan vuln DB; run-host-scan.sh loads api-keys and passes --api-token), X_API_KEY (optional bearer token; Audit Config → X only, not the APIs panel), GITHUB_TOKEN (optional; threat-feed GitHub search only, not the APIs panel). Saving APIs leaves both of those keys unchanged.
• Lookup order: shell export → ~/.discover/api-keys (first non-empty wins per key).
• Update: misc/update.sh ensures the file exists — if missing, copies resource/api-keys.example → ~/.discover/api-keys (chmod 600). Under sudo, seeds the invoking user’s home (SUDO_USER), not only root. If already present, only re-applies chmod 600 (quiet).
• Auto-migrate: if $DISCOVER/.env or ~/.discover/.env still exist, Discover merges them into ~/.discover/api-keys (existing api-keys values win) and removes the legacy files. Implemented in software-cve.migrate_legacy_api_key_files() (also run from Active / Shodan shell loaders).
• Agent / automation must never destroy live secrets:
  • Do not run discover-config.py set-api-keys, statusd POST /config/api-keys, or any write to ~/.discover/api-keys (or legacy .env key files) with dummy/test values.
  • Do not “verify” save paths against the operator’s real key file. Use a temp file, dry logic, or read-only checks.
  • Do not clear, overwrite, or restore keys unless the operator explicitly asks to save or change a specific key.
  • Tests may read presence only (configured / empty); never paste or rewrite real key material for convenience.

Report Export (Audit only)

• No Domain menu item for export. UI is an Export button at the top of Report → Audit only when the page is Discover-hosted (http://127.0.0.1:17322/…). Not shown on Passive or Active. Manual file:// never shows the button.
• Modal: Client / Defender / Operator radios + Export + Cancel; success shows the output path ($HOME/data by default).
• Defender CSV headers: Time UTC, Operator, Operator IP, Target, Action.
• Backend: statusd POST /export → recon/export-report.sh --kind … --report … --out-dir … --quiet (JSON path on stdout).
• Client pages/audit.htm: opens on All (the Audit default) and omits Scans per day and By operator. Operator export and the live report keep both panes.
• Client Subdomains: staged package shows the host-scan tool menu without Run and without the ⓘ button. TXT, HTM, and WEB are ordinary links. WEB opens web.htm. Operator export and the live report keep Run, the ⓘ button, and chevrons only on statusd.
• Client omits tools/gowitness/gowitness.jsonl. Yes links already point at the screenshot JPEGs. Operator export keeps the JSONL so an operator import can rebuild those links.
• Client omits assets/report-mode.json. The HTML report does not read it. Operator export keeps the operator stamp. Import → Operator scans refuses a client package when export-meta.json kind is client, assets/report-mode.json mode is client, or pages/subdomains.htm contains window.DISCOVER_CLIENT_VIEW=true.
• Client omits export-meta.json. The export is still recorded on the live engagement in tools/exports/log.jsonl and the audit log. Operator export keeps the file.
• Client omits tools/exports/ (log.jsonl). The Exports table is already in pages/audit.htm. Operator export keeps the ledger. The live engagement still appends each export to tools/exports/log.jsonl.
• Client omits raw Passive dumps at the top of tools/ whose names start with z (zsubfinder, zcrtsh, zarin-emails, zarin-names, and the other theHarvester z* files). Passive already merged them into emails, names, subdomains, hosts, and the document lists. The HTML report does not read them. Operator export keeps them.
• Client omits source files the HTML pages already contain: whatweb.json, httpx.jsonl, subdomains, emails, names, names-phones.tsv, names-manual.tsv, hosts, public-ips, private-ips, private-subs, dns-private.tsv, squatting, records, whois-domain, whois-ip, active-alive.tsv, active.txt, active-targets.txt, import-batch-hosts.txt, subdomains-import.tsv, passive-scan-date, and .DS_Store. Operator export keeps them.
• Client omits Shodan summary.json, summary.tsv, and index.json. The panel uses index.js, kev-ids.js, and hosts/*.json.
• Client host-scans keep output.txt, robots.txt, nikto.htm, transom.htm, web.htm, and status.js. Other run files (ffuf.json, ferox.json, nikto.conf, meta.json, latest.json, status.json, nmap.raw, nuclei.txt, nuclei-pass2.txt, and the raw logs) are omitted after web.htm is written. Operator export keeps them.
• Client keeps operator IPs in the Audit page table. Import events that already store a dash stay a dash. The package omits tools/audit/log.txt and log.txt.bak-*. The live engagement still appends to that log. Operator export keeps it.
• Assets: inc-report-export.js, modern.css (export classes); import-report syncs JS. Bust inc-report-export.js?v=… after changes.

Import subdomains (Audit Import hub / CLI)

• Two modes: existing (Firefox/Pentest-Tools/TSV) or team-csv (subdomain,ip,category). UI: Audit → Import → Subdomains. CLI: import-subdomains.sh --report … --mode … --import … [--run-active] [--json].
• CSV list: one IPv4 per host (empty IP → dig). Skip hosts already in tools/subdomains (no overwrite). Category = Discover rules first, else CSV. Never write recon/subdomain-categories.tsv.
• A host whose address is 1.1.1.1, or whose missing address resolves to 1.1.1.1, is left out. A listed 1.1.1.1 is not replaced by another lookup.
• After import: refresh pages/subdomains.htm and pages/hosts.htm (unique public IPs). CSV list also writes tools/import-batch-hosts.txt (new public hosts only) and may offer Active on that batch only (DISCOVER_ACTIVE_SCOPE=import-batch).
• After import + Active (full or import-batch): rebuild the entire Active page from merged tools/ artifacts — Scope (public/private/responding), status codes, alive-by-category, CMS, web servers, technologies, software versions + CVE enrichment, Login pages, and scan date. Batch Active must merge httpx/whatweb/gowitness into the engagement files first, then call the same full pages/active.htm rebuild (not a batch-only summary). Scan date = latest httpx timestamp, not the first line.
• Active public-DNS check: before httpx, dig +timeout=2 +tries=1 +short A <host> @1.1.1.1 (parallel). RFC1918, 127.0.0.0/8, 169.254.0.0/16, or 0.0.0.0 drops that host from httpx and records host<TAB>ip in tools/dns-private.tsv. tools/private-subs is stored-private rows plus those leaks (dig IP wins). Do not rewrite the public IP in tools/subdomains. Do not use the operator resolver (VPN split-horizon is not a finding).

Active Login pages (by category)

• Login pages table on Active (under CMS when any category exists). The Source / Type column icon (same look as sort) switches By source (default, ▲) · By type (▼). It does not sort that column. Unicode ⓘ next to the heading explains overlapping By source counts vs By type. Asset: inc-active-login-view.js.
• By source column Source: Path, Title, Tech, Status with host counts; each links to Subdomains ?login=path|title|tech|status.
• By type column Type: Basic (HTTP challenge: Basic / Digest / NTLM / Negotiate, including 401 with empty title) · Form (Path / Title / Tech, or WhatWeb PasswordField). Generic 401 Authorization Required titles with HTTP-challenge do not count as Form. Host may be both. Links: ?login=basic|form. PasswordField, PoweredBy, Index Of, Object (HTML tag), header/protocol noise (HSTS, HTTP/3, X-UA-Compatible), and Windows Server stay out of Technologies / Software versions (NOISE_PLUGINS).
• Source categories (host may match more than one; counts are unique hosts per category, alive public hosts):
  • Path — newest tools/host-scans/<host>/ffuf/*/ffuf.json FUZZ path matches high-confidence login paths (login, wp-login.php, oauth, sso, …), or httpx final_url / url path (/login.jsp, …). Avoid bare admin. SPA filter: when a run has many hits sharing one body length (≥50 results, mode count ≥20, mode ≥50% of results), login-named paths with that same length are ignored (soft-200 app shell).
  • Title — httpx/page title matches login / sign-in / SSO / unauthorized / password phrases, plus common non-English titles (Zaloguj, Anmelden, Connexion, …).
  • Tech — fingerprint includes products that typically expose a login UI (CMS, Grafana, Kibana, GitLab/Gitea, Citrix, Keycloak, Jira, Confluence, Argo CD, Harbor, MinIO, Rancher, UiPath, pgAdmin, Control-M, …). Match the product as a word so Atlassian Jira counts as Jira. httpx tech tokens, page title, and hostname all count (SPA consoles often only have HSTS in tech).
  • Status — HTTP 401 from httpx and an auth-related page title, or 401 with HTTP-challenge tech (Basic / Digest / NTLM / Negotiate) even if the title is empty. Bare 403 and bare 401 with empty title and no challenge are noise.
  • Skip Microsoft SSO: if httpx final_url / url contains login.microsoftonline.com (or related Microsoft IdP hosts), the host gets no Login pages signals (all four). Citrix: short live probe POST /p/u/doAuthentication.do → follow doSaml when present; if the SAML hop is Microsoft, skip (covers NetScaler AAA that only show LogonPoint in httpx). RNAS: title Unified Access RNAS or hostname rnas-* / .rnas. — remote network access gateways, skip (not app logins).
• Subdomains public rows get data-login-path|title|tech|status="1" when rebuilt (Active / write_subdomains_active_page / software-cve refresh). Filter: inc-subdomains-filter.js (?login=). Bust filter ?v= after changes.

Threat intel (Active Software Versions)

• Alerts is a full-size header button on Active, immediately left of Enrich, statusd only. The label shows the unreviewed count (Alerts 3). It opens the Threat intel list for software in this report. Opening the report, opening Active, and opening this list do not start a check. The list shows Last check as mm/dd/yyyy - hh:mm in the view timezone (Z when that zone is UTC). The stored stamp stays UTC. No check yet says No check yet. Check now is the only way a check starts. While a check runs, the status lists one line per source as that source starts: [*] Checking NVD. then KEV, advisories, GitHub, and Metasploit. The X line includes the configured account count, for example [*] Checking 4 X accounts. One account uses the singular. Check now stays disabled. Close stays available. When the check finishes, a blank line and then Check finished are added under those lines, with the new-alert count or the note from a first check. The source lines stay. An empty list says No alerts yet. A row opens that alert. Back (or the header close, or Escape) marks that alert reviewed and returns to the list. file:// and Client export do not show the button. Client and Defender exports omit tools/threat-feed/. Asset: inc-threat-feed.js. Bust ?v= and modern.css on Active after changes.
• The check watches Software versions in tools/software-cves-cache.json. Sources: NVD (new CVEs for those CPEs since the last check), CISA KEV, Exploit-DB, CISA advisories, WordPress and Drupal security feeds when that product is present, GitHub repositories for CVEs already on that software and for CVEs or KEV hits the check just found, local Metasploit exploit modules for those CVEs, and X accounts from Config. The check reads the installed framework catalog. The first check only records what is already public. The first read of the module list on an engagement that already has a baseline records the current modules and adds no Metasploit alerts. Later checks add an alert when a new exploit module matches a CVE on the report. Queries use product and CVE identifiers. Nothing is written to the audit log.
• There is no daily clock and no check when the report opens. POST /threat-feed/check runs only from Check now. POST /threat-feed/reviewed marks one alert reviewed. GET /threat-feed returns the list and does not start a check. Optional GITHUB_TOKEN in ~/.discover/api-keys raises the GitHub search limit. X uses the Config bearer token and account list. Restart statusd after endpoint changes.

Active page Enrich (Shodan + Software CVEs)

• Enrich button on Active (statusd only). Modal title Enrich Engagement Data. Checkboxes, default all on, top to bottom: KEVs, Shodan, Software CVEs. When more than one option runs and the Active page was rebuilt, the finished modal replaces Cancel and Enrich with Reload Active page. One option leaves Cancel and Enrich, and shows Reload Active page in the results when the page was rebuilt. With results showing, the dialog grows to the viewport and the summary pane grows with it, so less of the list sits in a scrollbar.
  • KEVs → POST /kev-refresh → active-tech.py <report> --refresh-kev --json. Downloads the CISA catalog to resource/kevs.json (same feed as Update), rewrites tools/shodan/kev-ids.js, and rebuilds Active so Top CVE badges use the new catalog. No NVD queries. No API key.
  • Shodan → POST /shodan-refresh-all → shodan-enrich.py <report> --force --json-summary
  • Software CVEs → POST /software-cve-refresh → active-tech.py <report> --refresh-cves --json (re-queries NVD for missing/empty cache entries by default; --force-all optional; rebuilds pages/active.htm Software versions table + cve-software-index.js)
• Software versions CVE data is NVD CPE, not Shodan host vulns. Permanent skips: Microsoft HTTPAPI, Java Servlet, JavaServer Pages (SKIP_PRODUCTS in software-cve.py). IIS uses microsoft:internet_information_services. IIS:10 stays out of this table. jQuery UI uses jqueryui:jquery_ui. WordPress, Slider Revolution, Yoast SEO, and WPML are mapped. SharePoint build numbers stay unmapped. The lookup asks NVD for a version range and drops a CVE whose version shape does not match the label (a year against a dotted build). Enrich lists each product that was not looked up.
• Software versions also read the newest transom.htm for each host. X-AspNet-Version 4.0.30319 and the older X-AspNet-Version discloses 4.0.30319 wording become Microsoft ASP.NET:4.0.30319. X-AspNetMvc-Version becomes ASP.NET MVC. An outdated Apache, PHP, nginx, or Microsoft-IIS banner, and a versioned X-Powered-By value, use the same Product:version labels. A host already counted for that label is not counted again. An unversioned product token on that host is replaced by the versioned label. The Subdomains Technologies column shows that label, and ?software= matches it. Every Software Versions label links to that filter, including a version with no CVEs. Missing headers, cookies, and other Transom findings stay out of this table.
• Top CVE on Active: on the Discover host (statusd, browser not Windows) the CVE id opens Firefox tabs for NVD, Rapid7, Tenable, Exploit-DB, Sploitus, CVEbase, and GitHub via discover-cve: → misc/open-cve-tabs.sh. In Edge on Windows that control opens assets/cve-links.htm, a page of those same links. The KEV badge stays a direct CISA catalog link. Bust inc-active-cve-tabs.js?v= after the click script changes. A package exported before cve-links.htm exists needs a new export.
• Assets: inc-active-refresh.js; bust ?v= and modern.css after UI changes. Restart statusd after endpoint changes.