CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Core Working Principles

Adapted from Andrej Karpathy's four CLAUDE.md principles — read these before touching code. They target reasoning failures (wrong assumptions, over-engineering, scope creep, weak success criteria), not formatting.

1. Think Before Coding — Don't assume. Don't hide confusion. Surface tradeoffs.
   • State assumptions explicitly; if uncertain, ask rather than guess.
   • Present competing interpretations instead of silently picking one.
   • Call out inconsistencies, confusion, and tradeoffs as you find them.
2. Simplicity First — Minimum code that solves the problem. Nothing speculative.
   • No features beyond what was asked.
   • No abstractions for single-use code.
   • No "flexibility" or configurability that wasn't requested.
   • No error handling for scenarios that cannot occur.
3. Surgical Changes — Touch only what you must. Clean up only your own mess.
   • Match the surrounding style and conventions.
   • Don't refactor unbroken adjacent code that's orthogonal to the task.
   • Only remove what your change rendered obsolete.
4. Goal-Driven Execution — Define success criteria. Loop until verified.
   • Turn tasks into verifiable goals: "add validation" → "write tests, then make them pass"; "fix the bug" → "reproduce it in a test, then fix"; "refactor X" → "ensure tests pass before and after".
   • Iterate against those criteria instead of asking for constant clarification.

Setup After Clone

git config core.hooksPath .githooks               # enable pre-commit hook (fmt + clippy + tests)

Build & Test Commands

cargo fmt                                         # auto-format code (always run before build)
cargo build                                       # debug build
cargo build --release                             # release build (~11s)
cargo test                                        # unit tests + offline integration tests (~300 tests)
cargo test --test integration_crawl -- --ignored --test-threads=1  # network integration tests (crawls crawler.siteone.io)
cargo test scoring::ci_gate::tests::all_checks_pass  # run a single test by name
cargo clippy -- -D warnings                       # lint (CI enforces zero warnings)
cargo fmt -- --check                              # format check

# Browser rendering (`--browser`, chromiumoxide/CDP) is a DEFAULT feature — `cargo build`,
# `cargo test`, `cargo clippy` all include it. Build/test the lean variant (no chromiumoxide):
cargo build --release --no-default-features        # lean build, ~6 MB smaller, no browser
cargo test --no-default-features                   # tests without the browser feature
cargo clippy --no-default-features -- -D warnings  # lint the lean variant

Quick Run

./target/release/siteone-crawler --url=https://example.com --single-page
./target/release/siteone-crawler --url=https://example.com --output=json --http-cache-dir=  # no cache
./target/release/siteone-crawler --html-to-markdown=page.html                               # convert local HTML to markdown (stdout)
./target/release/siteone-crawler --html-to-markdown=page.html --html-to-markdown-output=page.md  # convert to file
./target/release/siteone-crawler --url=https://example.com --ai-provider=openai-compatible \
  --ai-endpoint=http://localhost:8000/v1 --ai-model=my-model --ai-profile           # chapter-based subject profile (MD+JSON+HTML)
./target/release/siteone-crawler --url=https://example.com --ai-profile --ai-dry-run # plan only, no API calls

Architecture

Crawl Lifecycle (in order)

1. CLI Parsing (Initiator → CoreOptions::parse_argv()): Parses 120+ CLI options, merges config file if present, validates. Exits with code 101 on error, code 2 on --help/--version. Non-crawl utility modes (--serve-markdown, --serve-offline, --html-to-markdown, --ai-list-models, --ai-check) exit early in main.rs before creating the Manager.
2. Analyzer Registration (Initiator::register_analyzers()): Creates all 18 analyzer instances (Accessibility, BestPractice, BrowserConsole, Caching, ContentType, DNS, ExternalLinks, Fastest, Headers, Page404, Redirects, Security, SeoAndOpenGraph, SkippedUrls, Slowest, SourceDomains, SslTls, Technologies) and registers them with AnalysisManager. Some analyzers receive config from CLI options (e.g. fastest_top_limit, max_heading_level); BrowserConsole is active only in --browser mode.
3. Manager Setup (Manager::run()): Creates Status (result storage), Output (text/json/multi), HttpClient (with optional proxy, auth, cache), ContentProcessorManager (HTML, CSS, JS, XML, Astro, Next.js, Svelte processors), and the Crawler instance. The crawl loop fetches through an Arc<dyn Fetcher> (src/engine/fetcher.rs): by default the HttpClient; with the browser feature + --browser, a BrowserRenderer that renders each HTML page in Chromium and returns the same HttpResponse (plus browser_diagnostics). Everything downstream is unchanged.
4. Robots.txt Fetch (Crawler::fetch_robots_txt()): Before crawling starts, fetches and parses /robots.txt from the initial domain. Respects --ignore-robots-txt option. It also records per origin how the file was fetched (Status::set_robots_txt_state: RobotsFetchState::{Ok, NotFound, Unavailable, Skipped, NotAttempted}, redirects followed for the state only), which --ai-geo reads; with --ai-geo the manager fetches, after the crawl, the robots.txt of the key-page origins the crawl did not read (≤ 10, e.g. the www origin the homepage redirects to; not in a dry run).
5. Crawl Loop (Crawler::run()): Breadth-first concurrent URL processing:
   • URL queue (DashMap) seeded with initial URL
   • Tokio tasks limited by Semaphore (= --workers count) + rate limiting (--max-reqs-per-sec)
   • Per-URL flow: check robots.txt → HTTP request → on error, store with negative status code → on success, run content processors → extract links from HTML → enqueue discovered URLs
   • Content processors (HtmlProcessor, CssProcessor, etc.) transform response bodies during crawl — used by offline/markdown exporters for URL rewriting
   • Each visited URL's response is stored in Status for post-crawl analysis
   • Per-URL data collected: status code, headers, body, response time, content type, size, redirects
6. Post-Crawl Analysis (Manager::run_post_crawl()): Sequential pipeline after crawling ends:
   • Transfer skipped URLs from crawler to Status
   • Run all registered analyzers (AnalysisManager::run_analyzers()): each analyzer gets read access to Status (all crawled data) and write access to Output (adds tables/findings)
   • Add content processor stats table
7. Exporters (Manager::run_exporters()): Generate output files based on CLI options:
   • SitemapExporter: XML/TXT sitemap files
   • OfflineWebsiteExporter: Static website copy with rewritten relative URLs
   • MarkdownExporter: HTML→Markdown conversion with relative .md links
   • FileExporter: Save text/JSON output to file
   • HtmlReport: Self-contained HTML report (also used by Mailer and Upload)
   • MailerExporter: Email HTML report via SMTP
   • UploadExporter: Upload report to remote server
   • AnimationExporter (feature browser): Assemble per-page screenshots into a GIF/MP4 animation (GIF via the embedded image crate, MP4 via external ffmpeg)
   • AiReportExporter (JSON + HTML): when the extract AI action (--ai-report) produced an AiReportModel, atomically writes a no-clobber pair named ai-report.<preset>.<host>.<run-id>.json + .html
   • AiElaborateExporter (Markdown + JSON + HTML): when --ai-elaborate produced a BrandDoc, atomically writes a no-clobber triple named ai-elaborate.<template>.<host>.<run-id>.md + .json + .html (rolls back earlier files if a later one fails)
   • AiConsistencyExporter (Markdown + JSON + HTML + CSV): when --ai-consistency produced a ConsistencyDoc, writes the no-clobber set ai-consistency.<host>.<run-id>.md + .json + .html + .csv via export/utils/atomic_files.rs::write_files_no_clobber (exclusive creation of the final paths; a failure removes only the files this call created)
   • AiGeoExporter (Markdown + JSON + HTML + a kit directory): when --ai-geo produced a GeoDoc, publishes the report ai-geo.<host>.<run-id>.{md,json,html} (via write_files_no_clobber) together with the kit ai-geo-kit.<host>.<run-id>/ (create_dir of the kit dir and create_new of every file, only into directories this call created; a failed report removes exactly the kit entries this call created). With --ai-consistency too, the manager exports consistency first and puts its HTML file name into the GeoDoc (ConsistencyLink) before this export
8. Scoring (scorer::calculate_scores()): Computes quality scores (0–10) across 5 weighted categories (Performance 20%, SEO 20%, Security 25%, Accessibility 20%, Best Practices 15%). Deductions come from summary findings (criticals, warnings) and stats (404s, 5xx, slow responses).
9. CI/CD Gate (ci_gate::evaluate()): When --ci is active, checks scores and stats against configurable thresholds (--ci-min-score, --ci-max-404, etc.). Returns exit code 10 on failure.
10. Summary & Output (Output::add_summary(), Output::end()): Prints summary table with OK/Warning/Critical counts, finalizes output. Exit code: 0 = success, 3 = no pages crawled, 10 = CI gate failed.

How Analyzers Work

Each analyzer implements the Analyzer trait (analysis/analyzer.rs). Analyzers are post-crawl only — they don't run during crawling. The AnalysisManager calls each analyzer's analyze(&Status, &mut Output) method after all URLs have been visited. Analyzers whose runs_without_working_urls() returns true (SSL/TLS) also run when no URL could be crawled at all, e.g. to explain why an HTTPS site failed. Analyzers read crawled data from Status (visited URLs, response headers, bodies, skipped URLs) and produce SuperTable instances that get added to Output. Analyzers also add Item entries to the Summary (OK, Warning, Critical, Info findings) which feed into scoring.

How Content Processors Work

Content processors implement ContentProcessor (content_processor/content_processor.rs) and run during crawl on each URL's response body. They serve two purposes: (1) transform content for offline/markdown export (rewrite URLs to relative paths), and (2) extract metadata (links, assets). Processors are type-specific: HtmlProcessor handles HTML, CssProcessor handles CSS url() references, etc. The ContentProcessorManager dispatches to the right processor based on content type.

Concurrency Model

The crawler uses tokio for async I/O with a semaphore-based worker pool (options.workers). Shared state uses:

• Arc<DashMap<...>> for lock-free concurrent maps (URL queue, visited URLs, skipped URLs)
• Arc<Mutex<...>> for sequential-access state (Status, Output, AnalysisManager)
• Arc<AtomicBool/AtomicUsize> for simple flags and counters

Key Traits

• Analyzer (analysis/analyzer.rs): Post-crawl analysis (SEO, security, headers, etc.). Each analyzer gets &Status and &mut Output.
• Exporter (export/exporter.rs): Output generators (HTML report, offline website, markdown, sitemap, mailer, upload).
• Output (output/output.rs): Formatting backend. Implementations: TextOutput, JsonOutput, MultiOutput.
• ContentProcessor (content_processor/content_processor.rs): Per-URL content transformation during crawl (HTML, JS, CSS, XML processors).

Options System

CLI options are defined in options/core_options.rs via get_options() which returns an Options struct with typed option groups. Parsing flow: parse_argv() → merge config file → parse flags → CoreOptions::from_options() → apply_option_value() for each option. New CLI options require: adding the field to CoreOptions, a case in apply_option_value(), and an entry in the appropriate option group.

Exit Codes

HTTP Response Body

HttpResponse.body is Option<Vec<u8>> (not String) to preserve binary data for images, fonts, etc. Use body_text() for string content. Failed HTTP requests return Ok(HttpResponse) with negative status codes (-1 connection error, -2 timeout, -4 send error), not Err.

The crawler's HTTP client has automatic decompression off (HttpClient::client_builder(), also for the fallback client): the body is decoded by http_client::decode_body, while the headers stay as received, including Content-Encoding and the compressed Content-Length (#107). Reported sizes are the decoded length (Crawler::get_body_size, which trusts an asset's Content-Length only without Content-Encoding). Never switch reqwest's automatic decompression back on: it would strip those headers.

utils::get_flat_response_headers joins the instances of a repeated header with ", ", except set-cookie and x-robots-tag, which it joins with "\n" so that each X-Robots-Tag instance (with its own googlebot: scope) survives; utils::header_value_for_display maps the "\n" to " | " wherever a header value is shown (header analyzer tables, --extra-columns, JSON extras).

Testing Structure

• Unit tests: In-file #[cfg(test)] mod tests blocks (standard Rust convention)
• Integration tests: tests/integration_crawl.rs with shared helpers in tests/common/mod.rs
• Network-dependent integration tests are #[ignore] — run explicitly with --ignored

Testing Complex Scenarios with Sample Websites

The crawler has a built-in HTTP server (--serve-offline=<dir>) that can serve any local directory as a static website. This enables efficient local testing of edge cases without deploying a real site:

1. Create a sample website directory, e.g. ./tmp/sample-website-xyz/
2. Add HTML files and assets simulating the desired scenario (spaces in filenames, special characters, redirect chains, broken links, specific heading structures, etc.)
3. Start the built-in server: ./target/release/siteone-crawler --serve-offline=./tmp/sample-website-xyz/ --serve-port=8888
4. In another terminal, crawl the local site: ./target/release/siteone-crawler --url=http://127.0.0.1:8888/
5. Verify the crawler handles the scenario correctly (output, offline export, analysis results)

This approach is useful for reproducing bug reports, testing regex edge cases (e.g. URLs with spaces, HTML entities, unusual attribute quoting), validating offline/markdown export for specific HTML structures, and any scenario that would be hard to find on a live website.

Key Files

• src/engine/crawler.rs (~1700 lines): Core crawl loop, URL queue management, HTML/content parsing
• src/options/core_options.rs (~2500 lines): All 120+ CLI options, parsing, validation
• src/export/utils/offline_url_converter.rs (~1400 lines): URL-to-file-path conversion for offline export
• src/export/html_report/report.rs: HTML report generation with embedded template
• src/scoring/scorer.rs: Quality score calculation from summary findings
• src/scoring/ci_gate.rs: CI/CD threshold evaluation
• src/engine/fetcher.rs: Fetcher trait — the single seam the crawl loop fetches through; HttpClient (direct HTTP) and BrowserRenderer both implement it
• src/browser/ (feature browser): BrowserRenderer (renderer.rs), Chromium detection/download/launch (launcher.rs), CDP diagnostics collection (diagnostics.rs), screenshots in every --screenshot-viewport size + pre-capture animation settling (screenshot.rs), --screenshot-viewport parsing: WxH/presets, up to 5 sizes (viewport.rs), pre-capture auto-scroll for lazy-loaded and scroll-triggered content (auto_scroll.rs, --browser-auto-scroll, on by default), cookie-banner dismissal/hiding (cookie_consent.rs). diagnostics.rs data types are always compiled so HttpResponse can carry an inert Option<BrowserDiagnostics>, and so is viewport.rs, because option validation runs in every build. With --ai-geo, BrowserRenderer::fetch records BrowserDiagnostics.raw_text_chars (the Main-region text of the HTML as fetched, ai::geo::render::own_text_chars, dialogs and cookie banners left out) before it replaces the body with the rendered DOM, so --ai-geo compares raw and rendered text without a refetch
• src/export/animation_exporter.rs (feature browser): builds GIF/MP4 animations from per-page screenshots (GIF via the embedded image crate, MP4 via external ffmpeg; frames streamed to disk for O(1) memory)
• src/analysis/browser_console_analyzer.rs: reports browser console/JS/network/security diagnostics (the browser-mode analyzer; active only in --browser mode)
• src/ai/report/ (--ai-report): first-class AI report engine — model.rs (AiReportModel: typed schema + per-page rows + derived rollups; FieldType::Findings + ReportCell::Findings for audit presets; ReportRow::errored carries an honest _error, never fabricated values, and is excluded from rollups), schema.rs (--ai-extract-fields DSL + --ai-schema-file → FieldSpecs + JSON Schema for native enforcement), coerce.rs (LLM-output → typed cells), presets.rs (ia, quality, topics, compliance — the last grounded in EU CCD2 loan-advertising rules). The extract action (src/ai/actions/extract.rs, dispatched in runner.rs::run_extract_action) runs it per page with up to 3 parse attempts (each first runs normalize::repair_json, a mechanical LLM-JSON repairer); src/export/ai_report_exporter.rs + ai_report_html.rs render the JSON + self-contained HTML. Native JSON-schema enforcement plugs into provider.rs::shape_request (ChatRequest.json_schema); auto only enforces for hosted OpenAI/Gemini (vLLM guided_json is unreliable — returns only the first field on some models)
• src/ai/elaborate/ (--ai-elaborate): builds one large, richly structured brand profile (Markdown + JSON + self-contained HTML). Post-crawl pipeline (mirrors src/ai/summary/), orchestrated by mod.rs::run: selection.rs::build_candidates (full ranked universe) → cluster.rs (mass-entity URL clustering, e.g. /blog/*) + nav.rs (global-nav detection) → optional gapfill.rs (bounded, robots-honoring, mask-respecting fetch of nav pages the crawl missed) → select.rs (two id-constrained LLM rounds: the model only ever returns integer ids into numbered lists, so hallucinated URLs are impossible; a deterministic floor/cluster-rep union guards a weak round) → extract.rs (per-page typed PageEssence, verbatim entities) → merge.rs (deterministic dedup into an un-hallucinatable BrandModel — the crawler renders the structured lists, the LLM never does) → synthesize.rs (prose-only, single-shot or output-cap-driven sectioned map-reduce) into a fixed per-templates.rs skeleton → optional correct.rs pass (safe {from,to} edits applied to PROSE ONLY, unique-match, never touching verbatim data). Output language reuses --ai-report-language + report::locale::ReportLocale. doc.rs (BrandDoc) renders MD/JSON/HTML; failed pages are counted, listed, and excluded — never fabricated. --ai-elaborate is its own pipeline (not an --ai-actions value): used alone it clears the default actions; combined with explicit --ai-actions both run. src/export/ai_elaborate_exporter.rs writes the paired MD+JSON+HTML
• src/ai/profile/ (--ai-profile): a second, independent elaborate-style pipeline that builds a polished, chapter-based subject profile (MD + JSON + self-contained HTML). Post-crawl (mod.rs::run): candidate universe → summarize.rs (≤1000-char site description) → classify.rs (id-constrained pick of one of 13 subject types; failure → general) → describe.rs (≤300-char per-page descriptions, parallel) → per-chapter select.rs (numbered-id page selection, un-hallucinatable) + synthesize.rs (chapter markdown) + correct.rs (universal {from,to} edits with exact/regex:/ellipsis forms and delete guards), all chapters in parallel under --ai-max-concurrency → executive summary. Prompt content is embedded from repo-root prompts/profile/** via include_str! (registry.rs/embedded.rs, generated by scripts/gen_profile_prompts.py; parser in promptpack.rs). Input budgets scale from --ai-context-window (reference 128K) in budget.rs. doc.rs (ProfileDoc) renders MD/JSON/HTML; src/export/ai_profile_exporter.rs writes the no-clobber triple ai-profile.<type>.<host>.<run-id>.{md,json,html}. Left the existing --ai-elaborate (v1) untouched.
• src/ai/consistency/ (--ai-consistency): finds possible contradictions in hard facts (contacts, prices, rates, fees, conditions, dates, figures, identifiers) across the crawled pages and their shared header/footer; Markdown + JSON + HTML + CSV. Post-crawl (mod.rs::run), fail-soft: sources.rs (numbered B12 evidence blocks per page, scored and cut to the extraction budget in document order; the unique fact-bearing header/footer lines L7 with their EXACT page sets, one-digit variants surviving the 300-line cap, ≤ 5 lines per chrome source) → extract.rs (one call per source; ≤ 5 facts, each kept only when it cites a supplied block, its quote is in that block and its value is token-bounded in the quote; the evidence is the crawler's copy, never model text) → keys.rs (per AttributeKey::bucket — fee shares price, other_figure shares spec — the model groups (subject, attribute) labels in chunks bounded by input bytes and output tokens, over up to 4 levels with rotating sort keys; incompleteness is recorded) → judge.rs (deterministic candidates at occurrence level on grounding::ValueKey — Uncertain is never "consistent"; cohorts of ≤ 6 values; round-robin review allocation capped at 250; batched review (≤ 8 groups and an output ceiling of max(8,000, 2,000 + 1,500 × groups) tokens per call, split on truncation, groups an answer left out asked once more, dated articles judged against the crawl date) that may return several results per group; code gates: critical only for likely + money/legal-identity keys, numbers in prose must occur in the group, forbidden words → deterministic prose) → doc.rs (ConsistencyDoc, schema siteone-crawler/ai-consistency/2: disclaimer and completeness state before the findings, a fixed caution line on each finding, by-page view, full audit trail; EN/CS fixed texts in text()). Static prompts in repo-root prompts/consistency/{extract,group,judge}.md (prompts.rs); all website data goes in the user message inside <page_data>/<chrome_data>/<labels>/<groups>. Budgets are bytes of the final escaped message from ContextBudget (--ai-context-window). Progress tasks consistency:extract|group|review; src/export/ai_consistency_exporter.rs writes the files
• src/ai/geo/ (--ai-geo): AI search readiness (GEO) — how well AI search and answer engines can reach, read, understand and quote the site; report (Markdown + JSON + HTML, schema siteone-crawler/ai-geo/2) + a deployable kit. Post-crawl (mod.rs::run, fail-soft; the deterministic part runs even when build_config fails, the LLM categories then "not assessed"): keys.rs (key pages: homepage, homepage-linked, sitemap-listed, ≤ 200 — independent of --ai-max-pages) → robots_ai.rs + agents.rs (RFC 9309 matcher per origin and per AI crawler with purposes/compliance, explain, policy_equivalent) → access.rs (4xx/5xx/transport, redirect chains, suspected challenge / soft 404 on short pages, slow pages; in --browser mode without the render time) → controls.rs (scoped snippet directives per engine from meta + each X-Robots-Tag instance, data-nosnippet, canonical) → render.rs (plain: app shell; --browser: raw vs rendered text from raw_text_chars, overlays and collapsed text left out; a failed render gets the plain check) → signals.rs, jsonld.rs (existing markup; typed kit templates built from verified blocks), discovery.rs (sitemaps, lastmod, hreflang; sitemap_proposal only for a crawl that followed every link it found — CrawlEnd from the manager (--max-visited-urls, --max-queue-length via Crawler::dropped_urls_at_queue_limit, --max-url-length via Crawler::dropped_urls_over_max_length, --max-non200-responses-per-basename via Crawler::dropped_urls_for_basename and the skipped (-6) links, interruption), no failed, refused (403) or dangling-redirect linked URL, no URL skipped for robots.txt that Googlebot or Bingbot may crawl (the manager passes the crawler's skipped list, which reaches Status only after the AI phase) — of a site without a declared or crawled sitemap) → analyze.rs (one call per page with numbered blocks in <page_data>, prompt prompts/geo/page.md; the model returns block ids, verify_analysis checks every id, lead numbers, FAQ pair order, byline position and authorship (the name of a name-level author node or after a byline label, which the kit publishes; otherwise review-only), entity values; a lead only for pages that do not state their offer early) → findings.rs (statuses, evidence Strong/Moderate/Weak, priorities ≤ 10) → kit.rs (robots snippet without Allow, robots.proposed.txt only under the safety rule, JSON-LD for indexable pages, FAQPage from ≥ 2 real questions, leads, entity drafts, llms.txt, sitemap) → doc.rs (GeoDoc, EN/CS fixed texts in text()). Progress task geo:pages, usage category AI GEO (page); src/export/ai_geo_exporter.rs writes the files
• src/ai/blocks.rs: DOM evidence blocks shared by AI pipelines — blocks_from_html walks the parsed HTML in document order into Block { region: Main|Chrome, kind, heading_path, text, collapsed, hidden, landmark } (landmark numbers the outermost chrome landmark of a Chrome block, 0 for Main, so a header/footer label never names a line of another landmark) (site header/footer/nav/aside = Chrome, but a header/footer inside article/main is content; table rows carry their column headers; hidden/collapsed content is flagged, and content no visitor can open ([hidden] but until-found, inline display:none or visibility:hidden, not on a tab panel) is also hidden, table rows, row groups and cells included, and hidden content inside a table cell becomes a block of its own, never text of its row, which GEO never builds markup from; each text node counted once)
• src/ai/grounding.rs: meaning-preserving matching shared by AI pipelines — normalize_for_match (with an offset map back to the original), contains_normalized, find_token_bounded (5 % never inside 15 %, 290 never inside 1 290), fact_signals, and value_key → ValueKey::{Exact, Uncertain} (full international phones, site_country for national ones, language-aware numbers keeping operators/ranges/units/time base, CS/EN dates; the LLM's normalized value is never used for the key). An uncertain equivalence never establishes "consistent"
• src/ai/telemetry.rs + src/ai/progress.rs: every LLM HTTP attempt and cache hit goes through AiClient::complete_cached (src/ai/client.rs), which builds one RequestRecord per attempt (tokens incl. reasoning/cached, send→body time, tok/s) and telemetry::reports it: a line on stderr ( AI ✓ #12 SEO 12/40 · /blog/post · 3,412 in · 812 out (540 reasoning) · 6.8 s · 119 tok/s, hidden by --hide-progress-bar) plus an aiRequest event. The page/stage comes from a tokio task-local set by telemetry::scope/progress::unit in each pipeline; progress counts units per task (seo, report:<preset>, profile:chapters, …, one advance per unit, failed ones too) for the 12/40 and the aiProgress events. Usage is parsed shape-first (provider::parse_usage, parse_reasoning); an unknown shape only means missing statistics, never a panic
• src/ai/tools.rs (--ai-list-models, --ai-check): one-shot utility modes for the GUI's model picker, run from main.rs before any crawl (no --url). Each prints exactly one JSON object on stdout, also on failure ({"ok":false,"error":…}, exit 1; exit 101 for a configuration error, answered in JSON without the help text) and never a credential. provider::models_request/parse_model_list shape the model-list GET and read data[]/models[] (context window from max_model_len/max_input_tokens/inputTokenLimit, …); --ai-check sends one prompt through the normal AiClient with the cache off
• src/events.rs + docs/EVENTS.md: the NDJSON event stream of --events-file (protocol 1, additive only): run/phase/URL/artifact/issue events and the AI events aiRequest, aiProgress, aiUsage. docs/EVENTS.md documents every event with an example line; keep it in step with events.rs

Edition & Rust Version

Project uses edition = "2024" (Rust 1.85+) with rust-version = "1.94". Edition 2024 features used throughout: unsafe extern blocks, if let chaining (if let ... && ...), unsafe { std::env::set_var() }.

Commit Policy

Never commit automatically. Commits are only allowed on explicit user request. Before every commit, always run git status, review the changes, and stage only the relevant files — never use git add -A or git add . blindly.

Commit Messages

Use Conventional Commits: feat:, fix:, refactor:, perf:, docs:, style:, ci:, chore:, test:. Examples:

• feat: add built-in HTTP server for markdown/offline exports
• fix: correct non-ASCII text corruption in heading ID generation
• perf: eliminate heap allocation in content_type_for_extension
• chore: bump version to 2.0.3

Releasing a New Version

1. Update version in Cargo.toml (version = "X.Y.Z")
2. Update version in src/version.rs (pub const CODE: &str = "X.Y.Z.YYYYMMDD";)
3. Run cargo check so that Cargo.lock is updated with the new version
4. Commit all three files (Cargo.toml, src/version.rs, Cargo.lock): git commit -m "chore: bump version to X.Y.Z"
5. Tag and push: git tag vX.Y.Z && git push && git push --tags

Important Conventions

• Tables, column order, and formatting must stay consistent across versions. The HTML parser uses the scraper crate.
• HTTP cache lives in tmp/http-client-cache/ by default. Delete it for fresh crawls or use --http-cache-dir= to disable.
• rustls requires explicit ring CryptoProvider installation in main.rs.