AGENTS.md

Overview

Local Deep Research is a Python/PDM application with a FastAPI backend and a Vite frontend. Backend source is in src/local_deep_research/; web code is in src/local_deep_research/web/; tests are in tests/. src/local_deep_research/web/app.py:main() is the process entry point and launches Uvicorn; src/local_deep_research/web/fastapi_app.py constructs the FastAPI application, mounts Socket.IO, and manages startup/shutdown through lifespan().

Source-development prerequisites: Python ≥3.12, <3.15; Node.js 24.0.0+ (frontend rebuild only), PDM, and SQLCipher; see docs/developing.md and docs/SQLCIPHER_INSTALL.md.

Architecture

Iterative research pipeline: a BaseSearchStrategy generates questions, runs parallel searches across BaseSearchEngine implementations, and synthesizes a report via report_generator.py. The web layer is FastAPI + Socket.IO (ASGI); each user has their own encrypted SQLCipher database (no shared application-data DB; a central ldr_auth.db stores only usernames). The main entry point is src/local_deep_research/search_system.py (AdvancedSearchSystem). New search engines require registration in ENGINE_REGISTRY (src/local_deep_research/web_search_engines/engine_registry.py) and the class allowlist in src/local_deep_research/security/module_whitelist.py; new strategies require wiring in create_strategy() (src/local_deep_research/search_system_factory.py) — only LLM providers are auto-discovered. See docs/architecture.md for the full thread lifecycle, QueuePool/FD budget, and cleanup model.

Where to look

Commands

pdm install            # installs the package editable; with --no-self, prefix commands with PYTHONPATH=src
npm install
pdm run pre-commit install && pdm run pre-commit install-hooks
pdm run python -m local_deep_research.web.app
pdm run pytest tests/security/test_repo_guardrails.py -q
npm test
npm run build
pdm run pre-commit run --all-files
pdm run mypy src/ --config-file=pyproject.toml --show-error-codes --pretty

Pre-commit includes auto-fix hooks; inspect the resulting diff after it runs.

Contributing

• Keep PRs small and atomic: one change per PR, with focused tests for behavior changes; see CONTRIBUTING.md.
• Before working on an issue, claim it with a comment and open an early (draft) PR; skip issues that already have a recent claim or linked PR.
• Discuss the approach in the issue first for architectural or cross-subsystem changes, and for anything that can shift research quality: LLM prompts, src/local_deep_research/advanced_search_system/, or search-engine query/parsing/ranking behavior (see docs/BENCHMARKING.md).
• Add a changelog fragment only for user-visible behavior: changelog.d/<PR number>.<category>.md (+<slug> instead of a number when there is no PR yet), with category one of breaking, security, feature, bugfix, removal, misc. The release workflow renders them; local preview is pdm run towncrier build --draft --version <X.Y.Z>.

Filing issues

Issues are the project's backlog, and most are filed by agents with gh rather than through the GitHub web UI, so the issue templates in .github/ISSUE_TEMPLATE/ are bypassed — these rules apply instead.

• Title: name the file or function and the symptom.
• Body: prose, not a template. State what is wrong, where (path and function), why it matters, and the fix options. Cross-reference the originating PR or review (Found during the review of #N) and any sibling issues.
• Labels: one primary area label from the existing set (database, python, javascript, ci-cd, security, documentation, tests, ui-ux, performance, llm-providers, rag, api, ...; list them with gh label list) plus one of priority: high, priority: medium, priority: low.
• Search first to avoid duplicates: gh issue list -R LearningCircuit/local-deep-research --search "<terms>" --state all.
• File with gh issue create --label <area> --label "priority: <level>" --title "..." --body-file <file>.
• Security vulnerabilities are never filed as public issues; follow SECURITY.md.

Invariants

• Test runs default LDR_TESTING_WITH_MOCKS to true when it is unset; tests marked requires_llm or integration are skipped in that mode. Set it to false only when exercising real LLM or network behavior.
• Do not hand-edit generated files. Committed — regenerate and commit the result: docs/CONFIGURATION.md (scripts/generate_config_docs.py), docs/ci/workflow-status.md (scripts/generate_workflow_status.py). Gitignored — never commit: src/local_deep_research/web/static/dist/ (Vite build) and src/local_deep_research/web/static/css/themes.css (written by lifespan() from src/local_deep_research/web/themes/<category>/*.css).
• User data is per-user SQLCipher data. Access it through get_user_db_session; background threads must pass their own db_session to get_settings_manager() because they lack authenticated request context — that path raises rather than silently using defaults. Workers create their own thread-local session — never receive or share a Session across threads, and never hold one across yield in SSE/streaming generators (convention, not hook-enforced).
• Database migrations: do NOT use the alembic CLI — src/local_deep_research/database/migrations/env.py rejects it. Use run_migrations(engine) from src/local_deep_research/database/alembic_runner.py; revision IDs are zero-padded 4-digit strings in src/local_deep_research/database/migrations/versions/NNNN_description.py, with down_revision set to the previous number (check the highest existing number first).
• New database models inherit src/local_deep_research/database/base.py:Base, are imported by src/local_deep_research/database/models/__init__.py, use UtcDateTime for datetime columns (enforced by scripts/pre_commit/check_datetime_timezone.py), and get a numbered migration. New FastAPI routers must be registered in _mount_all() in src/local_deep_research/web/fastapi_app.py.
• DB-backed settings keys must use the namespaces in ALLOWED_SETTING_PREFIXES in src/local_deep_research/web/routers/settings.py; BLOCKED_SETTING_PREFIXES (auth., bootstrap., db_config., security., server., testing.) are not web-editable, and env-only settings are defined in src/local_deep_research/settings/env_definitions/. Use context managers for resource-owning services and sessions; all journal-quality DB consumers except build_db() open it read-only.
• Test logging: the package disables loguru at import; use the loguru_caplog fixture (or loguru_caplog_full for tracebacks) — standard pytest caplog captures nothing.
• Pre-commit runs 60+ custom hooks from .pre-commit-hooks/ (most block commits; the recommend-* hooks are advisory and return 0, and fix-exception-logging.py is an auto-fixer). Common gotchas: tests must import from local_deep_research (or use # allow: no-sut-import — <reason>); direct os.environ access is limited to the paths allowlisted in .pre-commit-hooks/check-env-vars.py; time.sleep(>0.1s) in tests requires @pytest.mark.slow; tautological tests are rejected (# allow: weak-test to opt out); silent except: pass, os.path, open() without encoding=, and literal settings keys outside the allowed namespaces are all rejected.
• Protected infrastructure packages (settings, utilities, security, config) must not import the web layer, except the narrow allowlists in .pre-commit-hooks/check-layer-imports.py. research_id is a string UUID, not an integer.
• Logging: use get_logs_directory() under LDR_DATA_DIR; never log secrets. In src/local_deep_research/llm/providers/, src/local_deep_research/embeddings/providers/ and src/local_deep_research/web_search_engines/, import logger from src/local_deep_research/security/secure_logging.py (hook-enforced: raw loguru there leaks exception tracebacks with credentials to all sinks); elsewhere plain from loguru import logger is the norm — do not mass-convert. Use pathlib.Path, explicit text encodings, and bundled local assets rather than remote resources.
• Persisted research logs capture only the rendered message and selected fields — diagnostic context needed in app_logs or exported JSONL must appear in the message text, not just Loguru extra.
• Vector-store writes must use the canonical reentrant per-(username, index_path) lock (_get_faiss_write_lock() in src/local_deep_research/research_library/services/library_rag_service.py) and BaseVectorStore.apply(); a plain threading.Lock self-deadlocks.
• report_content is answer-only; sources are structured separately. Detailed reports append the sources sentinel; raw/quick LLM output must call format_document_split(..., trust_sentinel=False).
• Treat .pre-commit-config.yaml and its executable checks as authoritative for configured behavior, while remembering that their enforcement is intentionally scoped.

Security invariants

See SECURITY.md for the full threat model; these are the rules code changes must keep.

• The repository is deny-by-default: every tracked file must match .file-whitelist.txt, so a new file type needs a deliberate whitelist entry. Symlinks whose target is missing or is a directory, submodule gitlinks and other non-regular entries are rejected rather than skipped. Never commit secrets, keys, databases or local data.
• All server-side outbound HTTP goes through safe_get, safe_post or SafeSession (src/local_deep_research/security/safe_requests.py), which validate every redirect hop and pin DNS. Raw requests calls are rejected by .pre-commit-hooks/check-safe-requests.py; do not route around it with another HTTP client.
• New network paths must also respect the egress policy (src/local_deep_research/security/egress/README.md): engines are created via create_search_engine(), LLMs via get_llm(), agent URL fetches call evaluate_url(). Operator-configurable LLM and embedding base URLs are checked with assert_base_url_safe().
• In src/local_deep_research/security/ssrf_validator.py, keep RFC_FORBIDDEN_URL_CHARS_RE and the urllib3.util.parse_url host extraction (removing either reopens GHSA-g23j-2vwm-5c25), and never remove entries from ALWAYS_BLOCKED_METADATA_IPS. Log rejected URLs only through redact_url_for_log().
• Switches that weaken a security control (outbound notifications, private-IP access, NAT64, unprotected egress, unencrypted databases) are env-only operator decisions and must never become web-editable settings. LDR_BOOTSTRAP_ALLOW_UNENCRYPTED is for isolated local development only: it disables password verification.
• GitHub Actions workflows pin actions to full commit SHAs, declare least-privilege permissions:, check out with persist-credentials: false, and follow the existing step-security/harden-runner pattern; zizmor audits workflows in CI.
• Hidden Unicode from the enforced set is rejected in tracked text by .pre-commit-hooks/check-zero-width-unicode.py. The enforced set is exactly: every Bidi_Control codepoint (U+061C, U+200E–U+200F, U+202A–U+202E, U+2066–U+2069), the tag block U+E0000–E007F, the variation-selector supplement U+E0100–E01EF, the hook's ZERO_WIDTH_CODEPOINTS constant (U+061C, U+180E, U+200B–U+200D, U+200E–U+200F, U+2060, U+FEFF), and the invisible operators / deprecated format controls U+2061–U+2064 and U+206A–U+206F. Write those characters in source as \uXXXX escapes (Python/JavaScript/TypeScript); in Markdown, which has no \uXXXX escape, use character references such as ‍. Input handling fails closed at file level: bytes that do not decode as UTF-8 are refused outright (enforced codepoints seen as UTF-8 byte sequences in the raw bytes are still reported), and so is any input containing a NUL byte; between them those two rules refuse every UTF-16/UTF-32 shape except byte soup that is simultaneously NUL-free and valid UTF-8 — out of scope because it is read and scanned as the UTF-8 text it already decodes to. Variation selectors U+FE00–U+FE0F (commonly part of emoji sequences) and the Mongolian free variation selectors U+180B–U+180D are not rejected: no file is refused for carrying them.
• Tracked PATHS are held to a stricter rule than tracked text: every tracked path must be pure printable ASCII (U+0020–U+007E). Hidden-Unicode or lookalike filenames (zero-width characters, bidi marks, tag characters, fullwidth/Cyrillic/Greek homoglyphs, control characters) are rejected outright by the index guard shared by .github/scripts/file-whitelist-check.sh and .pre-commit-hooks/file-whitelist-check.sh (agent_guidance_index_filter.py), which also rejects AGENT(S).md / CLAUDE.md / GEMINI.md basenames at any depth in any letter case (Unicode caseless matching) except the exact root paths. The root AGENTS.md, CLAUDE.md and GEMINI.md must be regular files (tracked mode 100644, no symlink), and the CLAUDE.md / GEMINI.md bridges' blob must be exactly @AGENTS.md plus a trailing newline.